Re: KASAN-related VMAP allocation errors in debug kernels with many logical CPUS

[Uladzislau Rezki <urezki@gmail.com>]

> > > 
> > OK. It is related to a module vmap space allocation when a module is
> > inserted. I wounder why it requires 2.5MB for a module? It seems a lot
> > to me.
> > 
> 
> Indeed. I assume KASAN can go wild when it instruments each and every memory
> access.
> 
> > > 
> > > Really looks like only module vmap space. ~ 1 GiB of vmap module space ...
> > > 
> > If an allocation request for a module is 2.5MB we can load ~400 modules
> > having 1GB address space.
> > 
> > "lsmod | wc -l"? How many modules your system has?
> > 
> 
> ~71, so not even close to 400.
> 
> > > What I find interesting is that we have these recurring allocations of similar sizes failing.
> > > I wonder if user space is capable of loading the same kernel module concurrently to
> > > trigger a massive amount of allocations, and module loading code only figures out
> > > later that it has already been loaded and backs off.
> > > 
> > If there is a request about allocating memory it has to be succeeded
> > unless there are some errors like no space no memory.
> 
> Yes. But as I found out we're really out of space because module loading
> code allocates module VMAP space first, before verifying if the module was
> already loaded or is concurrently getting loaded.
> 
> See below.
> 
> [...]
> 
> > I wrote a small patch to dump a modules address space when a fail occurs:
> > 
> > <snip v6.0>
> > diff --git a/mm/vmalloc.c b/mm/vmalloc.c
> > index 83b54beb12fa..88d323310df5 100644
> > --- a/mm/vmalloc.c
> > +++ b/mm/vmalloc.c
> > @@ -1580,6 +1580,37 @@ preload_this_cpu_lock(spinlock_t *lock, gfp_t gfp_mask, int node)
> >   		kmem_cache_free(vmap_area_cachep, va);
> >   }
> > +static void
> > +dump_modules_free_space(unsigned long vstart, unsigned long vend)
> > +{
> > +	unsigned long va_start, va_end;
> > +	unsigned int total = 0;
> > +	struct vmap_area *va;
> > +
> > +	if (vend != MODULES_END)
> > +		return;
> > +
> > +	trace_printk("--- Dump a modules address space: 0x%lx - 0x%lx\n", vstart, vend);
> > +
> > +	spin_lock(&free_vmap_area_lock);
> > +	list_for_each_entry(va, &free_vmap_area_list, list) {
> > +		va_start = (va->va_start > vstart) ? va->va_start:vstart;
> > +		va_end = (va->va_end < vend) ? va->va_end:vend;
> > +
> > +		if (va_start >= va_end)
> > +			continue;
> > +
> > +		if (va_start >= vstart && va_end <= vend) {
> > +			trace_printk(" va_free: 0x%lx - 0x%lx size=%lu\n",
> > +				va_start, va_end, va_end - va_start);
> > +			total += (va_end - va_start);
> > +		}
> > +	}
> > +
> > +	spin_unlock(&free_vmap_area_lock);
> > +	trace_printk("--- Total free: %u ---\n", total);
> > +}
> > +
> >   /*
> >    * Allocate a region of KVA of the specified size and alignment, within the
> >    * vstart and vend.
> > @@ -1663,10 +1694,13 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
> >   		goto retry;
> >   	}
> > -	if (!(gfp_mask & __GFP_NOWARN) && printk_ratelimit())
> > +	if (!(gfp_mask & __GFP_NOWARN) && printk_ratelimit()) {
> >   		pr_warn("vmap allocation for size %lu failed: use vmalloc=<size> to increase size\n",
> >   			size);
> > +		dump_modules_free_space();
> > +	}
> > +
> >   	kmem_cache_free(vmap_area_cachep, va);
> >   	return ERR_PTR(-EBUSY);
> >   }
> 
> Thanks!
> 
> I can spot the same module getting loaded over and over again concurrently
> from user space, only failing after all the allocations when realizing that
> the module is in fact already loaded in add_unformed_module(), failing with
> -EEXIST.
> 
> That looks quite inefficient. Here is how often user space tries to load the
> same module on that system. Note that I print *after* allocating module VMAP
> space.
> 
OK. It explains the problem :) Indeed it is inefficient. Allocating and later
on figuring out that a module is already there looks weird. Furthermore an
attacking from the user space can be organized.


> # dmesg | grep Loading | cut -d" " -f5 | sort | uniq -c
>     896 acpi_cpufreq
>       1 acpi_pad
>       1 acpi_power_meter
>       2 ahci
>       1 cdrom
>       2 compiled-in
>       1 coretemp
>      15 crc32c_intel
>     307 crc32_pclmul
>       1 crc64
>       1 crc64_rocksoft
>       1 crc64_rocksoft_generic
>      12 crct10dif_pclmul
>      16 dca
>       1 dm_log
>       1 dm_mirror
>       1 dm_mod
>       1 dm_region_hash
>       1 drm
>       1 drm_kms_helper
>       1 drm_shmem_helper
>       1 fat
>       1 fb_sys_fops
>      14 fjes
>       1 fuse
>     205 ghash_clmulni_intel
>       1 i2c_algo_bit
>       1 i2c_i801
>       1 i2c_smbus
>       4 i40e
>       4 ib_core
>       1 ib_uverbs
>       4 ice
>     403 intel_cstate
>       1 intel_pch_thermal
>       1 intel_powerclamp
>       1 intel_rapl_common
>       1 intel_rapl_msr
>     399 intel_uncore
>       1 intel_uncore_frequency
>       1 intel_uncore_frequency_common
>      64 ioatdma
>       1 ipmi_devintf
>       1 ipmi_msghandler
>       1 ipmi_si
>       1 ipmi_ssif
>       4 irdma
>     406 irqbypass
>       1 isst_if_common
>     165 isst_if_mbox_msr
>     300 kvm
>     408 kvm_intel
>       1 libahci
>       2 libata
>       1 libcrc32c
>     409 libnvdimm
>       8 Loading
>       1 lpc_ich
>       1 megaraid_sas
>       1 mei
>       1 mei_me
>       1 mgag200
>       1 nfit
>       1 pcspkr
>       1 qrtr
>     405 rapl
>       1 rfkill
>       1 sd_mod
>       2 sg
>     409 skx_edac
>       1 sr_mod
>       1 syscopyarea
>       1 sysfillrect
>       1 sysimgblt
>       1 t10_pi
>       1 uas
>       1 usb_storage
>       1 vfat
>       1 wmi
>       1 x86_pkg_temp_thermal
>       1 xfs
> 
> 
> For each if these loading request, we'll reserve module VMAP space, and free
> it once we realize later that the module was already previously loaded.
> 
> So with a lot of CPUs we might end up trying to load the same module that
> often at the same time that we actually run out of module VMAP space.
> 
> I have a prototype patch that seems to fix this in module loading code.
> 
Good! I am glad the problem can be solved :)

--
Uladzislau Rezki