kvm-kmod and kernels 3.11.*

kvm-kmod and kernels 3.11.*

[Gabriel L. Somlo]

Jan,

After upgrading my F18 box to 3.11.10-100.fc18.x86_64 I noticed the
"version fence" on kvm-kmod is set to 3.10, and that the current
"source link" for the kvm subproject is set to commit
bf640876e21fe603f7f52b0c27d66b7716da0384


After some testing, kvm-kmod works fine without any changes beyond
bumping "max_kernel_version" to 11 in configure, and advancing the
kvm source link to a6be8569b6705cbc26e7ae1a8be476067cc5a78b


Any further than that, and I start getting errors like this:

  CC [M]  /home/somlo/KVM-OSX/SCRATCH/kvm-kmod/x86/svm.o
In file included from
/home/somlo/KVM-OSX/SCRATCH/kvm-kmod/include/linux/kvm_para.h:61:0,
                 from
/home/somlo/KVM-OSX/SCRATCH/kvm-kmod/x86/../external-module-compat-comm.h:14,
                 from
/home/somlo/KVM-OSX/SCRATCH/kvm-kmod/x86/external-module-compat.h:26,
                 from <command-line>:1:
/home/somlo/KVM-OSX/SCRATCH/kvm-kmod/include/asm/kvm_para.h: In
function `kvm_cpuid_base':
/home/somlo/KVM-OSX/SCRATCH/kvm-kmod/include/asm/kvm_para.h:128:3:
error: implicit declaration of function `hypervisor_cpuid_base'
[-Werror=implicit-function-declaration]


I tried copying and/or cut'n'pasting from additional source files in
the current kvm git tree, but haven't totally figured out how the magic
of "make sync" works yet :) Do you have any notes or any other
documentation one could use to be able to usefully hack on kvm-kmod
when it starts lagging behind the latest kvm master branch ?

Thanks much,
--Gabriel

Re: kvm-kmod and kernels 3.11.*

[Paolo Bonzini]

Il 12/12/2013 22:27, Gabriel L. Somlo ha scritto:
> I tried copying and/or cut'n'pasting from additional source files in
> the current kvm git tree, but haven't totally figured out how the magic
> of "make sync" works yet :) Do you have any notes or any other
> documentation one could use to be able to usefully hack on kvm-kmod
> when it starts lagging behind the latest kvm master branch ?

You need to "hack" the ./sync script to generate the appropriate 
#ifdefs.

It looks like the code to support newer releases is currently in the 
next branch of kvm-kmod.git.

For completeness, here is the patch I am using right now.

Paolo


diff --git a/sync b/sync
index 273ec5b..0e7ab14 100755
--- a/sync
+++ b/sync
@@ -196,7 +196,7 @@ def hack_content(fname, data):
             w('\tkvm_xstate_size_init();\n')
         if match_block_end(r'case CPU_STARTING:', r'hardware_enable'):
             w('#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,28)')
-            w('\t\thardware_enable(NULL);')
+            w('\t\thardware_enable();')
             w('#else')
             w('\t\tsmp_call_function_single(cpu, hardware_enable, NULL, 1);')
             line = '#endif'
@@ -321,6 +321,13 @@ def hack_content(fname, data):
             w('#else')
             w('\tif (!kvm_cpu_has_amd_erratum(kvm_amd_erratum_383))')
             line = '#endif'
+        if line == '\t.count_objects = mmu_shrink_count,':
+            w('#if LINUX_VERSION_CODE >= KERNEL_VERSION(3,12,0)')
+        if line == '\t.scan_objects = mmu_shrink_scan,':
+            w(line)
+            w('#else')
+            w('\t.shrink = mmu_shrink,')
+            line = '#endif'
         w(line)
         if line == '\tkvm_arch_vcpu_put(vcpu);':
             w('\tkvm_fire_urn();')
@@ -358,6 +365,14 @@ def hack_content(fname, data):
             w('#endif')
         if line == '#define _ASM_X86_KVM_HOST_H':
             w('#include <linux/clocksource.h>')
+        if match_block_end(r'^mmu_shrink_count', r'^}'):
+            w('#if LINUX_VERSION_CODE <= KERNEL_VERSION(3,12,0)')
+            w('static int mmu_shrink(struct shrinker *shrink, struct shrink_control *sc)')
+            w('{')
+            w('\tif (sc->nr_to_scan != 0) mmu_shrink_scan(shrink, sc);');
+            w('\treturn mmu_shrink_count(shrink, sc);');
+            w('}')
+            w('#endif')
     if eventfd_file:
         result.append('#else\n'
                       'void kvm_eventfd_init(struct kvm *kvm) { }\n'
@@ -443,6 +458,7 @@ def header_sync(arch):
          for dir in ['%(linux)s/arch/%(arch)s/include/uapi/asm/kvm*.h',
                      '%(linux)s/arch/%(arch)s/include/uapi/asm/vmx.h',
                      '%(linux)s/arch/%(arch)s/include/uapi/asm/svm.h',
+                     '%(linux)s/arch/%(arch)s/include/uapi/asm/msr-index.h',
                      '%(linux)s/arch/%(arch)s/include/uapi/asm/hyperv.h']
          for x in glob(dir % { 'arch': arch, 'linux': linux })
          ])
diff --git a/x86/external-module-compat.h b/x86/external-module-compat.h
index e5039a6..7acd97e 100644
--- a/x86/external-module-compat.h
+++ b/x86/external-module-compat.h
@@ -23,6 +23,13 @@ typedef u64 phys_addr_t;
 #define cpu_has_hypervisor boot_cpu_has(X86_FEATURE_HYPERVISOR)
 #endif
 
+#if LINUX_VERSION_CODE < KERNEL_VERSION(3,12,0)
+static inline int hypervisor_cpuid_base(const char *sig, uint32_t leaves)
+{
+       return 0;
+}
+#endif
+
 #include "../external-module-compat-comm.h"
 
 #include <asm/msr.h>

Re: kvm-kmod and kernels 3.11.*

[Jan Kiszka]

On 2013-12-12 22:59, Paolo Bonzini wrote:
> Il 12/12/2013 22:27, Gabriel L. Somlo ha scritto:
>> I tried copying and/or cut'n'pasting from additional source files in
>> the current kvm git tree, but haven't totally figured out how the magic
>> of "make sync" works yet :) Do you have any notes or any other
>> documentation one could use to be able to usefully hack on kvm-kmod
>> when it starts lagging behind the latest kvm master branch ?
> 
> You need to "hack" the ./sync script to generate the appropriate 
> #ifdefs.
> 
> It looks like the code to support newer releases is currently in the 
> next branch of kvm-kmod.git.

Yes, kvm-kmod is starting to bitrot. I just noticed I messed up the last
stable release back in August, and no one seems to have noticed it.
There were no complains about the missing releases for 3.11 and 3.12, so
it the relevance must have dropped significantly.

kvm-kmod is still useful for hacking, so I'm considering to stop
providing releases (except for a final 3.10.21), just updates in git to
allow wrapping of latest kvm head on recent kernels. There are breakages
with kernels < 3.0 now, about which I personally do not care anymore, so
lifting the entrance level to 3.0 seems reasonable.

> 
> For completeness, here is the patch I am using right now.

Is there anything in your patch that is still required with current
kvm-kmod next? Then I would be happy about a patch!

Jan

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: kvm-kmod and kernels 3.11.*

[Paolo Bonzini]

Il 13/12/2013 16:04, Jan Kiszka ha scritto:
> Yes, kvm-kmod is starting to bitrot. I just noticed I messed up the last
> stable release back in August, and no one seems to have noticed it.
> There were no complains about the missing releases for 3.11 and 3.12, so
> it the relevance must have dropped significantly.

I think the relevance of releases is ~zero.  kvm-kmod still remains
extremely useful IMHO as a hacking tool.

> kvm-kmod is still useful for hacking, so I'm considering to stop
> providing releases (except for a final 3.10.21), just updates in git to
> allow wrapping of latest kvm head on recent kernels.

Yes, git can be good enough for distribution.

> There are breakages
> with kernels < 3.0 now, about which I personally do not care anymore, so
> lifting the entrance level to 3.0 seems reasonable.

Agreed on this as well.  Personally I would be fine even with only 3-4
releases backwards compatibility (e.g. 3.8) right now.

>> > For completeness, here is the patch I am using right now.
> Is there anything in your patch that is still required with current
> kvm-kmod next? Then I would be happy about a patch!

Your next branch works for me.  The only difference between my patch and
your origin/next is that I'm copying

	%(linux)s/arch/%(arch)s/include/uapi/asm/msr-index.h'

and you have

+
+#ifndef MSR_IA32_VMX_MISC_PREEMPTION_TIMER_SCALE
+#define MSR_IA32_VMX_MISC_PREEMPTION_TIMER_SCALE   0x1F
+#endif

Everything else is roughly the same.  Thanks for maintaining kvm-kmod!

Paolo

Re: kvm-kmod and kernels 3.11.*

[Jan Kiszka]

On 2013-12-13 16:12, Paolo Bonzini wrote:
> Il 13/12/2013 16:04, Jan Kiszka ha scritto:
>> Yes, kvm-kmod is starting to bitrot. I just noticed I messed up the last
>> stable release back in August, and no one seems to have noticed it.
>> There were no complains about the missing releases for 3.11 and 3.12, so
>> it the relevance must have dropped significantly.
> 
> I think the relevance of releases is ~zero.  kvm-kmod still remains
> extremely useful IMHO as a hacking tool.

Exactly.

> 
>> kvm-kmod is still useful for hacking, so I'm considering to stop
>> providing releases (except for a final 3.10.21), just updates in git to
>> allow wrapping of latest kvm head on recent kernels.
> 
> Yes, git can be good enough for distribution.
> 
>> There are breakages
>> with kernels < 3.0 now, about which I personally do not care anymore, so
>> lifting the entrance level to 3.0 seems reasonable.
> 
> Agreed on this as well.  Personally I would be fine even with only 3-4
> releases backwards compatibility (e.g. 3.8) right now.
> 
>>>> For completeness, here is the patch I am using right now.
>> Is there anything in your patch that is still required with current
>> kvm-kmod next? Then I would be happy about a patch!
> 
> Your next branch works for me.  The only difference between my patch and
> your origin/next is that I'm copying
> 
> 	%(linux)s/arch/%(arch)s/include/uapi/asm/msr-index.h'
> 
> and you have
> 
> +
> +#ifndef MSR_IA32_VMX_MISC_PREEMPTION_TIMER_SCALE
> +#define MSR_IA32_VMX_MISC_PREEMPTION_TIMER_SCALE   0x1F
> +#endif

Indeed, will change this.

Thanks!
Jan

> 
> Everything else is roughly the same.  Thanks for maintaining kvm-kmod!
> 
> Paolo
> 


-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: kvm-kmod and kernels 3.11.*

[Gabriel L. Somlo]

On Fri, Dec 13, 2013 at 04:36:07PM +0100, Jan Kiszka wrote:
> On 2013-12-13 16:12, Paolo Bonzini wrote:
>> I think the relevance of releases is ~zero.  kvm-kmod still remains
>> extremely useful IMHO as a hacking tool.
> 
> Exactly.

I agree. No idea how "Real KVM Hackers (tm)" do it, but this wannabe
amateur KVM hacker couldn't pull it off without the safety blanket of
his stock distro kernel (and kvm-kmod) :) So thanks again for
maintaining it !

>> Agreed on this as well.  Personally I would be fine even with only 3-4
>> releases backwards compatibility (e.g. 3.8) right now.

Same here. I tend to run up-to-date stock Fedoras that are still
maintained (which right now means as old as F18, with its current
kernel (which is 3.11.10). What's important to me is the ability
to track the latest-and-greatest KVM master, but compile it against
such a distro kernel. I've never had a need to use the releases.

Paolo, thanks for pointing out the "next" branch of kvm-kmod.

Using that, I still got an error when building against the latest KVM
(commit 2e7babfa892a55588467ef03b545002e32f31528):

  CC [M]  /home/somlo/KVM-OSX/SCRATCH/kvm-kmod/x86/x86.o
/home/somlo/KVM-OSX/SCRATCH/kvm-kmod/x86/x86.c: In function 'vcpu_enter_guest':
/home/somlo/KVM-OSX/SCRATCH/kvm-kmod/x86/x86.c:6042:2: error: implicit declaration of function 'smp_mb__after_srcu_read_unlock' [-Werror=implicit-function-declaration]


Adding the following string (cut'n'paste from kvm's srchu.h):

static inline void smp_mb__after_srcu_read_unlock(void) {}

to include-compat/linux/srcu.h didn't work, as apparently
x86/x86.c doesn't want to include *our* srcu.h :(

So I tried to mimic the existing "sed" functionality of the "sync"
script, and was was left deciding between excising the call to         smp_mb__after_srcu_read_unlock() from line 6009 of x86.c (since it's
defined as a no-op in the header), or sed-ing in the declaration from
the kvm srcu.h via the sync script. I think the latter is slightly less
cheesy, but I'm really curious what you think would have been the
"canonical" way to do this :)

With the patch below, kvm-kmod:next can build kvm modules from the
latest/current kvm tree. I don't know if/how I can "update source link"
by way of a patch, but 2e7babfa892a55588467ef03b545002e32f31528 works
fine after applying the enclosed fix.

Thanks again,
--Gabriel


diff --git a/sync b/sync
index c22ee9a..1a941b6 100755
--- a/sync
+++ b/sync
@@ -337,6 +337,12 @@ def hack_content(fname, data):
             w('\t.shrink = mmu_shrink,')
             line = '#else'
 
+        if line == 'static int vcpu_enter_guest(struct kvm_vcpu *vcpu)':
+            w('#if LINUX_VERSION_CODE < KERNEL_VERSION(3,12,0)')
+            w('static inline void smp_mb__after_srcu_read_unlock(void) {}')
+            w('#endif')
+            w('')
+
         w(line)
 
         if line == '\t.scan_objects = mmu_shrink_scan,':

Re: kvm-kmod and kernels 3.11.*

[Paolo Bonzini]

Il 13/12/2013 17:19, Gabriel L. Somlo ha scritto:
> So I tried to mimic the existing "sed" functionality of the "sync"
> script, and was was left deciding between excising the call to         smp_mb__after_srcu_read_unlock() from line 6009 of x86.c (since it's
> defined as a no-op in the header), or sed-ing in the declaration from
> the kvm srcu.h via the sync script. I think the latter is slightly less
> cheesy, but I'm really curious what you think would have been the
> "canonical" way to do this :)

You can also do this:

diff --git a/external-module-compat-comm.h b/external-module-compat-comm.h
index 34fb320..5fa075e 100644
--- a/external-module-compat-comm.h
+++ b/external-module-compat-comm.h
@@ -1408,3 +1408,7 @@ static inline void guest_exit(void)
 }
 #endif /* !CONFIG_CONTEXT_TRACKING */
 #endif /* < 3.10 */
+
+#if LINUX_VERSION_CODE < KERNEL_VERSION(3,13,0)
+#define smp_mb__after_srcu_read_unlock() smp_mb()
+#endif

Paolo

Re: kvm-kmod and kernels 3.11.*

[Jan Kiszka]

On 2013-12-13 17:19, Gabriel L. Somlo wrote:
> On Fri, Dec 13, 2013 at 04:36:07PM +0100, Jan Kiszka wrote:
>> On 2013-12-13 16:12, Paolo Bonzini wrote:
>>> I think the relevance of releases is ~zero.  kvm-kmod still remains
>>> extremely useful IMHO as a hacking tool.
>>
>> Exactly.
> 
> I agree. No idea how "Real KVM Hackers (tm)" do it, but this wannabe
> amateur KVM hacker couldn't pull it off without the safety blanket of
> his stock distro kernel (and kvm-kmod) :) So thanks again for
> maintaining it !
> 
>>> Agreed on this as well.  Personally I would be fine even with only 3-4
>>> releases backwards compatibility (e.g. 3.8) right now.
> 
> Same here. I tend to run up-to-date stock Fedoras that are still
> maintained (which right now means as old as F18, with its current
> kernel (which is 3.11.10). What's important to me is the ability
> to track the latest-and-greatest KVM master, but compile it against
> such a distro kernel. I've never had a need to use the releases.
> 
> Paolo, thanks for pointing out the "next" branch of kvm-kmod.
> 
> Using that, I still got an error when building against the latest KVM
> (commit 2e7babfa892a55588467ef03b545002e32f31528):
> 
>   CC [M]  /home/somlo/KVM-OSX/SCRATCH/kvm-kmod/x86/x86.o
> /home/somlo/KVM-OSX/SCRATCH/kvm-kmod/x86/x86.c: In function 'vcpu_enter_guest':
> /home/somlo/KVM-OSX/SCRATCH/kvm-kmod/x86/x86.c:6042:2: error: implicit declaration of function 'smp_mb__after_srcu_read_unlock' [-Werror=implicit-function-declaration]
> 
> 
> Adding the following string (cut'n'paste from kvm's srchu.h):
> 
> static inline void smp_mb__after_srcu_read_unlock(void) {}
> 
> to include-compat/linux/srcu.h didn't work, as apparently
> x86/x86.c doesn't want to include *our* srcu.h :(
> 
> So I tried to mimic the existing "sed" functionality of the "sync"
> script, and was was left deciding between excising the call to         smp_mb__after_srcu_read_unlock() from line 6009 of x86.c (since it's
> defined as a no-op in the header), or sed-ing in the declaration from
> the kvm srcu.h via the sync script. I think the latter is slightly less
> cheesy, but I'm really curious what you think would have been the
> "canonical" way to do this :)

In this case it is rather via external-module-compat-comm.h. I already
had a patch for this here, will now push it very soon.

sync is already pretty slow, we need to keep it short, possibly even
clean it up in the light of the new entry level (3.0).

Jan

> 
> With the patch below, kvm-kmod:next can build kvm modules from the
> latest/current kvm tree. I don't know if/how I can "update source link"
> by way of a patch, but 2e7babfa892a55588467ef03b545002e32f31528 works
> fine after applying the enclosed fix.
> 
> Thanks again,
> --Gabriel
> 
> 
> diff --git a/sync b/sync
> index c22ee9a..1a941b6 100755
> --- a/sync
> +++ b/sync
> @@ -337,6 +337,12 @@ def hack_content(fname, data):
>              w('\t.shrink = mmu_shrink,')
>              line = '#else'
>  
> +        if line == 'static int vcpu_enter_guest(struct kvm_vcpu *vcpu)':
> +            w('#if LINUX_VERSION_CODE < KERNEL_VERSION(3,12,0)')
> +            w('static inline void smp_mb__after_srcu_read_unlock(void) {}')
> +            w('#endif')
> +            w('')
> +
>          w(line)
>  
>          if line == '\t.scan_objects = mmu_shrink_scan,':
> 

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux