Re: [PATCH] module,bug: Add TAINT_OOT_MODULE flag for modules not built in-tree

Re: [PATCH] module,bug: Add TAINT_OOT_MODULE flag for modules not built in-tree

[Luis R. Rodriguez]

On Mon, Oct 24, 2011 at 6:12 AM, Ben Hutchings <ben@decadent.org.uk> wrote:
> Use of the GPL or a compatible licence doesn't necessarily make the code
> any good.  We already consider staging modules to be suspect, and this
> should also be true for out-of-tree modules which may receive very
> little review.
>
> Signed-off-by: Ben Hutchings <ben@decadent.org.uk>
> ---
> Debian has been carrying this for the last few kernel versions.  The
> recent thread '[RFC] virtualbox tainting.' and discussions at KS suggest
> that this might be more generally useful.

This indeed seems like a good idea to advocate getting things upstream
(not just staging) but what about the case where we have upstream
drivers from future kernels backported to older kernels and the newer
driver is simply provided as a feature for users who may need new
features / chipset support on their old distribution kernel?

It seems this taint flag will be used for driers backported through
compat-wireless, the compat kernel module or any other backported
driver, even if it is indeed upstream and whereby kernel developer
*do* commit to actually fixing issues. In our experience
compat-wireless bugs *are real bugs*, not backport bugs so we do look
into them. In our latest linux-next.git based release for example
backport code consists only of 1.3804% of the code.

  Luis

Re: [PATCH] module,bug: Add TAINT_OOT_MODULE flag for modules not built in-tree

[Ben Hutchings]

On Mon, Dec 12, 2011 at 01:40:44PM -0800, Luis R. Rodriguez wrote:
> On Mon, Oct 24, 2011 at 6:12 AM, Ben Hutchings <ben@decadent.org.uk> wrote:
> > Use of the GPL or a compatible licence doesn't necessarily make the code
> > any good.  We already consider staging modules to be suspect, and this
> > should also be true for out-of-tree modules which may receive very
> > little review.
> >
> > Signed-off-by: Ben Hutchings <ben@decadent.org.uk>
> > ---
> > Debian has been carrying this for the last few kernel versions.  The
> > recent thread '[RFC] virtualbox tainting.' and discussions at KS suggest
> > that this might be more generally useful.
> 
> This indeed seems like a good idea to advocate getting things upstream
> (not just staging) but what about the case where we have upstream
> drivers from future kernels backported to older kernels and the newer
> driver is simply provided as a feature for users who may need new
> features / chipset support on their old distribution kernel?

They continue to work without any loss of functionality.  (After the
follow-up patches to keep dynamic debugging and lock debugging
working.)

> It seems this taint flag will be used for driers backported through
> compat-wireless, the compat kernel module or any other backported
> driver, even if it is indeed upstream and whereby kernel developer
> *do* commit to actually fixing issues. In our experience
> compat-wireless bugs *are real bugs*, not backport bugs so we do look
> into them. In our latest linux-next.git based release for example
> backport code consists only of 1.3804% of the code.

Now you can look for (O) after the module name in a BUG/Oops message
and you can tell whether the user really had the original or
compat-wireless version of the driver.

It is really up to each distributor or developer how they treat
bug reports with the O taint.  When handling Debian bug reports I
won't automatically reject such a tainted kernel but I will look
carefully at the module list.

Ben.

-- 
Ben Hutchings
We get into the habit of living before acquiring the habit of thinking.
                                                              - Albert Camus

Re: [PATCH] module,bug: Add TAINT_OOT_MODULE flag for modules not built in-tree

[Luis R. Rodriguez]

On Mon, Dec 12, 2011 at 1:58 PM, Ben Hutchings <ben@decadent.org.uk> wrote:
> On Mon, Dec 12, 2011 at 01:40:44PM -0800, Luis R. Rodriguez wrote:
>> On Mon, Oct 24, 2011 at 6:12 AM, Ben Hutchings <ben@decadent.org.uk> wrote:
>> > Use of the GPL or a compatible licence doesn't necessarily make the code
>> > any good.  We already consider staging modules to be suspect, and this
>> > should also be true for out-of-tree modules which may receive very
>> > little review.
>> >
>> > Signed-off-by: Ben Hutchings <ben@decadent.org.uk>
>> > ---
>> > Debian has been carrying this for the last few kernel versions.  The
>> > recent thread '[RFC] virtualbox tainting.' and discussions at KS suggest
>> > that this might be more generally useful.
>>
>> This indeed seems like a good idea to advocate getting things upstream
>> (not just staging) but what about the case where we have upstream
>> drivers from future kernels backported to older kernels and the newer
>> driver is simply provided as a feature for users who may need new
>> features / chipset support on their old distribution kernel?
>
> They continue to work without any loss of functionality.  (After the
> follow-up patches to keep dynamic debugging and lock debugging
> working.)

Great!

>> It seems this taint flag will be used for driers backported through
>> compat-wireless, the compat kernel module or any other backported
>> driver, even if it is indeed upstream and whereby kernel developer
>> *do* commit to actually fixing issues. In our experience
>> compat-wireless bugs *are real bugs*, not backport bugs so we do look
>> into them. In our latest linux-next.git based release for example
>> backport code consists only of 1.3804% of the code.
>
> Now you can look for (O) after the module name in a BUG/Oops message
> and you can tell whether the user really had the original or
> compat-wireless version of the driver.
>
> It is really up to each distributor or developer how they treat
> bug reports with the O taint.  When handling Debian bug reports I
> won't automatically reject such a tainted kernel but I will look
> carefully at the module list.

I'm working on getting my companies to abandon 802.11 proprietary
drivers for good. For Station mode of operation this is pretty much
mission complete. For AP products.. this is work in progress. The out
of tree flag is a good utility one can use to help justify working
upstream but if we treat any future-kernel-backported-driver equally
to any out of tree crap piece of shit driver, it seems to do unjustice
to the value of a properly upstream backported driver. I will note
that I put a lot of effort to ensure that the backport effort is
upstream-centric in an *extremely* upstream-biased way, see how I
label extra patches for tarballs [1]. If your patches are not upstream
the only way you get into these tarballs are by providing patches into
these directories:

  * pending-stable/ stable fixes from linux-next.git not yet on a stable release
  * linux-next-cherry-picks/ patches upstream but that won't go to the
stable release that we want to cherry pick
  * linux-next-pending/ patches posted on the public development
mailing list, patch not yet merged due to the maintainer being away on
vacation or whatever
  * crap/ patches not even posted publicly yet

Each tarball used also gets pegged with a letter if *any* patch from
any of these directories gets applied. The compat module, upon being
loaded, will also print the kernel ring buffer the exact release,
whether extra patches were provided, the upstream git tree used as
base and so on.

So -- although from a technical perspective this may mean Debian /
other kernel developers may ignore the taint flag for compat-wireless
it'd sure be nice to avoid it for them all together. Just can't think
of a way to do it yet... If you agree should we continue to think of a
way if its possible?

[1] http://wireless.kernel.org/en/users/Download/stable#Legend

  Luis

Re: [PATCH] module,bug: Add TAINT_OOT_MODULE flag for modules not built in-tree

[Luis R. Rodriguez]

On Mon, Dec 12, 2011 at 2:47 PM, Luis R. Rodriguez <mcgrof@frijolero.org> wrote:
> On Mon, Dec 12, 2011 at 1:58 PM, Ben Hutchings <ben@decadent.org.uk> wrote:
>> On Mon, Dec 12, 2011 at 01:40:44PM -0800, Luis R. Rodriguez wrote:
>>> On Mon, Oct 24, 2011 at 6:12 AM, Ben Hutchings <ben@decadent.org.uk> wrote:
>>> > Use of the GPL or a compatible licence doesn't necessarily make the code
>>> > any good.  We already consider staging modules to be suspect, and this
>>> > should also be true for out-of-tree modules which may receive very
>>> > little review.
>>> >
>>> > Signed-off-by: Ben Hutchings <ben@decadent.org.uk>
>>> > ---
>>> > Debian has been carrying this for the last few kernel versions.  The
>>> > recent thread '[RFC] virtualbox tainting.' and discussions at KS suggest
>>> > that this might be more generally useful.
>>>
>>> This indeed seems like a good idea to advocate getting things upstream
>>> (not just staging) but what about the case where we have upstream
>>> drivers from future kernels backported to older kernels and the newer
>>> driver is simply provided as a feature for users who may need new
>>> features / chipset support on their old distribution kernel?
>>
>> They continue to work without any loss of functionality.  (After the
>> follow-up patches to keep dynamic debugging and lock debugging
>> working.)
>
> Great!
>
>>> It seems this taint flag will be used for driers backported through
>>> compat-wireless, the compat kernel module or any other backported
>>> driver, even if it is indeed upstream and whereby kernel developer
>>> *do* commit to actually fixing issues. In our experience
>>> compat-wireless bugs *are real bugs*, not backport bugs so we do look
>>> into them. In our latest linux-next.git based release for example
>>> backport code consists only of 1.3804% of the code.
>>
>> Now you can look for (O) after the module name in a BUG/Oops message
>> and you can tell whether the user really had the original or
>> compat-wireless version of the driver.
>>
>> It is really up to each distributor or developer how they treat
>> bug reports with the O taint.  When handling Debian bug reports I
>> won't automatically reject such a tainted kernel but I will look
>> carefully at the module list.
>
> I'm working on getting my companies to abandon 802.11 proprietary
> drivers for good. For Station mode of operation this is pretty much
> mission complete. For AP products.. this is work in progress. The out
> of tree flag is a good utility one can use to help justify working
> upstream but if we treat any future-kernel-backported-driver equally
> to any out of tree crap piece of shit driver, it seems to do unjustice
> to the value of a properly upstream backported driver. I will note
> that I put a lot of effort to ensure that the backport effort is
> upstream-centric in an *extremely* upstream-biased way, see how I
> label extra patches for tarballs [1]. If your patches are not upstream
> the only way you get into these tarballs are by providing patches into
> these directories:
>
>  * pending-stable/ stable fixes from linux-next.git not yet on a stable release
>  * linux-next-cherry-picks/ patches upstream but that won't go to the
> stable release that we want to cherry pick
>  * linux-next-pending/ patches posted on the public development
> mailing list, patch not yet merged due to the maintainer being away on
> vacation or whatever
>  * crap/ patches not even posted publicly yet
>
> Each tarball used also gets pegged with a letter if *any* patch from
> any of these directories gets applied. The compat module, upon being
> loaded, will also print the kernel ring buffer the exact release,
> whether extra patches were provided, the upstream git tree used as
> base and so on.
>
> So -- although from a technical perspective this may mean Debian /
> other kernel developers may ignore the taint flag for compat-wireless
> it'd sure be nice to avoid it for them all together. Just can't think
> of a way to do it yet... If you agree should we continue to think of a
> way if its possible?
>
> [1] http://wireless.kernel.org/en/users/Download/stable#Legend

How about a way to peg a driver as a backport from future kernels?
Like maybe MODULE_COMPAT() ?

  Luis

Re: [PATCH] module,bug: Add TAINT_OOT_MODULE flag for modules not built in-tree

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 3675 bytes --]

On Mon, 2011-12-12 at 14:47 -0800, Luis R. Rodriguez wrote:
> On Mon, Dec 12, 2011 at 1:58 PM, Ben Hutchings <ben@decadent.org.uk> wrote:
> > On Mon, Dec 12, 2011 at 01:40:44PM -0800, Luis R. Rodriguez wrote:
[...]
> >> It seems this taint flag will be used for driers backported through
> >> compat-wireless, the compat kernel module or any other backported
> >> driver, even if it is indeed upstream and whereby kernel developer
> >> *do* commit to actually fixing issues. In our experience
> >> compat-wireless bugs *are real bugs*, not backport bugs so we do look
> >> into them. In our latest linux-next.git based release for example
> >> backport code consists only of 1.3804% of the code.
> >
> > Now you can look for (O) after the module name in a BUG/Oops message
> > and you can tell whether the user really had the original or
> > compat-wireless version of the driver.
> >
> > It is really up to each distributor or developer how they treat
> > bug reports with the O taint.  When handling Debian bug reports I
> > won't automatically reject such a tainted kernel but I will look
> > carefully at the module list.
> 
> I'm working on getting my companies to abandon 802.11 proprietary
> drivers for good. For Station mode of operation this is pretty much
> mission complete. For AP products.. this is work in progress. The out
> of tree flag is a good utility one can use to help justify working
> upstream but if we treat any future-kernel-backported-driver equally
> to any out of tree crap piece of shit driver, it seems to do unjustice
> to the value of a properly upstream backported driver.

Well, it's a warning that not all the kernel code comes from the
original source tree that the version string identifies.  It's not a
value judgement (unlike TAINT_CRAP).

> I will note
> that I put a lot of effort to ensure that the backport effort is
> upstream-centric in an *extremely* upstream-biased way, see how I
> label extra patches for tarballs [1]. If your patches are not upstream
> the only way you get into these tarballs are by providing patches into
> these directories:
> 
>   * pending-stable/ stable fixes from linux-next.git not yet on a stable release
>   * linux-next-cherry-picks/ patches upstream but that won't go to the
> stable release that we want to cherry pick
>   * linux-next-pending/ patches posted on the public development
> mailing list, patch not yet merged due to the maintainer being away on
> vacation or whatever
>   * crap/ patches not even posted publicly yet
> 
> Each tarball used also gets pegged with a letter if *any* patch from
> any of these directories gets applied. The compat module, upon being
> loaded, will also print the kernel ring buffer the exact release,
> whether extra patches were provided, the upstream git tree used as
> base and so on.

Thanks, I appreciate that.

> So -- although from a technical perspective this may mean Debian /
> other kernel developers may ignore the taint flag for compat-wireless
> it'd sure be nice to avoid it for them all together. Just can't think
> of a way to do it yet... If you agree should we continue to think of a
> way if its possible?

Maybe we should be talking about updating the distribution packages
instead.  For the Debian kernel packages, we backport various drivers to
the stable distribution to add support for new hardware.  Please mail
debian-kernel@lists.debian.org if you would like to work with us on
that.

Ben.

> [1] http://wireless.kernel.org/en/users/Download/stable#Legend
> 
>   Luis

-- 
Ben Hutchings
Computers are not intelligent.	They only think they are.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

[RFC] modpost: add option to allow external modules to avoid taint

[John W. Linville]

In some cases, it might be desirable to package a module from an
external source tree alongside the base kernel.  In those cases, it
might also be desirable to not have those modules tainting the kernel.

This patch provides a mechanism for an external module build to declare
itself as an "integrated build".  Such a module is then treated the same
as an intree module.

Signed-off-by: John W. Linville <linville@tuxdriver.com>
---
Any thoughts on this?  I'm thinking of adding this to Fedora kernels,
where I have been working to integrate the compat-wireless package as
part of the base kernel RPM.

 scripts/Makefile.modpost |    1 +
 scripts/mod/modpost.c    |   10 ++++++++--
 2 files changed, 9 insertions(+), 2 deletions(-)

diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost
index 08dce14..160c6fb 100644
--- a/scripts/Makefile.modpost
+++ b/scripts/Makefile.modpost
@@ -81,6 +81,7 @@ modpost = scripts/mod/modpost                    \
  $(if $(KBUILD_EXTMOD),-o $(modulesymfile))      \
  $(if $(CONFIG_DEBUG_SECTION_MISMATCH),,-S)      \
  $(if $(KBUILD_EXTMOD)$(KBUILD_MODPOST_WARN),-w) \
+ $(if $(INTEGRATED_BUILD),-B) \
  $(if $(cross_build),-c)
 
 quiet_cmd_modpost = MODPOST $(words $(filter-out vmlinux FORCE, $^)) modules
diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
index 2bd594e..5d077f9 100644
--- a/scripts/mod/modpost.c
+++ b/scripts/mod/modpost.c
@@ -43,6 +43,9 @@ static int warn_unresolved = 0;
 static int sec_mismatch_count = 0;
 static int sec_mismatch_verbose = 1;
 
+/* Is this a module being built as part of an integrated package? */
+static int integrated_build = 0;
+
 enum export {
 	export_plain,      export_unused,     export_gpl,
 	export_unused_gpl, export_gpl_future, export_unknown
@@ -1851,7 +1854,7 @@ static void add_header(struct buffer *b, struct module *mod)
 
 static void add_intree_flag(struct buffer *b, int is_intree)
 {
-	if (is_intree)
+	if (is_intree || integrated_build)
 		buf_printf(b, "\nMODULE_INFO(intree, \"Y\");\n");
 }
 
@@ -2101,7 +2104,7 @@ int main(int argc, char **argv)
 	struct ext_sym_list *extsym_iter;
 	struct ext_sym_list *extsym_start = NULL;
 
-	while ((opt = getopt(argc, argv, "i:I:e:cmsSo:awM:K:")) != -1) {
+	while ((opt = getopt(argc, argv, "i:I:e:cmsSo:awM:K:B")) != -1) {
 		switch (opt) {
 		case 'i':
 			kernel_read = optarg;
@@ -2139,6 +2142,9 @@ int main(int argc, char **argv)
 		case 'w':
 			warn_unresolved = 1;
 			break;
+		case 'B':
+			integrated_build = 1;
+			break;
 		default:
 			exit(1);
 		}
-- 
1.7.4.4

Re: [RFC] modpost: add option to allow external modules to avoid taint

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

On Wed, 2011-12-14 at 11:20 -0500, John W. Linville wrote:
> In some cases, it might be desirable to package a module from an
> external source tree alongside the base kernel.  In those cases, it
> might also be desirable to not have those modules tainting the kernel.
> 
> This patch provides a mechanism for an external module build to declare
> itself as an "integrated build".  Such a module is then treated the same
> as an intree module.
> 
> Signed-off-by: John W. Linville <linville@tuxdriver.com>
> ---
> Any thoughts on this?  I'm thinking of adding this to Fedora kernels,
> where I have been working to integrate the compat-wireless package as
> part of the base kernel RPM.
[...]

If you're integrating it then why can't you *really* build it in-tree?

Ben.

-- 
Ben Hutchings
Computers are not intelligent.	They only think they are.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [RFC] modpost: add option to allow external modules to avoid taint

[John W. Linville]

On Wed, Dec 14, 2011 at 04:52:00PM +0000, Ben Hutchings wrote:
> On Wed, 2011-12-14 at 11:20 -0500, John W. Linville wrote:
> > In some cases, it might be desirable to package a module from an
> > external source tree alongside the base kernel.  In those cases, it
> > might also be desirable to not have those modules tainting the kernel.
> > 
> > This patch provides a mechanism for an external module build to declare
> > itself as an "integrated build".  Such a module is then treated the same
> > as an intree module.
> > 
> > Signed-off-by: John W. Linville <linville@tuxdriver.com>
> > ---
> > Any thoughts on this?  I'm thinking of adding this to Fedora kernels,
> > where I have been working to integrate the compat-wireless package as
> > part of the base kernel RPM.
> [...]
> 
> If you're integrating it then why can't you *really* build it in-tree?

Well, of course, I could.  But it would be a _huge_ waste of my time
doing so.  The compat-wireless package is already sitting there,
complete with all the necessary backporting infrastructure.  It's all
fat and juicy, waiting to be used.  Wasting time replicating that
just doesn't make any sense to my mind.

John
-- 
John W. Linville		Someday the world will need a hero, and you
linville@tuxdriver.com			might be all we have.  Be ready.

Re: [RFC] modpost: add option to allow external modules to avoid taint

[Ben Hutchings]

[-- Attachment #1: Type: text/plain, Size: 2511 bytes --]

On Fri, 2011-12-16 at 14:26 +1030, Rusty Russell wrote:
> On Wed, 14 Dec 2011 11:20:03 -0500, "John W. Linville" <linville@tuxdriver.com> wrote:
> > In some cases, it might be desirable to package a module from an
> > external source tree alongside the base kernel.  In those cases, it
> > might also be desirable to not have those modules tainting the kernel.
> > 
> > This patch provides a mechanism for an external module build to declare
> > itself as an "integrated build".  Such a module is then treated the same
> > as an intree module.
> > 
> > Signed-off-by: John W. Linville <linville@tuxdriver.com>
> > ---
> > Any thoughts on this?  I'm thinking of adding this to Fedora kernels,
> > where I have been working to integrate the compat-wireless package as
> > part of the base kernel RPM.
> 
> I don't think the feature is useful it it's too easy to disable.
> Experience has shown this with license tags.
> 
> We really want to indicate "out-of-support" which is only a 1:1
> mapping to out-of-tree for upstream kernels.

Who are 'we' in this instance?

> How does Debian handle this?

All the modules in Debian's kernel binary packages are built in-tree.
Backported modules are patched in as necessary.

Debian includes many packages of OOT modules, but those are supported by
their respective maintainers and not the kernel team.  So for the kernel
team, the 'O' flag does not mean 'unsupported' but may indicate that
another maintainer should handle the bug (or it may also be irrelevant
to the bug).

> Perhaps it makes more sense to use the proposed module signing stuff in
> a simplified mode to mark built-with-kernel modules (eg. just put the
> sha of known modules inside the kernel).

Unlike commercial distributions, no-one is paying Debian for support
contracts and no-one can game the system by hiding OOT modules.  So it's
probably not worthwhile for us to use module signing at all.

However, supposing we did go down this route, I would guess that
checksums for ~3000 modules take up more space than the signature
checking code.  Instead, we could perhaps generate a key pair during
build, include the public key in the kernel and then discard the private
key.  (But getting entropy would likely be a problem for the key
generation.)

Ben.

> I think we should revert this change, meanwhile, and figure out what to
> do.
> 
> Cheers,
> Rusty.
> 

-- 
Ben Hutchings
Computers are not intelligent.	They only think they are.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [RFC] modpost: add option to allow external modules to avoid taint

[Rusty Russell]

On Fri, 16 Dec 2011 04:39:49 +0000, Ben Hutchings <ben@decadent.org.uk> wrote:
Non-text part: multipart/signed
> On Fri, 2011-12-16 at 14:26 +1030, Rusty Russell wrote:
> > On Wed, 14 Dec 2011 11:20:03 -0500, "John W. Linville" <linville@tuxdriver.com> wrote:
> > We really want to indicate "out-of-support" which is only a 1:1
> > mapping to out-of-tree for upstream kernels.
> 
> Who are 'we' in this instance?

Whoever turns this flag on.  I was using friendly, inclusive language :)

> > How does Debian handle this?
> 
> All the modules in Debian's kernel binary packages are built in-tree.
> Backported modules are patched in as necessary.
> 
> Debian includes many packages of OOT modules, but those are supported by
> their respective maintainers and not the kernel team.  So for the kernel
> team, the 'O' flag does not mean 'unsupported' but may indicate that
> another maintainer should handle the bug (or it may also be irrelevant
> to the bug).

So, in your case, the kernel team want to know what's outside their
support, so this flag works well for you.

As John pointed out, it's a bit useless for them.  We could enable it
with a config option, or they could ignore it, since they're going to
module-signing route anyway.

> > Perhaps it makes more sense to use the proposed module signing stuff in
> > a simplified mode to mark built-with-kernel modules (eg. just put the
> > sha of known modules inside the kernel).
> 
> Unlike commercial distributions, no-one is paying Debian for support
> contracts and no-one can game the system by hiding OOT modules.  So it's
> probably not worthwhile for us to use module signing at all.
> 
> However, supposing we did go down this route, I would guess that
> checksums for ~3000 modules take up more space than the signature
> checking code.  Instead, we could perhaps generate a key pair during
> build, include the public key in the kernel and then discard the private
> key.  (But getting entropy would likely be a problem for the key
> generation.)

Agreed, 60k is a bit expensive for this minor feature.  

Thanks,
Rusty.