* Re: [PATCH v3 0/7] User namespace mount updates
From: Seth Forshee @ 2015-11-17 22:01 UTC (permalink / raw)
To: Al Viro
Cc: Austin S Hemmelgarn, Eric W. Biederman, linux-bcache, dm-devel,
linux-raid, linux-mtd, linux-fsdevel, linux-security-module,
selinux, Serge Hallyn, Andy Lutomirski, linux-kernel,
Theodore Ts'o
In-Reply-To: <20151117210542.GY22011@ZenIV.linux.org.uk>
On Tue, Nov 17, 2015 at 09:05:42PM +0000, Al Viro wrote:
> On Tue, Nov 17, 2015 at 03:39:16PM -0500, Austin S Hemmelgarn wrote:
>
> > >This is absolutely insane, no matter how much LSM snake oil you slatter on
> > >the whole thing. All of a sudden you are exposing a huge attack surface
> > >in the place where it would hurt most and as the consolation we are offered
> > >basically "Ted is willing to fix holes when they are found".
None of the LSM changes are intended to protect against attacks from
these sorts of attacks at all, so that's irrelevant.
As I said before, I'm also working to find holes up front. That plus a
commitment from the maintainer seems like a good start at least. What
bar would you set for a given filesystem to be considered "safe enough"?
> > For the context of static image attacks, anything that's found
> > _needs_ to be fixed regardless, and unless you can find some way to
> > actually prevent attacks on mounted filesystems that doesn't involve
> > a complete re-write of the filesystem drivers, then there's not much
> > we can do about it. Yes, unprivileged mounts expose an attack
> > surface, but so does userspace access to the network stack, and so
> > do a lot of other features that are considered essential in a modern
> > general purpose operating system.
>
> "X is exposes an attack surface. Y exposes a diferent attack surface.
> Y is considered important. Therefore X is important enough to implement it"
>
> Right...
That isn't the argument he made. I would summarize the argument as,
"Saying that X exposes an attack surface isn't by itself enough to
reject X, otherwise we wouldn't expose anything (such as example Y)."
You believe that the attack surface is too large, and that's
understandable. Is it your opinion that this is a fundamental problem
for an in-kernel filesystem driver, i.e. that we can never be confident
enough in an in-kernel filesystem parser to allow untrusted data? If
not, what would it take to establish a level of confidence that you
would be comfortable with?
^ permalink raw reply
* Re: [PATCH v3 4/7] fs: Treat foreign mounts as nosuid
From: James Morris @ 2015-11-18 0:00 UTC (permalink / raw)
To: Seth Forshee
Cc: Eric W. Biederman, Alexander Viro, Serge Hallyn, James Morris,
Serge E. Hallyn, Paul Moore, Stephen Smalley, Eric Paris,
Andy Lutomirski, linux-kernel, linux-bcache, dm-devel, linux-raid,
linux-mtd, linux-fsdevel, linux-security-module, selinux
In-Reply-To: <1447778351-118699-5-git-send-email-seth.forshee@canonical.com>
On Tue, 17 Nov 2015, Seth Forshee wrote:
> From: Andy Lutomirski <luto@amacapital.net>
>
> If a process gets access to a mount from a different user
> namespace, that process should not be able to take advantage of
> setuid files or selinux entrypoints from that filesystem. Prevent
> this by treating mounts from other mount namespaces and those not
> owned by current_user_ns() or an ancestor as nosuid.
>
> This will make it safer to allow more complex filesystems to be
> mounted in non-root user namespaces.
>
> This does not remove the need for MNT_LOCK_NOSUID. The setuid,
> setgid, and file capability bits can no longer be abused if code in
> a user namespace were to clear nosuid on an untrusted filesystem,
> but this patch, by itself, is insufficient to protect the system
> from abuse of files that, when execed, would increase MAC privilege.
>
> As a more concrete explanation, any task that can manipulate a
> vfsmount associated with a given user namespace already has
> capabilities in that namespace and all of its descendents. If they
> can cause a malicious setuid, setgid, or file-caps executable to
> appear in that mount, then that executable will only allow them to
> elevate privileges in exactly the set of namespaces in which they
> are already privileges.
>
> On the other hand, if they can cause a malicious executable to
> appear with a dangerous MAC label, running it could change the
> caller's security context in a way that should not have been
> possible, even inside the namespace in which the task is confined.
>
> As a hardening measure, this would have made CVE-2014-5207 much
> more difficult to exploit.
>
> Signed-off-by: Andy Lutomirski <luto@amacapital.net>
> Signed-off-by: Seth Forshee <seth.forshee@canonical.com>
Acked-by: James Morris <james.l.morris@oracle.com>
--
James Morris
<jmorris@namei.org>
^ permalink raw reply
* Re: [PATCH v3 5/7] selinux: Add support for unprivileged mounts from user namespaces
From: James Morris @ 2015-11-18 0:02 UTC (permalink / raw)
To: Seth Forshee
Cc: Eric W. Biederman, Paul Moore, Stephen Smalley, Eric Paris,
Alexander Viro, Serge Hallyn, Andy Lutomirski, linux-kernel,
linux-bcache, dm-devel, linux-raid, linux-mtd, linux-fsdevel,
linux-security-module, selinux, James Morris, Serge E. Hallyn
In-Reply-To: <1447778351-118699-6-git-send-email-seth.forshee@canonical.com>
On Tue, 17 Nov 2015, Seth Forshee wrote:
> Security labels from unprivileged mounts in user namespaces must
> be ignored. Force superblocks from user namespaces whose labeling
> behavior is to use xattrs to use mountpoint labeling instead.
> For the mountpoint label, default to converting the current task
> context into a form suitable for file objects, but also allow the
> policy writer to specify a different label through policy
> transition rules.
>
> Pieced together from code snippets provided by Stephen Smalley.
>
> Signed-off-by: Seth Forshee <seth.forshee@canonical.com>
> Acked-by: Stephen Smalley <sds@tycho.nsa.gov>
Acked-by: James Morris <james.l.morris@oracle.com>
--
James Morris
<jmorris@namei.org>
^ permalink raw reply
* Re: [PATCH v3 6/7] userns: Replace in_userns with current_in_userns
From: James Morris @ 2015-11-18 0:03 UTC (permalink / raw)
To: Seth Forshee
Cc: Eric W. Biederman, Alexander Viro, Serge Hallyn, James Morris,
Serge E. Hallyn, Andy Lutomirski, linux-kernel, linux-bcache,
dm-devel, linux-raid, linux-mtd, linux-fsdevel,
linux-security-module, selinux
In-Reply-To: <1447778351-118699-7-git-send-email-seth.forshee@canonical.com>
On Tue, 17 Nov 2015, Seth Forshee wrote:
> All current callers of in_userns pass current_user_ns as the
> first argument. Simplify by replacing in_userns with
> current_in_userns which checks whether current_user_ns is in the
> namespace supplied as an argument.
>
> Signed-off-by: Seth Forshee <seth.forshee@canonical.com>
Nice cleanup.
Acked-by: James Morris <james.l.morris@oracle.com>
--
James Morris
<jmorris@namei.org>
^ permalink raw reply
* Re: [PATCH v3 7/7] Smack: Handle labels consistently in untrusted mounts
From: James Morris @ 2015-11-18 0:12 UTC (permalink / raw)
To: Seth Forshee
Cc: Eric W. Biederman, Casey Schaufler, Alexander Viro, Serge Hallyn,
Andy Lutomirski, linux-kernel, linux-bcache, dm-devel, linux-raid,
linux-mtd, linux-fsdevel, linux-security-module, selinux,
James Morris, Serge E. Hallyn
In-Reply-To: <1447778351-118699-8-git-send-email-seth.forshee@canonical.com>
On Tue, 17 Nov 2015, Seth Forshee wrote:
> + sbsp = inode->i_sb->s_security;
> + if ((sbsp->smk_flags & SMK_SB_UNTRUSTED) &&
Where is SMK_SB_UNTRUSTED defined?
I can't see it in this patch series, mainline or security next.
--
James Morris
<jmorris@namei.org>
^ permalink raw reply
* Re: [PATCH v3 7/7] Smack: Handle labels consistently in untrusted mounts
From: Seth Forshee @ 2015-11-18 0:50 UTC (permalink / raw)
To: James Morris
Cc: Eric W. Biederman, Casey Schaufler, Alexander Viro, Serge Hallyn,
Andy Lutomirski, linux-kernel, linux-bcache, dm-devel, linux-raid,
linux-mtd, linux-fsdevel, linux-security-module, selinux,
James Morris, Serge E. Hallyn
In-Reply-To: <alpine.LRH.2.20.1511181112090.1460@namei.org>
On Wed, Nov 18, 2015 at 11:12:51AM +1100, James Morris wrote:
> On Tue, 17 Nov 2015, Seth Forshee wrote:
>
> > + sbsp = inode->i_sb->s_security;
> > + if ((sbsp->smk_flags & SMK_SB_UNTRUSTED) &&
>
> Where is SMK_SB_UNTRUSTED defined?
>
> I can't see it in this patch series, mainline or security next.
Eric's already applied a few of my patches to a branch in his tree,
including the one that defines this. See the for-testing branch of
git://git.kernel.org/pub/scm/linux/kernel/git/ebiederm/user-namespace.git,
specifically
https://git.kernel.org/cgit/linux/kernel/git/ebiederm/user-namespace.git/commit/?id=36979551f2345d4e20714c1fc53d6a15c87f8b64.
Thanks,
Seth
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Austin S Hemmelgarn @ 2015-11-18 12:23 UTC (permalink / raw)
To: Seth Forshee
Cc: Al Viro, Eric W. Biederman, linux-bcache, dm-devel, linux-raid,
linux-mtd, linux-fsdevel, linux-security-module, selinux,
Serge Hallyn, Andy Lutomirski, linux-kernel, Theodore Ts'o
In-Reply-To: <20151117213255.GE108807@ubuntu-hedt>
[-- Attachment #1: Type: text/plain, Size: 2942 bytes --]
On 2015-11-17 16:32, Seth Forshee wrote:
> On Tue, Nov 17, 2015 at 03:54:50PM -0500, Austin S Hemmelgarn wrote:
>> On 2015-11-17 14:16, Seth Forshee wrote:
>>> On Tue, Nov 17, 2015 at 02:02:09PM -0500, Austin S Hemmelgarn wrote:
>>>> On 2015-11-17 12:55, Al Viro wrote:
>>>>> On Tue, Nov 17, 2015 at 11:25:51AM -0600, Seth Forshee wrote:
>>>>>
>>>>>> Shortly after that I plan to follow with support for ext4. I've been
>>>>>> fuzzing ext4 for a while now and it has held up well, and I'm currently
>>>>>> working on hand-crafted attacks. Ted has commented privately (to others,
>>>>>> not to me personally) that he will fix bugs for such attacks, though I
>>>>>> haven't seen any public comments to that effect.
>>>>>
>>>>> _Static_ attacks, or change-image-under-mounted-fs attacks?
>>>> To properly protect against attacks on mounted filesystems, we'd
>>>> need some new concept of a userspace immutable file (that is, one
>>>> where nobody can write to it except the kernel, and only the kernel
>>>> can change it between regular access and this new state), and then
>>>> have the kernel set an image (or block device) to this state when a
>>>> filesystem is mounted from it (this introduces all kinds of other
>>>> issues too however, for example stuff that allows an online fsck on
>>>> the device will stop working, as will many un-deletion tools).
>>>
>>> Yeah, Serge and I were just tossing that idea around on irc. If we can
>>> make that work then it's probably the best solution.
>>>
>>> From a naive perspective it seems like all we really have to do is make
>>> the block device inode immutable to userspace when it is mounted. And
>>> the parent block device if it's a partition, which might be a bit
>>> troublesome. We'd have to ensure that writes couldn't happen via any fds
>>> already open when the device was mounted too.
>>>
>>> We'd need some cooperation from the loop driver too I suppose, to make
>>> sure the file backing the loop device is also immutable.
>>>
>> From a completeness perspective, you'd also need to hook into DM,
>> MD, and bcache to handle their backing devices. There's not much we
>> could do about iSCSI/ATAoE/NBD devices, and I think being able to
>
> But really no one would be able to mount any of those without
> intervention from a privileged user anyway. The same is true today of
> loop devices, but I have some patches to change that.
Um, no, depending on how your system is configured, it's fully possible
to mount those as a regular user with no administrative interaction
required. All that needs done is some udev rules (or something
equivalent) to add ACL's to the device nodes allowing regular users to
access them (and there are systems I've seen that are naive enough to do
this). And I can almost assure you that there will be someone out there
who decides to directly expose such block devices to a user namespace.
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Austin S Hemmelgarn @ 2015-11-18 12:46 UTC (permalink / raw)
To: Seth Forshee, Al Viro
Cc: Eric W. Biederman, linux-bcache, dm-devel, linux-raid, linux-mtd,
linux-fsdevel, linux-security-module, selinux, Serge Hallyn,
Andy Lutomirski, linux-kernel, Theodore Ts'o
In-Reply-To: <20151117220125.GF108807@ubuntu-hedt>
[-- Attachment #1: Type: text/plain, Size: 3254 bytes --]
On 2015-11-17 17:01, Seth Forshee wrote:
> On Tue, Nov 17, 2015 at 09:05:42PM +0000, Al Viro wrote:
>> On Tue, Nov 17, 2015 at 03:39:16PM -0500, Austin S Hemmelgarn wrote:
>>
>>>> This is absolutely insane, no matter how much LSM snake oil you slatter on
>>>> the whole thing. All of a sudden you are exposing a huge attack surface
>>>> in the place where it would hurt most and as the consolation we are offered
>>>> basically "Ted is willing to fix holes when they are found".
>
> None of the LSM changes are intended to protect against attacks from
> these sorts of attacks at all, so that's irrelevant.
>
> As I said before, I'm also working to find holes up front. That plus a
> commitment from the maintainer seems like a good start at least. What
> bar would you set for a given filesystem to be considered "safe enough"?
>
>>> For the context of static image attacks, anything that's foun
>>> _needs_ to be fixed regardless, and unless you can find some way to
>>> actually prevent attacks on mounted filesystems that doesn't involve
>>> a complete re-write of the filesystem drivers, then there's not much
>>> we can do about it. Yes, unprivileged mounts expose an attack
>>> surface, but so does userspace access to the network stack, and so
>>> do a lot of other features that are considered essential in a modern
>>> general purpose operating system.
>>
>> "X is exposes an attack surface. Y exposes a diferent attack surface.
>> Y is considered important. Therefore X is important enough to implement it"
>>
>> Right...
>
> That isn't the argument he made. I would summarize the argument as,
> "Saying that X exposes an attack surface isn't by itself enough to
> reject X, otherwise we wouldn't expose anything (such as example Y)."
It's good to see someone understood my meaning...
>
> You believe that the attack surface is too large, and that's
> understandable. Is it your opinion that this is a fundamental problem
> for an in-kernel filesystem driver, i.e. that we can never be confident
> enough in an in-kernel filesystem parser to allow untrusted data? If
> not, what would it take to establish a level of confidence that you
> would be comfortable with?
While I can't speak for Al's opinion on this, I would like to point out
my earlier comment:
> It's unfeasible from a practical standpoint to expect filesystems to
> assume that stuff they write might change under them due to malicious
> intent of a third party.
We can't protect against everything, not without making the system
completely unusable for general purpose computing. There is always some
degree of trust involved in usage of a computer, the OS has to trust
that the hardware works correctly, the administrator has to trust the OS
to behave correctly, and the users have to trust the administrator. The
administrator also needs to have at least some trust in the users,
otherwise he shouldn't be allowing them to use the system.
Perhaps we should have an option that can only be enabled on creation of
the userns that would allow it to use regular kernel mounts, and without
that option we default to only allowing FUSE and a couple of virtual
filesystems (like /proc and devtmpfs).
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Seth Forshee @ 2015-11-18 14:22 UTC (permalink / raw)
To: Austin S Hemmelgarn
Cc: Al Viro, Eric W. Biederman, linux-bcache, dm-devel, linux-raid,
linux-mtd, linux-fsdevel, linux-security-module, selinux,
Serge Hallyn, Andy Lutomirski, linux-kernel, Theodore Ts'o
In-Reply-To: <564C6DD4.6090308@gmail.com>
On Wed, Nov 18, 2015 at 07:23:48AM -0500, Austin S Hemmelgarn wrote:
> On 2015-11-17 16:32, Seth Forshee wrote:
> >On Tue, Nov 17, 2015 at 03:54:50PM -0500, Austin S Hemmelgarn wrote:
> >>On 2015-11-17 14:16, Seth Forshee wrote:
> >>>On Tue, Nov 17, 2015 at 02:02:09PM -0500, Austin S Hemmelgarn wrote:
> >>>>On 2015-11-17 12:55, Al Viro wrote:
> >>>>>On Tue, Nov 17, 2015 at 11:25:51AM -0600, Seth Forshee wrote:
> >>>>>
> >>>>>>Shortly after that I plan to follow with support for ext4. I've been
> >>>>>>fuzzing ext4 for a while now and it has held up well, and I'm currently
> >>>>>>working on hand-crafted attacks. Ted has commented privately (to others,
> >>>>>>not to me personally) that he will fix bugs for such attacks, though I
> >>>>>>haven't seen any public comments to that effect.
> >>>>>
> >>>>>_Static_ attacks, or change-image-under-mounted-fs attacks?
> >>>>To properly protect against attacks on mounted filesystems, we'd
> >>>>need some new concept of a userspace immutable file (that is, one
> >>>>where nobody can write to it except the kernel, and only the kernel
> >>>>can change it between regular access and this new state), and then
> >>>>have the kernel set an image (or block device) to this state when a
> >>>>filesystem is mounted from it (this introduces all kinds of other
> >>>>issues too however, for example stuff that allows an online fsck on
> >>>>the device will stop working, as will many un-deletion tools).
> >>>
> >>>Yeah, Serge and I were just tossing that idea around on irc. If we can
> >>>make that work then it's probably the best solution.
> >>>
> >>> From a naive perspective it seems like all we really have to do is make
> >>>the block device inode immutable to userspace when it is mounted. And
> >>>the parent block device if it's a partition, which might be a bit
> >>>troublesome. We'd have to ensure that writes couldn't happen via any fds
> >>>already open when the device was mounted too.
> >>>
> >>>We'd need some cooperation from the loop driver too I suppose, to make
> >>>sure the file backing the loop device is also immutable.
> >>>
> >> From a completeness perspective, you'd also need to hook into DM,
> >>MD, and bcache to handle their backing devices. There's not much we
> >>could do about iSCSI/ATAoE/NBD devices, and I think being able to
> >
> >But really no one would be able to mount any of those without
> >intervention from a privileged user anyway. The same is true today of
> >loop devices, but I have some patches to change that.
> Um, no, depending on how your system is configured, it's fully
> possible to mount those as a regular user with no administrative
> interaction required. All that needs done is some udev rules (or
> something equivalent) to add ACL's to the device nodes allowing
> regular users to access them (and there are systems I've seen that
> are naive enough to do this). And I can almost assure you that
> there will be someone out there who decides to directly expose such
> block devices to a user namespace.
But it still requires the admin set it up that way, no? And aren't
privileges required to set up those devices in the first place?
I'm not saying that it wouldn't be a good idea to lock down the backing
stores for those types of devices too, just that it isn't something that
a regular user could exploit without an admin doing something to
facilitate it.
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Seth Forshee @ 2015-11-18 14:30 UTC (permalink / raw)
To: Austin S Hemmelgarn
Cc: Al Viro, Eric W. Biederman, linux-bcache, dm-devel, linux-raid,
linux-mtd, linux-fsdevel, linux-security-module, selinux,
Serge Hallyn, Andy Lutomirski, linux-kernel, Theodore Ts'o
In-Reply-To: <564C733D.7010601@gmail.com>
On Wed, Nov 18, 2015 at 07:46:53AM -0500, Austin S Hemmelgarn wrote:
> On 2015-11-17 17:01, Seth Forshee wrote:
> >On Tue, Nov 17, 2015 at 09:05:42PM +0000, Al Viro wrote:
> >>On Tue, Nov 17, 2015 at 03:39:16PM -0500, Austin S Hemmelgarn wrote:
> >>
> >>>>This is absolutely insane, no matter how much LSM snake oil you slatter on
> >>>>the whole thing. All of a sudden you are exposing a huge attack surface
> >>>>in the place where it would hurt most and as the consolation we are offered
> >>>>basically "Ted is willing to fix holes when they are found".
> >
> >None of the LSM changes are intended to protect against attacks from
> >these sorts of attacks at all, so that's irrelevant.
> >
> >As I said before, I'm also working to find holes up front. That plus a
> >commitment from the maintainer seems like a good start at least. What
> >bar would you set for a given filesystem to be considered "safe enough"?
> >
> >>>For the context of static image attacks, anything that's foun
> >>>_needs_ to be fixed regardless, and unless you can find some way to
> >>>actually prevent attacks on mounted filesystems that doesn't involve
> >>>a complete re-write of the filesystem drivers, then there's not much
> >>>we can do about it. Yes, unprivileged mounts expose an attack
> >>>surface, but so does userspace access to the network stack, and so
> >>>do a lot of other features that are considered essential in a modern
> >>>general purpose operating system.
> >>
> >>"X is exposes an attack surface. Y exposes a diferent attack surface.
> >>Y is considered important. Therefore X is important enough to implement it"
> >>
> >>Right...
> >
> >That isn't the argument he made. I would summarize the argument as,
> >"Saying that X exposes an attack surface isn't by itself enough to
> >reject X, otherwise we wouldn't expose anything (such as example Y)."
> It's good to see someone understood my meaning...
> >
> >You believe that the attack surface is too large, and that's
> >understandable. Is it your opinion that this is a fundamental problem
> >for an in-kernel filesystem driver, i.e. that we can never be confident
> >enough in an in-kernel filesystem parser to allow untrusted data? If
> >not, what would it take to establish a level of confidence that you
> >would be comfortable with?
> While I can't speak for Al's opinion on this, I would like to point
> out my earlier comment:
> > It's unfeasible from a practical standpoint to expect filesystems
> to > assume that stuff they write might change under them due to
> malicious > intent of a third party.
So maybe the first requirement is that the user cannot modify the
backing store directly while the device is mounted.
> We can't protect against everything, not without making the system
> completely unusable for general purpose computing. There is always
> some degree of trust involved in usage of a computer, the OS has to
> trust that the hardware works correctly, the administrator has to
> trust the OS to behave correctly, and the users have to trust the
> administrator. The administrator also needs to have at least some
> trust in the users, otherwise he shouldn't be allowing them to use
> the system.
>
> Perhaps we should have an option that can only be enabled on
> creation of the userns that would allow it to use regular kernel
> mounts, and without that option we default to only allowing FUSE and
> a couple of virtual filesystems (like /proc and devtmpfs).
I've considered the idea of something more global like a sysctl, or a
per-filesystem knob in sysfs. I guess a per-container knob is another
option, I'm not sure what interface we use to expose it though.
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Al Viro @ 2015-11-18 14:58 UTC (permalink / raw)
To: Seth Forshee
Cc: Austin S Hemmelgarn, Eric W. Biederman, linux-bcache, dm-devel,
linux-raid, linux-mtd, linux-fsdevel, linux-security-module,
selinux, Serge Hallyn, Andy Lutomirski, linux-kernel,
Theodore Ts'o
In-Reply-To: <20151118142238.GB134139@ubuntu-hedt>
On Wed, Nov 18, 2015 at 08:22:38AM -0600, Seth Forshee wrote:
> But it still requires the admin set it up that way, no? And aren't
> privileges required to set up those devices in the first place?
>
> I'm not saying that it wouldn't be a good idea to lock down the backing
> stores for those types of devices too, just that it isn't something that
> a regular user could exploit without an admin doing something to
> facilitate it.
Sigh... If it boils down to "all admins within all containers must be
trusted not to try and break out" (along with "roothole in any container
escalates to kernel-mode code execution on host"), then what the fuck
is the *point* of bothering with containers, userns, etc. in the first
place? If your model is basically "you want isolation, just use kvm",
fine, but where's the place for userns in all that?
And if you are talking about the _host_ admin, then WTF not have him just
mount what's needed as part of setup and to hell with mounting those
inside the container?
Look at that from the hosting company POV - they are offering a bunch of
virtual machines on one physical system. And you want the admins on those
virtual machines independent from the host admin. Fine, but then you
really need to keep them unable to screw each other or gain kernel-mode
execution on the host.
Again, what's the point of all that? I assumed the model where containers
do, you know, contain what's in them, regardless of trust. You guys seem
to assume something different and I really wonder what it _is_...
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Seth Forshee @ 2015-11-18 15:05 UTC (permalink / raw)
To: Al Viro
Cc: Austin S Hemmelgarn, Eric W. Biederman, linux-bcache, dm-devel,
linux-raid, linux-mtd, linux-fsdevel, linux-security-module,
selinux, Serge Hallyn, Andy Lutomirski, linux-kernel,
Theodore Ts'o
In-Reply-To: <20151118145818.GC22011@ZenIV.linux.org.uk>
On Wed, Nov 18, 2015 at 02:58:18PM +0000, Al Viro wrote:
> On Wed, Nov 18, 2015 at 08:22:38AM -0600, Seth Forshee wrote:
>
> > But it still requires the admin set it up that way, no? And aren't
> > privileges required to set up those devices in the first place?
> >
> > I'm not saying that it wouldn't be a good idea to lock down the backing
> > stores for those types of devices too, just that it isn't something that
> > a regular user could exploit without an admin doing something to
> > facilitate it.
>
> Sigh... If it boils down to "all admins within all containers must be
> trusted not to try and break out" (along with "roothole in any container
> escalates to kernel-mode code execution on host"), then what the fuck
> is the *point* of bothering with containers, userns, etc. in the first
> place? If your model is basically "you want isolation, just use kvm",
> fine, but where's the place for userns in all that?
>
> And if you are talking about the _host_ admin, then WTF not have him just
> mount what's needed as part of setup and to hell with mounting those
> inside the container?
Yes, the host admin. I'm not talking about trusting the admin inside the
container at all.
From my perspective the idea is essentially to allow mounting with fuse
or with ext4 using "mount -o loop ..." within a container.
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Al Viro @ 2015-11-18 15:13 UTC (permalink / raw)
To: Seth Forshee
Cc: Austin S Hemmelgarn, Eric W. Biederman, linux-bcache, dm-devel,
linux-raid, linux-mtd, linux-fsdevel, linux-security-module,
selinux, Serge Hallyn, Andy Lutomirski, linux-kernel,
Theodore Ts'o
In-Reply-To: <20151118150512.GE134139@ubuntu-hedt>
On Wed, Nov 18, 2015 at 09:05:12AM -0600, Seth Forshee wrote:
> Yes, the host admin. I'm not talking about trusting the admin inside the
> container at all.
Then why not have the same host admin just plain mount it when setting the
container up and be done with that? From the host namespace, before spawning
the docker instance or whatever framework you are using. IDGI...
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Richard Weinberger @ 2015-11-18 15:19 UTC (permalink / raw)
To: Al Viro
Cc: Seth Forshee, Austin S Hemmelgarn, Eric W. Biederman,
linux-bcache, device-mapper development,
linux-raid@vger.kernel.org, linux-mtd@lists.infradead.org,
linux-fsdevel, LSM, selinux, Serge Hallyn, Andy Lutomirski, LKML,
Theodore Ts'o
In-Reply-To: <20151118151335.GD22011@ZenIV.linux.org.uk>
On Wed, Nov 18, 2015 at 4:13 PM, Al Viro <viro@zeniv.linux.org.uk> wrote:
> On Wed, Nov 18, 2015 at 09:05:12AM -0600, Seth Forshee wrote:
>
>> Yes, the host admin. I'm not talking about trusting the admin inside the
>> container at all.
>
> Then why not have the same host admin just plain mount it when setting the
> container up and be done with that? From the host namespace, before spawning
> the docker instance or whatever framework you are using. IDGI...
Because hosting companies sell containers as "full virtual machines"
and customers expect to be able mount stuff like disk images they upload.
--
Thanks,
//richard
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Austin S Hemmelgarn @ 2015-11-18 15:34 UTC (permalink / raw)
To: Al Viro, Seth Forshee
Cc: Eric W. Biederman, linux-bcache, dm-devel, linux-raid, linux-mtd,
linux-fsdevel, linux-security-module, selinux, Serge Hallyn,
Andy Lutomirski, linux-kernel, Theodore Ts'o
In-Reply-To: <20151118145818.GC22011@ZenIV.linux.org.uk>
[-- Attachment #1: Type: text/plain, Size: 2774 bytes --]
On 2015-11-18 09:58, Al Viro wrote:
> On Wed, Nov 18, 2015 at 08:22:38AM -0600, Seth Forshee wrote:
>
>> But it still requires the admin set it up that way, no? And aren't
>> privileges required to set up those devices in the first place?
>>
>> I'm not saying that it wouldn't be a good idea to lock down the backing
>> stores for those types of devices too, just that it isn't something that
>> a regular user could exploit without an admin doing something to
>> facilitate it.
>
> Sigh... If it boils down to "all admins within all containers must be
> trusted not to try and break out" (along with "roothole in any container
> escalates to kernel-mode code execution on host"), then what the fuck
> is the *point* of bothering with containers, userns, etc. in the first
> place? If your model is basically "you want isolation, just use kvm",
> fine, but where's the place for userns in all that?
In this case, Seth is referring to the host admin, not the container admin.
>
> And if you are talking about the _host_ admin, then WTF not have him just
> mount what's needed as part of setup and to hell with mounting those
> inside the container?
This is decidedly non-trivial to handle in some cases. IIRC, one of the
particular things that sparked this in the first place was the Chrome
Native Client having to have CAP_SYS_ADMIN or SUID set on it to handle
setting up it's own sandbox, which is not something that should ever be
set on an executable that runs untrusted code (which is the whole point
of NaCl).
>
> Look at that from the hosting company POV - they are offering a bunch of
> virtual machines on one physical system. And you want the admins on those
> virtual machines independent from the host admin. Fine, but then you
> really need to keep them unable to screw each other or gain kernel-mode
> execution on the host.
>
> Again, what's the point of all that? I assumed the model where containers
> do, you know, contain what's in them, regardless of trust. You guys seem
> to assume something different and I really wonder what it _is_...
Yes, hosting and isolation of untrusted code are valid uses for
containers, which is why I suggested the ability to disallow mounts
other than FUSE, and make that the default state. There are other
perfectly valid uses for them as well, and for me the two I'm
particularly interested in are safe deployment of a new system from an
existing system (ala Gentoo or Arch installation, or manual installation
of *BSD), and running non-native distros without virtualization (On a
single user system, virtualization is overkill when all you want is a
Debian or Fedora or Arch testing environment and don't care about their
specific kernel features).
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Nikolay Borisov @ 2015-11-18 15:36 UTC (permalink / raw)
To: Al Viro, Seth Forshee
Cc: Austin S Hemmelgarn, Eric W. Biederman, linux-bcache, dm-devel,
linux-raid, linux-mtd, linux-fsdevel, linux-security-module,
selinux, Serge Hallyn, Andy Lutomirski, linux-kernel,
Theodore Ts'o
In-Reply-To: <20151118145818.GC22011@ZenIV.linux.org.uk>
On 11/18/2015 04:58 PM, Al Viro wrote:
> On Wed, Nov 18, 2015 at 08:22:38AM -0600, Seth Forshee wrote:
>
>> But it still requires the admin set it up that way, no? And aren't
>> privileges required to set up those devices in the first place?
>>
>> I'm not saying that it wouldn't be a good idea to lock down the backing
>> stores for those types of devices too, just that it isn't something that
>> a regular user could exploit without an admin doing something to
>> facilitate it.
>
> Sigh... If it boils down to "all admins within all containers must be
> trusted not to try and break out" (along with "roothole in any container
> escalates to kernel-mode code execution on host"), then what the fuck
> is the *point* of bothering with containers, userns, etc. in the first
> place? If your model is basically "you want isolation, just use kvm",
> fine, but where's the place for userns in all that?
>
> And if you are talking about the _host_ admin, then WTF not have him just
> mount what's needed as part of setup and to hell with mounting those
> inside the container?
>
> Look at that from the hosting company POV - they are offering a bunch of
> virtual machines on one physical system. And you want the admins on those
> virtual machines independent from the host admin. Fine, but then you
> really need to keep them unable to screw each other or gain kernel-mode
> execution on the host.
Actually from the POV of a hosting company there's also the use case of
wanting to use container as substitutes for virtual machines (of course
we are a long way from that). But being able to do what those patches
enable (i.e. what Seth has pointed to with mount -o loop) is beneficial
and desirable.
>
> Again, what's the point of all that? I assumed the model where containers
> do, you know, contain what's in them, regardless of trust. You guys seem
> to assume something different and I really wonder what it _is_...
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Austin S Hemmelgarn @ 2015-11-18 15:38 UTC (permalink / raw)
To: Seth Forshee
Cc: Al Viro, Eric W. Biederman, linux-bcache, dm-devel, linux-raid,
linux-mtd, linux-fsdevel, linux-security-module, selinux,
Serge Hallyn, Andy Lutomirski, linux-kernel, Theodore Ts'o
In-Reply-To: <20151118143022.GC134139@ubuntu-hedt>
[-- Attachment #1: Type: text/plain, Size: 4256 bytes --]
On 2015-11-18 09:30, Seth Forshee wrote:
> On Wed, Nov 18, 2015 at 07:46:53AM -0500, Austin S Hemmelgarn wrote:
>> On 2015-11-17 17:01, Seth Forshee wrote:
>>> On Tue, Nov 17, 2015 at 09:05:42PM +0000, Al Viro wrote:
>>>> On Tue, Nov 17, 2015 at 03:39:16PM -0500, Austin S Hemmelgarn wrote:
>>>>
>>>>>> This is absolutely insane, no matter how much LSM snake oil you slatter on
>>>>>> the whole thing. All of a sudden you are exposing a huge attack surface
>>>>>> in the place where it would hurt most and as the consolation we are offered
>>>>>> basically "Ted is willing to fix holes when they are found".
>>>
>>> None of the LSM changes are intended to protect against attacks from
>>> these sorts of attacks at all, so that's irrelevant.
>>>
>>> As I said before, I'm also working to find holes up front. That plus a
>>> commitment from the maintainer seems like a good start at least. What
>>> bar would you set for a given filesystem to be considered "safe enough"?
>>>
>>>>> For the context of static image attacks, anything that's foun
>>>>> _needs_ to be fixed regardless, and unless you can find some way to
>>>>> actually prevent attacks on mounted filesystems that doesn't involve
>>>>> a complete re-write of the filesystem drivers, then there's not much
>>>>> we can do about it. Yes, unprivileged mounts expose an attack
>>>>> surface, but so does userspace access to the network stack, and so
>>>>> do a lot of other features that are considered essential in a modern
>>>>> general purpose operating system.
>>>>
>>>> "X is exposes an attack surface. Y exposes a diferent attack surface.
>>>> Y is considered important. Therefore X is important enough to implement it"
>>>>
>>>> Right...
>>>
>>> That isn't the argument he made. I would summarize the argument as,
>>> "Saying that X exposes an attack surface isn't by itself enough to
>>> reject X, otherwise we wouldn't expose anything (such as example Y)."
>> It's good to see someone understood my meaning...
>>>
>>> You believe that the attack surface is too large, and that's
>>> understandable. Is it your opinion that this is a fundamental problem
>>> for an in-kernel filesystem driver, i.e. that we can never be confident
>>> enough in an in-kernel filesystem parser to allow untrusted data? If
>>> not, what would it take to establish a level of confidence that you
>>> would be comfortable with?
>> While I can't speak for Al's opinion on this, I would like to point
>> out my earlier comment:
>>> It's unfeasible from a practical standpoint to expect filesystems
>> to > assume that stuff they write might change under them due to
>> malicious > intent of a third party.
>
> So maybe the first requirement is that the user cannot modify the
> backing store directly while the device is mounted.
>
>> We can't protect against everything, not without making the system
>> completely unusable for general purpose computing. There is always
>> some degree of trust involved in usage of a computer, the OS has to
>> trust that the hardware works correctly, the administrator has to
>> trust the OS to behave correctly, and the users have to trust the
>> administrator. The administrator also needs to have at least some
>> trust in the users, otherwise he shouldn't be allowing them to use
>> the system.
>>
>> Perhaps we should have an option that can only be enabled on
>> creation of the userns that would allow it to use regular kernel
>> mounts, and without that option we default to only allowing FUSE and
>> a couple of virtual filesystems (like /proc and devtmpfs).
>
> I've considered the idea of something more global like a sysctl, or a
> per-filesystem knob in sysfs. I guess a per-container knob is another
> option, I'm not sure what interface we use to expose it though.
>
The most useful way I can see of implementing this would be to have an
option on container creation that controls whether kernel mounts are
allowed or not (possibly have it allow any of {no mounts, only FUSE
mounts, all mounts}), and then have a sysctl to set the default for
containers created without this option (and possibly one to force all
containers to ignore the option, and just use the default).
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 3019 bytes --]
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Daniel J Walsh @ 2015-11-18 18:33 UTC (permalink / raw)
To: Austin S Hemmelgarn, Seth Forshee
Cc: linux-bcache-u79uwXL29TY76Z2rM5mHXA, Serge Hallyn,
Theodore Ts'o, dm-devel-H+wXaHxf7aLQT0dZR+AlfA,
linux-kernel-u79uwXL29TY76Z2rM5mHXA, Andy Lutomirski,
linux-raid-u79uwXL29TY76Z2rM5mHXA,
linux-security-module-u79uwXL29TY76Z2rM5mHXA,
linux-mtd-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, Al Viro,
selinux-+05T5uksL2qpZYMLLGbcSA,
linux-fsdevel-u79uwXL29TY76Z2rM5mHXA
In-Reply-To: <564C9B92.5080107-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>
[-- Attachment #1.1: Type: text/plain, Size: 5066 bytes --]
On 11/18/2015 10:38 AM, Austin S Hemmelgarn wrote:
> On 2015-11-18 09:30, Seth Forshee wrote:
>> On Wed, Nov 18, 2015 at 07:46:53AM -0500, Austin S Hemmelgarn wrote:
>>> On 2015-11-17 17:01, Seth Forshee wrote:
>>>> On Tue, Nov 17, 2015 at 09:05:42PM +0000, Al Viro wrote:
>>>>> On Tue, Nov 17, 2015 at 03:39:16PM -0500, Austin S Hemmelgarn wrote:
>>>>>
>>>>>>> This is absolutely insane, no matter how much LSM snake oil you
>>>>>>> slatter on
>>>>>>> the whole thing. All of a sudden you are exposing a huge attack
>>>>>>> surface
>>>>>>> in the place where it would hurt most and as the consolation we
>>>>>>> are offered
>>>>>>> basically "Ted is willing to fix holes when they are found".
>>>>
>>>> None of the LSM changes are intended to protect against attacks from
>>>> these sorts of attacks at all, so that's irrelevant.
>>>>
>>>> As I said before, I'm also working to find holes up front. That plus a
>>>> commitment from the maintainer seems like a good start at least. What
>>>> bar would you set for a given filesystem to be considered "safe
>>>> enough"?
>>>>
>>>>>> For the context of static image attacks, anything that's foun
>>>>>> _needs_ to be fixed regardless, and unless you can find some way to
>>>>>> actually prevent attacks on mounted filesystems that doesn't involve
>>>>>> a complete re-write of the filesystem drivers, then there's not much
>>>>>> we can do about it. Yes, unprivileged mounts expose an attack
>>>>>> surface, but so does userspace access to the network stack, and so
>>>>>> do a lot of other features that are considered essential in a modern
>>>>>> general purpose operating system.
>>>>>
>>>>> "X is exposes an attack surface. Y exposes a diferent attack
>>>>> surface.
>>>>> Y is considered important. Therefore X is important enough to
>>>>> implement it"
>>>>>
>>>>> Right...
>>>>
>>>> That isn't the argument he made. I would summarize the argument as,
>>>> "Saying that X exposes an attack surface isn't by itself enough to
>>>> reject X, otherwise we wouldn't expose anything (such as example Y)."
>>> It's good to see someone understood my meaning...
>>>>
>>>> You believe that the attack surface is too large, and that's
>>>> understandable. Is it your opinion that this is a fundamental problem
>>>> for an in-kernel filesystem driver, i.e. that we can never be
>>>> confident
>>>> enough in an in-kernel filesystem parser to allow untrusted data? If
>>>> not, what would it take to establish a level of confidence that you
>>>> would be comfortable with?
>>> While I can't speak for Al's opinion on this, I would like to point
>>> out my earlier comment:
>>>> It's unfeasible from a practical standpoint to expect filesystems
>>> to > assume that stuff they write might change under them due to
>>> malicious > intent of a third party.
>>
>> So maybe the first requirement is that the user cannot modify the
>> backing store directly while the device is mounted.
>>
>>> We can't protect against everything, not without making the system
>>> completely unusable for general purpose computing. There is always
>>> some degree of trust involved in usage of a computer, the OS has to
>>> trust that the hardware works correctly, the administrator has to
>>> trust the OS to behave correctly, and the users have to trust the
>>> administrator. The administrator also needs to have at least some
>>> trust in the users, otherwise he shouldn't be allowing them to use
>>> the system.
>>>
>>> Perhaps we should have an option that can only be enabled on
>>> creation of the userns that would allow it to use regular kernel
>>> mounts, and without that option we default to only allowing FUSE and
>>> a couple of virtual filesystems (like /proc and devtmpfs).
>>
>> I've considered the idea of something more global like a sysctl, or a
>> per-filesystem knob in sysfs. I guess a per-container knob is another
>> option, I'm not sure what interface we use to expose it though.
>>
> The most useful way I can see of implementing this would be to have an
> option on container creation that controls whether kernel mounts are
> allowed or not (possibly have it allow any of {no mounts, only FUSE
> mounts, all mounts}), and then have a sysctl to set the default for
> containers created without this option (and possibly one to force all
> containers to ignore the option, and just use the default).
>
>
>
> _______________________________________________
> Selinux mailing list
> Selinux-+05T5uksL2qpZYMLLGbcSA@public.gmane.org
> To unsubscribe, send email to Selinux-leave-+05T5uksL2pAGbPMOrvdOA@public.gmane.org
> To get help, send an email containing "help" to Selinux-request-+05T5uksL2pAGbPMOrvdOA@public.gmane.org
From a safely/security point of view. I think you should focus on the
clear linux containers model for allowing containers on hosting
providers. If we need "guaranteed" separation you really need to take
advantage of kvm and each container using its own kernel. In standard
containers if a process can talk directly to the host kernel, any kernel
bug could potentially allow a break out.
[-- Attachment #1.2: Type: text/html, Size: 8694 bytes --]
[-- Attachment #2: Type: text/plain, Size: 304 bytes --]
_______________________________________________
Selinux mailing list
Selinux-+05T5uksL2qpZYMLLGbcSA@public.gmane.org
To unsubscribe, send email to Selinux-leave-+05T5uksL2pAGbPMOrvdOA@public.gmane.org
To get help, send an email containing "help" to Selinux-request-+05T5uksL2pAGbPMOrvdOA@public.gmane.org
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: J. Bruce Fields @ 2015-11-18 18:44 UTC (permalink / raw)
To: Al Viro
Cc: Austin S Hemmelgarn, Seth Forshee, Eric W. Biederman,
linux-bcache, dm-devel, linux-raid, linux-mtd, linux-fsdevel,
linux-security-module, selinux, Serge Hallyn, Andy Lutomirski,
linux-kernel, Theodore Ts'o
In-Reply-To: <20151117193012.GX22011@ZenIV.linux.org.uk>
On Tue, Nov 17, 2015 at 07:30:12PM +0000, Al Viro wrote:
> On Tue, Nov 17, 2015 at 02:02:09PM -0500, Austin S Hemmelgarn wrote:
>
> > >_Static_ attacks, or change-image-under-mounted-fs attacks?
> > To properly protect against attacks on mounted filesystems, we'd
> > need some new concept of a userspace immutable file (that is, one
> > where nobody can write to it except the kernel, and only the kernel
> > can change it between regular access and this new state), and then
> > have the kernel set an image (or block device) to this state when a
> > filesystem is mounted from it (this introduces all kinds of other
> > issues too however, for example stuff that allows an online fsck on
> > the device will stop working, as will many un-deletion tools).
> >
> > The only other option would be to force the FS to cache all metadata
> > in memory, and validate between the cache and what's on disk on
> > every access, which is not realistic for any real world system.
>
> Doctor, it hurt when I do it...
>
> IOW, the other option is to refuse attempting this insanity. Fuse probably
> can be handled, but being able to mount (with kernel-space drivera) an
> arbitrary ext4 image is equivalent to being able to do anything and it's
> going to stay that way for the forseeable future.
What about the filesystems that desktop users commonly mount? (fat,
isofs, udf?)
--b.
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Theodore Ts'o @ 2015-11-18 19:10 UTC (permalink / raw)
To: Seth Forshee
Cc: Al Viro, Eric W. Biederman, linux-bcache, dm-devel, linux-raid,
linux-mtd, linux-fsdevel, linux-security-module, selinux,
Serge Hallyn, Andy Lutomirski, linux-kernel
In-Reply-To: <20151117183444.GB108807@ubuntu-hedt>
On Tue, Nov 17, 2015 at 12:34:44PM -0600, Seth Forshee wrote:
> On Tue, Nov 17, 2015 at 05:55:06PM +0000, Al Viro wrote:
> > On Tue, Nov 17, 2015 at 11:25:51AM -0600, Seth Forshee wrote:
> >
> > > Shortly after that I plan to follow with support for ext4. I've been
> > > fuzzing ext4 for a while now and it has held up well, and I'm currently
> > > working on hand-crafted attacks. Ted has commented privately (to others,
> > > not to me personally) that he will fix bugs for such attacks, though I
> > > haven't seen any public comments to that effect.
> >
> > _Static_ attacks, or change-image-under-mounted-fs attacks?
>
> Right now only static attacks, change-image-under-mounted-fs attacks
> will be next.
I will fix bugs about static attacks. That is, it's interesting to me
that a buggy file system (no matter how it is created), not cause the
kernel to crash --- and privilege escalation attacks tend to be
strongly related to those bugs where we're not doing strong enough
checking.
Protecting against a malicious user which changes the image under the
file system is a whole other kettle of fish. I am not at all user you
can do this without completely sacrificing performance or making the
code impossible to maintain. So my comments do *not* extend to
protecting against a malicious user who is changing the block device
underneath the kernel.
If you want to submit patches to make the kernel more robust against
these attacks, I'm certainly willing to look at the patches. But I'm
certainly not guaranteeing that they will go in, and I'm certainly not
promising to fix all vulnerabilities that you might find that are
caused by a malicious block device. Sorry, that's too much buying a
pig in a poke....
- Ted
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Seth Forshee @ 2015-11-18 19:28 UTC (permalink / raw)
To: Theodore Ts'o, Al Viro, Eric W. Biederman, linux-bcache,
dm-devel, linux-raid, linux-mtd, linux-fsdevel,
linux-security-module, selinux, Serge Hallyn, Andy Lutomirski,
linux-kernel
In-Reply-To: <20151118191045.GB3434@thunk.org>
On Wed, Nov 18, 2015 at 02:10:45PM -0500, Theodore Ts'o wrote:
> On Tue, Nov 17, 2015 at 12:34:44PM -0600, Seth Forshee wrote:
> > On Tue, Nov 17, 2015 at 05:55:06PM +0000, Al Viro wrote:
> > > On Tue, Nov 17, 2015 at 11:25:51AM -0600, Seth Forshee wrote:
> > >
> > > > Shortly after that I plan to follow with support for ext4. I've been
> > > > fuzzing ext4 for a while now and it has held up well, and I'm currently
> > > > working on hand-crafted attacks. Ted has commented privately (to others,
> > > > not to me personally) that he will fix bugs for such attacks, though I
> > > > haven't seen any public comments to that effect.
> > >
> > > _Static_ attacks, or change-image-under-mounted-fs attacks?
> >
> > Right now only static attacks, change-image-under-mounted-fs attacks
> > will be next.
>
> I will fix bugs about static attacks. That is, it's interesting to me
> that a buggy file system (no matter how it is created), not cause the
> kernel to crash --- and privilege escalation attacks tend to be
> strongly related to those bugs where we're not doing strong enough
> checking.
>
> Protecting against a malicious user which changes the image under the
> file system is a whole other kettle of fish. I am not at all user you
> can do this without completely sacrificing performance or making the
> code impossible to maintain. So my comments do *not* extend to
> protecting against a malicious user who is changing the block device
> underneath the kernel.
>
> If you want to submit patches to make the kernel more robust against
> these attacks, I'm certainly willing to look at the patches. But I'm
> certainly not guaranteeing that they will go in, and I'm certainly not
> promising to fix all vulnerabilities that you might find that are
> caused by a malicious block device. Sorry, that's too much buying a
> pig in a poke....
Thanks Ted. My plan right now is to explore the possibility of blocking
writes to the backing store from userspace while it's mounted.
^ permalink raw reply
* Re: [PATCH v3 0/7] User namespace mount updates
From: Serge Hallyn @ 2015-11-18 19:32 UTC (permalink / raw)
To: Theodore Ts'o, Seth Forshee, Al Viro, Eric W. Biederman,
linux-bcache, dm-devel, linux-raid, linux-mtd, linux-fsdevel,
linux-security-module, selinux, Serge Hallyn, Andy Lutomirski,
linux-kernel
In-Reply-To: <20151118191045.GB3434@thunk.org>
Quoting Theodore Ts'o (tytso@mit.edu):
> On Tue, Nov 17, 2015 at 12:34:44PM -0600, Seth Forshee wrote:
> > On Tue, Nov 17, 2015 at 05:55:06PM +0000, Al Viro wrote:
> > > On Tue, Nov 17, 2015 at 11:25:51AM -0600, Seth Forshee wrote:
> > >
> > > > Shortly after that I plan to follow with support for ext4. I've been
> > > > fuzzing ext4 for a while now and it has held up well, and I'm currently
> > > > working on hand-crafted attacks. Ted has commented privately (to others,
> > > > not to me personally) that he will fix bugs for such attacks, though I
> > > > haven't seen any public comments to that effect.
> > >
> > > _Static_ attacks, or change-image-under-mounted-fs attacks?
> >
> > Right now only static attacks, change-image-under-mounted-fs attacks
> > will be next.
>
> I will fix bugs about static attacks. That is, it's interesting to me
> that a buggy file system (no matter how it is created), not cause the
> kernel to crash --- and privilege escalation attacks tend to be
> strongly related to those bugs where we're not doing strong enough
> checking.
>
> Protecting against a malicious user which changes the image under the
> file system is a whole other kettle of fish. I am not at all user you
> can do this without completely sacrificing performance or making the
> code impossible to maintain. So my comments do *not* extend to
> protecting against a malicious user who is changing the block device
> underneath the kernel.
Yup, thanks, Ted. I think the only sane thing to do is work on making the
mounted files immutable. Guarding against under-mounted-writes seems
crazy. Well, actually it seems like a fascinating problem, and maybe
solvable without fs changes, but not in scope here.
> If you want to submit patches to make the kernel more robust against
> these attacks, I'm certainly willing to look at the patches. But I'm
> certainly not guaranteeing that they will go in, and I'm certainly not
> promising to fix all vulnerabilities that you might find that are
> caused by a malicious block device. Sorry, that's too much buying a
> pig in a poke....
>
> - Ted
>
^ permalink raw reply
* Re: RAID 6 Reshape Woes
From: Phil Turmel @ 2015-11-19 1:23 UTC (permalink / raw)
To: Francisco Parada, linux-raid
In-Reply-To: <41BC47FD-C02B-4DDA-BF1C-75032831AA29@abitofthisabitofthat.com>
On 11/18/2015 08:07 PM, Francisco Parada wrote:
> Resending, previous message got rejected due to “HTML”. Damn Apple Mail ;-)
Heh, but let me fix that typo: Damn Apple ;-)
> Hi all,
>
> I thought I had corrected all the flaws in my setup, but I was mistaken. I took care of my hard drive timeout mismatch encountered via a thread a little over a week ago, subjected “RAID 6 Not Mounting (Block device is empty), by adding “smartctl -l scterc,70,70 /dev/sdX” and “for x in /sys/block/*/device/timeout ; do echo 180 > $x ; done” to my boot scripts. I took care of my PSU issue, by replacing my enclosure’s defective PSU, with a new PSU which tested out OK with a multimeter. Today, however, I report some bad news once again.
Ugly.
> After having stressed my rebuilt array for a few days, by adding large sums of data and noting no further syslog errors, I decided that I could not live with 18GB of disk space remaining. Since my last post, I’ve accumulated an additional Terrabyte, and so I ran out of space. At the ready, I had a spare drive, so I decided to run "mdadm --grow --raid-devices=7 --backup-file=/root/grow_md126.bak /dev/md126”, to go from a 6 drive RAID 6 array to my 7 drive array. All was good for about a minute, and then my nightmare began. Luckily, I have a backup of prior to my Terrabyte, which is alright if I lose, just rather not.
Time to toss some enclosures and/or cables.
> mdstat output:
> ====================================================================================
> Every 1.0s: cat /proc/mdstat Wed Nov 18 19:25:02 2015
>
> Personalities : [raid0] [linear] [multipath] [raid1] [raid6] [raid5] [raid4] [raid10]
> md126 : active raid6 sdh[0](F) sdk[6] sdg[5](F) sdf[4](F) sde[3](F) sdj[2] sdi[1]
> 11720540160 blocks super 1.2 level 6, 512k chunk, algorithm 2 [7/3] [_UU___U]
> [>....................] reshape = 0.0% (2726560/2930135040) finish=193325.8min speed=252K/sec
> bitmap: 1/22 pages [4KB], 65536KB chunk
Hmmm. Slow as molasses.
> The device is still mounted and I can access all the data in it.
Probably not. You are just seeing kernel block cache effects, I suspect.
> At 18:55:24, I started my rebuild:
> =====================================================================================================
> Nov 18 18:55:24 DoctorBanner mdadm[1127]: RebuildStarted event detected on md device /dev/md126
> =====================================================================================================
Uhm, what? What command or action did you take? Or are you simply
doing a "flashback" to the start of this process?
> Then 3 seconds later (18:55:27), the first “reshape interrupted” message appeared, but I didn’t notice, because the array was chugging along at 9KB/s according to /proc/mdstat:
> =====================================================================================================
> Nov 18 18:55:27 DoctorBanner kernel: [77563.553030] md: md126: reshape interrupted.
> =====================================================================================================
>
> At some point before the following entries and after starting the reshape, I ran “echo 50000 > /proc/sys/dev/raid/speed_limit_min” to help speed up the reshape, and so I think this is what started causing the issue.
>
> It continued to reshape for about 5 minutes, and then things got really ugly:
> =====================================================================================================
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163377] ata7.00: failed to read SCR 1 (Emask=0x40)
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163382] ata7.01: failed to read SCR 1 (Emask=0x40)
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163384] ata7.02: failed to read SCR 1 (Emask=0x40)
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163385] ata7.03: failed to read SCR 1 (Emask=0x40)
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163386] ata7.04: failed to read SCR 1 (Emask=0x40)
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163388] ata7.05: failed to read SCR 1 (Emask=0x40)
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163392] ata7.15: exception Emask 0x10 SAct 0x0 SErr 0x400000 action 0x6 frozen
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163394] ata7.15: irq_stat 0x08000000, interface fatal error
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163397] ata7.15: SError: { Handshk }
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163399] ata7.00: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163402] ata7.00: failed command: WRITE DMA EXT
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163406] ata7.00: cmd 35/00:40:40:fd:56/00:05:00:00:00/e0 tag 23 dma 688128 out
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163406] res 50/00:00:7f:6b:6c/00:00:00:00:00/e0 Emask 0x100 (unknown error)
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163408] ata7.00: status: { DRDY }
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163410] ata7.01: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163412] ata7.02: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163414] ata7.03: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163416] ata7.04: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163418] ata7.05: exception Emask 0x100 SAct 0x0 SErr 0x0 action 0x6 frozen
> Nov 18 19:00:31 DoctorBanner kernel: [77868.163422] ata7.15: hard resetting link
> Nov 18 19:00:41 DoctorBanner kernel: [77878.160885] ata7.15: softreset failed (1st FIS failed)
> Nov 18 19:00:41 DoctorBanner kernel: [77878.160893] ata7.15: hard resetting link
> Nov 18 19:00:51 DoctorBanner kernel: [77888.162415] ata7.15: softreset failed (1st FIS failed)
> Nov 18 19:00:51 DoctorBanner kernel: [77888.162423] ata7.15: hard resetting link
> Nov 18 19:01:26 DoctorBanner kernel: [77923.153671] ata7.15: softreset failed (1st FIS failed)
> Nov 18 19:01:26 DoctorBanner kernel: [77923.153679] ata7.15: limiting SATA link speed to 1.5 Gbps
> Nov 18 19:01:26 DoctorBanner kernel: [77923.153683] ata7.15: hard resetting link
> Nov 18 19:01:31 DoctorBanner kernel: [77928.160337] ata7.15: softreset failed (1st FIS failed)
> Nov 18 19:01:31 DoctorBanner kernel: [77928.160344] ata7.15: failed to reset PMP, giving up
> Nov 18 19:01:31 DoctorBanner kernel: [77928.160347] ata7.15: Port Multiplier detaching
> =====================================================================================================
>
>
> Which then proceeded to rejecting I/O and offlining devices (full syslog attached).
>
> I’m kind of alright with losing this one, since now I have a decent backup. But is it even possible to recover from something like a failure this while it’s reshaping?
Stop the array completely. Use --assemble --force with all of the
drives, including the new one. Include the same --backup-file.
> I’m going to start chalking it up to the PCIe Port Multiplier being the root of the problem.
Likely. Are the port multipliers capable of the same speeds as the
drives and controllers?
> What do you guys think?
New enclosures & controllers so you can ditch the port multipliers?
Phil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply
* Re: RAID 6 Reshape Woes
From: Francisco Parada @ 2015-11-19 2:45 UTC (permalink / raw)
To: Phil Turmel; +Cc: linux-raid
In-Reply-To: <564D249C.308@turmel.org>
> Heh, but let me fix that typo: Damn Apple ;-)
:-D I set myself up for that one!
> Time to toss some enclosures and/or cables.
Yeah, going to have to get some new toys for the holidays :-)
> Uhm, what? What command or action did you take? Or are you simply
> doing a "flashback" to the start of this process?
Yeah, I was just narrating my actions along with my syslog events.
> Stop the array completely. Use --assemble --force with all of the
> drives, including the new one. Include the same --backup-file.
Thanks, I unmounted, but my "mdadm --stop /dev/md126" command isn't exiting ... Going to give it a few more minutes. According to "ps -ef" the "/lib/systemd/systemd-udev" commanf is keeping it busy. I tried killing that PID with the "-9" argument as the root user, but no such luck.
> Likely. Are the port multipliers capable of the same speeds as the
> drives and controllers?
I believe the enclosure's backplanes are 1.5Gbps, and maybe mismatched with my 3Gbps multiplier might be the bane of my issue.
>> What do you guys think?
>
> New enclosures & controllers so you can ditch the port multipliers?
Yeah, I'm thinking of just building a 4U and using an LSI RAID controller I have laying around, use it as JBOD letting mdadm do the lifting, and stop with these damn cases already. I was just trying to do the best with what I had, but going this Port Multiplier route is what's getting me into this mess. I have a couple of SunFire x2270 1U systems fully spec'ed out that I wanted to take advantage of, and by using these external enclosures, I was getting around the 4 hard drive bay limitation. I guess I'll build something that's cheap just to mount the drives and share via Samba and NFS onto the SunFire systems, then do the heavy CPU lifting from there instead.
Thanks,
Cisco
^ permalink raw reply
* Re: RAID 6 Reshape Woes
From: Adam Goryachev @ 2015-11-19 4:23 UTC (permalink / raw)
To: Francisco Parada; +Cc: linux-raid
In-Reply-To: <6D622406-5CAB-4096-987B-AA45AF9F448A@abitofthisabitofthat.com>
On 18/11/2015 21:45, Francisco Parada wrote:
>
> What do you guys think?
>> New enclosures & controllers so you can ditch the port multipliers?
> Yeah, I'm thinking of just building a 4U and using an LSI RAID controller I have laying around, use it as JBOD letting mdadm do the lifting, and stop with these damn cases already. I was just trying to do the best with what I had, but going this Port Multiplier route is what's getting me into this mess. I have a couple of SunFire x2270 1U systems fully spec'ed out that I wanted to take advantage of, and by using these external enclosures, I was getting around the 4 hard drive bay limitation. I guess I'll build something that's cheap just to mount the drives and share via Samba and NFS onto the SunFire systems, then do the heavy CPU lifting from there instead.
I've used nbd before to export a drive from one machine to use as a
RAID1 member, added with a local block device (mdadm) and that worked
really reliably for me (the remote nbd was also set as write-mostly).
Since then I've also used DRBD to manage both parts of that.
However, in your case, I think there is a more modern version of nbd
which basically exports a block device as a SATA device over the
network. Could you utilise your "couple" of 4 bay 1RU servers, and most
of them use a single drive (or partition from a couple of drives in
RAID1), and export the other partition (95% disk space) to the one
"server" which will then combine them all with mdadm RAID6 or whatever
is needed ?
One problem you have here is that you will lose too many "members"
whenever one of your machines reboots.
Other options come to mind, but they certainly become more complex, and
further away from where you are now.
Not sure if any of the above is useful, or if anyone else can comment on
how well those options work, especially the SATA export for use with
mdadm would be interesting (better if each server was exporting only 1
disk, then you can lose a machine (or disk) without data loss.
Regards,
Adam
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox