Re: Security policy ambiguities - XSA-108 process post-mortem

[George Dunlap <george.dunlap@eu.citrix.com>]

On 10/10/2014 03:25 PM, Jan Beulich wrote:
>>>> On 09.10.14 at 13:24, <George.Dunlap@eu.citrix.com> wrote:
>> I think that the security team should attempt to determine whether
>> pre-disclosure deployment might give away too much information, and
>> specifically say in each advisory whether early deployment is allowed
>> or not, potentially with specifications about what kind of deployments
>> will be allowed (if necessary).  Most of the time this will just be,
>> "Rebooting servers to deploy this fix is allowed", but it leaves the
>> option open to change it if necessary.
> We're sometimes already struggling determining the set of
> consequences a certain issue may have (see statements like
> "... cannot be excluded"). I think anticipating what sufficiently
> "qualified" people may be able to guess from early deployment
> would end up being rather difficult.

Perhaps -- but it seems to me to be the least risky of all the alternatives.

As far as I can tell we basically have the following options:

1. Never allow people to deploy during the embargo period.

This eliminates the possibility that someone will be helped to gain 
information about the vulnerability by comparing a patched system to an 
unpatched one.  However, it means that cloud operators may spend a short 
amount of time "publicly vulnerable" while they reboot their systems.  I 
assume it would significantly increase the difficulty of coordinating 
such a deployment, as you would have to reboot *all* your servers in the 
smallest time window possible, instead of being able to stage it over 
several days.

It also increases the temptation for operators to "cheat" by starting 
the process a little bit early.  This I think could lead to more 
significant problems community-wise: with incentive to break the rules, 
enforcement becomes an issue -- and I'm sure none of the team want to 
have to deal with that.

2. Always allow people to deploy during the embargo period.

This is simple on the security team, and minimizes the "publicly 
vulnerable time".  It makes deployments easier to coordinate, and avoids 
adding an incentive for "bending" the rules.

However, it does in theory allow an attacker the ability to gain 
information about the vulnerability by comparing patched systems to 
unpatched systems.  In practice, the vast majority of the time the 
measurable difference in functionality will be like finding a needle in 
a haystack; and if the attacker had ever even thought to try the 
functionality (e.g., XSA-7), she would have already known about the 
bug.  But that may not always be the case.

3. Have the security team attempt to evaluate the risk.

This adds work to the security team.  But it at least allows the 
possibility that, in cases where it's pretty clear that early deployment 
*would* give clear hints to an attacker, they could decide to include 
deployment in the embargo.

4. Have individual cloud operators evaluate the risk.

This seems like a recipe for disaster.

I think #1 is too risky, particularly from a community perspective. But 
maybe I'm being a bit pessimistic.

In my mind, #3 is basically exactly the same as #2, except that in cases 
where there would clearly be a problem, the team has an option of 
embargoing deployment as well.

I just spent about 10 minutes skimming through the 80 or so XSAs on 
http://xenbits.xen.org/xsa/.  In most of the cases, it was pretty clear 
that the only behavior that changed would be behavior which would 
already have clued an attacker into the existence of a potential 
vulnerability.  For example, in XSA-108, beforehand some reads from the 
reserved x2apic range would have returned values whereas now they would 
#GP.  But if the attacker knew that they returned values, it already 
would have occurred to her to see if they returned anything of interest.

I didn't notice a single exception to this, though admittedly I didn't 
spend much time looking at it.

This suggests to me that #2 is probably not that dangerous the majority 
of the time.  It also suggests that there may be a simple criteria that 
can be applied for #3 that can eliminate most of the guesswork: Is 
anything in the original behavior being changed likely to lead an 
attacker to think that there may be a vulnerability there?  In the case 
of basically all of them, I think the answer there would be "yes".

So at the moment I would favor #3, then #2; I'm not in favor of #1 but I 
wouldn't strenuously argue against it.  But the main thing is that we  
have a clear policy.

  -George