[RFC] IMA Log Snapshotting Design Proposal

[RFC] IMA Log Snapshotting Design Proposal

[Sush Shringarputale]

================================================
| A. Problem Statement                         |
================================================
Depending on the IMA policy, the IMA log can consume a lot of Kernel 
memory on
the device.  For instance, the events for the following IMA policy 
entries may
need to be measured in certain scenarios, but they can also lead to a 
verbose
IMA log when the device is running for a long period of time.
┌───────────────────────────────────────┐
│# PROC_SUPER_MAGIC                     │
│measure fsmagic=0x9fa0                 │
│# SYSFS_MAGIC                          │
│measure fsmagic=0x62656572             │
│# DEBUGFS_MAGIC                        │
│measure fsmagic=0x64626720             │
│# TMPFS_MAGIC                          │
│measure fsmagic=0x01021994             │
│# RAMFS_MAGIC                          │
│measure fsmagic=0x858458f6             │
│# SECURITYFS_MAGIC                     │
│measure fsmagic=0x73636673             │
│# OVERLAYFS_MAGIC                      │
│measure fsmagic=0x794c7630             │
│# log, audit or tmp files              │
│measure obj_type=var_log_t             │
│measure obj_type=auditd_log_t          │
│measure obj_type=tmp_t                 │
└───────────────────────────────────────┘

Secondly, certain devices are configured to take Kernel updates using Kexec
soft-boot.  The IMA log from the previous Kernel gets carried over and the
Kernel memory consumption problem worsens when such devices undergo multiple
Kexec soft-boots over a long period of time.

The above two scenarios can cause IMA log to grow and consume Kernel memory.

In addition, a large IMA log can add pressure on the network bandwidth when
the attestation client sends it to remote-attestation-service.

Truncating IMA log to reclaim memory is not feasible, since it makes the 
log go
out of sync with the TPM PCR quote making remote attestation fail.

A sophisticated solution is required which will help relieve the memory
pressure on the device and continue supporting remote attestation without
disruptions.

-------------------------------------------------------------------------------
================================================
| B. Proposed Solution                         |
================================================
In this document, we propose an enhancement to the IMA subsystem to improve
the long-running performance by snapshotting the IMA log, while still
providing mechanisms to verify its integrity using the PCR quotes.

The remainder of the document describes details of the proposed solution 
in the
following sub-sections.
  - High-level Work-flow
  - Snapshot Triggering Mechanism
  - Design Choices for Storing Snapshots
  - Attestation-Client and Remote-Attestation-Service Side Changes
  - Example Walk-through
  - Open Questions

-------------------------------------------------------------------------------
================================================
| B.1 High-level Work-flow                     |
================================================
Pre-requisites:
- IMA Integrity guarantees are maintained.

The proposed high level work-flow of IMA log snapshotting is as follows:
- A user-mode process will trigger the snapshot by opening a file in SysFS
   say /sys/kernel/security/ima/snapshot (referred to as 
sysk_ima_snapshot_file
   here onwards).
- The Kernel will get the current TPM PCR values and PCR update counter [2]
   and store them as template data in a new IMA event "snapshot_aggregate".
   This event will be measured by IMA using critical data measurement
   functionality [1].  Recording regular IMA events will be paused while
   "snapshot_aggregate" is being computed using the existing IMA mutex lock.
- Once the "snapshot_aggregate" is computed and measured in IMA log, the 
prior
   IMA events will be made available in the sysk_ima_snapshot_file.
- The UM process will copy those IMA events from sysk_ima_snapshot_file to a
   snapshot file on disk chosen by UM (referred to as UM_snapshot_file here
   onwards).  The location, file-system type, access permissions etc. of the
   UM_snapshot_file would be controlled by UM process itself.
- Once UM is done copying the IMA events from sysk_ima_snapshot_file to
   UM_snapshot_file, it will indicate to the Kernel that the snapshot can be
   finalized by triggering a write with any data to the 
sysk_ima_snapshot_file.
   UM process cannot prevent the IMA log purge operation after this point.
- The Kernel will truncate the current IMA log and and clear HTable up 
to the
   "snapshot_aggregate" marker.
- The Kernel will measure the PCR update counter as part of measuring
   snapshot_aggregate, so that it can be used by the remote attestation 
service
   for detecting missing events.
- UM can prevent the IMA log purge by closing the sysk_ima_snapshot_file
   without performing a write operation on it.  In this case, while the
   "snapshot_aggregate" marker may still be in the log, the event can be 
ignored
   since the previous entries in the IMA log will not be purged.

Note:
- This work-flow should work when interleaved with Kexec 'load' and 
'execute'
   events and should not cause IMA log + snapshot to go out of sync with PCR
   quotes. The implementation details are omitted from this document for
   brevity.
-------------------------------------------------------------------------------
================================================
| B.2 Snapshot Triggering Mechanism            |
================================================
The IMA subsystem will create a new file: /sys/kernel/security/ima/snapshot
(a.k.a. sysk_ima_snapshot_file). A UM process opening this file will trigger
the snapshot.  The file will be opened exclusively, so only one UM 
process can
trigger the snapshot at a time.
Once the kernel has written the snapshot_aggregate marker to the IMA 
log, the
IMA log prior to that can be read by UM on the same FD.  When UM writes some
data to sysk_ima_snapshot_file, the kernel will finalize the snapshot.  
If UM
closes sysk_ima_snapshot_file without writing to it, the Kernel will not
purge the IMA log.

If a system administrator requires that only a specific client process 
should
trigger the snapshot, this capability can be set as an SeLinux policy.

-------------------------------------------------------------------------------
================================================
| B.3 Design Choices for Storing Snapshots     |
================================================
Choosing the snapshot location is handled by the UM process.  The Kernel 
will
wait for the UM process to indicate that the current IMA log has been 
written to
UM_snapshot_file and then the Kernel will truncate the IMA log. UM is
responsible to clear any existing UM_snapshot_file(s) on system start or 
on a
hard reboot/power cycle.

-------------------------------------------------------------------------------
======================================================================
| B.4 Attestation-Client and Remote-Attestation-Service Side Changes |
======================================================================
A remote attestation client and service will need to be aware of the
snapshot_aggregate marker and how it should be handled.  A typical 
attestation
path on the device will remain the same - send the current IMA log along 
with
the signed TPM PCR quotes to the remote attestation-service.  But the
attestation clients and remote services need to be aware that they need 
to use
UM_snapshot_file(s) too, along with PCR quotes and the data in
"snapshot_aggregate" event in IMA log to re-establish the chain of trust if
the feature is enabled and a snapshot is taken.

-------------------------------------------------------------------------------
================================================
| B.4.1 Attestation-Client side changes        |
================================================
To use snapshot feature, the attestation client is required to know
     - if snapshotting is supported/enabled by the Kernel
     - if snapshotting is supported/enabled by the 
remote-attestation-service
       it interacts with
     - the correct order of UM_snapshot_file(s) for backtracking through the
       IMA log

The protocol between the attestation-client and remote-attestation-service
needs to be updated to send previous snapshots to the service as requested.

For service transition, clients connected to a service unaware of
snapshot_aggregate should not write any data to the sysk_ima_snapshot_file
before closing it.  This will add the snapshot_aggregate marker to the IMA
log without purging the log.  Once the service implements snapshot_aggregate
parsing, the client can implement the functionality to  write to the
sysk_ima_snapshot_file each time it triggers the snapshot.  This would 
indicate
to the Kernel that the log should be purged.

-------------------------------------------------------------------------------
================================================
| B.4.2 Remote-Attestation-Service side changes|
================================================
For remote attestation to work, the service will need to know how to 
validate
the snapshot_aggregate entry in the IMA log.  It will have to read the PCR
values present in the template data of snapshot_aggregate event in the 
latest
IMA log, and ensure that the PCR quotes align with the contents of the past
UM_snapshot_file(s).  This will re-establish the chain of trust needed 
for the
device to pass remote attestation.  This will also maintain the ability of
the remote-attestation-service to seal the secrets, if the client-server use
TPM unseal mechanism to attest the state of the device.

The client-service protocol will need an implementation for requesting
previous snapshots of the IMA logs.  There may be various scenarios when
such a request is made such as:
     - the service requires all the data since boot based on admin decisions
     - linked IMA events are split across a snapshot boundary
     - any other protocol level decision

The service will need to request the client to send the old 
UM_snapshot_file(s)
and ensure that the log replay still generates the expected PCR values 
provided
in the quote.

To avoid asking for previous snapshot chunks from the device, the service
may maintain the past attestation status of the device at a given snapshot
checkpoint.  This would require the service to be stateful, but it will
relieve the storage pressure on the device as well as the network bandwidth
once the attestation-client starts taking snapshots.
-------------------------------------------------------------------------------
================================================
| B.5 Example Walk-through                     |
================================================
For illustration, we provide an example of the snapshot scenario. Assume
the below is the IMA log on a system before a snapshot is taken.
┌───┬─────────────┬────────┬───────────────┬─────────────────┬────────────┐
│PCR│Template Hash│Template│   Data Hash   │    Data File │TemplateData│
├───┼─────────────┼────────┼───────────────┼─────────────────┼────────────┤
│10 │322a847385...│ima-sig │sha256:309c... │boot_aggregate │            │
├───┼─────────────┼────────┼───────────────┼─────────────────┼────────────┤
│10 │92dbf55061...│ima-buf │sha256:b93e... │kernel_version...│352342347...│
├───┼─────────────┼────────┼───────────────┼─────────────────┼────────────┤
│11 │e8e12d9532...│ima-buf │sha256:cd00... │dm_table_load....│568956899...│
├───┼─────────────┼────────┼───────────────┼─────────────────┼────────────┤
│12 │e8e12d9532...│ima-sig │sha256:5601... │/usr/lib/modul...│            │
└───┴─────────────┴────────┴───────────────┴─────────────────┴────────────┘

For this example, the PCR values in sha256 bank are as follows:

     sha256:
         0 : 
0xDA009CB9DDBF2DF2FD5982A8E44492718D3CAE516A99847262790479368F82B6
         ...
         ...
         10: 
0xB93EBF68FC66C6B6F5E2D0E34E113D84303BC0D5DA0B419F059DE27EAE3BAA29
         11: 
0xCD00ABB3D84DB0F0419F2064ECD8AD4C01A1ADCDCADB15DEED47BA6FCE40D420
         12: 
0x560153FB6A0CC6033475994ECA726765892BB48772682F692E44A0393281DB45
         ...
         ...
         23: 
0x0000000000000000000000000000000000000000000000000000000000000000

If a snapshot is taken at this point, the current IMA log will be written to
disk.  No additional events will be measured in the log until the 
snapshot is
complete.

The state of the IMA log after the snapshot:
┌───┬─────────┬────────┬─────────────┬──────────────────┬─────────────────────┐
│PCR│Template │Template│ Data Hash   │     Data File    │ Template      │
│   │ Hash    │        │             │     Data File    │ Data        │
├───┼─────────┼────────┼─────────────┼──────────────────┼─────────────────────┤
│10 │e55cba...│ima-buf │Sha256:30a...│snapshot_aggregate│ 
<TEMPLATE_DATA>   │
│   │         │        │             │                  │   see the 
grammar   │
│   │         │        │             │                  │ below          │
└───┴─────────┴────────┴─────────────┴──────────────────┴─────────────────────┘
  where the Data Hash of the log line is -
  Append(H(snapshot_file),PCR0,PCR1,...PCR23).  All available PCR banks 
will be
  included in the template data in a known structure.

The Template Data will follow the grammar below:
     TEMPLATE_DATA := 
<PCR_Update_Counter>";"<Snapshot_Counter>";"<PCR_Banks>";"
     PCR_Update_Counter := "PCR_Counter="<PCR update counter value from TPM>
     Snapshot_Counter := "Snapshot_Attempt_Count=" <num. of snapshots 
attempted>
     PCR_Banks          := <PCR_Bank>";"|<PCR_Banks> "," <PCR_Bank>";"
     PCR_Bank           := <PCR_Hash Type> ":" <PCRn>
     PCR_Hash_Type      := "sha1"|"sha256"|"sha384"
     PCRn               := "PCR"<N>":"<PCR_Hash>
     N                  := [0-23]
     PCR_Hash           := <hash_of_the_pcr_N>

The state of the PCRs after the snapshot:
     (Only PCR 10 will change, since snapshot_aggregate is being extended
      in that PCR.)
     sha256:
         0 : 
0xDA009CB9DDBF2DF2FD5982A8E44492718D3CAE516A99847262790479368F82B6
         ...
         ...
         10 : 
0x30A1B69F09A9599ED45FFEA5C94000852518394A8E1A3B8D343D4458E6FB0B04
         11 : 
0xCD00ABB3D84DB0F0419F2064ECD8AD4C01A1ADCDCADB15DEED47BA6FCE40D420
         12 : 
0x560153FB6A0CC6033475994ECA726765892BB48772682F692E44A0393281DB45
         ...
         ...
         23: 
0x0000000000000000000000000000000000000000000000000000000000000000

An example of "snapshot_aggregate" template data is given below.
10 e55cba... ima-buf  Sha256:30a... snapshot_aggregate
     PCR_Counter=11;
     Snapshot_Attempt_Count=7;
sha256:PCR0:0xDA009CB9DDBF2DF2FD5982A8E44492718D3CAE516A99847262790479368F82B6,
sha256:PCR1:0x30A1B69F09A9599ED45FFEA5C94000852518394A8E1A3B8D343D4458E6FB0B04,
     ...
sha256:PCR23:0x30A1B69F09A9599ED45FFEA5C94000852518394A8E1A3B8D343D4458E6FB0B04;

Future logs can then continue to extend the assigned PCRs.  Since IMA
measurements extend the TPM-PCRs and computing snapshot_aggregate involves
reading TPM PCR banks, IMA measurements must be suspended until the
snapshot_aggregate is computed and measured.  Otherwise, these two 
operations
may interfere with each other compromising the integrity of the system.

The remote-attestation-service can verify the contents of the past (N-1)
UM_snapshot_file(s) by replaying the events in them and comparing them
with the PCR values stored in template data of the first 
"snapshot_aggregate"
event of subsequent IMA log.

-------------------------------------------------------------------------------
================================================
| C. Open Questions                            |
================================================
1. There is a known issue with PCRs being reset after hibernate [3].  Is 
there
   any proposed solution for the hibernate problem? Can we reuse parts 
of it for
   IMA log snapshotting?

-------------------------------------------------------------------------------
We appreciate your comments on this proposal as well as pointers to any
prior efforts made in addressing this issue.

Thanks!
Sush & Tushar

-------------------------------------------------------------------------------
================================================
| D. References                                |
================================================
[1] IMA: support for measuring kernel integrity critical data
https://patchwork.kernel.org/project/linux-integrity/cover/20210108040708.8389-1-tusharsu@linux.microsoft.com/

[2] Measuring TPM update counter in IMA
https://patchwork.kernel.org/project/linux-integrity/cover/20230801181917.8535-1-tusharsu@linux.microsoft.com/

[3] TPM: Hibernate with IMA PCR 10
https://lore.kernel.org/linux-integrity/2bc656bf67af52e0b9a68e91c5b574e0ab4ffa8e.camel@linux.ibm.com/

Re: [RFC] IMA Log Snapshotting Design Proposal

[James Bottomley]

On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
[...]
> Truncating IMA log to reclaim memory is not feasible, since it makes
> the log go out of sync with the TPM PCR quote making remote
> attestation fail.

This assumption isn't entirely true.  It's perfectly possible to shard
an IMA log using two TPM2_Quote's for the beginning and end PCR values
to validate the shard.  The IMA log could be truncated in the same way
(replace the removed part of the log with a TPM2_Quote and AK, so the
log still validates from the beginning quote to the end).

If you use a TPM2_Quote mechanism to save the log, all you need to do
is have the kernel generate the quote with an internal AK.  You can
keep a record of the quote and the AK at the beginning of the truncated
kernel log.  If the truncated entries are saved in a file shard it
should have a beginning and end quote and a record of the AK used. 
Since verifiers like Keylime are already using this beginning and end
quote for sharded logs, it's the most natural format to feed to
something externally for verification and it means you don't have to
invent a new format to do the same thing.

Regards,

James

Re: [RFC] IMA Log Snapshotting Design Proposal

[Stefan Berger]

On 8/1/23 17:21, James Bottomley wrote:
> On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
> [...]
>> Truncating IMA log to reclaim memory is not feasible, since it makes
>> the log go out of sync with the TPM PCR quote making remote
>> attestation fail.
> 
> This assumption isn't entirely true.  It's perfectly possible to shard
> an IMA log using two TPM2_Quote's for the beginning and end PCR values
> to validate the shard.  The IMA log could be truncated in the same way
> (replace the removed part of the log with a TPM2_Quote and AK, so the
> log still validates from the beginning quote to the end).
> 
> If you use a TPM2_Quote mechanism to save the log, all you need to do
> is have the kernel generate the quote with an internal AK.  You can
> keep a record of the quote and the AK at the beginning of the truncated
> kernel log.  If the truncated entries are saved in a file shard it

The truncation seems dangerous to me. Maybe not all the scenarios with an attestation
client (client = reading logs and quoting) are possible then anymore, such as starting
an attestation client only after truncation but a verifier must have witnessed the
system's PCRs and log state before the truncation occurred.

I think an ima-buf (or similar) log entry in IMA log would have to appear at the beginning of the
truncated log stating the value of all PCRs that IMA touched (typically only PCR 10
but it can be others). The needs to be done since the quote itself doesn't
provide the state of the individual PCRs. This would at least allow an attestation
client to re-read the log from the beginning (when it is re-start or started for the
first time after the truncation). However, this alone (without the
internal AK quoting the old state) could lead to abuse where I could create totally
fake IMA logs stating the state of the PCRs at the beginning (so the verifier
syncs its internal PCR state to this state). Further, even with the AK-quote that
you propose I may be able to create fake logs and trick a verifier into
trusting the machine IFF it doesn't know what kernel this system was booted with
that I may have hacked to provide a fake AK-quote that just happens to match the
PCR state presented at the beginning of the log.

=> Can a truncated log be made safe for attestation when the attestation starts
only after the truncation occurred?

=> Even if attestation was occurring 'what' state does an attestation server
need to carry around for an attested-to system so that the truncation is 'safe'
and I cannot create fake AK-quotes and fake IMA logs with initial PCR states?
Can I ever restart the client and have it read the truncated log from the
beginning and what type of verification needs to happen on the server then?
It seems like the server would have to remember the state of the IMA PCRs upon
last truncation to detect a possible attack. This would make staring to monitor
a system after truncation impossible -- would be good to know these details.




> should have a beginning and end quote and a record of the AK used.
> Since verifiers like Keylime are already using this beginning and end
> quote for sharded logs, it's the most natural format to feed to
> something externally for verification and it means you don't have to
> invent a new format to do the same thing.
> 
> Regards,
> 
> James
> 
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

Re: [RFC] IMA Log Snapshotting Design Proposal

[James Bottomley]

On Mon, 2023-08-07 at 18:49 -0400, Stefan Berger wrote:
> 
> 
> On 8/1/23 17:21, James Bottomley wrote:
> > On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
> > [...]
> > > Truncating IMA log to reclaim memory is not feasible, since it
> > > makes the log go out of sync with the TPM PCR quote making remote
> > > attestation fail.
> > 
> > This assumption isn't entirely true.  It's perfectly possible to
> > shard an IMA log using two TPM2_Quote's for the beginning and end
> > PCR values to validate the shard.  The IMA log could be truncated
> > in the same way (replace the removed part of the log with a
> > TPM2_Quote and AK, so the log still validates from the beginning
> > quote to the end).
> > 
> > If you use a TPM2_Quote mechanism to save the log, all you need to
> > do is have the kernel generate the quote with an internal AK.  You
> > can keep a record of the quote and the AK at the beginning of the
> > truncated kernel log.  If the truncated entries are saved in a file
> > shard it
> 
> The truncation seems dangerous to me. Maybe not all the scenarios
> with an attestation client (client = reading logs and quoting) are
> possible then anymore, such as starting an attestation client only
> after truncation but a verifier must have witnessed the system's PCRs
> and log state before the truncation occurred.

That's not exactly correct.  Nothing needs to have "witnessed" the
starting PCR value because the quote vouches for it (and can vouch for
it after the fact).  The only thing you need to verify the quote is the
attestation key and the only thing you need to do to trust the
attestation key is ensure it was TPM created.  All of that can be
verified after the fact as well.  The only thing that can be done to
disrupt this is to destroy the TPM (or re-own it).

Remember the assumption is you *also* have the removed log shard to
present.  From that the PCR state of the starting quote can be
calculated and checked for matching the quote.  If you lose that, it's
equivalent to the log being tampered with and all bets are off.  The
assumption is that because of the impossibility of engineering TPM
extensions, it should be impossible to come up with a fake log that
produces the PCRs of the real one.  If that's violated, then IMA itself
becomes useless.

James

Re: [RFC] IMA Log Snapshotting Design Proposal

[Stefan Berger]

On 8/8/23 08:35, James Bottomley wrote:
> On Mon, 2023-08-07 at 18:49 -0400, Stefan Berger wrote:
>>
>>
>> On 8/1/23 17:21, James Bottomley wrote:
>>> On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
>>> [...]
>>>> Truncating IMA log to reclaim memory is not feasible, since it
>>>> makes the log go out of sync with the TPM PCR quote making remote
>>>> attestation fail.
>>>
>>> This assumption isn't entirely true.  It's perfectly possible to
>>> shard an IMA log using two TPM2_Quote's for the beginning and end
>>> PCR values to validate the shard.  The IMA log could be truncated
>>> in the same way (replace the removed part of the log with a
>>> TPM2_Quote and AK, so the log still validates from the beginning
>>> quote to the end).
>>>
>>> If you use a TPM2_Quote mechanism to save the log, all you need to
>>> do is have the kernel generate the quote with an internal AK.  You
>>> can keep a record of the quote and the AK at the beginning of the
>>> truncated kernel log.  If the truncated entries are saved in a file
>>> shard it
>>
>> The truncation seems dangerous to me. Maybe not all the scenarios
>> with an attestation client (client = reading logs and quoting) are
>> possible then anymore, such as starting an attestation client only
>> after truncation but a verifier must have witnessed the system's PCRs
>> and log state before the truncation occurred.
> 
> That's not exactly correct.  Nothing needs to have "witnessed" the
> starting PCR value because the quote vouches for it (and can vouch for
> it after the fact).  The only thing you need to verify the quote is the
> attestation key and the only thing you need to do to trust the> attestation key is ensure it was TPM created.  All of that can be
> verified after the fact as well.  The only thing that can be done to
> disrupt this is to destroy the TPM (or re-own it).> 
> Remember the assumption is you *also* have the removed log shard to
> present.  From that the PCR state of the starting quote can be

Yes, the whole sequence of old logs needs to be available. IF that's the
case and the logs can be stitched together seamlessly, who then looks at the
kernel AK quote and under what circumstances?

> calculated and checked for matching the quote.  If you lose that, it's
> equivalent to the log being tampered with and all bets are off.  The
> assumption is that because of the impossibility of engineering TPM
> extensions, it should be impossible to come up with a fake log that
> produces the PCRs of the real one.  If that's violated, then IMA itself
> becomes useless.
> 
> James
> 
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

Re: [RFC] IMA Log Snapshotting Design Proposal

[James Bottomley]

On Tue, 2023-08-08 at 09:31 -0400, Stefan Berger wrote:
> 
> 
> On 8/8/23 08:35, James Bottomley wrote:
> > On Mon, 2023-08-07 at 18:49 -0400, Stefan Berger wrote:
> > > 
> > > 
> > > On 8/1/23 17:21, James Bottomley wrote:
> > > > On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
> > > > [...]
> > > > > Truncating IMA log to reclaim memory is not feasible, since
> > > > > it makes the log go out of sync with the TPM PCR quote making
> > > > > remote attestation fail.
> > > > 
> > > > This assumption isn't entirely true.  It's perfectly possible
> > > > to shard an IMA log using two TPM2_Quote's for the beginning
> > > > and end PCR values to validate the shard.  The IMA log could be
> > > > truncated in the same way (replace the removed part of the log
> > > > with a TPM2_Quote and AK, so the log still validates from the
> > > > beginning quote to the end).
> > > > 
> > > > If you use a TPM2_Quote mechanism to save the log, all you need
> > > > to do is have the kernel generate the quote with an internal
> > > > AK.  You can keep a record of the quote and the AK at the
> > > > beginning of the truncated kernel log.  If the truncated
> > > > entries are saved in a file shard it
> > > 
> > > The truncation seems dangerous to me. Maybe not all the scenarios
> > > with an attestation client (client = reading logs and quoting)
> > > are possible then anymore, such as starting an attestation client
> > > only after truncation but a verifier must have witnessed the
> > > system's PCRs and log state before the truncation occurred.
> > 
> > That's not exactly correct.  Nothing needs to have "witnessed" the
> > starting PCR value because the quote vouches for it (and can vouch
> > for it after the fact).  The only thing you need to verify the
> > quote is the attestation key and the only thing you need to do to
> > trust the attestation key is ensure it was TPM created.  All of
> > that can be verified after the fact as well.  The only thing that
> > can be done to disrupt this is to destroy the TPM (or re-own it).> 
> > Remember the assumption is you *also* have the removed log shard to
> > present.  From that the PCR state of the starting quote can be
> 
> Yes, the whole sequence of old logs needs to be available.

Yes and no.  If the person relying on the logs is happy they've
extracted all the evidentiary value from the log itself then they can
reduce the preceding log shard to simply the PCR values that match the
quote and discard the rest.

>  IF that's the case and the logs can be stitched together seamlessly,
> who then looks at the kernel AK quote and under what circumstances?

For incremental attestation.  Each log shard can be verified using the
base PCR values corresponding to the bottom quote then replayed and the
top quote verified.  This means that logs that aren't needed anymore
can be discarded, which, I recall, was the base reason for this
proposal: reducing IMA memory consumption.  Although all you need to do
is extract the shards from kernel memory to file space and free the
kernel memory.  Since that scheme can keep all logs intact, there's no
reason to further reduce them unless the filesystem is running out of
space.

James

Re: [RFC] IMA Log Snapshotting Design Proposal

[Stefan Berger]

On 8/8/23 14:26, James Bottomley wrote:
> On Tue, 2023-08-08 at 09:31 -0400, Stefan Berger wrote:
>>
>>
>> On 8/8/23 08:35, James Bottomley wrote:
>>> On Mon, 2023-08-07 at 18:49 -0400, Stefan Berger wrote:
>>>>
>>>>
>>>> On 8/1/23 17:21, James Bottomley wrote:
>>>>> On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
>>>>> [...]
>>>>>> Truncating IMA log to reclaim memory is not feasible, since
>>>>>> it makes the log go out of sync with the TPM PCR quote making
>>>>>> remote attestation fail.
>>>>>
>>>>> This assumption isn't entirely true.  It's perfectly possible
>>>>> to shard an IMA log using two TPM2_Quote's for the beginning
>>>>> and end PCR values to validate the shard.  The IMA log could be
>>>>> truncated in the same way (replace the removed part of the log
>>>>> with a TPM2_Quote and AK, so the log still validates from the
>>>>> beginning quote to the end).
>>>>>
>>>>> If you use a TPM2_Quote mechanism to save the log, all you need
>>>>> to do is have the kernel generate the quote with an internal
>>>>> AK.  You can keep a record of the quote and the AK at the
>>>>> beginning of the truncated kernel log.  If the truncated
>>>>> entries are saved in a file shard it
>>>>
>>>> The truncation seems dangerous to me. Maybe not all the scenarios
>>>> with an attestation client (client = reading logs and quoting)
>>>> are possible then anymore, such as starting an attestation client
>>>> only after truncation but a verifier must have witnessed the
>>>> system's PCRs and log state before the truncation occurred.
>>>
>>> That's not exactly correct.  Nothing needs to have "witnessed" the
>>> starting PCR value because the quote vouches for it (and can vouch
>>> for it after the fact).  The only thing you need to verify the
>>> quote is the attestation key and the only thing you need to do to
>>> trust the attestation key is ensure it was TPM created.  All of
>>> that can be verified after the fact as well.  The only thing that
>>> can be done to disrupt this is to destroy the TPM (or re-own it).>
>>> Remember the assumption is you *also* have the removed log shard to
>>> present.  From that the PCR state of the starting quote can be
>>
>> Yes, the whole sequence of old logs needs to be available.
> 
> Yes and no.  If the person relying on the logs is happy they've
> extracted all the evidentiary value from the log itself then they can
> reduce the preceding log shard to simply the PCR values that match the
> quote and discard the rest.
> 
>>   IF that's the case and the logs can be stitched together seamlessly,
>> who then looks at the kernel AK quote and under what circumstances?
> 
> For incremental attestation.  Each log shard can be verified using the
> base PCR values corresponding to the bottom quote then replayed and the


Somehow you have to tell a verifier to take a snapshot of the current state
of the PCRs when it replays the logs to be able to truncate the log. Whether
the state of the PCRs is in the log itself or it's just some sort of entry in
the log indicating a truncation probably doesn't matter for as long as the
verifying side keeps state of the PCRs at point of truncatiokn.

Also, the verifying side needs to take notice of the trustworthiness of the
system at the time the log was truncated in case the attestation client is
restarted and starts out sending the log with the first entry. The PCR state
shown at the beginning of the truncated log (when restarting the attestation
client) must then match when the 'notice' was taken and that determines its
trustworthiness at this point in the log.

That there's a kernel AK signature at this point doesn't seem necessary since one
presumably can verify the log and PCR states at the end with the 'regular' quote.
Nobody should ever trust a system by starting to look at the beginning of a
truncated log. You have to have evaluated all the entries in the log before and
determined whether the system was trustworthy. I don't think the kernel AK
quote buys much - at least not from what I can see.


> top quote verified.  This means that logs that aren't needed anymore
> can be discarded, which, I recall, was the base reason for this
> proposal: reducing IMA memory consumption.  Although all you need to do
> is extract the shards from kernel memory to file space and free the
> kernel memory.  Since that scheme can keep all logs intact, there's no
> reason to further reduce them unless the filesystem is running out of
> space.
> 
> James
> 
> 
> _______________________________________________
> kexec mailing list
> kexec@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/kexec

Re: [RFC] IMA Log Snapshotting Design Proposal

[James Bottomley]

On Tue, 2023-08-08 at 16:09 -0400, Stefan Berger wrote:
> 
> 
> On 8/8/23 14:26, James Bottomley wrote:
> > On Tue, 2023-08-08 at 09:31 -0400, Stefan Berger wrote:
> > > 
> > > 
> > > On 8/8/23 08:35, James Bottomley wrote:
> > > > On Mon, 2023-08-07 at 18:49 -0400, Stefan Berger wrote:
> > > > > 
> > > > > 
> > > > > On 8/1/23 17:21, James Bottomley wrote:
> > > > > > On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale
> > > > > > wrote:
> > > > > > [...]
> > > > > > > Truncating IMA log to reclaim memory is not feasible,
> > > > > > > since
> > > > > > > it makes the log go out of sync with the TPM PCR quote
> > > > > > > making
> > > > > > > remote attestation fail.
> > > > > > 
> > > > > > This assumption isn't entirely true.  It's perfectly
> > > > > > possible
> > > > > > to shard an IMA log using two TPM2_Quote's for the
> > > > > > beginning
> > > > > > and end PCR values to validate the shard.  The IMA log
> > > > > > could be
> > > > > > truncated in the same way (replace the removed part of the
> > > > > > log
> > > > > > with a TPM2_Quote and AK, so the log still validates from
> > > > > > the
> > > > > > beginning quote to the end).
> > > > > > 
> > > > > > If you use a TPM2_Quote mechanism to save the log, all you
> > > > > > need
> > > > > > to do is have the kernel generate the quote with an
> > > > > > internal
> > > > > > AK.  You can keep a record of the quote and the AK at the
> > > > > > beginning of the truncated kernel log.  If the truncated
> > > > > > entries are saved in a file shard it
> > > > > 
> > > > > The truncation seems dangerous to me. Maybe not all the
> > > > > scenarios
> > > > > with an attestation client (client = reading logs and
> > > > > quoting)
> > > > > are possible then anymore, such as starting an attestation
> > > > > client
> > > > > only after truncation but a verifier must have witnessed the
> > > > > system's PCRs and log state before the truncation occurred.
> > > > 
> > > > That's not exactly correct.  Nothing needs to have "witnessed"
> > > > the
> > > > starting PCR value because the quote vouches for it (and can
> > > > vouch
> > > > for it after the fact).  The only thing you need to verify the
> > > > quote is the attestation key and the only thing you need to do
> > > > to
> > > > trust the attestation key is ensure it was TPM created.  All of
> > > > that can be verified after the fact as well.  The only thing
> > > > that
> > > > can be done to disrupt this is to destroy the TPM (or re-own
> > > > it).>
> > > > Remember the assumption is you *also* have the removed log
> > > > shard to
> > > > present.  From that the PCR state of the starting quote can be
> > > 
> > > Yes, the whole sequence of old logs needs to be available.
> > 
> > Yes and no.  If the person relying on the logs is happy they've
> > extracted all the evidentiary value from the log itself then they
> > can
> > reduce the preceding log shard to simply the PCR values that match
> > the
> > quote and discard the rest.
> > 
> > >   IF that's the case and the logs can be stitched together
> > > seamlessly, who then looks at the kernel AK quote and under what
> > > circumstances?
> > 
> > For incremental attestation.  Each log shard can be verified using
> > the base PCR values corresponding to the bottom quote then replayed
> > and the
> 
> 
> Somehow you have to tell a verifier to take a snapshot of the current
> state of the PCRs when it replays the logs to be able to truncate the
> log.

No, the verifier is server side.  It would be the agent or the kernel,
client side, which gets the quote and shards the log.  That way the
operation can be done in such a way as to make sure the quote and the
shard point match.  I'd imagine the verifier can provide some sort of
guide as to how big it wants the shards to be and the client complies.

> Whether the state of the PCRs is in the log itself or it's just some
> sort of entry in the log indicating a truncation probably doesn't
> matter for as long as the verifying side keeps state of the PCRs at
> point of truncatiokn.

The idea would be the log shard would be self attesting, that's why
quote at beginning and end (and probably PCR state at beginning), so no
verifier required until someone wants to check the log.

> Also, the verifying side needs to take notice of the trustworthiness
> of the system at the time the log was truncated in case the
> attestation client is restarted and starts out sending the log with
> the first entry.

That's usually what a runtime verification system is actually
verifying, yes.

>  The PCR state shown at the beginning of the truncated log (when
> restarting the attestation client) must then match when the 'notice'
> was taken and that determines its trustworthiness at this point in
> the log.
> 
> That there's a kernel AK signature

A quote is a signature over PCR state signed by an AK, yes.

>  at this point doesn't seem necessary since one presumably can verify
> the log and PCR states at the end with the 'regular' quote.

I don't understand this.  A regular quote is a signature over PCR state
by an AK.  The point about saving the AK in the log for the original is
that if the *kernel* truncates the log and saves it to a file, it needs
to generate both the AK and the quote for the top of the file shard. 
That means the AK/EK binding is unverified, but can be verified by
loading the AK and running the usual tests, which can only be done if
you have the loadable AK, which is why you need it as part of the log
saving proposal.

> Nobody should ever trust a system by starting to look at the
> beginning of a truncated log. You have to have evaluated all the
> entries in the log before and determined whether the system was
> trustworthy. I don't think the kernel AK quote buys much - at least
> not from what I can see.

You have a log shard with PCR state at the beginning and calculated at
the end, both of which you can verify.  How you establish trust in the
starting PCR values of the shard is a policy decision, but the policy
doesn't have to be you see every shard up to boot.  You could take the
word of the system owner on a handoff of responsibility for instance.

James

Re: [RFC] IMA Log Snapshotting Design Proposal

[Tushar Sugandhi]

On 8/8/23 14:41, James Bottomley wrote:
> On Tue, 2023-08-08 at 16:09 -0400, Stefan Berger wrote:
>>
>> On 8/8/23 14:26, James Bottomley wrote:
>>> On Tue, 2023-08-08 at 09:31 -0400, Stefan Berger wrote:
>>>>
>>>> On 8/8/23 08:35, James Bottomley wrote:
>>>>> On Mon, 2023-08-07 at 18:49 -0400, Stefan Berger wrote:
>>>>>>
>>>>>> On 8/1/23 17:21, James Bottomley wrote:
>>>>>>> On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale
>>>>>>> wrote:
>>>>>>> [...]
>>>>>>>> Truncating IMA log to reclaim memory is not feasible,
>>>>>>>> since
>>>>>>>> it makes the log go out of sync with the TPM PCR quote
>>>>>>>> making
>>>>>>>> remote attestation fail.
>>>>>>> This assumption isn't entirely true.  It's perfectly
>>>>>>> possible
>>>>>>> to shard an IMA log using two TPM2_Quote's for the
>>>>>>> beginning
>>>>>>> and end PCR values to validate the shard.  The IMA log
>>>>>>> could be
>>>>>>> truncated in the same way (replace the removed part of the
>>>>>>> log
>>>>>>> with a TPM2_Quote and AK, so the log still validates from
>>>>>>> the
>>>>>>> beginning quote to the end).
>>>>>>>
>>>>>>> If you use a TPM2_Quote mechanism to save the log, all you
>>>>>>> need
>>>>>>> to do is have the kernel generate the quote with an
>>>>>>> internal
>>>>>>> AK.  You can keep a record of the quote and the AK at the
>>>>>>> beginning of the truncated kernel log.  If the truncated
>>>>>>> entries are saved in a file shard it
>>>>>> The truncation seems dangerous to me. Maybe not all the
>>>>>> scenarios
>>>>>> with an attestation client (client = reading logs and
>>>>>> quoting)
>>>>>> are possible then anymore, such as starting an attestation
>>>>>> client
>>>>>> only after truncation but a verifier must have witnessed the
>>>>>> system's PCRs and log state before the truncation occurred.
>>>>> That's not exactly correct.  Nothing needs to have "witnessed"
>>>>> the
>>>>> starting PCR value because the quote vouches for it (and can
>>>>> vouch
>>>>> for it after the fact).  The only thing you need to verify the
>>>>> quote is the attestation key and the only thing you need to do
>>>>> to
>>>>> trust the attestation key is ensure it was TPM created.  All of
>>>>> that can be verified after the fact as well.  The only thing
>>>>> that
>>>>> can be done to disrupt this is to destroy the TPM (or re-own
>>>>> it).>
>>>>> Remember the assumption is you *also* have the removed log
>>>>> shard to
>>>>> present.  From that the PCR state of the starting quote can be
>>>> Yes, the whole sequence of old logs needs to be available.
>>> Yes and no.  If the person relying on the logs is happy they've
>>> extracted all the evidentiary value from the log itself then they
>>> can
>>> reduce the preceding log shard to simply the PCR values that match
>>> the
>>> quote and discard the rest.
>>>
>>>>    IF that's the case and the logs can be stitched together
>>>> seamlessly, who then looks at the kernel AK quote and under what
>>>> circumstances?
>>> For incremental attestation.  Each log shard can be verified using
>>> the base PCR values corresponding to the bottom quote then replayed
>>> and the
>>
>> Somehow you have to tell a verifier to take a snapshot of the current
>> state of the PCRs when it replays the logs to be able to truncate the
>> log.
> No, the verifier is server side.  It would be the agent or the kernel,
> client side, which gets the quote and shards the log.  That way the
> operation can be done in such a way as to make sure the quote and the
> shard point match.  I'd imagine the verifier can provide some sort of
> guide as to how big it wants the shards to be and the client complies.
Yup. Agreed.
>> Whether the state of the PCRs is in the log itself or it's just some
>> sort of entry in the log indicating a truncation probably doesn't
>> matter for as long as the verifying side keeps state of the PCRs at
>> point of truncatiokn.
> The idea would be the log shard would be self attesting, that's why
> quote at beginning and end (and probably PCR state at beginning), so no
> verifier required until someone wants to check the log.
>
>> Also, the verifying side needs to take notice of the trustworthiness
>> of the system at the time the log was truncated in case the
>> attestation client is restarted and starts out sending the log with
>> the first entry.
> That's usually what a runtime verification system is actually
> verifying, yes.
>
>>   The PCR state shown at the beginning of the truncated log (when
>> restarting the attestation client) must then match when the 'notice'
>> was taken and that determines its trustworthiness at this point in
>> the log.
>>
>> That there's a kernel AK signature
> A quote is a signature over PCR state signed by an AK, yes.
>
>>   at this point doesn't seem necessary since one presumably can verify
>> the log and PCR states at the end with the 'regular' quote.
> I don't understand this.  A regular quote is a signature over PCR state
> by an AK.  The point about saving the AK in the log for the original is
> that if the *kernel* truncates the log and saves it to a file, it needs
> to generate both the AK and the quote for the top of the file shard.
> That means the AK/EK binding is unverified, but can be verified by
> loading the AK and running the usual tests, which can only be done if
> you have the loadable AK, which is why you need it as part of the log
> saving proposal.
I had this question about the usability of AK/EK in this
context. Although AK/EK + PCR quote is needed to verify the snapshot
shards / IMA logs are not tampered with, I am still not sure why AK/EK
needs to be part of the shard/IMA log. The client sending AK/EK to
attestation service separately would still serve the purpose, right?

For instance, PCR quotes will be signed by AK. So as long as the verifier
trusts the AK/EK, it can verify the quotes are not tampered with.
Replaying IMA log/snapshot can produce the PCR quotes which can be
matched with signed PCR quotes. If they match, then the verifier can
conclude that the IMA log is not tampered with. So AK doesn't need to
be part of the log/snapshot.

BTW, in this proposal, kernel is truncating the log and passing the
truncated buffer to UM.  UM client need to save it to the disk location
of it's choice.


>> Nobody should ever trust a system by starting to look at the
>> beginning of a truncated log. You have to have evaluated all the
>> entries in the log before and determined whether the system was
>> trustworthy. I don't think the kernel AK quote buys much - at least
>> not from what I can see.
> You have a log shard with PCR state at the beginning and calculated at
> the end, both of which you can verify.  How you establish trust in the
> starting PCR values of the shard is a policy decision, but the policy
> doesn't have to be you see every shard up to boot.  You could take the
> word of the system owner on a handoff of responsibility for instance.
Agreed. You don’t have to go all the way to the boot snapshot
(shard).  It is up to the attestation service which last-known-good PCR
quotes (sometimes referred as golden PCR quotes) it trusts, and verify
the snapshots / logs after that.

~Tushar

> James

Re: [RFC] IMA Log Snapshotting Design Proposal

[James Bottomley]

On Wed, 2023-08-09 at 21:43 -0700, Tushar Sugandhi wrote:
> On 8/8/23 14:41, James Bottomley wrote:
> > On Tue, 2023-08-08 at 16:09 -0400, Stefan Berger wrote:
[...]
> > >   at this point doesn't seem necessary since one presumably can
> > > verify the log and PCR states at the end with the 'regular'
> > > quote.
> >  
> > I don't understand this.  A regular quote is a signature over PCR
> > state by an AK.  The point about saving the AK in the log for the
> > original is that if the *kernel* truncates the log and saves it to
> > a file, it needs to generate both the AK and the quote for the top
> > of the file shard. That means the AK/EK binding is unverified, but
> > can be verified by loading the AK and running the usual tests,
> > which can only be done if you have the loadable AK, which is why
> > you need it as part of the log saving proposal.
>  
> I had this question about the usability of AK/EK in this
> context. Although AK/EK + PCR quote is needed to verify the snapshot
> shards / IMA logs are not tampered with, I am still not sure why
> AK/EK needs to be part of the shard/IMA log. The client sending AK/EK
> to attestation service separately would still serve the purpose,
> right?

Well, the EK doesn't need to be part of the log: it's just a permanent
part of the TPM identity.  To verify the log, you need access to the
TPM that was used to create it, so that's the point at which you get
the EK.

An AK is simply a TPM generated signing key (meaning the private part
of the key is secured by the TPM and known to no-one else).  In the
literature a TPM generated signing key doesn't become an Attestation
Key until it's been verified using an EK property (either a certify for
a signing EK or a make/activate credential round trip for the more
usual encryption EK.

So the proposal is for each quote that's used to verify a log shard is
that the TPM simply generate a random signing key and use that to sign
the quote.  You need to save the TPM form of the generated key so it
can be loaded later and the reason for that is you can do the EK
verification at any time after the quote was given by loading the saved
key and running the verification protocol.  In the normal attestation
you do the EK verification of the AK *before* the quote, but there's no
property of the quote that depends on this precedence provided you do
the quote with a TPM generated signing key.

The underlying point is that the usual way an EK verifies an AK
requires a remote observer, which the kernel won't have, so the kernel
must do all its stuff locally (generate key, get quote) and then at
some point later the system can become remote connected and prove to
whatever external entity that the log shard is valid.  So we have to
have all the components necessary for that proof: the log shard, the
quote and the TPM form of the AK.

> For instance, PCR quotes will be signed by AK. So as long as the
> verifier trusts the AK/EK,

Right, but if you're sharding a log, the kernel doesn't know if a
verifier has been in contact yet.  The point of the protocol above is
to make that not matter.  The verifier can contact the system after the
log has been saved and the verification will still work.

>  it can verify the quotes are not tampered with.
> Replaying IMA log/snapshot can produce the PCR quotes which can be
> matched with signed PCR quotes. If they match, then the verifier can
> conclude that the IMA log is not tampered with. So AK doesn't need to
> be part of the log/snapshot.

Only if the system is currently in contact with the verifier and the
verifier has created the AK.  That may not have happened.

> BTW, in this proposal, kernel is truncating the log and passing the
> truncated buffer to UM.  UM client need to save it to the disk
> location of it's choice.

Yes, but I was assuming tampering with or discarding the log file would
be treated in exactly the same way as an in-kernel IMA log tamper.

James

Re: [RFC] IMA Log Snapshotting Design Proposal

[Tushar Sugandhi]

On 8/10/23 04:43, James Bottomley wrote:
> On Wed, 2023-08-09 at 21:43 -0700, Tushar Sugandhi wrote:
>> On 8/8/23 14:41, James Bottomley wrote:
>>> On Tue, 2023-08-08 at 16:09 -0400, Stefan Berger wrote:
> [...]
>>>>    at this point doesn't seem necessary since one presumably can
>>>> verify the log and PCR states at the end with the 'regular'
>>>> quote.
>>>   
>>> I don't understand this.  A regular quote is a signature over PCR
>>> state by an AK.  The point about saving the AK in the log for the
>>> original is that if the *kernel* truncates the log and saves it to
>>> a file, it needs to generate both the AK and the quote for the top
>>> of the file shard. That means the AK/EK binding is unverified, but
>>> can be verified by loading the AK and running the usual tests,
>>> which can only be done if you have the loadable AK, which is why
>>> you need it as part of the log saving proposal.
>>   
>> I had this question about the usability of AK/EK in this
>> context. Although AK/EK + PCR quote is needed to verify the snapshot
>> shards / IMA logs are not tampered with, I am still not sure why
>> AK/EK needs to be part of the shard/IMA log. The client sending AK/EK
>> to attestation service separately would still serve the purpose,
>> right?
> 
> Well, the EK doesn't need to be part of the log: it's just a permanent
> part of the TPM identity.  To verify the log, you need access to the
> TPM that was used to create it, so that's the point at which you get
> the EK.
> 
Agreed. EK is part of TPM identity. But to verify the log,
you don’t need to have physical access to the TPM. You need to have
access to just public part of EK and AK/AIK certs (TPM on the system
would sign the quote using the private AK).
I believe you already know this, just stating for the sake of
completing the conversation. :)
> An AK is simply a TPM generated signing key (meaning the private part
> of the key is secured by the TPM and known to no-one else).  In the
> literature a TPM generated signing key doesn't become an Attestation
> Key until it's been verified using an EK property (either a certify for
> a signing EK or a make/activate credential round trip for the more
> usual encryption EK.
> 
Yes. That aligns with my understanding of EK/AK in general.
Thanks for describing.

> So the proposal is for each quote that's used to verify a log shard is
> that the TPM simply generate a random signing key and use that to sign
I believe you are suggesting creating a new AK each time you
want to sign a PCR quote. It is doable in TPM 2.0, and it provides
benefits like privacy and untraceability. But it comes with it’s own
costs – cost of generating new AK each time you want to sign,
maintaining mapping of AK and it’s signed quotes, maintaining
multiple public AK certs etc.

> the quote.  You need to save the TPM form of the generated key so it
> can be loaded later and the reason for that is you can do the EK
> verification at any time after the quote was given by loading the saved
> key and running the verification protocol.  In the normal attestation
> you do the EK verification of the AK *before* the quote, but there's no
> property of the quote that depends on this precedence provided you do
> the quote with a TPM generated signing key.
Yes.

> 
> The underlying point is that the usual way an EK verifies an AK
> requires a remote observer, which the kernel won't have, so the kernel
Agreed.

> must do all its stuff locally (generate key, get quote) and then at
I believe the Kernel doesn’t have to generate key while
taking the snapshot. In the current proposal, Kernel can simply get
the (unsigned) PCR quote and log it in IMA log as part of the
snapshot_aggregate event. We don’t need to sign the quote while
logging it in the IMA log as snapshot_aggregate. And the act of
logging that event in IMA log extends the PCR bank. Sometime later,
when a remote observer wants to validate the log – it can do it by
comparing against the PCR quote that was signed at that point.

> some point later the system can become remote connected and prove to
> whatever external entity that the log shard is valid.  So we have to
> have all the components necessary for that proof: the log shard, the
> quote and the TPM form of the AK.
> 
>> For instance, PCR quotes will be signed by AK. So as long as the
>> verifier trusts the AK/EK,
> 
> Right, but if you're sharding a log, the kernel doesn't know if a
> verifier has been in contact yet.  The point of the protocol above is
> to make that not matter.  The verifier can contact the system after the
> log has been saved and the verification will still work.
> 
The Kernel doesn’t need to know. And it still doesn’t matter.
The benefit of our approach is the PCR values that represent the
previous snapshot(shard) is now logged in the IMA log as
snapshot_aggregate, and the PCRs are extended again as part of
logging that event in IMA log.

>>   it can verify the quotes are not tampered with.
>> Replaying IMA log/snapshot can produce the PCR quotes which can be
>> matched with signed PCR quotes. If they match, then the verifier can
>> conclude that the IMA log is not tampered with. So AK doesn't need to
>> be part of the log/snapshot.
> 
> Only if the system is currently in contact with the verifier and the
> verifier has created the AK.  That may not have happened.
> 
Hope my above explanation addresses this point.

>> BTW, in this proposal, kernel is truncating the log and passing the
>> truncated buffer to UM.  UM client need to save it to the disk
>> location of it's choice.
> 
> Yes, but I was assuming tampering with or discarding the log file would
> be treated in exactly the same way as an in-kernel IMA log tamper.
> 
Hope my above explanation addresses this point.

~Tushar

> James

Re: [RFC] IMA Log Snapshotting Design Proposal

[Tushar Sugandhi]

On 8/8/23 11:26, James Bottomley wrote:
> On Tue, 2023-08-08 at 09:31 -0400, Stefan Berger wrote:
>>
>> On 8/8/23 08:35, James Bottomley wrote:
>>> On Mon, 2023-08-07 at 18:49 -0400, Stefan Berger wrote:
>>>>
>>>> On 8/1/23 17:21, James Bottomley wrote:
>>>>> On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
>>>>> [...]
>>>>>> Truncating IMA log to reclaim memory is not feasible, since
>>>>>> it makes the log go out of sync with the TPM PCR quote making
>>>>>> remote attestation fail.
>>>>> This assumption isn't entirely true.  It's perfectly possible
>>>>> to shard an IMA log using two TPM2_Quote's for the beginning
>>>>> and end PCR values to validate the shard.  The IMA log could be
>>>>> truncated in the same way (replace the removed part of the log
>>>>> with a TPM2_Quote and AK, so the log still validates from the
>>>>> beginning quote to the end).
>>>>>
>>>>> If you use a TPM2_Quote mechanism to save the log, all you need
>>>>> to do is have the kernel generate the quote with an internal
>>>>> AK.  You can keep a record of the quote and the AK at the
>>>>> beginning of the truncated kernel log.  If the truncated
>>>>> entries are saved in a file shard it
>>>> The truncation seems dangerous to me. Maybe not all the scenarios
>>>> with an attestation client (client = reading logs and quoting)
>>>> are possible then anymore, such as starting an attestation client
>>>> only after truncation but a verifier must have witnessed the
>>>> system's PCRs and log state before the truncation occurred.
>>> That's not exactly correct.  Nothing needs to have "witnessed" the
>>> starting PCR value because the quote vouches for it (and can vouch
>>> for it after the fact).  The only thing you need to verify the
>>> quote is the attestation key and the only thing you need to do to
>>> trust the attestation key is ensure it was TPM created.  All of
>>> that can be verified after the fact as well.  The only thing that
>>> can be done to disrupt this is to destroy the TPM (or re-own it).>
>>> Remember the assumption is you *also* have the removed log shard to
>>> present.  From that the PCR state of the starting quote can be
>> Yes, the whole sequence of old logs needs to be available.
> Yes and no.  If the person relying on the logs is happy they've
> extracted all the evidentiary value from the log itself then they can
> reduce the preceding log shard to simply the PCR values that match the
> quote and discard the rest.
>
>>   IF that's the case and the logs can be stitched together seamlessly,
>> who then looks at the kernel AK quote and under what circumstances?
> For incremental attestation.  Each log shard can be verified using the
> base PCR values corresponding to the bottom quote then replayed and the
> top quote verified.  This means that logs that aren't needed anymore
> can be discarded, which, I recall, was the base reason for this
> proposal: reducing IMA memory consumption.  Although all you need to do
> is extract the shards from kernel memory to file space and free the
> kernel memory.  Since that scheme can keep all logs intact, there's no
> reason to further reduce them unless the filesystem is running out of
> space.
>
> James
Thank you James for addressing Stefan’s concerns here.
Appreciate it.


~Tushar

Re: [RFC] IMA Log Snapshotting Design Proposal

[Tushar Sugandhi]

On 8/8/23 06:31, Stefan Berger wrote:
>
>
> On 8/8/23 08:35, James Bottomley wrote:
>> On Mon, 2023-08-07 at 18:49 -0400, Stefan Berger wrote:
>>>
>>>
>>> On 8/1/23 17:21, James Bottomley wrote:
>>>> On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
>>>> [...]
>>>>> Truncating IMA log to reclaim memory is not feasible, since it
>>>>> makes the log go out of sync with the TPM PCR quote making remote
>>>>> attestation fail.
>>>>
>>>> This assumption isn't entirely true.  It's perfectly possible to
>>>> shard an IMA log using two TPM2_Quote's for the beginning and end
>>>> PCR values to validate the shard.  The IMA log could be truncated
>>>> in the same way (replace the removed part of the log with a
>>>> TPM2_Quote and AK, so the log still validates from the beginning
>>>> quote to the end).
>>>>
>>>> If you use a TPM2_Quote mechanism to save the log, all you need to
>>>> do is have the kernel generate the quote with an internal AK.  You
>>>> can keep a record of the quote and the AK at the beginning of the
>>>> truncated kernel log.  If the truncated entries are saved in a file
>>>> shard it
>>>
>>> The truncation seems dangerous to me. Maybe not all the scenarios
>>> with an attestation client (client = reading logs and quoting) are
>>> possible then anymore, such as starting an attestation client only
>>> after truncation but a verifier must have witnessed the system's PCRs
>>> and log state before the truncation occurred.
>>
>> That's not exactly correct.  Nothing needs to have "witnessed" the
>> starting PCR value because the quote vouches for it (and can vouch for
>> it after the fact).  The only thing you need to verify the quote is the
>> attestation key and the only thing you need to do to trust the> 
>> attestation key is ensure it was TPM created.  All of that can be
>> verified after the fact as well.  The only thing that can be done to
>> disrupt this is to destroy the TPM (or re-own it).> Remember the 
>> assumption is you *also* have the removed log shard to
>> present.  From that the PCR state of the starting quote can be
>
> Yes, the whole sequence of old logs needs to be available. IF that's the
> case and the logs can be stitched together seamlessly, who then looks 
> at the
> kernel AK quote and under what circumstances?
>
James has addressed this question in his response to this message [1].

Thanks James.

[1] 
https://lore.kernel.org/all/350ecdcbf7796f488807fcd7983414a02dd71be4.camel@HansenPartnership.com/#r 


~Tushar

>> calculated and checked for matching the quote.  If you lose that, it's
>> equivalent to the log being tampered with and all bets are off. The
>> assumption is that because of the impossibility of engineering TPM
>> extensions, it should be impossible to come up with a fake log that
>> produces the PCRs of the real one.  If that's violated, then IMA itself
>> becomes useless.
>>
>> James
>>
>>
>> _______________________________________________
>> kexec mailing list
>> kexec@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/kexec

Re: [RFC] IMA Log Snapshotting Design Proposal

[Tushar Sugandhi]

hi James,
Thanks for addressing Stefan's concerns.

On 8/8/23 05:35, James Bottomley wrote:
> On Mon, 2023-08-07 at 18:49 -0400, Stefan Berger wrote:
>>
>> On 8/1/23 17:21, James Bottomley wrote:
>>> On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
>>> [...]
>>>> Truncating IMA log to reclaim memory is not feasible, since it
>>>> makes the log go out of sync with the TPM PCR quote making remote
>>>> attestation fail.
>>> This assumption isn't entirely true.  It's perfectly possible to
>>> shard an IMA log using two TPM2_Quote's for the beginning and end
>>> PCR values to validate the shard.  The IMA log could be truncated
>>> in the same way (replace the removed part of the log with a
>>> TPM2_Quote and AK, so the log still validates from the beginning
>>> quote to the end).
>>>
>>> If you use a TPM2_Quote mechanism to save the log, all you need to
>>> do is have the kernel generate the quote with an internal AK.  You
>>> can keep a record of the quote and the AK at the beginning of the
>>> truncated kernel log.  If the truncated entries are saved in a file
>>> shard it
>> The truncation seems dangerous to me. Maybe not all the scenarios
>> with an attestation client (client = reading logs and quoting) are
>> possible then anymore, such as starting an attestation client only
>> after truncation but a verifier must have witnessed the system's PCRs
>> and log state before the truncation occurred.
> That's not exactly correct.  Nothing needs to have "witnessed" the
> starting PCR value because the quote vouches for it (and can vouch for
> it after the fact).  The only thing you need to verify the quote is the
> attestation key and the only thing you need to do to trust the
> attestation key is ensure it was TPM created.  All of that can be
> verified after the fact as well.  The only thing that can be done to
> disrupt this is to destroy the TPM (or re-own it).
Agreed.
>
> Remember the assumption is you *also* have the removed log shard to
> present.  From that the PCR state of the starting quote can be
> calculated and checked for matching the quote.  If you lose that, it's
> equivalent to the log being tampered with and all bets are off.  The
> assumption is that because of the impossibility of engineering TPM
> extensions, it should be impossible to come up with a fake log that
> produces the PCRs of the real one.  If that's violated, then IMA itself
> becomes useless.
Totally agreed. I believe our proposal is consistent with your
comments. Please let us know if you see any inconsistencies or technical
blockers.

~Tushar

>
> James

Re: [RFC] IMA Log Snapshotting Design Proposal

[Tushar Sugandhi]

Thanks a lot Stefan for looking into this proposal,
and providing your feedback. We really appreciate it.

On 8/7/23 15:49, Stefan Berger wrote:
>
>
> On 8/1/23 17:21, James Bottomley wrote:
>> On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
>> [...]
>>> Truncating IMA log to reclaim memory is not feasible, since it makes
>>> the log go out of sync with the TPM PCR quote making remote
>>> attestation fail.
>>
>> This assumption isn't entirely true.  It's perfectly possible to shard
>> an IMA log using two TPM2_Quote's for the beginning and end PCR values
>> to validate the shard.  The IMA log could be truncated in the same way
>> (replace the removed part of the log with a TPM2_Quote and AK, so the
>> log still validates from the beginning quote to the end).
>>
>> If you use a TPM2_Quote mechanism to save the log, all you need to do
>> is have the kernel generate the quote with an internal AK.  You can
>> keep a record of the quote and the AK at the beginning of the truncated
>> kernel log.  If the truncated entries are saved in a file shard it
>
> The truncation seems dangerous to me. Maybe not all the scenarios with 
> an attestation
> client (client = reading logs and quoting) are possible then anymore, 
> such as starting
> an attestation client only after truncation but a verifier must have 
> witnessed the
> system's PCRs and log state before the truncation occurred.
You are correct that truncation on it’s own is dangerous. It needs to be
accompanied by (a) saving the IMA log data to disk as snapshots, (b) 
adding the
necessary TPM PCR quotes to the current IMA log (as James mentioned above),
(c) attestation clients having an ability to send the past snapshots to the
remote-attestation-service (verifiers), (d) and verifiers having an ability
to use the snapshots along with current IMA logs for the purpose of 
attestation.
All these points are explained in the original RFC email in sections B.1 
through B.5 [1].
>
> I think an ima-buf (or similar) log entry in IMA log would have to 
> appear at the beginning of the
> truncated log stating the value of all PCRs that IMA touched 
> (typically only PCR 10
> but it can be others). The needs to be done since the quote itself 
> doesn't
> provide the state of the individual PCRs. This would at least allow an 
> attestation
> client to re-read the log from the beginning (when it is re-start or 
> started for the
> first time after the truncation). 
  Agreed. See the description of snapshot_aggregate in Section B.5 in the
original RFC email [1].
> However, this alone (without the
> internal AK quoting the old state) could lead to abuse where I could 
> create totally
> fake IMA logs stating the state of the PCRs at the beginning (so the 
> verifier
> syncs its internal PCR state to this state). 
Yes, the PCR quotes sent to the verifier must be signed by the AK that
is trusted by the verifier. That assumption is true regardless of IMA log
snapshotting feature.
> Further, even with the AK-quote that
> you propose I may be able to create fake logs and trick a verifier into
> trusting the machine IFF it doesn't know what kernel this system was 
> booted with
> that I may have hacked to provide a fake AK-quote that just happens to 
> match the
> PCR state presented at the beginning of the log.
>
If the Kernel is compromised, then all-bets are off.
(Regardless of IMA log snapshotting feature.)
> => Can a truncated log be made safe for attestation when the 
> attestation starts
> only after the truncation occurred?
>
Yes. If the “PCR quotes in the snapshot_aggregate event in IMA log”
+ "replay of rest of the events in IMA log" results in the “final PCR 
quotes”
that matches with the “AK signed PCR quotes” sent by the client, then 
the truncated
IMA log can be trusted. The verifier can either ‘trust’ the “PCR quotes 
in the
snapshot_aggregate event in IMA log” or it can ask for the (n-1)th 
snapshot shard
to check the past events.

> => Even if attestation was occurring 'what' state does an attestation 
> server
> need to carry around for an attested-to system so that the truncation 
> is 'safe'
> and I cannot create fake AK-quotes and fake IMA logs with initial PCR 
> states?
Assuming most of the client devices take a snapshot at specific checkpoints,
the “PCR quotes in the snapshot_aggregate event in IMA log” will be the 
same for them.
The remote attestation server will have to remember these golden PCR quotes.
It doesn't have to remember the state of each client device.

> Can I ever restart the client and have it read the truncated log from the
> beginning and what type of verification needs to happen on the server 
> then?
>
Yes, restarting the client should be possible.
> It seems like the server would have to remember the state of the IMA 
> PCRs upon
> last truncation to detect a possible attack. This would make staring 
> to monitor
> a system after truncation impossible -- would be good to know these 
> details.
>
The server is not forced to remember the state of IMA PCRs. It can
always ask for the last n snapshot files (shards) and replay the events. 
Even
though the data is truncated from the IMA log, it is not totally lost. It is
simply being transferred to the disk. It is saved by UM as snapshot 
files/shards.
The goal of IMA snapshotting is to reduce the Kernel memory pressure on the
client devices - to save them from out-of-memory errors which are harder 
to manage
on long running clients. It comes with a cost of additional work on the 
server
side to attest those clients.


Being said that, in the current proposal, taking a snapshots is totally 
optional
and controlled by UM attestation clients. If the 
attestation-clients/services are
not-ready/don’t-want to take advantage of IMA log snapshotting, they 
don’t have to.

No snapshot will be taken, and the client-service can process the 
monolithic IMA
log just like they do today.

[1] 
https://lore.kernel.org/all/c5737141-7827-1c83-ab38-0119dcfea485@linux.microsoft.com/#t 


>
>
>
>> should have a beginning and end quote and a record of the AK used.
>> Since verifiers like Keylime are already using this beginning and end
>> quote for sharded logs, it's the most natural format to feed to
>> something externally for verification and it means you don't have to
>> invent a new format to do the same thing.
>>
>> Regards,
>>
>> James
>>
>>
>> _______________________________________________
>> kexec mailing list
>> kexec@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/kexec

Re: [RFC] IMA Log Snapshotting Design Proposal

[Stefan Berger]

On 8/9/23 21:15, Tushar Sugandhi wrote:
> Thanks a lot Stefan for looking into this proposal,
> and providing your feedback. We really appreciate it.
> 
> On 8/7/23 15:49, Stefan Berger wrote:
>>
>>
>> On 8/1/23 17:21, James Bottomley wrote:
>>> On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
>>> [...]
>>>> Truncating IMA log to reclaim memory is not feasible, since it makes
>>>> the log go out of sync with the TPM PCR quote making remote
>>>> attestation fail.
>>>
>>> This assumption isn't entirely true.  It's perfectly possible to shard
>>> an IMA log using two TPM2_Quote's for the beginning and end PCR values
>>> to validate the shard.  The IMA log could be truncated in the same way
>>> (replace the removed part of the log with a TPM2_Quote and AK, so the
>>> log still validates from the beginning quote to the end).
>>>
>>> If you use a TPM2_Quote mechanism to save the log, all you need to do
>>> is have the kernel generate the quote with an internal AK.  You can
>>> keep a record of the quote and the AK at the beginning of the truncated
>>> kernel log.  If the truncated entries are saved in a file shard it
>>
>> The truncation seems dangerous to me. Maybe not all the scenarios with an attestation
>> client (client = reading logs and quoting) are possible then anymore, such as starting
>> an attestation client only after truncation but a verifier must have witnessed the
>> system's PCRs and log state before the truncation occurred.
> You are correct that truncation on it’s own is dangerous. It needs to be
> accompanied by (a) saving the IMA log data to disk as snapshots, (b) adding the
> necessary TPM PCR quotes to the current IMA log (as James mentioned above),
> (c) attestation clients having an ability to send the past snapshots to the
> remote-attestation-service (verifiers), (d) and verifiers having an ability
> to use the snapshots along with current IMA logs for the purpose of attestation.
> All these points are explained in the original RFC email in sections B.1 through B.5 [1].

I read it.

Maybe you have dismissed the PCR update counter already...
I am not sure what the PCR update counter is supposed to help with. It won't allow you to detect
missing log events but rather will confuse anyone looking at it when my application extends PCR 12
for example, which also affects the update counter. It's a global counter that increases with every
PCR extension (except PCR 16, 21, 22, 23) and if used as proposed would prevent any application from
extending PCRs.

https://github.com/stefanberger/libtpms/blob/master/src/tpm2/PCR.c#L667
https://github.com/stefanberger/libtpms/blob/master/src/tpm2/PCR.c#L629
https://github.com/stefanberger/libtpms/blob/master/src/tpm2/PCR.c#L161


The shards should will need to be written into some sort of standard location or a config file needs to
be defined, so that everyone knows where to find them and how they are named.


>>
>> I think an ima-buf (or similar) log entry in IMA log would have to appear at the beginning of the
>> truncated log stating the value of all PCRs that IMA touched (typically only PCR 10
>> but it can be others). The needs to be done since the quote itself doesn't
>> provide the state of the individual PCRs. This would at least allow an attestation
>> client to re-read the log from the beginning (when it is re-start or started for the
>> first time after the truncation). 
>   Agreed. See the description of snapshot_aggregate in Section B.5 in the
> original RFC email [1].
>> However, this alone (without the
>> internal AK quoting the old state) could lead to abuse where I could create totally
>> fake IMA logs stating the state of the PCRs at the beginning (so the verifier
>> syncs its internal PCR state to this state). 
> Yes, the PCR quotes sent to the verifier must be signed by the AK that
> is trusted by the verifier. That assumption is true regardless of IMA log
> snapshotting feature.
>> Further, even with the AK-quote that
>> you propose I may be able to create fake logs and trick a verifier into
>> trusting the machine IFF it doesn't know what kernel this system was booted with
>> that I may have hacked to provide a fake AK-quote that just happens to match the
>> PCR state presented at the beginning of the log.
>>
> If the Kernel is compromised, then all-bets are off.
> (Regardless of IMA log snapshotting feature.)
>> => Can a truncated log be made safe for attestation when the attestation starts
>> only after the truncation occurred?
>>
> Yes. If the “PCR quotes in the snapshot_aggregate event in IMA log”

PCR quote or 'quotes'? Why multiple?

Form your proposal but you may have changed your opinion  following what I see in other messages:
"- The Kernel will get the current TPM PCR values and PCR update counter [2]
    and store them as template data in a new IMA event "snapshot_aggregate"."

Afaik TPM quote's don't give you the state of the individual PCR values, therefore
I would expect to at least find the 'PCR values' of all the PCRs that IMA touched to
be in the snapshot_aggregate so I can replay all the following events on top of these
PCR values and come up with the values that were used in the "final PCR quote". This
is unless you expect the server to take an automatic snapshot of the values of the
PCRs  that it computed while evaluating the log in case it ever needs to go back.

> + "replay of rest of the events in IMA log" results in the “final PCR quotes”
> that matches with the “AK signed PCR quotes” sent by the client, then the truncated
> IMA log can be trusted. The verifier can either ‘trust’ the “PCR quotes in the
> snapshot_aggregate event in IMA log” or it can ask for the (n-1)th snapshot shard
> to check the past events.

For anything regarding determining the 'trustworthiness of a system' one would have to
be able to go back to the very beginning of the log *or* remember in what state a
system was when the latest snapshot was taken so that if a restart happens it can resume
with that assumption about state of trustworthiness and know what the values of the PCRs
were at that time so it can resume replaying the log (or the server would get these
values from the log).

The AK quotes by the kernel (which adds a 2nd AK key) that James is proposing
could be useful if the entire log, consisting of multiple shards, is very large and
cannot be transferred from the client to the server in one go so that the server could
evaluate the 'final PCR quote' immediately . However, if a client can indicated 'I will
send more the next time and I have this much more to transfer' and the server allows
this multiple times (until all the 1MB shards of the 20MB log are transferred) then that
kernel AK key would not be necessary since presumably the "final PCR quote", created
by a user space client, would resolve whether the entire log is trustworthy.

> 
>> => Even if attestation was occurring 'what' state does an attestation server
>> need to carry around for an attested-to system so that the truncation is 'safe'
>> and I cannot create fake AK-quotes and fake IMA logs with initial PCR states?
> Assuming most of the client devices take a snapshot at specific checkpoints,
> the “PCR quotes in the snapshot_aggregate event in IMA log” will be the same for them.
> The remote attestation server will have to remember these golden PCR quotes.

I thought maybe 'golden PCR values'... because those let me replay PCR extensions from
a previous point.

> It doesn't have to remember the state of each client device.

Can you give a reason for this? You mean the state doesn't need to be remembered for client
devices whose log hasn't been truncated?

  
>> Can I ever restart the client and have it read the truncated log from the
>> beginning and what type of verification needs to happen on the server then?
>>
> Yes, restarting the client should be possible.

Yes, this must be possible.

>> It seems like the server would have to remember the state of the IMA PCRs upon
>> last truncation to detect a possible attack. This would make staring to monitor
>> a system after truncation impossible -- would be good to know these details.
>>
> The server is not forced to remember the state of IMA PCRs. It can
> always ask for the last n snapshot files (shards) and replay the events. Even
> though the data is truncated from the IMA log, it is not totally lost. It is
> simply being transferred to the disk. It is saved by UM as snapshot files/shards.
> The goal of IMA snapshotting is to reduce the Kernel memory pressure on the
> client devices - to save them from out-of-memory errors which are harder to manage
> on long running clients. It comes with a cost of additional work on the server
> side to attest those clients.

Agreed.
> 
> 
> Being said that, in the current proposal, taking a snapshots is totally optional
> and controlled by UM attestation clients. If the attestation-clients/services are
> not-ready/don’t-want to take advantage of IMA log snapshotting, they don’t have to.

Agreed.

> 
> No snapshot will be taken, and the client-service can process the monolithic IMA
> log just like they do today.
> 

Agreed.

> [1] https://lore.kernel.org/all/c5737141-7827-1c83-ab38-0119dcfea485@linux.microsoft.com/#t
> 
>>
>>
>>
>>> should have a beginning and end quote and a record of the AK used.
>>> Since verifiers like Keylime are already using this beginning and end
>>> quote for sharded logs, it's the most natural format to feed to
>>> something externally for verification and it means you don't have to
>>> invent a new format to do the same thing.
>>>
>>> Regards,
>>>
>>> James
>>>
>>>
>>> _______________________________________________
>>> kexec mailing list
>>> kexec@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/kexec
> 

Re: [RFC] IMA Log Snapshotting Design Proposal

[Tushar Sugandhi]

On 8/10/23 07:12, Stefan Berger wrote:
> 
> 
> On 8/9/23 21:15, Tushar Sugandhi wrote:
>> Thanks a lot Stefan for looking into this proposal,
>> and providing your feedback. We really appreciate it.
>>
>> On 8/7/23 15:49, Stefan Berger wrote:
>>>
>>>
>>> On 8/1/23 17:21, James Bottomley wrote:
>>>> On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
>>>> [...]
>>>>> Truncating IMA log to reclaim memory is not feasible, since it makes
>>>>> the log go out of sync with the TPM PCR quote making remote
>>>>> attestation fail.
>>>>
>>>> This assumption isn't entirely true.  It's perfectly possible to shard
>>>> an IMA log using two TPM2_Quote's for the beginning and end PCR values
>>>> to validate the shard.  The IMA log could be truncated in the same way
>>>> (replace the removed part of the log with a TPM2_Quote and AK, so the
>>>> log still validates from the beginning quote to the end).
>>>>
>>>> If you use a TPM2_Quote mechanism to save the log, all you need to do
>>>> is have the kernel generate the quote with an internal AK.  You can
>>>> keep a record of the quote and the AK at the beginning of the truncated
>>>> kernel log.  If the truncated entries are saved in a file shard it
>>>
>>> The truncation seems dangerous to me. Maybe not all the scenarios 
>>> with an attestation
>>> client (client = reading logs and quoting) are possible then anymore, 
>>> such as starting
>>> an attestation client only after truncation but a verifier must have 
>>> witnessed the
>>> system's PCRs and log state before the truncation occurred.
>> You are correct that truncation on it’s own is dangerous. It needs to be
>> accompanied by (a) saving the IMA log data to disk as snapshots, (b) 
>> adding the
>> necessary TPM PCR quotes to the current IMA log (as James mentioned 
>> above),
>> (c) attestation clients having an ability to send the past snapshots 
>> to the
>> remote-attestation-service (verifiers), (d) and verifiers having an 
>> ability
>> to use the snapshots along with current IMA logs for the purpose of 
>> attestation.
>> All these points are explained in the original RFC email in sections 
>> B.1 through B.5 [1].
> 
> I read it.
> 
> Maybe you have dismissed the PCR update counter already...
> I am not sure what the PCR update counter is supposed to help with. It 
> won't allow you to detect
> missing log events but rather will confuse anyone looking at it when my 
> application extends PCR 12
> for example, which also affects the update counter. It's a global 
> counter that increases with every
> PCR extension (except PCR 16, 21, 22, 23) and if used as proposed would 
> prevent any application from
> extending PCRs.
> 
> https://github.com/stefanberger/libtpms/blob/master/src/tpm2/PCR.c#L667
> https://github.com/stefanberger/libtpms/blob/master/src/tpm2/PCR.c#L629
> https://github.com/stefanberger/libtpms/blob/master/src/tpm2/PCR.c#L161
> 
> 
Agree with your point about TPM PCR update counter Stefan.
I will bring it up in the update counter patch series discussion [1].

[1] 
https://patchwork.kernel.org/project/linux-integrity/cover/20230801181917.8535-1-tusharsu@linux.microsoft.com/ 


> The shards should will need to be written into some sort of standard 
> location or a config file needs to
> be defined, so that everyone knows where to find them and how they are 
> named.
> 
We thought about well known standard location earlier.
Letting the Kernel choose the name/location of the snapshot
file comes with its own complexity. Our initial stance is we don’t
want to handle that at Kernel level, and let the UM client choose
the location/naming of the snapshot files. But we are happy to
reconsider if the community requests it.
> 
>>>
>>> I think an ima-buf (or similar) log entry in IMA log would have to 
>>> appear at the beginning of the
>>> truncated log stating the value of all PCRs that IMA touched 
>>> (typically only PCR 10
>>> but it can be others). The needs to be done since the quote itself 
>>> doesn't
>>> provide the state of the individual PCRs. This would at least allow 
>>> an attestation
>>> client to re-read the log from the beginning (when it is re-start or 
>>> started for the
>>> first time after the truncation). 
>>   Agreed. See the description of snapshot_aggregate in Section B.5 in the
>> original RFC email [1].
>>> However, this alone (without the
>>> internal AK quoting the old state) could lead to abuse where I could 
>>> create totally
>>> fake IMA logs stating the state of the PCRs at the beginning (so the 
>>> verifier
>>> syncs its internal PCR state to this state). 
>> Yes, the PCR quotes sent to the verifier must be signed by the AK that
>> is trusted by the verifier. That assumption is true regardless of IMA log
>> snapshotting feature.
>>> Further, even with the AK-quote that
>>> you propose I may be able to create fake logs and trick a verifier into
>>> trusting the machine IFF it doesn't know what kernel this system was 
>>> booted with
>>> that I may have hacked to provide a fake AK-quote that just happens 
>>> to match the
>>> PCR state presented at the beginning of the log.
>>>
>> If the Kernel is compromised, then all-bets are off.
>> (Regardless of IMA log snapshotting feature.)
>>> => Can a truncated log be made safe for attestation when the 
>>> attestation starts
>>> only after the truncation occurred?
>>>
>> Yes. If the “PCR quotes in the snapshot_aggregate event in IMA log”
> 
> PCR quote or 'quotes'? Why multiple?
> 
> Form your proposal but you may have changed your opinion  following what 
> I see in other messages:
> "- The Kernel will get the current TPM PCR values and PCR update counter 
> [2]
>     and store them as template data in a new IMA event 
> "snapshot_aggregate"."
> 
> Afaik TPM quote's don't give you the state of the individual PCR values, 
> therefore
> I would expect to at least find the 'PCR values' of all the PCRs that 
> IMA touched to
> be in the snapshot_aggregate so I can replay all the following events on 
> top of these
> PCR values and come up with the values that were used in the "final PCR 
> quote". This
> is unless you expect the server to take an automatic snapshot of the 
> values of the
> PCRs  that it computed while evaluating the log in case it ever needs to 
> go back.
> 
I meant a single set of PCR values captured when snapshot_aggregate
is logged. Sorry for the confusion.

>> + "replay of rest of the events in IMA log" results in the “final PCR 
>> quotes”
>> that matches with the “AK signed PCR quotes” sent by the client, then 
>> the truncated
>> IMA log can be trusted. The verifier can either ‘trust’ the “PCR 
>> quotes in the
>> snapshot_aggregate event in IMA log” or it can ask for the (n-1)th 
>> snapshot shard
>> to check the past events.
> 
> For anything regarding determining the 'trustworthiness of a system' one 
> would have to
> be able to go back to the very beginning of the log *or* remember in 
> what state a
> system was when the latest snapshot was taken so that if a restart 
> happens it can resume
> with that assumption about state of trustworthiness and know what the 
> values of the PCRs
> were at that time so it can resume replaying the log (or the server 
> would get these
> values from the log).
> 
Correct. We intend to support the above. I hope our proposal
description captures it. BTW, when you say ‘restart’, you mean the UM
process restart, right? Because in case of a Kernel restart
(i.e. cold-boot) the past IMA log (and the TPM state) is lost,
and old snapshots (if any) are useless.

> The AK quotes by the kernel (which adds a 2nd AK key) that James is 
> proposing
> could be useful if the entire log, consisting of multiple shards, is 
> very large and
> cannot be transferred from the client to the server in one go so that 
> the server could
> evaluate the 'final PCR quote' immediately . However, if a client can 
> indicated 'I will
> send more the next time and I have this much more to transfer' and the 
> server allows
> this multiple times (until all the 1MB shards of the 20MB log are 
> transferred) then that
> kernel AK key would not be necessary since presumably the "final PCR 
> quote", created
> by a user space client, would resolve whether the entire log is 
> trustworthy.
> 
See my responses to James today [2]

[2] 
https://lore.kernel.org/all/72e39852-1ff1-c7f6-ac7e-593e8142dbe8@linux.microsoft.com/
>>
>>> => Even if attestation was occurring 'what' state does an attestation 
>>> server
>>> need to carry around for an attested-to system so that the truncation 
>>> is 'safe'
>>> and I cannot create fake AK-quotes and fake IMA logs with initial PCR 
>>> states?
>> Assuming most of the client devices take a snapshot at specific 
>> checkpoints,
>> the “PCR quotes in the snapshot_aggregate event in IMA log” will be 
>> the same for them.
>> The remote attestation server will have to remember these golden PCR 
>> quotes.
> 
> I thought maybe 'golden PCR values'... because those let me replay PCR 
> extensions from
> a previous point.
> 
>> It doesn't have to remember the state of each client device.
> 
> Can you give a reason for this? You mean the state doesn't need to be 
> remembered for client
> devices whose log hasn't been truncated?
> 
I meant it doesn’t have to be remembered for each individual
client device. Majority of the client devices will be in one of the few
golden-PCR-states when the snapshots are captured.

~ Tushar

> 
>>> Can I ever restart the client and have it read the truncated log from 
>>> the
>>> beginning and what type of verification needs to happen on the server 
>>> then?
>>>
>> Yes, restarting the client should be possible.
> 
> Yes, this must be possible.
> 
>>> It seems like the server would have to remember the state of the IMA 
>>> PCRs upon
>>> last truncation to detect a possible attack. This would make staring 
>>> to monitor
>>> a system after truncation impossible -- would be good to know these 
>>> details.
>>>
>> The server is not forced to remember the state of IMA PCRs. It can
>> always ask for the last n snapshot files (shards) and replay the 
>> events. Even
>> though the data is truncated from the IMA log, it is not totally lost. 
>> It is
>> simply being transferred to the disk. It is saved by UM as snapshot 
>> files/shards.
>> The goal of IMA snapshotting is to reduce the Kernel memory pressure 
>> on the
>> client devices - to save them from out-of-memory errors which are 
>> harder to manage
>> on long running clients. It comes with a cost of additional work on 
>> the server
>> side to attest those clients.
> 
> Agreed.
>>
>>
>> Being said that, in the current proposal, taking a snapshots is 
>> totally optional
>> and controlled by UM attestation clients. If the 
>> attestation-clients/services are
>> not-ready/don’t-want to take advantage of IMA log snapshotting, they 
>> don’t have to.
> 
> Agreed.
> 
>>
>> No snapshot will be taken, and the client-service can process the 
>> monolithic IMA
>> log just like they do today.
>>
> 
> Agreed.
> 
>> [1] 
>> https://lore.kernel.org/all/c5737141-7827-1c83-ab38-0119dcfea485@linux.microsoft.com/#t
>>
>>>
>>>
>>>
>>>> should have a beginning and end quote and a record of the AK used.
>>>> Since verifiers like Keylime are already using this beginning and end
>>>> quote for sharded logs, it's the most natural format to feed to
>>>> something externally for verification and it means you don't have to
>>>> invent a new format to do the same thing.
>>>>
>>>> Regards,
>>>>
>>>> James
>>>>
>>>>
>>>> _______________________________________________
>>>> kexec mailing list
>>>> kexec@lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/kexec
>>

Re: [RFC] IMA Log Snapshotting Design Proposal

[Stefan Berger]

On 8/11/23 11:57, Tushar Sugandhi wrote:
> 
> 
> 
> [1] https://patchwork.kernel.org/project/linux-integrity/cover/20230801181917.8535-1-tusharsu@linux.microsoft.com/
> 
>> The shards should will need to be written into some sort of standard location or a config file needs to
>> be defined, so that everyone knows where to find them and how they are named.
>>
> We thought about well known standard location earlier.
> Letting the Kernel choose the name/location of the snapshot
> file comes with its own complexity. Our initial stance is we don’t
> want to handle that at Kernel level, and let the UM client choose
> the location/naming of the snapshot files. But we are happy to
> reconsider if the community requests it.

I would also let user space do the snapshotting but all applications
relying on shards should know where they are located on the system
and what the naming scheme is so they can be  process in proper order.
evmctl for example would have to know where the shards are if keylime
agent had taken snapshots.



>>> Yes. If the “PCR quotes in the snapshot_aggregate event in IMA log”
>>
>> PCR quote or 'quotes'? Why multiple?
>>
>> Form your proposal but you may have changed your opinion  following what I see in other messages:
>> "- The Kernel will get the current TPM PCR values and PCR update counter [2]
>>     and store them as template data in a new IMA event "snapshot_aggregate"."
>>
>> Afaik TPM quote's don't give you the state of the individual PCR values, therefore
>> I would expect to at least find the 'PCR values' of all the PCRs that IMA touched to
>> be in the snapshot_aggregate so I can replay all the following events on top of these
>> PCR values and come up with the values that were used in the "final PCR quote". This
>> is unless you expect the server to take an automatic snapshot of the values of the
>> PCRs  that it computed while evaluating the log in case it ever needs to go back.
>>
> I meant a single set of PCR values captured when snapshot_aggregate
> is logged. Sorry for the confusion.

Ok.

> 
>>> + "replay of rest of the events in IMA log" results in the “final PCR quotes”
>>> that matches with the “AK signed PCR quotes” sent by the client, then the truncated
>>> IMA log can be trusted. The verifier can either ‘trust’ the “PCR quotes in the
>>> snapshot_aggregate event in IMA log” or it can ask for the (n-1)th snapshot shard
>>> to check the past events.
>>
>> For anything regarding determining the 'trustworthiness of a system' one would have to
>> be able to go back to the very beginning of the log *or* remember in what state a
>> system was when the latest snapshot was taken so that if a restart happens it can resume
>> with that assumption about state of trustworthiness and know what the values of the PCRs
>> were at that time so it can resume replaying the log (or the server would get these
>> values from the log).
>>
> Correct. We intend to support the above. I hope our proposal
> description captures it. BTW, when you say ‘restart’, you mean the UM
> process restart, right? Because in case of a Kernel restart

Yes, client restart not reboot.

> (i.e. cold-boot) the past IMA log (and the TPM state) is lost,
> and old snapshots (if any) are useless.

Right. Some script should run on boot and delete all contents of the directory where the log
shards are.

> 
>> The AK quotes by the kernel (which adds a 2nd AK key) that James is proposing
>> could be useful if the entire log, consisting of multiple shards, is very large and
>> cannot be transferred from the client to the server in one go so that the server could
>> evaluate the 'final PCR quote' immediately . However, if a client can indicated 'I will
>> send more the next time and I have this much more to transfer' and the server allows
>> this multiple times (until all the 1MB shards of the 20MB log are transferred) then that
>> kernel AK key would not be necessary since presumably the "final PCR quote", created
>> by a user space client, would resolve whether the entire log is trustworthy.
>>
> See my responses to James today [2]
> 
> [2] https://lore.kernel.org/all/72e39852-1ff1-c7f6-ac7e-593e8142dbe8@linux.microsoft.com/

I think James was proposing one AK, possibly persisted in the TPM's NVRAM. Still, the less keys
that are involved in this the better...

    Stefan

Re: [RFC] IMA Log Snapshotting Design Proposal

[Tushar Sugandhi]

Thanks a lot James for looking at this proposal,
and sharing your thoughts. Really appreciate it.

On 8/1/23 14:21, James Bottomley wrote:
> On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
> [...]
>> Truncating IMA log to reclaim memory is not feasible, since it makes
>> the log go out of sync with the TPM PCR quote making remote
>> attestation fail.
> This assumption isn't entirely true.  It's perfectly possible to shard
> an IMA log using two TPM2_Quote's for the beginning and end PCR values
> to validate the shard.  The IMA log could be truncated in the same way
> (replace the removed part of the log with a TPM2_Quote and AK, so the
> log still validates from the beginning quote to the end).
Here we meant just truncating IMA log is not a complete
solution in itself. As you said, we have to take additional steps
like logging TPM2_Quotes etc. Logging AK is an interesting proposal
which we didn’t consider earlier. I am not sure if embedding AK to IMA
log/snapshot is needed. If the client sends them separately with "signed
PCR quotes" + "IMA log" + snapshots, it should still serve the purpose,
right?

>
> If you use a TPM2_Quote mechanism to save the log, all you need to do
> is have the kernel generate the quote with an internal AK.  You can
> keep a record of the quote and the AK at the beginning of the truncated
> kernel log.  If the truncated entries are saved in a file shard it
> should have a beginning and end quote and a record of the AK used.
A new IMA log snapshot file (or shard as you call it) will have
the TPM2_Quote record (plus some additional metadata) at the beginning.
I don't believe it needs to be logged at the end of the snapshot (since 
it can
be computed by replaying the remaining entries in the snapshot).

See the snapshot_aggregate field in section B.5 in the original RFC mail 
[1].
> Since verifiers like Keylime are already using this beginning and end
> quote for sharded logs, it's the most natural format to feed to
> something externally for verification and it means you don't have to
> invent a new format to do the same thing.
Could you please point to the Keylime source and/or documentation
which explains the use of beginning and end quotes? We would like to
understand how the verifiers are addressing this problem currently.


[1] 
https://lore.kernel.org/all/c5737141-7827-1c83-ab38-0119dcfea485@linux.microsoft.com/#t

~Tushar
>
> Regards,
>
> James

Re: [RFC] IMA Log Snapshotting Design Proposal

[Mimi Zohar]

Hi Sush, Tushar,

On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
> ================================================
> | A. Problem Statement                         |
> ================================================
> Depending on the IMA policy, the IMA log can consume a lot of Kernel 
> memory on
> the device.  For instance, the events for the following IMA policy 
> entries may
> need to be measured in certain scenarios, but they can also lead to a 
> verbose
> IMA log when the device is running for a long period of time.
> ┌───────────────────────────────────────┐
> │# PROC_SUPER_MAGIC                     │
> │measure fsmagic=0x9fa0                 │
> │# SYSFS_MAGIC                          │
> │measure fsmagic=0x62656572             │
> │# DEBUGFS_MAGIC                        │
> │measure fsmagic=0x64626720             │
> │# TMPFS_MAGIC                          │
> │measure fsmagic=0x01021994             │
> │# RAMFS_MAGIC                          │
> │measure fsmagic=0x858458f6             │
> │# SECURITYFS_MAGIC                     │
> │measure fsmagic=0x73636673             │
> │# OVERLAYFS_MAGIC                      │
> │measure fsmagic=0x794c7630             │
> │# log, audit or tmp files              │
> │measure obj_type=var_log_t             │
> │measure obj_type=auditd_log_t          │
> │measure obj_type=tmp_t                 │
> └───────────────────────────────────────┘
> 
> Secondly, certain devices are configured to take Kernel updates using Kexec
> soft-boot.  The IMA log from the previous Kernel gets carried over and the
> Kernel memory consumption problem worsens when such devices undergo multiple
> Kexec soft-boots over a long period of time.
> 
> The above two scenarios can cause IMA log to grow and consume Kernel memory.
> 
> In addition, a large IMA log can add pressure on the network bandwidth when
> the attestation client sends it to remote-attestation-service.
> 
> Truncating IMA log to reclaim memory is not feasible, since it makes the 
> log go
> out of sync with the TPM PCR quote making remote attestation fail.
> 
> A sophisticated solution is required which will help relieve the memory
> pressure on the device and continue supporting remote attestation without
> disruptions.

If the problem is kernel memory, then using a single tmpfs file has
already been proposed [1].  As entries are added to the measurement
list, they are copied to the tmpfs file and removed from kernel memory.
Userspace would still access the measurement list via the existing
securityfs file.

The IMA measurement list is a sequential file, allowing it to be read
from an offset.  How much or how little of the measuremnt list is read
by the attestation client and sent to the attestation server is up to
the attestation client/server.

If the problem is not kernel memory, but memory pressure in general,
then instead of a tmpfs file, the measurement list could similarly be
copied to a single persistent file [1].

> 
> -------------------------------------------------------------------------------
> ================================================
> | B. Proposed Solution                         |
> ================================================
> In this document, we propose an enhancement to the IMA subsystem to improve
> the long-running performance by snapshotting the IMA log, while still
> providing mechanisms to verify its integrity using the PCR quotes.
> 
> The remainder of the document describes details of the proposed solution 
> in the
> following sub-sections.
>   - High-level Work-flow
>   - Snapshot Triggering Mechanism
>   - Design Choices for Storing Snapshots
>   - Attestation-Client and Remote-Attestation-Service Side Changes
>   - Example Walk-through
>   - Open Questions
> -------------------------------------------------------------------------------
> ================================================
> | B.1 High-level Work-flow                     |
> ================================================
> Pre-requisites:
> - IMA Integrity guarantees are maintained.
> 
> The proposed high level work-flow of IMA log snapshotting is as follows:
> - A user-mode process will trigger the snapshot by opening a file in SysFS
>    say /sys/kernel/security/ima/snapshot (referred to as 
> sysk_ima_snapshot_file
>    here onwards).

Please fix the mailer so that it doesn't wrap sentences.   Adding blank
lines between bullets would improve readability.

> - The Kernel will get the current TPM PCR values and PCR update counter [2]
>    and store them as template data in a new IMA event "snapshot_aggregate".
>    This event will be measured by IMA using critical data measurement
>    functionality [1].  Recording regular IMA events will be paused while
>    "snapshot_aggregate" is being computed using the existing IMA mutex lock.

> - Once the "snapshot_aggregate" is computed and measured in IMA log, the 
> prior
>    IMA events will be made available in the sysk_ima_snapshot_file.

> - The UM process will copy those IMA events from sysk_ima_snapshot_file to a
>    snapshot file on disk chosen by UM (referred to as UM_snapshot_file here
>    onwards).  The location, file-system type, access permissions etc. of the
>    UM_snapshot_file would be controlled by UM process itself.

> - Once UM is done copying the IMA events from sysk_ima_snapshot_file to
>    UM_snapshot_file, it will indicate to the Kernel that the snapshot can be
>    finalized by triggering a write with any data to the 
> sysk_ima_snapshot_file.

>    UM process cannot prevent the IMA log purge operation after this point.
> - The Kernel will truncate the current IMA log and and clear HTable up 
> to the
>    "snapshot_aggregate" marker.

> - The Kernel will measure the PCR update counter as part of measuring
>    snapshot_aggregate, so that it can be used by the remote attestation 
> service
>    for detecting missing events.

> - UM can prevent the IMA log purge by closing the sysk_ima_snapshot_file
>    without performing a write operation on it.  In this case, while the
>    "snapshot_aggregate" marker may still be in the log, the event can be 
> ignored
>    since the previous entries in the IMA log will not be purged.
> 
> Note:
> - This work-flow should work when interleaved with Kexec 'load' and 
> 'execute'
>    events and should not cause IMA log + snapshot to go out of sync with PCR
>    quotes. The implementation details are omitted from this document for
>    brevity.

This design seems overly complex and requires synchronization between
the "snapshot" record and exporting the records from the measurement
list.  None of this would be necessary if the measurements were copied
from kernel memory to a backing file (e.g. tmpfs), as described in [1].

What is the real problem - kernel memory pressure, memory pressure in
general, or disk space?  Is the intention to remove or offload the
exported measurements?

Concerns:
- Pausing extending the measurement list.

[1] 
https://lore.kernel.org/linux-integrity/CAOQ4uxj4Pv2Wr1wgvBCDR-tnA5dsZT3rvdDzKgAH1aEV_-r9Qg@mail.gmail.com/#t

-- 
thanks,

Mimi

Re: [RFC] IMA Log Snapshotting Design Proposal

[Sush Shringarputale]

Hello Mimi,

Thanks for your feedback on this.


On 8/11/2023 6:14 AM, Mimi Zohar wrote:
> Hi Sush, Tushar,
>
> On Tue, 2023-08-01 at 12:12 -0700, Sush Shringarputale wrote:
>> ================================================
>> | A. Problem Statement                         |
>> ================================================
>> Depending on the IMA policy, the IMA log can consume a lot of Kernel
>> memory on
>> the device.  For instance, the events for the following IMA policy
>> entries may
>> need to be measured in certain scenarios, but they can also lead to a
>> verbose
>> IMA log when the device is running for a long period of time.
>> ┌───────────────────────────────────────┐
>> │# PROC_SUPER_MAGIC                     │
>> │measure fsmagic=0x9fa0                 │
>> │# SYSFS_MAGIC                          │
>> │measure fsmagic=0x62656572             │
>> │# DEBUGFS_MAGIC                        │
>> │measure fsmagic=0x64626720             │
>> │# TMPFS_MAGIC                          │
>> │measure fsmagic=0x01021994             │
>> │# RAMFS_MAGIC                          │
>> │measure fsmagic=0x858458f6             │
>> │# SECURITYFS_MAGIC                     │
>> │measure fsmagic=0x73636673             │
>> │# OVERLAYFS_MAGIC                      │
>> │measure fsmagic=0x794c7630             │
>> │# log, audit or tmp files              │
>> │measure obj_type=var_log_t             │
>> │measure obj_type=auditd_log_t          │
>> │measure obj_type=tmp_t                 │
>> └───────────────────────────────────────┘
>>
>> Secondly, certain devices are configured to take Kernel updates using Kexec
>> soft-boot.  The IMA log from the previous Kernel gets carried over and the
>> Kernel memory consumption problem worsens when such devices undergo multiple
>> Kexec soft-boots over a long period of time.
>>
>> The above two scenarios can cause IMA log to grow and consume Kernel memory.
>>
>> In addition, a large IMA log can add pressure on the network bandwidth when
>> the attestation client sends it to remote-attestation-service.
>>
>> Truncating IMA log to reclaim memory is not feasible, since it makes the
>> log go
>> out of sync with the TPM PCR quote making remote attestation fail.
>>
>> A sophisticated solution is required which will help relieve the memory
>> pressure on the device and continue supporting remote attestation without
>> disruptions.
> If the problem is kernel memory, then using a single tmpfs file has
> already been proposed [1].  As entries are added to the measurement
> list, they are copied to the tmpfs file and removed from kernel memory.
> Userspace would still access the measurement list via the existing
> securityfs file.
>
> The IMA measurement list is a sequential file, allowing it to be read
> from an offset.  How much or how little of the measuremnt list is read
> by the attestation client and sent to the attestation server is up to
> the attestation client/server.
>
> If the problem is not kernel memory, but memory pressure in general,
> then instead of a tmpfs file, the measurement list could similarly be
> copied to a single persistent file [1].
The suggested approach in this RFC discussion using a vfs_tmpfile was
only discussed but no prototype was created back then.  We are
discussing the approach internally now and will respond with more
details about it.
>> -------------------------------------------------------------------------------
>> ================================================
>> | B. Proposed Solution                         |
>> ================================================
>> In this document, we propose an enhancement to the IMA subsystem to improve
>> the long-running performance by snapshotting the IMA log, while still
>> providing mechanisms to verify its integrity using the PCR quotes.
>>
>> The remainder of the document describes details of the proposed solution
>> in the
>> following sub-sections.
>>    - High-level Work-flow
>>    - Snapshot Triggering Mechanism
>>    - Design Choices for Storing Snapshots
>>    - Attestation-Client and Remote-Attestation-Service Side Changes
>>    - Example Walk-through
>>    - Open Questions
>> -------------------------------------------------------------------------------
>> ================================================
>> | B.1 High-level Work-flow                     |
>> ================================================
>> Pre-requisites:
>> - IMA Integrity guarantees are maintained.
>>
>> The proposed high level work-flow of IMA log snapshotting is as follows:
>> - A user-mode process will trigger the snapshot by opening a file in SysFS
>>     say /sys/kernel/security/ima/snapshot (referred to as
>> sysk_ima_snapshot_file
>>     here onwards).
> Please fix the mailer so that it doesn't wrap sentences.   Adding blank
> lines between bullets would improve readability.
Noted, will do.
>> - The Kernel will get the current TPM PCR values and PCR update counter [2]
>>     and store them as template data in a new IMA event "snapshot_aggregate".
>>     This event will be measured by IMA using critical data measurement
>>     functionality [1].  Recording regular IMA events will be paused while
>>     "snapshot_aggregate" is being computed using the existing IMA mutex lock.
>> - Once the "snapshot_aggregate" is computed and measured in IMA log, the
>> prior
>>     IMA events will be made available in the sysk_ima_snapshot_file.
>> - The UM process will copy those IMA events from sysk_ima_snapshot_file to a
>>     snapshot file on disk chosen by UM (referred to as UM_snapshot_file here
>>     onwards).  The location, file-system type, access permissions etc. of the
>>     UM_snapshot_file would be controlled by UM process itself.
>> - Once UM is done copying the IMA events from sysk_ima_snapshot_file to
>>     UM_snapshot_file, it will indicate to the Kernel that the snapshot can be
>>     finalized by triggering a write with any data to the
>> sysk_ima_snapshot_file.
>>     UM process cannot prevent the IMA log purge operation after this point.
>> - The Kernel will truncate the current IMA log and and clear HTable up
>> to the
>>     "snapshot_aggregate" marker.
>> - The Kernel will measure the PCR update counter as part of measuring
>>     snapshot_aggregate, so that it can be used by the remote attestation
>> service
>>     for detecting missing events.
>> - UM can prevent the IMA log purge by closing the sysk_ima_snapshot_file
>>     without performing a write operation on it.  In this case, while the
>>     "snapshot_aggregate" marker may still be in the log, the event can be
>> ignored
>>     since the previous entries in the IMA log will not be purged.
>>
>> Note:
>> - This work-flow should work when interleaved with Kexec 'load' and
>> 'execute'
>>     events and should not cause IMA log + snapshot to go out of sync with PCR
>>     quotes. The implementation details are omitted from this document for
>>     brevity.
> This design seems overly complex and requires synchronization between
> the "snapshot" record and exporting the records from the measurement
> list.  None of this would be necessary if the measurements were copied
> from kernel memory to a backing file (e.g. tmpfs), as described in [1].
>
> What is the real problem - kernel memory pressure, memory pressure in
> general, or disk space?  Is the intention to remove or offload the
> exported measurements?
The main concern is the memory pressure on both the kernel and the 
attestation client
when it sends the request.  The concern you bring up is valid and we are 
working on
creating a prototype.  There is no intention to remove the exported 
measurements.
- Sush
> Concerns:
> - Pausing extending the measurement list.
>
> [1]
> https://lore.kernel.org/linux-integrity/CAOQ4uxj4Pv2Wr1wgvBCDR-tnA5dsZT3rvdDzKgAH1aEV_-r9Qg@mail.gmail.com/#t
>

Re: [RFC] IMA Log Snapshotting Design Proposal

[Mimi Zohar]

On Mon, 2023-08-14 at 14:42 -0700, Sush Shringarputale wrote:
> > This design seems overly complex and requires synchronization between
> > the "snapshot" record and exporting the records from the measurement
> > list.  None of this would be necessary if the measurements were copied
> > from kernel memory to a backing file (e.g. tmpfs), as described in [1].
> >
> > What is the real problem - kernel memory pressure, memory pressure in
> > general, or disk space?  Is the intention to remove or offload the
> > exported measurements?

> The main concern is the memory pressure on both the kernel and the 
> attestation client
> when it sends the request.  The concern you bring up is valid and we are 
> working on
> creating a prototype.  There is no intention to remove the exported 
> measurements.

Glad to hear that you're not intending to remove the exported
measurements.

Defining and including a new record in the measurement list measurement
is fine, if it helps with attestation and doesn't require pausing the
measurements.

-- 
thanks,

Mimi

Re: [RFC] IMA Log Snapshotting Design Proposal

[Sush Shringarputale]

On 8/14/2023 3:02 PM, Mimi Zohar wrote:
> On Mon, 2023-08-14 at 14:42 -0700, Sush Shringarputale wrote:
>>> This design seems overly complex and requires synchronization between
>>> the "snapshot" record and exporting the records from the measurement
>>> list.  None of this would be necessary if the measurements were copied
>>> from kernel memory to a backing file (e.g. tmpfs), as described in [1].
Even if the Kernel maintains the link between a tmpfs exported and an
in-memory IMA log - it still has to copy the tmpfs portion to the
Kernel memory during kexec soft boot.  tmpfs is cleared during kexec,
so this copying of tmpfs back to kernel memory is necessary to preserve
the integrity of the log during kexec.  But the copying would add back
the memory pressure on the node during kexec (which may result in
out-of-memory), defeating the purpose of the overall effort/feature.
Copying to a regular *persistent* protected file seems a cleaner
approach, compared to tmpfs.  We prototyped this solution, however it
does not seem to be a common pattern within the Kernel to write state
directly to files on disk file systems.  We considered two potential
options:

Option (A): (RECOMMENDED)
             Let Kernel write the file using KM file APIs
--------------------------------------------
     Use Kernel mode file APIs such as _file_open_root_, _vfs_llseek_,
     and _vfs_write_ to use a persistent file on disk.  There is not
     sufficient precedent for this pattern in the Kernel currently, so
     we need guidance from area experts on the best mechanism to
     implement this.

     As for the location of the file, we suggest setting this in
     KConfig. The file will be created by the Kernel, so it should be
     protected from UM access.  Additionally, on a full boot, the
     file should be cleared by the Kernel.

   POTENTIAL ISSUES AND MITIGATIONS
     - handling IO errors from KM

       A potential mitigation for this is to retry the failed write.
       This assumes that the vfs_write Kernel method can handle any
       failures gracefully without causing crashes.


     - using file system paths to resolve the file from KM

       Using the file_open_root seems to provide sufficient resiliency
       against this.  The file can be located at a well known location
       to minimize potential concerns.  However, any guidance in
       minimizing squatting risks would be greatly appreciated.

     - the file could be tampered by UM

       There needs to be a lock/kernel-only-permission on the file
       so that a UM process cannot tamper with the file.  A description
       on how this file would be protected was provided at [2].


Option (B): (NOT RECOMMENDED)
             Hand over the file writing to UM
--------------------------------------------
     Alternately, UM could write the file contents and decide the
     location of the file on disk.  If we want to preserve the Kernel
     behavior of rendering the IMA log as a single monolithic log on
     client (as ascii/binary_runtime_measurements file), the
     Kernel will have to read from a file written by UM and combine it
     with the remaining in-kernel-memory IMA log events.

     This approach of a Kernel reading from a UM written file as an
     IMA log violates the call stack, and thus it is not recommended.

If UM is to handle writing the IMA log to disk, the Kernel cannot read
those snapshotted events again.  Integrity of the IMA log can still be
maintained - a solution implementing this was in the original proposal
we submitted [1].

We appreciate the community's feedback on helping mold this feature to a
suitable implementation.

Thanks,
Sush and Tushar.


References:
[1]
https://lore.kernel.org/linux-integrity/c5737141-7827-1c83-ab38-0119dcfea485@linux.microsoft.com/ 


[2]
https://lore.kernel.org/linux-integrity/CAOQ4uxiBAGKco1BKgyLOMY54r_Ck2jnvz8RCFODD-V87CGqLEw@mail.gmail.com/ 

>>>
>>> What is the real problem - kernel memory pressure, memory pressure in
>>> general, or disk space?  Is the intention to remove or offload the
>>> exported measurements?
>> The main concern is the memory pressure on both the kernel and the
>> attestation client
>> when it sends the request.  The concern you bring up is valid and we are
>> working on
>> creating a prototype.  There is no intention to remove the exported
>> measurements.
> Glad to hear that you're not intending to remove the exported
> measurements.
>
> Defining and including a new record in the measurement list measurement
> is fine, if it helps with attestation and doesn't require pausing the
> measurements.
>

Re: [RFC] IMA Log Snapshotting Design Proposal

[Mimi Zohar]

On Mon, 2023-08-21 at 15:05 -0700, Sush Shringarputale wrote:
> 
> On 8/14/2023 3:02 PM, Mimi Zohar wrote:
> > On Mon, 2023-08-14 at 14:42 -0700, Sush Shringarputale wrote:
> >>> This design seems overly complex and requires synchronization between
> >>> the "snapshot" record and exporting the records from the measurement
> >>> list.  None of this would be necessary if the measurements were copied
> >>> from kernel memory to a backing file (e.g. tmpfs), as described in [1].
> Even if the Kernel maintains the link between a tmpfs exported and an
> in-memory IMA log - it still has to copy the tmpfs portion to the
> Kernel memory during kexec soft boot.  tmpfs is cleared during kexec,
> so this copying of tmpfs back to kernel memory is necessary to preserve
> the integrity of the log during kexec.  But the copying would add back
> the memory pressure on the node during kexec (which may result in
> out-of-memory), defeating the purpose of the overall effort/feature.
> Copying to a regular *persistent* protected file seems a cleaner
> approach, compared to tmpfs.

From a kernel perspective, it doesn't make a difference if userspace
provides a tmpfs or persistent file.  As per the discussion 
https://lore.kernel.org/linux-integrity/CAOQ4uxj4Pv2Wr1wgvBCDR-tnA5dsZT3rvdDzKgAH1aEV_-r9Qg@mail.gmail.com/#t
, userspace provides the kernel with the file descriptor of the opened
file.

> We prototyped this solution, however it
> does not seem to be a common pattern within the Kernel to write state
> directly to files on disk file systems.  We considered two potential
> options:

If no file descriptor is provided, then the measurements aren't copied
and removed from the securityfs file.  If there are write errors, the
measurements aren't removed from the securityfs file until the write
errors are resolved.

Mimi

> 
> Option (A): (RECOMMENDED)
>              Let Kernel write the file using KM file APIs
> --------------------------------------------
>      Use Kernel mode file APIs such as _file_open_root_, _vfs_llseek_,
>      and _vfs_write_ to use a persistent file on disk.  There is not
>      sufficient precedent for this pattern in the Kernel currently, so
>      we need guidance from area experts on the best mechanism to
>      implement this.
> 
>      As for the location of the file, we suggest setting this in
>      KConfig. The file will be created by the Kernel, so it should be
>      protected from UM access.  Additionally, on a full boot, the
>      file should be cleared by the Kernel.
> 
>    POTENTIAL ISSUES AND MITIGATIONS
>      - handling IO errors from KM
> 
>        A potential mitigation for this is to retry the failed write.
>        This assumes that the vfs_write Kernel method can handle any
>        failures gracefully without causing crashes.
> 
> 
>      - using file system paths to resolve the file from KM
> 
>        Using the file_open_root seems to provide sufficient resiliency
>        against this.  The file can be located at a well known location
>        to minimize potential concerns.  However, any guidance in
>        minimizing squatting risks would be greatly appreciated.
> 
>      - the file could be tampered by UM
> 
>        There needs to be a lock/kernel-only-permission on the file
>        so that a UM process cannot tamper with the file.  A description
>        on how this file would be protected was provided at [2].
> 
> Option (B): (NOT RECOMMENDED)
>              Hand over the file writing to UM
> --------------------------------------------
>      Alternately, UM could write the file contents and decide the
>      location of the file on disk.  If we want to preserve the Kernel
>      behavior of rendering the IMA log as a single monolithic log on
>      client (as ascii/binary_runtime_measurements file), the
>      Kernel will have to read from a file written by UM and combine it
>      with the remaining in-kernel-memory IMA log events.
> 
>      This approach of a Kernel reading from a UM written file as an
>      IMA log violates the call stack, and thus it is not recommended.
> 
> If UM is to handle writing the IMA log to disk, the Kernel cannot read
> those snapshotted events again.  Integrity of the IMA log can still be
> maintained - a solution implementing this was in the original proposal
> we submitted [1].
> 
> We appreciate the community's feedback on helping mold this feature to a
> suitable implementation.
> 
> Thanks,
> Sush and Tushar.
> 
> 
> References:
> [1]
> https://lore.kernel.org/linux-integrity/c5737141-7827-1c83-ab38-0119dcfea485@linux.microsoft.com/ 
> 
> 
> [2]
> https://lore.kernel.org/linux-integrity/CAOQ4uxiBAGKco1BKgyLOMY54r_Ck2jnvz8RCFODD-V87CGqLEw@mail.gmail.com/ 
> 
> >>>
> >>> What is the real problem - kernel memory pressure, memory pressure in
> >>> general, or disk space?  Is the intention to remove or offload the
> >>> exported measurements?
> >> The main concern is the memory pressure on both the kernel and the
> >> attestation client
> >> when it sends the request.  The concern you bring up is valid and we are
> >> working on
> >> creating a prototype.  There is no intention to remove the exported
> >> measurements.
> > Glad to hear that you're not intending to remove the exported
> > measurements.
> >
> > Defining and including a new record in the measurement list measurement
> > is fine, if it helps with attestation and doesn't require pausing the
> > measurements.
> >
> 

Re: [RFC] IMA Log Snapshotting Design Proposal

[Paul Moore]

On Mon, Aug 21, 2023 at 7:08 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> On Mon, 2023-08-21 at 15:05 -0700, Sush Shringarputale wrote:
> > On 8/14/2023 3:02 PM, Mimi Zohar wrote:
> > > On Mon, 2023-08-14 at 14:42 -0700, Sush Shringarputale wrote:
> > >>> This design seems overly complex and requires synchronization between
> > >>> the "snapshot" record and exporting the records from the measurement
> > >>> list.  None of this would be necessary if the measurements were copied
> > >>> from kernel memory to a backing file (e.g. tmpfs), as described in [1].
> > Even if the Kernel maintains the link between a tmpfs exported and an
> > in-memory IMA log - it still has to copy the tmpfs portion to the
> > Kernel memory during kexec soft boot.  tmpfs is cleared during kexec,
> > so this copying of tmpfs back to kernel memory is necessary to preserve
> > the integrity of the log during kexec.  But the copying would add back
> > the memory pressure on the node during kexec (which may result in
> > out-of-memory), defeating the purpose of the overall effort/feature.
> > Copying to a regular *persistent* protected file seems a cleaner
> > approach, compared to tmpfs.
>
> From a kernel perspective, it doesn't make a difference if userspace
> provides a tmpfs or persistent file.  As per the discussion
> https://lore.kernel.org/linux-integrity/CAOQ4uxj4Pv2Wr1wgvBCDR-tnA5dsZT3rvdDzKgAH1aEV_-r9Qg@mail.gmail.com/#t
> , userspace provides the kernel with the file descriptor of the opened
> file.
>
> > We prototyped this solution, however it
> > does not seem to be a common pattern within the Kernel to write state
> > directly to files on disk file systems.  We considered two potential
> > options:
>
> If no file descriptor is provided, then the measurements aren't copied
> and removed from the securityfs file.  If there are write errors, the
> measurements aren't removed from the securityfs file until the write
> errors are resolved.

It sounds like this approach would require the file/filesystem to be
continuously available for the life of the system once the log was
snapshotted/overflowed to persistent storage, yes?  Assuming that is
the case, what happens if the file/filesystem becomes inaccessible at
some point and an attestation client attempts to read the entire log?

-- 
paul-moore.com

Re: [RFC] IMA Log Snapshotting Design Proposal

[Mimi Zohar]

On Tue, 2023-08-29 at 15:34 -0400, Paul Moore wrote:
> On Mon, Aug 21, 2023 at 7:08 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > On Mon, 2023-08-21 at 15:05 -0700, Sush Shringarputale wrote:
> > > On 8/14/2023 3:02 PM, Mimi Zohar wrote:
> > > > On Mon, 2023-08-14 at 14:42 -0700, Sush Shringarputale wrote:
> > > >>> This design seems overly complex and requires synchronization between
> > > >>> the "snapshot" record and exporting the records from the measurement
> > > >>> list.  None of this would be necessary if the measurements were copied
> > > >>> from kernel memory to a backing file (e.g. tmpfs), as described in [1].
> > > Even if the Kernel maintains the link between a tmpfs exported and an
> > > in-memory IMA log - it still has to copy the tmpfs portion to the
> > > Kernel memory during kexec soft boot.  tmpfs is cleared during kexec,
> > > so this copying of tmpfs back to kernel memory is necessary to preserve
> > > the integrity of the log during kexec.  But the copying would add back
> > > the memory pressure on the node during kexec (which may result in
> > > out-of-memory), defeating the purpose of the overall effort/feature.
> > > Copying to a regular *persistent* protected file seems a cleaner
> > > approach, compared to tmpfs.
> >
> > From a kernel perspective, it doesn't make a difference if userspace
> > provides a tmpfs or persistent file.  As per the discussion
> > https://lore.kernel.org/linux-integrity/CAOQ4uxj4Pv2Wr1wgvBCDR-tnA5dsZT3rvdDzKgAH1aEV_-r9Qg@mail.gmail.com/#t
> > , userspace provides the kernel with the file descriptor of the opened
> > file.
> >
> > > We prototyped this solution, however it
> > > does not seem to be a common pattern within the Kernel to write state
> > > directly to files on disk file systems.  We considered two potential
> > > options:
> >
> > If no file descriptor is provided, then the measurements aren't copied
> > and removed from the securityfs file.  If there are write errors, the
> > measurements aren't removed from the securityfs file until the write
> > errors are resolved.
> 
> It sounds like this approach would require the file/filesystem to be
> continuously available for the life of the system once the log was
> snapshotted/overflowed to persistent storage, yes?  Assuming that is
> the case, what happens if the file/filesystem becomes inaccessible at
> some point and an attestation client attempts to read the entire log?

The main purpose of the change is to addres kernel memory pressure. 
Two designs are being discussed: Sush's "snapshotting" design and
Amir's original suggestion of continously exporting the measurement
records to a tmpfs or regular file.  Both designs require verifying the
initial attestation quote by walking the entire measurement list,
calculating the expected TPM PCR value(s).  That doesn't change.

-- 
thanks,

Mimi

Re: [RFC] IMA Log Snapshotting Design Proposal

[Paul Moore]

On Tue, Aug 29, 2023 at 5:05 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> On Tue, 2023-08-29 at 15:34 -0400, Paul Moore wrote:
> > On Mon, Aug 21, 2023 at 7:08 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > On Mon, 2023-08-21 at 15:05 -0700, Sush Shringarputale wrote:
> > > > On 8/14/2023 3:02 PM, Mimi Zohar wrote:
> > > > > On Mon, 2023-08-14 at 14:42 -0700, Sush Shringarputale wrote:
> > > > >>> This design seems overly complex and requires synchronization between
> > > > >>> the "snapshot" record and exporting the records from the measurement
> > > > >>> list.  None of this would be necessary if the measurements were copied
> > > > >>> from kernel memory to a backing file (e.g. tmpfs), as described in [1].
> > > > Even if the Kernel maintains the link between a tmpfs exported and an
> > > > in-memory IMA log - it still has to copy the tmpfs portion to the
> > > > Kernel memory during kexec soft boot.  tmpfs is cleared during kexec,
> > > > so this copying of tmpfs back to kernel memory is necessary to preserve
> > > > the integrity of the log during kexec.  But the copying would add back
> > > > the memory pressure on the node during kexec (which may result in
> > > > out-of-memory), defeating the purpose of the overall effort/feature.
> > > > Copying to a regular *persistent* protected file seems a cleaner
> > > > approach, compared to tmpfs.
> > >
> > > From a kernel perspective, it doesn't make a difference if userspace
> > > provides a tmpfs or persistent file.  As per the discussion
> > > https://lore.kernel.org/linux-integrity/CAOQ4uxj4Pv2Wr1wgvBCDR-tnA5dsZT3rvdDzKgAH1aEV_-r9Qg@mail.gmail.com/#t
> > > , userspace provides the kernel with the file descriptor of the opened
> > > file.
> > >
> > > > We prototyped this solution, however it
> > > > does not seem to be a common pattern within the Kernel to write state
> > > > directly to files on disk file systems.  We considered two potential
> > > > options:
> > >
> > > If no file descriptor is provided, then the measurements aren't copied
> > > and removed from the securityfs file.  If there are write errors, the
> > > measurements aren't removed from the securityfs file until the write
> > > errors are resolved.
> >
> > It sounds like this approach would require the file/filesystem to be
> > continuously available for the life of the system once the log was
> > snapshotted/overflowed to persistent storage, yes?  Assuming that is
> > the case, what happens if the file/filesystem becomes inaccessible at
> > some point and an attestation client attempts to read the entire log?
>
> The main purpose of the change is to addres kernel memory pressure.
> Two designs are being discussed: Sush's "snapshotting" design and
> Amir's original suggestion of continously exporting the measurement
> records to a tmpfs or regular file.  Both designs require verifying the
> initial attestation quote by walking the entire measurement list,
> calculating the expected TPM PCR value(s).  That doesn't change.

Sure, but my question is about what happens if portions of the
measurement list disappear due to file/filesystem problems?  How is
that handled?

-- 
paul-moore.com

Re: [RFC] IMA Log Snapshotting Design Proposal

[Mimi Zohar]

On Tue, 2023-08-29 at 17:30 -0400, Paul Moore wrote:
> On Tue, Aug 29, 2023 at 5:05 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > On Tue, 2023-08-29 at 15:34 -0400, Paul Moore wrote:
> > > On Mon, Aug 21, 2023 at 7:08 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > On Mon, 2023-08-21 at 15:05 -0700, Sush Shringarputale wrote:
> > > > > On 8/14/2023 3:02 PM, Mimi Zohar wrote:
> > > > > > On Mon, 2023-08-14 at 14:42 -0700, Sush Shringarputale wrote:
> > > > > >>> This design seems overly complex and requires synchronization between
> > > > > >>> the "snapshot" record and exporting the records from the measurement
> > > > > >>> list.  None of this would be necessary if the measurements were copied
> > > > > >>> from kernel memory to a backing file (e.g. tmpfs), as described in [1].
> > > > > Even if the Kernel maintains the link between a tmpfs exported and an
> > > > > in-memory IMA log - it still has to copy the tmpfs portion to the
> > > > > Kernel memory during kexec soft boot.  tmpfs is cleared during kexec,
> > > > > so this copying of tmpfs back to kernel memory is necessary to preserve
> > > > > the integrity of the log during kexec.  But the copying would add back
> > > > > the memory pressure on the node during kexec (which may result in
> > > > > out-of-memory), defeating the purpose of the overall effort/feature.
> > > > > Copying to a regular *persistent* protected file seems a cleaner
> > > > > approach, compared to tmpfs.
> > > >
> > > > From a kernel perspective, it doesn't make a difference if userspace
> > > > provides a tmpfs or persistent file.  As per the discussion
> > > > https://lore.kernel.org/linux-integrity/CAOQ4uxj4Pv2Wr1wgvBCDR-tnA5dsZT3rvdDzKgAH1aEV_-r9Qg@mail.gmail.com/#t
> > > > , userspace provides the kernel with the file descriptor of the opened
> > > > file.
> > > >
> > > > > We prototyped this solution, however it
> > > > > does not seem to be a common pattern within the Kernel to write state
> > > > > directly to files on disk file systems.  We considered two potential
> > > > > options:
> > > >
> > > > If no file descriptor is provided, then the measurements aren't copied
> > > > and removed from the securityfs file.  If there are write errors, the
> > > > measurements aren't removed from the securityfs file until the write
> > > > errors are resolved.
> > >
> > > It sounds like this approach would require the file/filesystem to be
> > > continuously available for the life of the system once the log was
> > > snapshotted/overflowed to persistent storage, yes?  Assuming that is
> > > the case, what happens if the file/filesystem becomes inaccessible at
> > > some point and an attestation client attempts to read the entire log?
> >
> > The main purpose of the change is to addres kernel memory pressure.
> > Two designs are being discussed: Sush's "snapshotting" design and
> > Amir's original suggestion of continously exporting the measurement
> > records to a tmpfs or regular file.  Both designs require verifying the
> > initial attestation quote by walking the entire measurement list,
> > calculating the expected TPM PCR value(s).  That doesn't change.
> 
> Sure, but my question is about what happens if portions of the
> measurement list disappear due to file/filesystem problems?  How is
> that handled?

With the "snapshotting" solution there could be multiple files, so
portions could be missing.  The other solution, the preferred solution,
would be one file.

Any suggestions?

-- 
thanks,

Mimi

Re: [RFC] IMA Log Snapshotting Design Proposal

[Paul Moore]

On Tue, Aug 29, 2023 at 5:54 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> On Tue, 2023-08-29 at 17:30 -0400, Paul Moore wrote:
> > On Tue, Aug 29, 2023 at 5:05 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > On Tue, 2023-08-29 at 15:34 -0400, Paul Moore wrote:
> > > > On Mon, Aug 21, 2023 at 7:08 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > > On Mon, 2023-08-21 at 15:05 -0700, Sush Shringarputale wrote:
> > > > > > On 8/14/2023 3:02 PM, Mimi Zohar wrote:
> > > > > > > On Mon, 2023-08-14 at 14:42 -0700, Sush Shringarputale wrote:
> > > > > > >>> This design seems overly complex and requires synchronization between
> > > > > > >>> the "snapshot" record and exporting the records from the measurement
> > > > > > >>> list.  None of this would be necessary if the measurements were copied
> > > > > > >>> from kernel memory to a backing file (e.g. tmpfs), as described in [1].
> > > > > > Even if the Kernel maintains the link between a tmpfs exported and an
> > > > > > in-memory IMA log - it still has to copy the tmpfs portion to the
> > > > > > Kernel memory during kexec soft boot.  tmpfs is cleared during kexec,
> > > > > > so this copying of tmpfs back to kernel memory is necessary to preserve
> > > > > > the integrity of the log during kexec.  But the copying would add back
> > > > > > the memory pressure on the node during kexec (which may result in
> > > > > > out-of-memory), defeating the purpose of the overall effort/feature.
> > > > > > Copying to a regular *persistent* protected file seems a cleaner
> > > > > > approach, compared to tmpfs.
> > > > >
> > > > > From a kernel perspective, it doesn't make a difference if userspace
> > > > > provides a tmpfs or persistent file.  As per the discussion
> > > > > https://lore.kernel.org/linux-integrity/CAOQ4uxj4Pv2Wr1wgvBCDR-tnA5dsZT3rvdDzKgAH1aEV_-r9Qg@mail.gmail.com/#t
> > > > > , userspace provides the kernel with the file descriptor of the opened
> > > > > file.
> > > > >
> > > > > > We prototyped this solution, however it
> > > > > > does not seem to be a common pattern within the Kernel to write state
> > > > > > directly to files on disk file systems.  We considered two potential
> > > > > > options:
> > > > >
> > > > > If no file descriptor is provided, then the measurements aren't copied
> > > > > and removed from the securityfs file.  If there are write errors, the
> > > > > measurements aren't removed from the securityfs file until the write
> > > > > errors are resolved.
> > > >
> > > > It sounds like this approach would require the file/filesystem to be
> > > > continuously available for the life of the system once the log was
> > > > snapshotted/overflowed to persistent storage, yes?  Assuming that is
> > > > the case, what happens if the file/filesystem becomes inaccessible at
> > > > some point and an attestation client attempts to read the entire log?
> > >
> > > The main purpose of the change is to addres kernel memory pressure.
> > > Two designs are being discussed: Sush's "snapshotting" design and
> > > Amir's original suggestion of continously exporting the measurement
> > > records to a tmpfs or regular file.  Both designs require verifying the
> > > initial attestation quote by walking the entire measurement list,
> > > calculating the expected TPM PCR value(s).  That doesn't change.
> >
> > Sure, but my question is about what happens if portions of the
> > measurement list disappear due to file/filesystem problems?  How is
> > that handled?
>
> With the "snapshotting" solution there could be multiple files, so
> portions could be missing.  The other solution, the preferred solution,
> would be one file.

With the snapshotting approach the kernel doesn't need to maintain
ongoing access to a file, that is left up to the user process
performing the attestation (or simply inspecting the logs).  I have to
ask, for the third time now in as many hours, how does the proposed
kernel-holds-an-fd-open solution handle the case where the
file/filesystem is no longer accessible?  The snapshotting approach
should be much more resilient here as the error recovery paths can be
much more involved than what we would have available in the kernel,
not to mention the flexibility of allowing a user process to determine
how to store and manage the snapshotted log.

Considering that the snapshotting approach is opt-in (userspace has to
initiate the snapshot), I'm not sure the concern over log offsets is a
significant objection: existing userspace will never trigger a
snapshot, and new userspace that could potentially trigger a snapshot
should be written to take that into account.

-- 
paul-moore.com

Re: [RFC] IMA Log Snapshotting Design Proposal

[Mimi Zohar]

On Tue, 2023-08-29 at 19:15 -0400, Paul Moore wrote:
> On Tue, Aug 29, 2023 at 5:54 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > On Tue, 2023-08-29 at 17:30 -0400, Paul Moore wrote:
> > > On Tue, Aug 29, 2023 at 5:05 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > On Tue, 2023-08-29 at 15:34 -0400, Paul Moore wrote:
> > > > > On Mon, Aug 21, 2023 at 7:08 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > > > On Mon, 2023-08-21 at 15:05 -0700, Sush Shringarputale wrote:
> > > > > > > On 8/14/2023 3:02 PM, Mimi Zohar wrote:
> > > > > > > > On Mon, 2023-08-14 at 14:42 -0700, Sush Shringarputale wrote:
> > > > > > > >>> This design seems overly complex and requires synchronization between
> > > > > > > >>> the "snapshot" record and exporting the records from the measurement
> > > > > > > >>> list.  None of this would be necessary if the measurements were copied
> > > > > > > >>> from kernel memory to a backing file (e.g. tmpfs), as described in [1].
> > > > > > > Even if the Kernel maintains the link between a tmpfs exported and an
> > > > > > > in-memory IMA log - it still has to copy the tmpfs portion to the
> > > > > > > Kernel memory during kexec soft boot.  tmpfs is cleared during kexec,
> > > > > > > so this copying of tmpfs back to kernel memory is necessary to preserve
> > > > > > > the integrity of the log during kexec.  But the copying would add back
> > > > > > > the memory pressure on the node during kexec (which may result in
> > > > > > > out-of-memory), defeating the purpose of the overall effort/feature.
> > > > > > > Copying to a regular *persistent* protected file seems a cleaner
> > > > > > > approach, compared to tmpfs.
> > > > > >
> > > > > > From a kernel perspective, it doesn't make a difference if userspace
> > > > > > provides a tmpfs or persistent file.  As per the discussion
> > > > > > https://lore.kernel.org/linux-integrity/CAOQ4uxj4Pv2Wr1wgvBCDR-tnA5dsZT3rvdDzKgAH1aEV_-r9Qg@mail.gmail.com/#t
> > > > > > , userspace provides the kernel with the file descriptor of the opened
> > > > > > file.
> > > > > >
> > > > > > > We prototyped this solution, however it
> > > > > > > does not seem to be a common pattern within the Kernel to write state
> > > > > > > directly to files on disk file systems.  We considered two potential
> > > > > > > options:
> > > > > >
> > > > > > If no file descriptor is provided, then the measurements aren't copied
> > > > > > and removed from the securityfs file.  If there are write errors, the
> > > > > > measurements aren't removed from the securityfs file until the write
> > > > > > errors are resolved.
> > > > >
> > > > > It sounds like this approach would require the file/filesystem to be
> > > > > continuously available for the life of the system once the log was
> > > > > snapshotted/overflowed to persistent storage, yes?  Assuming that is
> > > > > the case, what happens if the file/filesystem becomes inaccessible at
> > > > > some point and an attestation client attempts to read the entire log?
> > > >
> > > > The main purpose of the change is to addres kernel memory pressure.
> > > > Two designs are being discussed: Sush's "snapshotting" design and
> > > > Amir's original suggestion of continously exporting the measurement
> > > > records to a tmpfs or regular file.  Both designs require verifying the
> > > > initial attestation quote by walking the entire measurement list,
> > > > calculating the expected TPM PCR value(s).  That doesn't change.
> > >
> > > Sure, but my question is about what happens if portions of the
> > > measurement list disappear due to file/filesystem problems?  How is
> > > that handled?
> >
> > With the "snapshotting" solution there could be multiple files, so
> > portions could be missing.  The other solution, the preferred solution,
> > would be one file.
> 
> With the snapshotting approach the kernel doesn't need to maintain
> ongoing access to a file, that is left up to the user process
> performing the attestation (or simply inspecting the logs).  I have to
> ask, for the third time now in as many hours, how does the proposed
> kernel-holds-an-fd-open solution handle the case where the
> file/filesystem is no longer accessible?  The snapshotting approach
> should be much more resilient here as the error recovery paths can be
> much more involved than what we would have available in the kernel,
> not to mention the flexibility of allowing a user process to determine
> how to store and manage the snapshotted log.
> 
> Considering that the snapshotting approach is opt-in (userspace has to
> initiate the snapshot), I'm not sure the concern over log offsets is a
> significant objection: existing userspace will never trigger a
> snapshot, and new userspace that could potentially trigger a snapshot
> should be written to take that into account.

Your initial question was "what happens if the file/filesystem becomes
inaccessible at some point and an attestation client attempts to read
the entire log?".  For what reason would it be inaccessible?  For the
original single tmpfs file, what would make it inaccessible?  In the
"snapshotting" design this problem becomes a userspace issue.

The first sentence of the cover letter is "Depending on the IMA policy,
the IMA log can consume a lot of Kernel memory on the device." 
Copying/exporting the measurements from kernel memory to a single
(tmpfs) file resolves the kernel memory pressure issue.  The entire
measurement list would remain accessible via the securityfs file.  As
previously mentioned in this thread, it wouldn't resolve the kexec
issue of allocating a buffer large enough for really large measurement
lists.

If the real motivation for this patch set is trimming the measurement
list, the cover letter needs to say that.  The existing "snapshotting"
design requires locking of the entire measurement list while copying
and trimming it.  The "snapshotting" design needs to be simplified. 
Perhaps a trusted userspace process could copy the measurements and
then trigger trimming of the in kernel memory/tmpfs measurement list.

-- 
thanks,

Mimi

Re: [RFC] IMA Log Snapshotting Design Proposal

[Paul Moore]

On Wed, Aug 30, 2023 at 4:25 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> Your initial question was "what happens if the file/filesystem becomes
> inaccessible at some point and an attestation client attempts to read
> the entire log?".  For what reason would it be inaccessible?  For the
> original single tmpfs file, what would make it inaccessible?

In your reply that I had responded to you had mentioned that the
kernel was simply being passed a fd and taking ownership of it, the fd
could either be a tmpfs backed file or some form of persistent storage
as both were discussed in this thread.  I imagine a tmpfs filesystem
could still be forcibly unmounted, resulting in problems, but I can't
say that for certain.  However, there are definitely cases where a fd
backed against an arbitrary filesystem could run into problems:
storage device issues for local filesystems, networking issues for
network filesystems, and good old fashioned user/admin intervention in
both cases.

> In the
> "snapshotting" design this problem becomes a userspace issue.

Yes, exactly.  Userspace is almost always going to have an easier time
recovering from these types of errors ... or at least I believe so,
perhaps you have a clever solution for how the kernel can handle the
file/filesystem disappearing from under the fd?

> The first sentence of the cover letter is "Depending on the IMA policy,
> the IMA log can consume a lot of Kernel memory on the device." ...

As I'm still looking for an answer to my question, let's stay focused
on that before we start worrying too much about the phrasing of the
design proposal that was submitted.

-- 
paul-moore.com

Re: [RFC] IMA Log Snapshotting Design Proposal

[Mimi Zohar]

On Wed, 2023-08-30 at 16:47 -0400, Paul Moore wrote:
> On Wed, Aug 30, 2023 at 4:25 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > Your initial question was "what happens if the file/filesystem becomes
> > inaccessible at some point and an attestation client attempts to read
> > the entire log?".  For what reason would it be inaccessible?  For the
> > original single tmpfs file, what would make it inaccessible?
> 
> In your reply that I had responded to you had mentioned that the
> kernel was simply being passed a fd and taking ownership of it, the fd
> could either be a tmpfs backed file or some form of persistent storage
> as both were discussed in this thread.  I imagine a tmpfs filesystem
> could still be forcibly unmounted, resulting in problems, but I can't
> say that for certain.  However, there are definitely cases where a fd
> backed against an arbitrary filesystem could run into problems:
> storage device issues for local filesystems, networking issues for
> network filesystems, and good old fashioned user/admin intervention in
> both cases.

"I imagine tmpfs filesystem could still be forcibly unmounted" sounds
like an attack. Not being able to verify the measurement list against a
quote is probably a good thing.

> 
> > In the
> > "snapshotting" design this problem becomes a userspace issue.
> 
> Yes, exactly.  Userspace is almost always going to have an easier time
> recovering from these types of errors ... or at least I believe so,
> perhaps you have a clever solution for how the kernel can handle the
> file/filesystem disappearing from under the fd?

Nothing changes.  New measurements are initially stored in kernel
memory, until they are successfully copied.

> > The first sentence of the cover letter is "Depending on the IMA policy,
> > the IMA log can consume a lot of Kernel memory on the device." ...
> 
> As I'm still looking for an answer to my question, let's stay focused
> on that before we start worrying too much about the phrasing of the
> design proposal that was submitted.

It's more than just phrasing, but the purpose/motivation of the
proposed changes.

--  
thanks,

Mimi

Re: [RFC] IMA Log Snapshotting Design Proposal

[Paul Moore]

On Wed, Aug 30, 2023 at 5:50 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> On Wed, 2023-08-30 at 16:47 -0400, Paul Moore wrote:
> > On Wed, Aug 30, 2023 at 4:25 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > Your initial question was "what happens if the file/filesystem becomes
> > > inaccessible at some point and an attestation client attempts to read
> > > the entire log?".  For what reason would it be inaccessible?  For the
> > > original single tmpfs file, what would make it inaccessible?
> >
> > In your reply that I had responded to you had mentioned that the
> > kernel was simply being passed a fd and taking ownership of it, the fd
> > could either be a tmpfs backed file or some form of persistent storage
> > as both were discussed in this thread.  I imagine a tmpfs filesystem
> > could still be forcibly unmounted, resulting in problems, but I can't
> > say that for certain.  However, there are definitely cases where a fd
> > backed against an arbitrary filesystem could run into problems:
> > storage device issues for local filesystems, networking issues for
> > network filesystems, and good old fashioned user/admin intervention in
> > both cases.
>
> "I imagine tmpfs filesystem could still be forcibly unmounted" sounds
> like an attack. Not being able to verify the measurement list against a
> quote is probably a good thing.

Okay, can you answer the question for an arbitrary persistent
filesystem?  That was always the more important question, and your
continued avoidance is getting me increasingly annoyed.

-- 
paul-moore.com

Re: [RFC] IMA Log Snapshotting Design Proposal

[Paul Moore]

On Wed, Aug 30, 2023 at 6:21 PM Paul Moore <paul@paul-moore.com> wrote:
> On Wed, Aug 30, 2023 at 5:50 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > On Wed, 2023-08-30 at 16:47 -0400, Paul Moore wrote:
> > > On Wed, Aug 30, 2023 at 4:25 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > Your initial question was "what happens if the file/filesystem becomes
> > > > inaccessible at some point and an attestation client attempts to read
> > > > the entire log?".  For what reason would it be inaccessible?  For the
> > > > original single tmpfs file, what would make it inaccessible?
> > >
> > > In your reply that I had responded to you had mentioned that the
> > > kernel was simply being passed a fd and taking ownership of it, the fd
> > > could either be a tmpfs backed file or some form of persistent storage
> > > as both were discussed in this thread.  I imagine a tmpfs filesystem
> > > could still be forcibly unmounted, resulting in problems, but I can't
> > > say that for certain.  However, there are definitely cases where a fd
> > > backed against an arbitrary filesystem could run into problems:
> > > storage device issues for local filesystems, networking issues for
> > > network filesystems, and good old fashioned user/admin intervention in
> > > both cases.
> >
> > "I imagine tmpfs filesystem could still be forcibly unmounted" sounds
> > like an attack. Not being able to verify the measurement list against a
> > quote is probably a good thing.
>
> Okay, can you answer the question for an arbitrary persistent
> filesystem?  That was always the more important question, and your
> continued avoidance is getting me increasingly annoyed.

Speaking of being annoyed, I'm fixing Tushar's email as the bounces
are also driving me nuts.

-- 
paul-moore.com

Re: [RFC] IMA Log Snapshotting Design Proposal

[Mimi Zohar]

On Wed, 2023-08-30 at 18:23 -0400, Paul Moore wrote:
> On Wed, Aug 30, 2023 at 6:21 PM Paul Moore <paul@paul-moore.com> wrote:
> > On Wed, Aug 30, 2023 at 5:50 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > On Wed, 2023-08-30 at 16:47 -0400, Paul Moore wrote:
> > > > On Wed, Aug 30, 2023 at 4:25 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > > Your initial question was "what happens if the file/filesystem becomes
> > > > > inaccessible at some point and an attestation client attempts to read
> > > > > the entire log?".  For what reason would it be inaccessible?  For the
> > > > > original single tmpfs file, what would make it inaccessible?
> > > >
> > > > In your reply that I had responded to you had mentioned that the
> > > > kernel was simply being passed a fd and taking ownership of it, the fd
> > > > could either be a tmpfs backed file or some form of persistent storage
> > > > as both were discussed in this thread.  I imagine a tmpfs filesystem
> > > > could still be forcibly unmounted, resulting in problems, but I can't
> > > > say that for certain.  However, there are definitely cases where a fd
> > > > backed against an arbitrary filesystem could run into problems:
> > > > storage device issues for local filesystems, networking issues for
> > > > network filesystems, and good old fashioned user/admin intervention in
> > > > both cases.
> > >
> > > "I imagine tmpfs filesystem could still be forcibly unmounted" sounds
> > > like an attack. Not being able to verify the measurement list against a
> > > quote is probably a good thing.
> >
> > Okay, can you answer the question for an arbitrary persistent
> > filesystem?  That was always the more important question, ...

The original proposal, not mine, suggested using a tmpfs file.   The
idea of writing the measurements to a file on a persistent filesystem
wasn't mine either.  Sush/Tushar were pushing for writing to a
persistent file(s).  No argument from me that writing to a file on an
arbitrary persistent filesystem is not a good idea.

-- 
thanks,

Mimi

Re: [RFC] IMA Log Snapshotting Design Proposal

[Paul Moore]

On Wed, Aug 30, 2023 at 7:07 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> On Wed, 2023-08-30 at 18:23 -0400, Paul Moore wrote:
> > On Wed, Aug 30, 2023 at 6:21 PM Paul Moore <paul@paul-moore.com> wrote:
> > > On Wed, Aug 30, 2023 at 5:50 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > On Wed, 2023-08-30 at 16:47 -0400, Paul Moore wrote:
> > > > > On Wed, Aug 30, 2023 at 4:25 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > > > Your initial question was "what happens if the file/filesystem becomes
> > > > > > inaccessible at some point and an attestation client attempts to read
> > > > > > the entire log?".  For what reason would it be inaccessible?  For the
> > > > > > original single tmpfs file, what would make it inaccessible?
> > > > >
> > > > > In your reply that I had responded to you had mentioned that the
> > > > > kernel was simply being passed a fd and taking ownership of it, the fd
> > > > > could either be a tmpfs backed file or some form of persistent storage
> > > > > as both were discussed in this thread.  I imagine a tmpfs filesystem
> > > > > could still be forcibly unmounted, resulting in problems, but I can't
> > > > > say that for certain.  However, there are definitely cases where a fd
> > > > > backed against an arbitrary filesystem could run into problems:
> > > > > storage device issues for local filesystems, networking issues for
> > > > > network filesystems, and good old fashioned user/admin intervention in
> > > > > both cases.
> > > >
> > > > "I imagine tmpfs filesystem could still be forcibly unmounted" sounds
> > > > like an attack. Not being able to verify the measurement list against a
> > > > quote is probably a good thing.
> > >
> > > Okay, can you answer the question for an arbitrary persistent
> > > filesystem?  That was always the more important question, ...
>
> The original proposal, not mine, suggested using a tmpfs file.   The
> idea of writing the measurements to a file on a persistent filesystem
> wasn't mine either.  Sush/Tushar were pushing for writing to a
> persistent file(s).  No argument from me that writing to a file on an
> arbitrary persistent filesystem is not a good idea.

Progress.  Okay, so we all now agree that the kernel writing to an
arbitrary filesystem is not a good idea, and using tmpfs doesn't solve
the problem of general memory pressure (from previously in this
thread), that's all helpful and good to clarify.

Assuming Sush and Tushar rework the document to clarify the
motivation/purpose for the work, as you suggested earlier, I'm
assuming we can revisit this problem and solutions?

-- 
paul-moore.com

Re: [RFC] IMA Log Snapshotting Design Proposal

[Mimi Zohar]

On Wed, 2023-08-30 at 19:22 -0400, Paul Moore wrote:
> On Wed, Aug 30, 2023 at 7:07 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > On Wed, 2023-08-30 at 18:23 -0400, Paul Moore wrote:
> > > On Wed, Aug 30, 2023 at 6:21 PM Paul Moore <paul@paul-moore.com> wrote:
> > > > On Wed, Aug 30, 2023 at 5:50 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > > On Wed, 2023-08-30 at 16:47 -0400, Paul Moore wrote:
> > > > > > On Wed, Aug 30, 2023 at 4:25 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > > > > Your initial question was "what happens if the file/filesystem becomes
> > > > > > > inaccessible at some point and an attestation client attempts to read
> > > > > > > the entire log?".  For what reason would it be inaccessible?  For the
> > > > > > > original single tmpfs file, what would make it inaccessible?
> > > > > >
> > > > > > In your reply that I had responded to you had mentioned that the
> > > > > > kernel was simply being passed a fd and taking ownership of it, the fd
> > > > > > could either be a tmpfs backed file or some form of persistent storage
> > > > > > as both were discussed in this thread.  I imagine a tmpfs filesystem
> > > > > > could still be forcibly unmounted, resulting in problems, but I can't
> > > > > > say that for certain.  However, there are definitely cases where a fd
> > > > > > backed against an arbitrary filesystem could run into problems:
> > > > > > storage device issues for local filesystems, networking issues for
> > > > > > network filesystems, and good old fashioned user/admin intervention in
> > > > > > both cases.
> > > > >
> > > > > "I imagine tmpfs filesystem could still be forcibly unmounted" sounds
> > > > > like an attack. Not being able to verify the measurement list against a
> > > > > quote is probably a good thing.
> > > >
> > > > Okay, can you answer the question for an arbitrary persistent
> > > > filesystem?  That was always the more important question, ...
> >
> > The original proposal, not mine, suggested using a tmpfs file.   The
> > idea of writing the measurements to a file on a persistent filesystem
> > wasn't mine either.  Sush/Tushar were pushing for writing to a
> > persistent file(s).  No argument from me that writing to a file on an
> > arbitrary persistent filesystem is not a good idea.
> 
> ...  Okay, so we all now agree that the kernel writing to an
> arbitrary filesystem is not a good idea, and using tmpfs doesn't solve
> the problem of general memory pressure (from previously in this
> thread), that's all helpful and good to clarify.

Do we also agree that the "tmpfs" solution would address the existing
design motivation - kernel memory pressure?

> 
> Assuming Sush and Tushar rework the document to clarify the
> motivation/purpose for the work, as you suggested earlier, I'm
> assuming we can revisit this problem and solutions?

In addition to the mismatch between the motivation and the design, the
design has some major flaws that first need to be addressed.

Assuming the design issues are addressed, please make sure the document
is written clearly and concisely.

-- 
thanks,

Mimi

Re: [RFC] IMA Log Snapshotting Design Proposal

[Paul Moore]

On Thu, Aug 31, 2023 at 10:07 AM Mimi Zohar <zohar@linux.ibm.com> wrote:
> On Wed, 2023-08-30 at 19:22 -0400, Paul Moore wrote:
> > On Wed, Aug 30, 2023 at 7:07 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > On Wed, 2023-08-30 at 18:23 -0400, Paul Moore wrote:
> > > > On Wed, Aug 30, 2023 at 6:21 PM Paul Moore <paul@paul-moore.com> wrote:
> > > > > On Wed, Aug 30, 2023 at 5:50 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > > > On Wed, 2023-08-30 at 16:47 -0400, Paul Moore wrote:
> > > > > > > On Wed, Aug 30, 2023 at 4:25 PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > > > > > Your initial question was "what happens if the file/filesystem becomes
> > > > > > > > inaccessible at some point and an attestation client attempts to read
> > > > > > > > the entire log?".  For what reason would it be inaccessible?  For the
> > > > > > > > original single tmpfs file, what would make it inaccessible?
> > > > > > >
> > > > > > > In your reply that I had responded to you had mentioned that the
> > > > > > > kernel was simply being passed a fd and taking ownership of it, the fd
> > > > > > > could either be a tmpfs backed file or some form of persistent storage
> > > > > > > as both were discussed in this thread.  I imagine a tmpfs filesystem
> > > > > > > could still be forcibly unmounted, resulting in problems, but I can't
> > > > > > > say that for certain.  However, there are definitely cases where a fd
> > > > > > > backed against an arbitrary filesystem could run into problems:
> > > > > > > storage device issues for local filesystems, networking issues for
> > > > > > > network filesystems, and good old fashioned user/admin intervention in
> > > > > > > both cases.
> > > > > >
> > > > > > "I imagine tmpfs filesystem could still be forcibly unmounted" sounds
> > > > > > like an attack. Not being able to verify the measurement list against a
> > > > > > quote is probably a good thing.
> > > > >
> > > > > Okay, can you answer the question for an arbitrary persistent
> > > > > filesystem?  That was always the more important question, ...
> > >
> > > The original proposal, not mine, suggested using a tmpfs file.   The
> > > idea of writing the measurements to a file on a persistent filesystem
> > > wasn't mine either.  Sush/Tushar were pushing for writing to a
> > > persistent file(s).  No argument from me that writing to a file on an
> > > arbitrary persistent filesystem is not a good idea.
> >
> > ...  Okay, so we all now agree that the kernel writing to an
> > arbitrary filesystem is not a good idea, and using tmpfs doesn't solve
> > the problem of general memory pressure (from previously in this
> > thread), that's all helpful and good to clarify.
>
> Do we also agree that the "tmpfs" solution would address the existing
> design motivation - kernel memory pressure?

I haven't really looked into that so I'm not comfortable commenting on
that.  I can say that in off-list discussions with Sush, Tushar, and
others the issue we are facing is general memory pressure so using any
type of memory backed filesystem doesn't really resolve our problem.

> > Assuming Sush and Tushar rework the document to clarify the
> > motivation/purpose for the work, as you suggested earlier, I'm
> > assuming we can revisit this problem and solutions?
>
> In addition to the mismatch between the motivation and the design, the
> design has some major flaws that first need to be addressed.
>
> Assuming the design issues are addressed, please make sure the document
> is written clearly and concisely.

That feels like a normal part of the design, review, and iterate cycle
to me.  Sush and Tushar are following along and I'm sure they will
take all of that into account in the next revision of the design
document.

-- 
paul-moore.com

Re: [RFC] IMA Log Snapshotting Design Proposal

[Dr. Greg]

On Wed, Aug 30, 2023 at 07:22:27PM -0400, Paul Moore wrote:

Good morning.

> On Wed, Aug 30, 2023 at 7:07???PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > On Wed, 2023-08-30 at 18:23 -0400, Paul Moore wrote:
> > > On Wed, Aug 30, 2023 at 6:21???PM Paul Moore <paul@paul-moore.com> wrote:
> > > > On Wed, Aug 30, 2023 at 5:50???PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > > On Wed, 2023-08-30 at 16:47 -0400, Paul Moore wrote:
> > > > > > On Wed, Aug 30, 2023 at 4:25???PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > > > > Your initial question was "what happens if the file/filesystem becomes
> > > > > > > inaccessible at some point and an attestation client attempts to read
> > > > > > > the entire log?".  For what reason would it be inaccessible?  For the
> > > > > > > original single tmpfs file, what would make it inaccessible?
> > > > > >
> > > > > > In your reply that I had responded to you had mentioned that the
> > > > > > kernel was simply being passed a fd and taking ownership of it, the fd
> > > > > > could either be a tmpfs backed file or some form of persistent storage
> > > > > > as both were discussed in this thread.  I imagine a tmpfs filesystem
> > > > > > could still be forcibly unmounted, resulting in problems, but I can't
> > > > > > say that for certain.  However, there are definitely cases where a fd
> > > > > > backed against an arbitrary filesystem could run into problems:
> > > > > > storage device issues for local filesystems, networking issues for
> > > > > > network filesystems, and good old fashioned user/admin intervention in
> > > > > > both cases.
> > > > >
> > > > > "I imagine tmpfs filesystem could still be forcibly unmounted" sounds
> > > > > like an attack. Not being able to verify the measurement list against a
> > > > > quote is probably a good thing.
> > > >
> > > > Okay, can you answer the question for an arbitrary persistent
> > > > filesystem?  That was always the more important question, ...
> >
> > The original proposal, not mine, suggested using a tmpfs file.   The
> > idea of writing the measurements to a file on a persistent filesystem
> > wasn't mine either.  Sush/Tushar were pushing for writing to a
> > persistent file(s).  No argument from me that writing to a file on an
> > arbitrary persistent filesystem is not a good idea.

> Progress.  Okay, so we all now agree that the kernel writing to an
> arbitrary filesystem is not a good idea, and using tmpfs doesn't
> solve the problem of general memory pressure (from previously in
> this thread), that's all helpful and good to clarify.
>
> Assuming Sush and Tushar rework the document to clarify the
> motivation/purpose for the work, as you suggested earlier, I'm
> assuming we can revisit this problem and solutions?

IMA will obviously go, with our blessings, in its own direction.

I would only call out, as I indicated in my reply to Ken Goldman, that
our 20+ year old integrity and attestation architectures and models
are now arguably challenged, given emerging technologies and their
requirements for integrity and confidentiality.

This thread highlights a lot of the issues that caused us to bring
TSEM to the table.

If we, for the moment, isolate discussion to the growth and/or
management of the event log.  The architecture of externally modeled
namespaces gets the events immediately out of the kernel for a
userspace Trusted Modeling Agent (TMA) to deal with, however it
chooses.

The TMA doesn't even need to be running on the machine itself.  We are
currently feeding security event streams to entirely different
platforms, some hypervisor based, particularly for the training and
enforcement of machine learning models.

There are obviously concessions that need to be made for LSM event
handlers running in atomic context, but that isn't an issue for a
security model based simply on file integrity.  IMA in atomic context
isn't possible, not only from the perspective of doing the file I/O
but the need to sleep while the TPM transaction runs.

Once again, no criticism of IMA, it is proven and has its own camp and
following.  From our perspective, and our collaborators, there is an
opportunity, and need, to enhance the options available.

> paul-moore.com

Best wishes for an end of summer holiday weekend.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity

Re: [RFC] IMA Log Snapshotting Design Proposal

[Paul Moore]

On Thu, Aug 31, 2023 at 12:47 PM Dr. Greg <greg@enjellic.com> wrote:
> On Wed, Aug 30, 2023 at 07:22:27PM -0400, Paul Moore wrote:
> > On Wed, Aug 30, 2023 at 7:07???PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > On Wed, 2023-08-30 at 18:23 -0400, Paul Moore wrote:
> > > > On Wed, Aug 30, 2023 at 6:21???PM Paul Moore <paul@paul-moore.com> wrote:
> > > > > On Wed, Aug 30, 2023 at 5:50???PM Mimi Zohar <zohar@linux.ibm.com> wrote:
> > > > > > On Wed, 2023-08-30 at 16:47 -0400, Paul Moore wrote:
> > > > > > > On Wed, Aug 30, 2023 at 4:25???PM Mimi Zohar <zohar@linux.ibm.com> wrote:

...

> > Assuming Sush and Tushar rework the document to clarify the
> > motivation/purpose for the work, as you suggested earlier, I'm
> > assuming we can revisit this problem and solutions?
>
> IMA will obviously go, with our blessings, in its own direction.
>
> I would only call out, as I indicated in my reply to Ken Goldman, that
> our 20+ year old integrity and attestation architectures and models
> are now arguably challenged, given emerging technologies and their
> requirements for integrity and confidentiality.
>
> This thread highlights a lot of the issues that caused us to bring
> TSEM to the table ...

It would be helpful if you could refrain from hijacking barely
tangentially related threads as a marketing pitch for TSEM.

Everyone is welcome, and encouraged (!), to join the upstream
discussion threads, especially when one has something relevant to
contribute which hasn't been previously discussed.  However, when
every other contribution an individual makes follows the pattern of
"we could solve all of this if we only had feature 'X' which I
helpfully posted here <lore link>" then those contributions grow very
tiring and I tend to start ignoring that individual.

-- 
paul-moore.com

Re: [RFC] IMA Log Snapshotting Design Proposal - network bandwidth

[Ken Goldman]

On 8/1/2023 3:12 PM, Sush Shringarputale wrote:
> In addition, a large IMA log can add pressure on the network bandwidth when
> the attestation client sends it to remote-attestation-service.

I would not worry too much about network bandwidth.

1. Every solution eventually realizes that sending the entire log each 
time hurts performance.  The verifier will ask the attestor, "give me 
everything since record n", and the number of new entries approaches zero.

2. My benchmarks show that

On the client, the TPM quote time swamps everything else.
On the server, verifying the IMA entry signatures swamps everything else.

The network transfer time is negligible.

Re: [RFC] IMA Log Snapshotting Design Proposal - network bandwidth

[Tushar Sugandhi]

Thanks a lot Ken for looking at the proposal, and sharing your thoughts.

On 8/30/23 11:06, Ken Goldman wrote:
> 
> 
> On 8/1/2023 3:12 PM, Sush Shringarputale wrote:
>> In addition, a large IMA log can add pressure on the network bandwidth 
>> when
>> the attestation client sends it to remote-attestation-service.
> 
> I would not worry too much about network bandwidth.
Our bandwidth concerns are about scaled out system.

When IMA log size increases in the range of megabytes, and when the
number of client devices increases, it makes an impact on the overall
network bandwidth.

> 
> 1. Every solution eventually realizes that sending the entire log each 
> time hurts performance.  The verifier will ask the attestor, "give me 
> everything since record n", and the number of new entries approaches zero.
> 
Completely agreed. IMA log snapshotting (this proposed feature) is a
solution in that direction.

> 2. My benchmarks show that
> 
> On the client, the TPM quote time swamps everything else.
> On the server, verifying the IMA entry signatures swamps everything else.
> 
> The network transfer time is negligible.
Agreed, it is true in the context of an individual client device.

Our network bandwidth concerns are for the overall traffic on the scaled
out system. It impacts the network bandwidth when the IMA log is large
(MBs).  And the issue is compounded when there are large number of
client devices.

Thanks,
Tushar

Re: [RFC] IMA Log Snapshotting Design Proposal - network bandwidth

[Ken Goldman]

On 9/1/2023 5:20 PM, Tushar Sugandhi wrote:
> Thanks a lot Ken for looking at the proposal, and sharing your thoughts.
> 
> On 8/30/23 11:06, Ken Goldman wrote:
>>
>>
>> On 8/1/2023 3:12 PM, Sush Shringarputale wrote:
>>> In addition, a large IMA log can add pressure on the network 
>>> bandwidth when
>>> the attestation client sends it to remote-attestation-service.
>>
>> I would not worry too much about network bandwidth.
> Our bandwidth concerns are about scaled out system.
> 
> When IMA log size increases in the range of megabytes, and when the
> number of client devices increases, it makes an impact on the overall
> network bandwidth.

It should not, because the client only sends new measurements.  It only 
sends the entire list once per boot.

Does a megabyte matter in a modern network? As for overall performance,
a megabyte may take 10 msec, while the TPM quote could take 1000 msec, 
and verifier hash and asymmetric signature checks are also slower.

> 
>>
>> 1. Every solution eventually realizes that sending the entire log each 
>> time hurts performance.  The verifier will ask the attestor, "give me 
>> everything since record n", and the number of new entries approaches 
>> zero.
>>
> Completely agreed. IMA log snapshotting (this proposed feature) is a
> solution in that direction.

Does snapshotting matter?  The first time, the attester sends the entire 
log, whether part is snapshotted or not.  Same with incremental attestation.

I don't understand how snapshotting would have any affect on network 
traffic.

Re: [RFC] IMA Log Snapshotting Design Proposal - network bandwidth

[Paul Moore]

On Wed, Sep 6, 2023 at 4:21 PM Ken Goldman <kgold@linux.ibm.com> wrote:
> On 9/1/2023 5:20 PM, Tushar Sugandhi wrote:
> > On 8/30/23 11:06, Ken Goldman wrote:
> >> On 8/1/2023 3:12 PM, Sush Shringarputale wrote:
> >>> In addition, a large IMA log can add pressure on the network
> >>> bandwidth when
> >>> the attestation client sends it to remote-attestation-service.
> >>
> >> I would not worry too much about network bandwidth.
> > Our bandwidth concerns are about scaled out system.
> >
> > When IMA log size increases in the range of megabytes, and when the
> > number of client devices increases, it makes an impact on the overall
> > network bandwidth.
>
> It should not, because the client only sends new measurements.  It only
> sends the entire list once per boot.
>
> Does a megabyte matter in a modern network? As for overall performance,
> a megabyte may take 10 msec, while the TPM quote could take 1000 msec,
> and verifier hash and asymmetric signature checks are also slower.

I think there are two issues here: the first is the attestation
methodology, the second is simply the size of the deployment.

There is rarely just one answer to a question, and in the case of
remote attestation I believe that holds true.  Sending some delta of
measurements to a remote node performing attestation does reduce the
amount of network traffic, but it does add an additional burden of
state tracking to the attestation node.  Sending the full measurement
log decreases this tracking burden, but it does result in more network
traffic.  Arguably the "best" choice is likely going to be dependent
on a number of complex factors including the size and complexity of
the deployment.

However, the snapshotting work is not about managing network traffic,
it is about mitigating an unbounded memory buffer that has been
causing problems in at least one real world deployment.  The IMA
measurement log snapshot is designed to allow an admin, or some other
privileged entity, to checkpoint the log and trim the old entries in
such a way as to preserve the ability to perform a meaningful
attestation without having to maintain the entire measurement log in a
memory buffer.

-- 
paul-moore.com

Re: [RFC] IMA Log Snapshotting Design Proposal - aggregate

[Ken Goldman]

On 8/1/2023 3:12 PM, Sush Shringarputale wrote:
> - A user-mode process will trigger the snapshot by opening a file in SysFS
>    say /sys/kernel/security/ima/snapshot (referred to as 
> sysk_ima_snapshot_file
>    here onwards).
> - The Kernel will get the current TPM PCR values and PCR update counter [2]
>    and store them as template data in a new IMA event "snapshot_aggregate".

If this is relying on a user-mode process, is there a concern that the 
process doesn't run. Might it be safer to have the kernel trigger the
snapshot.

PCR reads are not atomic, with each other and with event log appends. 
Is this an issue?

The PCR update counter can change between PCR reads.  What is its purpose?

What is the purpose of the snapshot aggregate?  Since the entire event 
log has to be retained and sent to the verifier, is the aggregate redundant?

Re: [RFC] IMA Log Snapshotting Design Proposal - aggregate

[Tushar Sugandhi]

On 8/30/23 11:12, Ken Goldman wrote:
> On 8/1/2023 3:12 PM, Sush Shringarputale wrote:
>> - A user-mode process will trigger the snapshot by opening a file in 
>> SysFS
>>    say /sys/kernel/security/ima/snapshot (referred to as 
>> sysk_ima_snapshot_file
>>    here onwards).
>> - The Kernel will get the current TPM PCR values and PCR update 
>> counter [2]
>>    and store them as template data in a new IMA event 
>> "snapshot_aggregate".
> 
> If this is relying on a user-mode process, is there a concern that the 
> process doesn't run. Might it be safer to have the kernel trigger the
> snapshot.
> 
The UM process here would be typically an attestation client
which passes on the IMA log to the remote service for attestation.
If the process doesn't run, the client will operate the same way as it
does currently.

Kernel triggering snapshot would come with its own issues of Kernel
storing the snapshot on some persistent file-system. They are being
discussed on the main thread [1].

> PCR reads are not atomic, with each other and with event log appends. Is 
> this an issue?
> 
In this design, reading the PCR plus adding the snapshot_aggregate
has to be an atomic operation.  Other IMA events shouldn't interfere
with this operation. Just like IMA ensures adding an entry to the log
plus PCR extension happens in an atomic way by holding the
ima_extend_list_mutex [2], we intend to use a similar mechanism to
ensure reading the PCR plus adding the snapshot_aggregate remains an
atomic operation.  And since taking a snapshot would be a rare event
compared to adding a generic event to IMA log - overall we expect a low
overhead in case of snapshotting.

However, please note that the event addition to IMA logs will *not*
be paused while the log is written out to disk by the UM process.

> The PCR update counter can change between PCR reads.  What is its purpose?
> 
Earlier we believed the PCR update counter will help with keeping track
of events in the IMA log snapshot. But I soon realized (and it was
also pointed out by Stefan Berger on the PCR update counter patch-series 
[3]) that the update counter gets incremented on updates to any PCR
(including the PCRs not touched by IMA).

I agree that update counter may not be a reliable marker for this
particular feature.

We have put that series [3] on hold for other higher priority work items
in the IMA/kexec space.

> What is the purpose of the snapshot aggregate?  Since the entire event 
> log has to be retained and sent to the verifier, is the aggregate 
> redundant?

The goals of snapshot_aggregate marker are:
     1. To allow the IMA log to be divided into multiple chunks and
        provide attestation service the ability to verify and use the
        latest chunk (i.e. snapshot ) for attestation.

     2. To indicate to the attestation service that the client device has
        IMA log snapshotting feature enabled, and at least one snapshot
        is taken.  So that the service can ask for previous snapshots
        as needed.

     3. In the event of multiple snapshots, the snapshot_aggregate
        marker has sufficient information to verify the integrity
        of latest subset of isolated snapshots (with the help of PCR
        quote of course)

     4. snapshot_aggregate helps both kernel and UM define clear
        boundaries between multiple snapshots.
        (each new snapshot starts with either the first boot_aggregate
         or a snapshot_aggregate event)

The overall goals of IMA log snapshotting feature are:
     a. to relieve memory pressure on the client device.

     b. to make attestation service side processing more efficient
        They don't have to deal with the entire log since boot,
        as you mentioned on

     c. Reduce network bandwidth usage by sending less data
        for each attestation request.

We missed stating them explicitly in the original RFC proposal we
sent. We will add them in the next version of the proposal.

References:

[1] Re: [RFC] IMA Log Snapshotting Design Proposal - Paul Moore 
(kernel.org)
https://lore.kernel.org/linux-integrity/CAHC9VhQbnyd2nvmL-t=3kXppsm985ps+NPJ5QDvM1WSS-Hd_Ew@mail.gmail.com/


[2] ima_extend_list_mutex
https://elixir.bootlin.com/linux/latest/source/security/integrity/ima/ima_queue.c#L159

[3] Re: [PATCH 0/6] Measuring TPM update counter in IMA - Stefan Berger 
(kernel.org)
https://lore.kernel.org/linux-integrity/a4a5e40b-abc1-27fa-3984-cee18fb4522c@linux.ibm.com/

Thanks,

Re: [RFC] IMA Log Snapshotting Design Proposal - aggregate

[Ken Goldman]

On 9/1/2023 6:06 PM, Tushar Sugandhi wrote:
> 
> 
> On 8/30/23 11:12, Ken Goldman wrote:
>> On 8/1/2023 3:12 PM, Sush Shringarputale wrote:
>>> - A user-mode process will trigger the snapshot by opening a file in 
>>> SysFS
>>>    say /sys/kernel/security/ima/snapshot (referred to as 
>>> sysk_ima_snapshot_file
>>>    here onwards).
>>> - The Kernel will get the current TPM PCR values and PCR update 
>>> counter [2]
>>>    and store them as template data in a new IMA event 
>>> "snapshot_aggregate".
>>
>> If this is relying on a user-mode process, is there a concern that the 
>> process doesn't run. Might it be safer to have the kernel trigger the
>> snapshot.
>>
> The UM process here would be typically an attestation client
> which passes on the IMA log to the remote service for attestation.
> If the process doesn't run, the client will operate the same way as it
> does currently.

I see.

1. Ensure that the attestation client stores the snapshot in a 
well-known and widely readable location.  There can be more than one 
attestation client, and all need access to the snapshot.

There is a privacy concern around making the snapshot world-read.

2. Is there a concern that, if the client doesn't run, it doesn't solve 
the kernel memory issue?  Is this relying on a UM process to solve a 
kernel issue?
> 
>> PCR reads are not atomic, with each other and with event log appends. 
>> Is this an issue?
>>
> In this design, reading the PCR plus adding the snapshot_aggregate
> has to be an atomic operation.  Other IMA events shouldn't interfere
> with this operation. Just like IMA ensures adding an entry to the log
> plus PCR extension happens in an atomic way by holding the
> ima_extend_list_mutex [2], we intend to use a similar mechanism to
> ensure reading the PCR plus adding the snapshot_aggregate remains an
> atomic operation.  And since taking a snapshot would be a rare event
> compared to adding a generic event to IMA log - overall we expect a low
> overhead in case of snapshotting.

How would that work?  The PCR read is UM, but IMA events are kernel. The 
UM operation cannot block the kernel or there can be a deadlock, right?

(UM) PCR reads can take multiple TPM commands, and they should not block 
an (kernel) extend.

>> What is the purpose of the snapshot aggregate?  Since the entire event 
>> log has to be retained and sent to the verifier, is the aggregate 
>> redundant?
> 
> The goals of snapshot_aggregate marker are:
>      1. To allow the IMA log to be divided into multiple chunks and
>         provide attestation service the ability to verify and use the
>         latest chunk (i.e. snapshot ) for attestation.

I believe that the verifier needs the entire log the first time, whether 
there is a snapshot or not.  Shouldn't the snapshot process be opaque to 
the verifier?

> 
>      2. To indicate to the attestation service that the client device has
>         IMA log snapshotting feature enabled, and at least one snapshot
>         is taken.  So that the service can ask for previous snapshots
>         as needed.

Why does the verifier need this?  The first time, it asks for events 
starting at #0.  Next time, it asks for what's new.  It's independent of 
__where__ the log comes from.

> 
>      3. In the event of multiple snapshots, the snapshot_aggregate
>         marker has sufficient information to verify the integrity
>         of latest subset of isolated snapshots (with the help of PCR
>         quote of course)

A new verifier needs the entire log, no matter how many snapshots have 
been taken.

> 
>      4. snapshot_aggregate helps both kernel and UM define clear
>         boundaries between multiple snapshots.
>         (each new snapshot starts with either the first boot_aggregate
>          or a snapshot_aggregate event)
> 
> The overall goals of IMA log snapshotting feature are:
>      a. to relieve memory pressure on the client device.
> 
>      b. to make attestation service side processing more efficient
>         They don't have to deal with the entire log since boot,
>         as you mentioned on

I don't think snapshotting affects the verifier at all. The attestor is 
a bit more complicated, but not significantly.

Re: [RFC] IMA Log Snapshotting Design Proposal - aggregate

[Paul Moore]

On Wed, Sep 6, 2023 at 4:49 PM Ken Goldman <kgold@linux.ibm.com> wrote:
> On 9/1/2023 6:06 PM, Tushar Sugandhi wrote:
> > On 8/30/23 11:12, Ken Goldman wrote:
> >> On 8/1/2023 3:12 PM, Sush Shringarputale wrote:
> >>> - A user-mode process will trigger the snapshot by opening a file in
> >>> SysFS
> >>>    say /sys/kernel/security/ima/snapshot (referred to as
> >>> sysk_ima_snapshot_file
> >>>    here onwards).
> >>> - The Kernel will get the current TPM PCR values and PCR update
> >>> counter [2]
> >>>    and store them as template data in a new IMA event
> >>> "snapshot_aggregate".
> >>
> >> If this is relying on a user-mode process, is there a concern that the
> >> process doesn't run. Might it be safer to have the kernel trigger the
> >> snapshot.
> >>
> > The UM process here would be typically an attestation client
> > which passes on the IMA log to the remote service for attestation.
> > If the process doesn't run, the client will operate the same way as it
> > does currently.
>
> I see.
>
> 1. Ensure that the attestation client stores the snapshot in a
> well-known and widely readable location.  There can be more than one
> attestation client, and all need access to the snapshot.

A few points:

* There is no requirement for an admin or solution provider to support
or otherwise use the IMA measurement log snapshotting functionality,
even if enabled in the kernel it is opt-in at runtime (i.e. simply
don't trigger the snapshot).  If the deployment in question is using
an attestation solution which is not compatible with the snapshot
concept then there is no need to perform a snapshot, the system will
behave just as it would today; nothing gained, but nothing lost.

* One of the benefits of initiating the snapshot in userspace is that
it affords solution providers a tremendous amount of flexibility with
respect to how to manage the IMA measurement log.  Not only can
different, and potentially more complex, logic be used to determine
the appropriate time to trigger the snapshot, but the trimmed/old log
can be processed any way the admin sees fit; writing and supporting
userspace code is many orders of magnitude easier than kernel code.

> There is a privacy concern around making the snapshot world-read.

See the above points.  If security requirements can not be satisfied
by any of the various solutions designed to protect the integrity of
system configuration and log data, the snapshotting functionality can
always be blocked at runtime by another collection of access control
solutions.

> 2. Is there a concern that, if the client doesn't run, it doesn't solve
> the kernel memory issue?

Yes, the design relies on a userspace process to initiate, and
complete the snapshotting process.  The good news is that if the
userspace process fails the system is no worse off than it is today,
but modern init-systems, e.g. systemd, should help ensure the
reliability of the userspace snapshot process/daemon.

> Is this relying on a UM process to solve a  kernel issue?

The ultimate problem is that we have an unbounded memory buffer that
we can't enforce limits on in the traditional sense.  The design here
proposes a checkpoint system which allows us to mitigate the
uncontrolled growth of this buffer while preserving the ability to
remotely attest the system (although perhaps with different, or
modified attestation tools).

I have not seen any other options for a viable, kernel driven solution
in any of the discussions thus far, but if you have any suggestions I
think we would all be very interested :)  The tmpfs based solution
doesn't solve the problem of system-wide memory pressure as tmpfs is
still a memory-backed filesystem.  Passing a fd to the kernel is still
a userspace initiated action, with the added problem of requiring the
kernel do the I/O itself.

-- 
paul-moore.com

Re: [RFC] IMA Log Snapshotting Design Proposal - unseal

[Ken Goldman]

On 8/1/2023 3:12 PM, Sush Shringarputale wrote:

> For remote attestation to work, the service will need to know how to
>  validate the snapshot_aggregate entry in the IMA log.  It will have
> to read the PCR values present in the template data of
> snapshot_aggregate event in the latest IMA log, and ensure that the
> PCR quotes align with the contents of the past UM_snapshot_file(s).
> This will re-establish the chain of trust needed for the device to
> pass remote attestation.  This will also maintain the ability of the
> remote-attestation-service to seal the secrets, if the client-server
>  use TPM unseal mechanism to attest the state of the device.

I think that seal/unseal to IMA PCRs is futile.  Since boot is
multi-threaded, the IMA PCR is unpredictable even when valid.

Re: [RFC] IMA Log Snapshotting Design Proposal - unseal

[Dr. Greg]

On Wed, Aug 30, 2023 at 03:12:39PM -0400, Ken Goldman wrote:

Good morning, I hope the day is going well for everyone.

> On 8/1/2023 3:12 PM, Sush Shringarputale wrote:
> 
> >For remote attestation to work, the service will need to know how to
> > validate the snapshot_aggregate entry in the IMA log.  It will have
> >to read the PCR values present in the template data of
> >snapshot_aggregate event in the latest IMA log, and ensure that the
> >PCR quotes align with the contents of the past UM_snapshot_file(s).
> >This will re-establish the chain of trust needed for the device to
> >pass remote attestation.  This will also maintain the ability of the
> >remote-attestation-service to seal the secrets, if the client-server
> > use TPM unseal mechanism to attest the state of the device.

> I think that seal/unseal to IMA PCRs is futile.  Since boot is
> multi-threaded, the IMA PCR is unpredictable even when valid.

Yes, unbiased observation calls into question the relevancy of the
2000-2002/Palladium model for TPM based integrity attestation, both
from a hardware and software perspective.  In addition to interference
by standard scheduling artifacts, the notion of everything being SMP
makes PCR invariancy a hopeless issue.

In fact, our trust orchestrators demonstrate that the very act of
modeling, ie computing file digests, causes an event ordering that
will be different from subsequent invocations when the digest value is
cached.  A concept that should be of no surprise to anyone fluent in
the writings of Heisenberg... :-)

We discuss this at some length, along with our proposed solution, in
the documentation that we provide with our TSEM LSM:

https://lore.kernel.org/linux-security-module/20230710102319.19716-1-greg@enjellic.com/T/#t

TSEM is a superset of IMA, given that file checksums are just one of
the components in the security state coefficients that our security
modeling is based on, but the issue is the same.

Having an invariant 'security state' value also simplifies the
attestation processw, since the relying party only needs to evaluate a
single signed value in order to verify that the workload has been
consistent with its desired security model.

It is also becoming increasingly apparent that advancing the art of
integrity and trusted systems will become problematic without the
ability to 'namespace' or partition integrity on a workload by
workload basis.  We also introduce support for that with our security
modeling namespaces.

I missed copying the LSM list on my reply, but I discussed some of the
challenges that TDX, and COCO in general, raises to current integrity
infrastructures in the following thread, which should be easily
locatable on the LKML or COCO lists:

tsm: Introduce a shared ABI for attestation reports

Given our experiences, it is difficult to understand how the notion of
'legitimate' confidential computing is going to grow from 7 billion
dollars to 55 billion dollars over the next five years without the
concept of better 'trust orchestration'.

Have a good remainder of the week.

As always,
Dr. Greg

The Quixote Project - Flailing at the Travails of Cybersecurity

Re: [RFC] IMA Log Snapshotting Design Proposal - unseal

[Tushar Sugandhi]

On 8/30/23 12:12, Ken Goldman wrote:
> On 8/1/2023 3:12 PM, Sush Shringarputale wrote:
> 
>> For remote attestation to work, the service will need to know how to
>>  validate the snapshot_aggregate entry in the IMA log.  It will have
>> to read the PCR values present in the template data of
>> snapshot_aggregate event in the latest IMA log, and ensure that the
>> PCR quotes align with the contents of the past UM_snapshot_file(s).
>> This will re-establish the chain of trust needed for the device to
>> pass remote attestation.  This will also maintain the ability of the
>> remote-attestation-service to seal the secrets, if the client-server
>>  use TPM unseal mechanism to attest the state of the device.
> 
> I think that seal/unseal to IMA PCRs is futile.  Since boot is
> multi-threaded, the IMA PCR is unpredictable even when valid.

True. But here we are talking about seal/unseal post boot when the
device is in a stable state, and there are relatively less number of
events extending IMA PCR. The value of the actual IMA PCR doesn't matter
in this context as long as it stays the same between seal-unseal window.

If it changes between that window, the clients typically retry by
sending the request to the service with a new stable PCR.

Seal-unseal is supported by TPM spec, and is used widely. So we had to
ensure that our proposed design wouldn't regress this existing
functionality.

~Tushar

Re: [RFC] IMA Log Snapshotting Design Proposal - unseal

[Ken Goldman]

On 9/1/2023 5:22 PM, Tushar Sugandhi wrote:
> 
> 
> On 8/30/23 12:12, Ken Goldman wrote:
>> On 8/1/2023 3:12 PM, Sush Shringarputale wrote:
>>
>>> For remote attestation to work, the service will need to know how to
>>>  validate the snapshot_aggregate entry in the IMA log.  It will have
>>> to read the PCR values present in the template data of
>>> snapshot_aggregate event in the latest IMA log, and ensure that the
>>> PCR quotes align with the contents of the past UM_snapshot_file(s).
>>> This will re-establish the chain of trust needed for the device to
>>> pass remote attestation.  This will also maintain the ability of the
>>> remote-attestation-service to seal the secrets, if the client-server
>>>  use TPM unseal mechanism to attest the state of the device.
>>
>> I think that seal/unseal to IMA PCRs is futile.  Since boot is
>> multi-threaded, the IMA PCR is unpredictable even when valid.
> 
> True. But here we are talking about seal/unseal post boot when the
> device is in a stable state, and there are relatively less number of
> events extending IMA PCR. The value of the actual IMA PCR doesn't matter
> in this context as long as it stays the same between seal-unseal window.

Does 'relatively less number of events' really matter?  Even if there 
are only two, if the order is unpredictable, unseal fails.

> 
> If it changes between that window, the clients typically retry by
> sending the request to the service with a new stable PCR.

Does a retry help? Once the PCR has been extended to the wrong value, it 
can never get back to the correct value without a reboot.

> 
> Seal-unseal is supported by TPM spec, and is used widely. So we had to
> ensure that our proposed design wouldn't regress this existing
> functionality.

It works in the pre-OS environment, where the firmware specification 
mandates that the PCR values are repeatable. Once you're post-OS, 
multi-threaded, an unseal that includes PCR 10 is infeasible.