* [PATCH 1/2] rust: zerocopy: fix link to upstream `README.md`
@ 2026-10-13 18:48 Miguel Ojeda
2026-10-13 18:48 ` [PATCH 2/2] rust: zerocopy: update to v0.8.62 Miguel Ojeda
0 siblings, 1 reply; 2+ messages in thread
From: Miguel Ojeda @ 2026-10-13 18:48 UTC (permalink / raw)
To: Miguel Ojeda
Cc: Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Andreas Hindborg, Alice Ryhl, Trevor Gross, Danilo Krummrich,
Daniel Almeida, Tamir Duberstein, Alexandre Courbot,
Onur Özkan, rust-for-linux
Upstream moved the `zerocopy` crate into a `zerocopy/` subdirectory
of their repository in v0.8.51 [1]. However, in our v0.8.52 update,
the link was not updated.
Fix it by adding the subdirectory.
Fixes: 2808a3963988 ("rust: zerocopy: update to v0.8.52")
Link: https://github.com/google/zerocopy/pull/3434 [1]
Assisted-by: LLM
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
---
rust/zerocopy-derive/README.md | 2 +-
rust/zerocopy/README.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/rust/zerocopy-derive/README.md b/rust/zerocopy-derive/README.md
index d62c79804342..49de31b8e082 100644
--- a/rust/zerocopy-derive/README.md
+++ b/rust/zerocopy-derive/README.md
@@ -8,7 +8,7 @@ non-ASCII identifiers.
For copyright details, please see:
- https://github.com/google/zerocopy/blob/v0.8.54/README.md?plain=1
+ https://github.com/google/zerocopy/blob/v0.8.54/zerocopy/README.md?plain=1
https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-BSD
https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-APACHE
https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-MIT
diff --git a/rust/zerocopy/README.md b/rust/zerocopy/README.md
index 3d11a6502cf0..8443f6166008 100644
--- a/rust/zerocopy/README.md
+++ b/rust/zerocopy/README.md
@@ -7,7 +7,7 @@ modified to tweak the SPDX license identifiers.
For copyright details, please see:
- https://github.com/google/zerocopy/blob/v0.8.54/README.md?plain=1
+ https://github.com/google/zerocopy/blob/v0.8.54/zerocopy/README.md?plain=1
https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-BSD
https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-APACHE
https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-MIT
base-commit: 3132a951a7e4ec2e18bc314877137deee96ab83e
--
2.56.0
^ permalink raw reply related [flat|nested] 2+ messages in thread
* [PATCH 2/2] rust: zerocopy: update to v0.8.62
2026-10-13 18:48 [PATCH 1/2] rust: zerocopy: fix link to upstream `README.md` Miguel Ojeda
@ 2026-10-13 18:48 ` Miguel Ojeda
0 siblings, 0 replies; 2+ messages in thread
From: Miguel Ojeda @ 2026-10-13 18:48 UTC (permalink / raw)
To: Miguel Ojeda
Cc: Boqun Feng, Gary Guo, Björn Roy Baron, Benno Lossin,
Andreas Hindborg, Alice Ryhl, Trevor Gross, Danilo Krummrich,
Daniel Almeida, Tamir Duberstein, Alexandre Courbot,
Onur Özkan, rust-for-linux, Joshua Liebow-Feeser, Jack Wrenn
Update our vendored copy of `zerocopy` (and `zerocopy-derive`) to v0.8.62.
This update advances 8 releases, with 3 of them carrying most of the
changes. It contains new features as well as fixes [1]. However, almost
nothing has changed that requires updates on our side: we only need to
move our non-ASCII change to the recently added `project.rs` file, and
to add an SPDX license identifier to the also new `invariant.rs` file
(I reported it to upstream [2]).
The following script may be used to check for the remaining differences:
for path in $(cd rust/zerocopy-derive/ && find . -type f ! -name README.md); do
curl --silent --show-error --location \
https://github.com/google/zerocopy/raw/v0.8.62/zerocopy/zerocopy-derive/src/$path |
git diff --no-index - rust/zerocopy-derive/$path &&
echo $path: OK
done
for path in $(cd rust/zerocopy/ && find . -type f ! -name README.md); do
curl --silent --show-error --location \
https://github.com/google/zerocopy/raw/v0.8.62/zerocopy/$path |
git diff --no-index - rust/zerocopy/$path &&
echo $path: OK
done
Comparing the output above with the one for the latest update (v0.8.54)
shows no new differences on our side apart from the two mentioned above.
Cc: Joshua Liebow-Feeser <joshlf@google.com>
Cc: Jack Wrenn <jswrenn@amazon.com>
Link: https://github.com/google/zerocopy/releases [1]
Link: https://github.com/google/zerocopy/issues/3858 [2]
Assisted-by: LLM
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
---
rust/Makefile | 2 +-
rust/zerocopy-derive/README.md | 12 +-
rust/zerocopy-derive/derive/from_bytes.rs | 12 +
rust/zerocopy-derive/derive/into_bytes.rs | 147 +-
rust/zerocopy-derive/derive/known_layout.rs | 4 +-
rust/zerocopy-derive/derive/mod.rs | 33 +-
rust/zerocopy-derive/derive/project.rs | 569 +++
rust/zerocopy-derive/derive/try_from_bytes.rs | 869 ++--
rust/zerocopy-derive/invariant.rs | 116 +
rust/zerocopy-derive/lib.rs | 118 +-
rust/zerocopy-derive/repr.rs | 46 +-
rust/zerocopy-derive/util.rs | 414 +-
rust/zerocopy/README.md | 12 +-
.../benches/as_bytes_dynamic_size.x86-64 | 3 +-
.../benches/as_bytes_dynamic_size.x86-64.mca | 20 +-
.../zerocopy/benches/extend_vec_zeroed.x86-64 | 4 +-
.../benches/extend_vec_zeroed.x86-64.mca | 8 +-
.../zerocopy/benches/insert_vec_zeroed.x86-64 | 4 +-
.../benches/insert_vec_zeroed.x86-64.mca | 8 +-
rust/zerocopy/benches/new_vec_zeroed.x86-64 | 3 +-
.../benches/new_vec_zeroed.x86-64.mca | 46 +-
.../ref_from_bytes_dynamic_padding.x86-64 | 31 +-
.../ref_from_bytes_dynamic_padding.x86-64.mca | 70 +-
.../ref_from_bytes_dynamic_size.x86-64 | 21 +-
.../ref_from_bytes_dynamic_size.x86-64.mca | 58 +-
.../benches/ref_from_bytes_static_size.x86-64 | 7 +-
.../ref_from_bytes_static_size.x86-64.mca | 26 +-
.../ref_from_prefix_dynamic_padding.x86-64 | 24 +-
...ref_from_prefix_dynamic_padding.x86-64.mca | 62 +-
.../ref_from_prefix_dynamic_size.x86-64 | 21 +-
.../ref_from_prefix_dynamic_size.x86-64.mca | 52 +-
.../ref_from_suffix_dynamic_padding.x86-64 | 29 +-
...ref_from_suffix_dynamic_padding.x86-64.mca | 68 +-
...m_suffix_with_elems_dynamic_padding.x86-64 | 36 +-
...ffix_with_elems_dynamic_padding.x86-64.mca | 84 +-
...from_suffix_with_elems_dynamic_size.x86-64 | 31 +-
..._suffix_with_elems_dynamic_size.x86-64.mca | 70 +-
.../benches/ref_type_bytes_dynamic_padding.rs | 19 +
.../ref_type_bytes_dynamic_padding.x86-64 | 4 +
.../ref_type_bytes_dynamic_padding.x86-64.mca | 45 +
.../benches/ref_type_bytes_dynamic_size.rs | 17 +
.../ref_type_bytes_dynamic_size.x86-64 | 4 +
.../ref_type_bytes_dynamic_size.x86-64.mca | 45 +
.../ref_type_bytes_mut_dynamic_padding.rs | 19 +
.../ref_type_bytes_mut_dynamic_padding.x86-64 | 4 +
..._type_bytes_mut_dynamic_padding.x86-64.mca | 45 +
.../ref_type_bytes_mut_dynamic_size.rs | 19 +
.../ref_type_bytes_mut_dynamic_size.x86-64 | 4 +
...ref_type_bytes_mut_dynamic_size.x86-64.mca | 45 +
.../benches/ref_type_bytes_mut_static_size.rs | 19 +
.../ref_type_bytes_mut_static_size.x86-64 | 4 +
.../ref_type_bytes_mut_static_size.x86-64.mca | 45 +
.../benches/ref_type_bytes_static_size.rs | 17 +
.../benches/ref_type_bytes_static_size.x86-64 | 4 +
.../ref_type_bytes_static_size.x86-64.mca | 45 +
.../benches/ref_type_deref_dynamic_padding.rs | 19 +
.../ref_type_deref_dynamic_padding.x86-64 | 29 +
.../ref_type_deref_dynamic_padding.x86-64.mca | 91 +
.../benches/ref_type_deref_dynamic_size.rs | 19 +
.../ref_type_deref_dynamic_size.x86-64 | 27 +
.../ref_type_deref_dynamic_size.x86-64.mca | 87 +
.../ref_type_deref_mut_dynamic_size.rs | 19 +
.../ref_type_deref_mut_dynamic_size.x86-64 | 27 +
...ref_type_deref_mut_dynamic_size.x86-64.mca | 87 +
.../benches/ref_type_deref_mut_static_size.rs | 19 +
.../ref_type_deref_mut_static_size.x86-64 | 3 +
.../ref_type_deref_mut_static_size.x86-64.mca | 43 +
.../benches/ref_type_deref_static_size.rs | 19 +
.../benches/ref_type_deref_static_size.x86-64 | 3 +
.../ref_type_deref_static_size.x86-64.mca | 43 +
.../ref_type_from_bytes_dynamic_padding.rs | 19 +
...ref_type_from_bytes_dynamic_padding.x86-64 | 25 +
...type_from_bytes_dynamic_padding.x86-64.mca | 85 +
.../ref_type_from_bytes_dynamic_size.rs | 17 +
.../ref_type_from_bytes_dynamic_size.x86-64 | 21 +
...ef_type_from_bytes_dynamic_size.x86-64.mca | 77 +
.../ref_type_from_bytes_static_size.rs | 17 +
.../ref_type_from_bytes_static_size.x86-64 | 15 +
...ref_type_from_bytes_static_size.x86-64.mca | 65 +
...e_from_bytes_with_elems_dynamic_padding.rs | 20 +
...om_bytes_with_elems_dynamic_padding.x86-64 | 26 +
...ytes_with_elems_dynamic_padding.x86-64.mca | 85 +
...type_from_bytes_with_elems_dynamic_size.rs | 20 +
..._from_bytes_with_elems_dynamic_size.x86-64 | 28 +
...m_bytes_with_elems_dynamic_size.x86-64.mca | 89 +
.../ref_type_from_prefix_dynamic_padding.rs | 22 +
...ef_type_from_prefix_dynamic_padding.x86-64 | 26 +
...ype_from_prefix_dynamic_padding.x86-64.mca | 87 +
.../ref_type_from_prefix_dynamic_size.rs | 22 +
.../ref_type_from_prefix_dynamic_size.x86-64 | 19 +
...f_type_from_prefix_dynamic_size.x86-64.mca | 73 +
.../ref_type_from_prefix_static_size.rs | 20 +
.../ref_type_from_prefix_static_size.x86-64 | 15 +
...ef_type_from_prefix_static_size.x86-64.mca | 65 +
..._from_prefix_with_elems_dynamic_padding.rs | 23 +
...m_prefix_with_elems_dynamic_padding.x86-64 | 26 +
...efix_with_elems_dynamic_padding.x86-64.mca | 85 +
...ype_from_prefix_with_elems_dynamic_size.rs | 23 +
...from_prefix_with_elems_dynamic_size.x86-64 | 27 +
..._prefix_with_elems_dynamic_size.x86-64.mca | 87 +
.../ref_type_from_suffix_dynamic_padding.rs | 22 +
...ef_type_from_suffix_dynamic_padding.x86-64 | 31 +
...ype_from_suffix_dynamic_padding.x86-64.mca | 97 +
.../ref_type_from_suffix_dynamic_size.rs | 22 +
.../ref_type_from_suffix_dynamic_size.x86-64 | 23 +
...f_type_from_suffix_dynamic_size.x86-64.mca | 81 +
.../ref_type_from_suffix_static_size.rs | 20 +
.../ref_type_from_suffix_static_size.x86-64 | 18 +
...ef_type_from_suffix_static_size.x86-64.mca | 71 +
..._from_suffix_with_elems_dynamic_padding.rs | 23 +
...m_suffix_with_elems_dynamic_padding.x86-64 | 27 +
...ffix_with_elems_dynamic_padding.x86-64.mca | 87 +
...ype_from_suffix_with_elems_dynamic_size.rs | 23 +
...from_suffix_with_elems_dynamic_size.x86-64 | 27 +
..._suffix_with_elems_dynamic_size.x86-64.mca | 87 +
.../benches/ref_type_into_mut_dynamic_size.rs | 19 +
.../ref_type_into_mut_dynamic_size.x86-64 | 27 +
.../ref_type_into_mut_dynamic_size.x86-64.mca | 87 +
.../benches/ref_type_into_mut_static_size.rs | 19 +
.../ref_type_into_mut_static_size.x86-64 | 3 +
.../ref_type_into_mut_static_size.x86-64.mca | 43 +
.../ref_type_into_ref_dynamic_padding.rs | 19 +
.../ref_type_into_ref_dynamic_padding.x86-64 | 29 +
...f_type_into_ref_dynamic_padding.x86-64.mca | 91 +
.../benches/ref_type_into_ref_dynamic_size.rs | 19 +
.../ref_type_into_ref_dynamic_size.x86-64 | 27 +
.../ref_type_into_ref_dynamic_size.x86-64.mca | 87 +
.../benches/ref_type_into_ref_static_size.rs | 19 +
.../ref_type_into_ref_static_size.x86-64 | 3 +
.../ref_type_into_ref_static_size.x86-64.mca | 43 +
.../benches/ref_type_read_static_size.rs | 17 +
.../benches/ref_type_read_static_size.x86-64 | 7 +
.../ref_type_read_static_size.x86-64.mca | 51 +
.../benches/ref_type_write_static_size.rs | 20 +
.../benches/ref_type_write_static_size.x86-64 | 6 +
.../ref_type_write_static_size.x86-64.mca | 49 +
...t_via_runtime_check_dynamic_padding.x86-64 | 5 +-
...a_runtime_check_dynamic_padding.x86-64.mca | 30 +-
.../benches/try_read_from_bytes.x86-64 | 28 +-
.../benches/try_read_from_bytes.x86-64.mca | 66 +-
.../benches/try_read_from_prefix.x86-64 | 19 +-
.../benches/try_read_from_prefix.x86-64.mca | 52 +-
.../benches/try_read_from_suffix.x86-64 | 21 +-
.../benches/try_read_from_suffix.x86-64.mca | 56 +-
.../try_ref_from_bytes_dynamic_padding.x86-64 | 34 +-
..._ref_from_bytes_dynamic_padding.x86-64.mca | 76 +-
.../try_ref_from_bytes_dynamic_size.x86-64 | 31 +-
...try_ref_from_bytes_dynamic_size.x86-64.mca | 70 +-
...try_ref_from_prefix_dynamic_padding.x86-64 | 21 +-
...ref_from_prefix_dynamic_padding.x86-64.mca | 54 +-
.../try_ref_from_prefix_dynamic_size.x86-64 | 16 +-
...ry_ref_from_prefix_dynamic_size.x86-64.mca | 56 +-
...try_ref_from_suffix_dynamic_padding.x86-64 | 29 +-
...ref_from_suffix_dynamic_padding.x86-64.mca | 74 +-
...m_suffix_with_elems_dynamic_padding.x86-64 | 31 +-
...ffix_with_elems_dynamic_padding.x86-64.mca | 80 +-
...from_suffix_with_elems_dynamic_size.x86-64 | 28 +-
..._suffix_with_elems_dynamic_size.x86-64.mca | 72 +-
.../benches/write_to_dynamic_size.x86-64 | 3 +-
.../benches/write_to_dynamic_size.x86-64.mca | 34 +-
.../write_to_prefix_dynamic_size.x86-64 | 3 +-
.../write_to_prefix_dynamic_size.x86-64.mca | 36 +-
.../write_to_suffix_dynamic_size.x86-64 | 3 +-
.../write_to_suffix_dynamic_size.x86-64.mca | 38 +-
.../zerocopy/benches/zero_dynamic_size.x86-64 | 3 +-
.../benches/zero_dynamic_size.x86-64.mca | 20 +-
rust/zerocopy/src/byte_slice.rs | 6 +-
rust/zerocopy/src/byteorder.rs | 41 +-
rust/zerocopy/src/error.rs | 14 +-
rust/zerocopy/src/impls.rs | 117 +-
rust/zerocopy/src/layout.rs | 3729 +++++++++++++++--
rust/zerocopy/src/lib.rs | 1081 ++++-
rust/zerocopy/src/macros.rs | 53 +-
rust/zerocopy/src/pointer/inner.rs | 35 +-
rust/zerocopy/src/pointer/invariant.rs | 61 +-
rust/zerocopy/src/pointer/mod.rs | 50 +-
rust/zerocopy/src/pointer/ptr.rs | 177 +-
rust/zerocopy/src/pointer/transmute.rs | 100 +-
rust/zerocopy/src/ref.rs | 762 +++-
rust/zerocopy/src/split_at.rs | 718 +++-
rust/zerocopy/src/util/macro_util.rs | 112 +-
rust/zerocopy/src/util/macros.rs | 127 +-
rust/zerocopy/src/util/mod.rs | 40 +-
rust/zerocopy/src/wrappers.rs | 262 +-
184 files changed, 12220 insertions(+), 3006 deletions(-)
create mode 100644 rust/zerocopy-derive/derive/project.rs
create mode 100644 rust/zerocopy-derive/invariant.rs
create mode 100644 rust/zerocopy/benches/ref_type_bytes_dynamic_padding.rs
create mode 100644 rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_bytes_dynamic_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.rs
create mode 100644 rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_bytes_mut_static_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_bytes_static_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_bytes_static_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_bytes_static_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_deref_dynamic_padding.rs
create mode 100644 rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_deref_dynamic_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_deref_mut_static_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_deref_static_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_deref_static_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_deref_static_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_static_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_static_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_static_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_into_mut_dynamic_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_into_mut_static_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.rs
create mode 100644 rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_into_ref_dynamic_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_into_ref_static_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_read_static_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_read_static_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_read_static_size.x86-64.mca
create mode 100644 rust/zerocopy/benches/ref_type_write_static_size.rs
create mode 100644 rust/zerocopy/benches/ref_type_write_static_size.x86-64
create mode 100644 rust/zerocopy/benches/ref_type_write_static_size.x86-64.mca
diff --git a/rust/Makefile b/rust/Makefile
index ddca144cb084..0ac9ab28cc2b 100644
--- a/rust/Makefile
+++ b/rust/Makefile
@@ -94,7 +94,7 @@ zerocopy-flags := \
$(call cfgs-to-flags,$(zerocopy-cfgs))
zerocopy-envs := \
- CARGO_PKG_VERSION=0.8.54
+ CARGO_PKG_VERSION=0.8.62
proc_macro2-cfgs := \
feature="proc-macro" \
diff --git a/rust/zerocopy-derive/README.md b/rust/zerocopy-derive/README.md
index 49de31b8e082..fab3779b9ed2 100644
--- a/rust/zerocopy-derive/README.md
+++ b/rust/zerocopy-derive/README.md
@@ -1,14 +1,14 @@
# `zerocopy-derive`
-These source files come from the Rust `zerocopy-derive` crate, version v0.8.54
-(released 2026-07-08), hosted in the <https://github.com/google/zerocopy>
+These source files come from the Rust `zerocopy-derive` crate, version v0.8.62
+(released 2026-10-08), hosted in the <https://github.com/google/zerocopy>
repository, licensed under "BSD-2-Clause OR Apache-2.0 OR MIT" and only
modified to tweak the SPDX license identifiers and to remove the generation of
non-ASCII identifiers.
For copyright details, please see:
- https://github.com/google/zerocopy/blob/v0.8.54/zerocopy/README.md?plain=1
- https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-BSD
- https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-APACHE
- https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-MIT
+ https://github.com/google/zerocopy/blob/v0.8.62/zerocopy/README.md?plain=1
+ https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-BSD
+ https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-APACHE
+ https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-MIT
diff --git a/rust/zerocopy-derive/derive/from_bytes.rs b/rust/zerocopy-derive/derive/from_bytes.rs
index 66d820f6ad4c..7fe505a86457 100644
--- a/rust/zerocopy-derive/derive/from_bytes.rs
+++ b/rust/zerocopy-derive/derive/from_bytes.rs
@@ -89,6 +89,12 @@ pub(crate) fn find_zero_variant(enm: &DataEnum) -> Result<usize, bool> {
Err(has_unknown_discriminants)
}
pub(crate) fn derive_from_zeros(ctx: &Ctx, top_level: Trait) -> Result<TokenStream, Error> {
+ if let Some(span) = ctx.invariant_span {
+ return ctx.error_or_skip(Error::new(
+ span,
+ "cannot derive `FromZeros` for a type with invariants",
+ ));
+ }
let try_from_bytes = derive_try_from_bytes(ctx, top_level)?;
let from_zeros = match &ctx.ast.data {
Data::Struct(strct) => derive_from_zeros_struct(ctx, strct),
@@ -98,6 +104,12 @@ pub(crate) fn derive_from_zeros(ctx: &Ctx, top_level: Trait) -> Result<TokenStre
Ok(IntoIterator::into_iter([try_from_bytes, from_zeros]).collect())
}
pub(crate) fn derive_from_bytes(ctx: &Ctx, top_level: Trait) -> Result<TokenStream, Error> {
+ if let Some(span) = ctx.invariant_span {
+ return ctx.error_or_skip(Error::new(
+ span,
+ "cannot derive `FromBytes` for a type with invariants",
+ ));
+ }
let from_zeros = derive_from_zeros(ctx, top_level)?;
let from_bytes = match &ctx.ast.data {
Data::Struct(strct) => derive_from_bytes_struct(ctx, strct),
diff --git a/rust/zerocopy-derive/derive/into_bytes.rs b/rust/zerocopy-derive/derive/into_bytes.rs
index 0103a78d087f..55c53de6b8c4 100644
--- a/rust/zerocopy-derive/derive/into_bytes.rs
+++ b/rust/zerocopy-derive/derive/into_bytes.rs
@@ -2,7 +2,7 @@
//
use proc_macro2::{Span, TokenStream};
use quote::quote;
-use syn::{Data, DataEnum, DataStruct, DataUnion, Error, Type};
+use syn::{parse_quote, Data, DataEnum, DataStruct, DataUnion, Error, Ident, Type, WherePredicate};
use crate::{
repr::{EnumRepr, StructUnionRepr},
@@ -18,6 +18,74 @@ pub(crate) fn derive_into_bytes(ctx: &Ctx, _top_level: Trait) -> Result<TokenStr
Data::Union(unn) => derive_into_bytes_union(ctx, unn),
}
}
+
+/// If every field is exactly `T`, `[T; _]`, or a final `[T]` for the same type
+/// parameter `T`, returns the bounds required to prove this to rustc.
+fn homogeneous_field_bounds(ctx: &Ctx, strct: &DataStruct) -> Option<Vec<WherePredicate>> {
+ fn strip_parens_and_groups(mut ty: &Type) -> &Type {
+ loop {
+ ty = match ty {
+ Type::Group(group) => &group.elem,
+ Type::Paren(paren) => &paren.elem,
+ ty => return ty,
+ };
+ }
+ }
+
+ fn type_is_parameter(ty: &Type, parameter: &Ident) -> bool {
+ match strip_parens_and_groups(ty) {
+ Type::Path(path) => path.qself.is_none() && path.path.is_ident(parameter),
+ _ => false,
+ }
+ }
+
+ let fields = strct.fields();
+ ctx.ast.generics.type_params().find_map(|parameter| {
+ let parameter = ¶meter.ident;
+ let elements = fields
+ .iter()
+ .enumerate()
+ .map(|(index, (_, _, ty))| match strip_parens_and_groups(ty) {
+ Type::Array(array) => Some(&*array.elem),
+ Type::Slice(slice) if index + 1 == fields.len() => Some(&*slice.elem),
+ Type::Slice(_) => None,
+ ty => Some(ty),
+ })
+ .collect::<Option<Vec<_>>>()?;
+
+ if !elements.iter().all(|element| type_is_parameter(element, parameter)) {
+ return None;
+ }
+
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let mut bounds = fields
+ .iter()
+ .map(|(_, _, ty)| parse_quote!(#ty: #zerocopy_crate::IntoBytes))
+ .collect::<Vec<WherePredicate>>();
+
+ // `Ident` equality is only a syntactic prefilter above. In
+ // macro-generated input, two identifiers can both print as `T` and
+ // compare equal here while having different hygiene contexts. For
+ // example, a call-site `T` could resolve to a `u8` type alias while a
+ // def-site `T` resolves to this struct's type parameter. Instantiating
+ // that parameter with `u16` would make the fields actually be `u8` and
+ // `u16`; treating them as homogeneous could overlook padding and cause
+ // us to emit an unsound `IntoBytes` impl.
+ //
+ // Quote every element type from its original field AST so that its
+ // hygiene context is retained, then use `Identity` to have rustc verify
+ // that it resolves to `parameter`. Do not deduplicate these predicates:
+ // identically-printed types may have different hygiene contexts and
+ // must each be checked by rustc.
+ bounds.extend(elements.into_iter().map(|element| {
+ parse_quote! {
+ #element: #zerocopy_crate::util::macro_util::Identity<Type = #parameter>
+ }
+ }));
+ Some(bounds)
+ })
+}
+
fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream, Error> {
let repr = StructUnionRepr::from_attrs(&ctx.ast.attrs)?;
@@ -25,8 +93,12 @@ fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream
let is_c = repr.is_c();
let is_packed_1 = repr.is_packed_1();
let num_fields = strct.fields().len();
+ let mut homogeneous_bounds =
+ if is_c && !repr.is_align_gt_1() { homogeneous_field_bounds(ctx, strct) } else { None };
- let (padding_check, require_unaligned_fields) = if is_transparent || is_packed_1 {
+ let (padding_check, require_unaligned_fields, explicit_field_bounds) = if is_transparent
+ || is_packed_1
+ {
// No padding check needed.
// - repr(transparent): The layout and ABI of the whole struct is the
// same as its only non-ZST field (meaning there's no padding outside
@@ -43,12 +115,12 @@ fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream
// An important consequence of these rules is that a type with
// `#[repr(packed(1))]`` (or `#[repr(packed)]``) will have no
// inter-field padding.
- (None, false)
+ (None, false, None)
} else if is_c && !repr.is_align_gt_1() && num_fields <= 1 {
// No padding check needed. A repr(C) struct with zero or one field has
// no padding unless #[repr(align)] explicitly adds padding, which we
// check for in this branch's condition.
- (None, false)
+ (None, false, None)
} else if ctx.ast.generics.params.is_empty() {
// Is the last field a syntactic slice, i.e., `[SomeType]`.
let is_syntactic_dst =
@@ -66,10 +138,31 @@ fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream
// 2. The fields do not overlap.
// ...
if is_c && is_syntactic_dst {
- (Some(PaddingCheck::ReprCStruct), false)
+ (Some(PaddingCheck::ReprCStruct), false, None)
} else {
- (Some(PaddingCheck::Struct), false)
+ (Some(PaddingCheck::Struct), false, None)
}
+ } else if let Some(bounds) = homogeneous_bounds.take() {
+ // Let `a` be the alignment of `T` and `s` be its size. Rust guarantees
+ // that `s` is a multiple of `a`. `T`, `[T; N]`, and `[T]` all have
+ // alignment `a`, and their sizes are `s`, `N * s`, and `len * s`,
+ // respectively.
+ //
+ // Without a `packed` modifier, each field and the struct have
+ // alignment `a`. `align(1)` does not change that because every Rust
+ // type already has alignment at least 1. With `packed(P)`, each field
+ // and the struct instead have alignment `b = min(a, P)`. Rust
+ // alignments and valid values of `P` are powers of two, so `b` divides
+ // `a`; every field size is therefore also a multiple of `b`.
+ //
+ // Thus, after placing any field, the running offset is already aligned
+ // for the next field, and the final offset is already aligned for the
+ // struct. A repr(C) without an `align` modifier greater than 1
+ // therefore introduces neither inter-field nor trailing padding,
+ // including for a slice DST. The bounds returned by
+ // `homogeneous_field_bounds` require rustc to verify that the element
+ // type of every field is the same type `T`.
+ (None, false, Some(bounds))
} else if is_c && !repr.is_align_gt_1() {
// We can't use a padding check since there are generic type arguments.
// Instead, we require all field types to implement `Unaligned`. This
@@ -79,7 +172,7 @@ fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream
//
// FIXME(#10): Support type parameters for non-transparent, non-packed
// structs without requiring `Unaligned`.
- (None, true)
+ (None, true, None)
} else {
return ctx.error_or_skip(Error::new(
Span::call_site(),
@@ -87,7 +180,9 @@ fn derive_into_bytes_struct(ctx: &Ctx, strct: &DataStruct) -> Result<TokenStream
));
};
- let field_bounds = if require_unaligned_fields {
+ let field_bounds = if let Some(bounds) = explicit_field_bounds {
+ FieldBounds::Explicit(bounds)
+ } else if require_unaligned_fields {
FieldBounds::All(&[TraitBound::Slf, TraitBound::Other(Trait::Unaligned)])
} else {
FieldBounds::ALL_SELF
@@ -162,3 +257,39 @@ fn derive_into_bytes_union(ctx: &Ctx, unn: &DataUnion) -> Result<TokenStream, Er
.build();
Ok(quote!(#cfg_compile_error #impl_block))
}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn derive(input: syn::DeriveInput) -> TokenStream {
+ let ctx = Ctx::try_from_derive_input(input).unwrap();
+ derive_into_bytes(&ctx, Trait::IntoBytes).unwrap()
+ }
+
+ #[test]
+ fn raw_repr_matches_ordinary_repr() {
+ let ordinary = derive(parse_quote! {
+ #[repr(C)]
+ #[repr(align(8))]
+ struct Response {
+ status: u8,
+ }
+ });
+ let raw = derive(parse_quote! {
+ #[repr(C)]
+ #[r#repr(r#align(8))]
+ struct Response {
+ status: u8,
+ }
+ });
+
+ let ordinary = ordinary.to_string();
+ let raw = raw.to_string();
+
+ // In particular, both expansions contain the padding check which
+ // prevents the resulting type from implementing `IntoBytes`.
+ assert!(ordinary.contains("PaddingFree"));
+ assert_eq!(ordinary, raw);
+ }
+}
diff --git a/rust/zerocopy-derive/derive/known_layout.rs b/rust/zerocopy-derive/derive/known_layout.rs
index d0c4cecfff15..d4f6ecff5730 100644
--- a/rust/zerocopy-derive/derive/known_layout.rs
+++ b/rust/zerocopy-derive/derive/known_layout.rs
@@ -81,7 +81,7 @@ fn raw_from_ptr_len(
#[inline(always)]
fn pointer_to_metadata(ptr: *mut Self) -> <Self as #zerocopy_crate::KnownLayout>::PointerMetadata {
- <#trailing_field_ty>::pointer_to_metadata(ptr as *mut _)
+ <#trailing_field_ty as #zerocopy_crate::KnownLayout>::pointer_to_metadata(ptr as *mut _)
}
}
};
@@ -144,6 +144,7 @@ fn pointer_to_metadata(ptr: *mut Self) -> <Self as #zerocopy_crate::KnownLayout>
// Define the collection of type-level field handles.
let field_defs = field_indices.iter().zip(fields).map(|(idx, (vis, _, _))| {
quote! {
+ #[allow(missing_debug_implementations)]
#vis struct #idx;
}
});
@@ -200,6 +201,7 @@ unsafe impl #impl_generics #zerocopy_crate::util::macro_util::Field<#idx> for #i
// `#ty`, not `__ZerocopyKnownLayoutMaybeUninit` (see #2116).
#repr
#[doc(hidden)]
+ #[allow(missing_debug_implementations)]
#vis struct __ZerocopyKnownLayoutMaybeUninit<#params> (
#(#core::mem::MaybeUninit<
<#ident #ty_generics as
diff --git a/rust/zerocopy-derive/derive/mod.rs b/rust/zerocopy-derive/derive/mod.rs
index b3839fcf73c9..4ccb71807ee7 100644
--- a/rust/zerocopy-derive/derive/mod.rs
+++ b/rust/zerocopy-derive/derive/mod.rs
@@ -3,6 +3,7 @@
pub mod from_bytes;
pub mod into_bytes;
pub mod known_layout;
+pub mod project;
pub mod try_from_bytes;
pub mod unaligned;
@@ -102,13 +103,6 @@ pub(crate) fn derive_split_at(ctx: &Ctx, _top_level: Trait) -> Result<TokenStrea
}
};
- if repr.get_packed().is_some() {
- return ctx.error_or_skip(Error::new(
- Span::call_site(),
- "must not have #[repr(packed)] attribute",
- ));
- }
-
if !(repr.is_c() || repr.is_transparent()) {
return ctx.error_or_skip(Error::new(
Span::call_site(),
@@ -124,13 +118,28 @@ pub(crate) fn derive_split_at(ctx: &Ctx, _top_level: Trait) -> Result<TokenStrea
};
let zerocopy_crate = &ctx.zerocopy_crate;
- // SAFETY: `#ty`, per the above checks, is `repr(C)` or `repr(transparent)`
- // and is not packed; its trailing field is guaranteed to be well-aligned
- // for its type. By invariant on `FieldBounds::TRAILING_SELF`, the trailing
- // slice of the trailing field is also well-aligned for its type.
+ let elem = quote!(<#trailing_field as #zerocopy_crate::SplitAt>::Elem);
+ let elem =
+ if repr.get_packed().is_some() { quote!(#zerocopy_crate::Unalign<#elem>) } else { elem };
+
+ // SAFETY: The checks above require `repr(C)` or `repr(transparent)`, and
+ // `FieldBounds::TRAILING_SELF` requires the trailing field to be `SplitAt`.
+ // Its `Elem` therefore has the size, bit validity, and `UnsafeCell`
+ // coverage of the actual trailing element. For an unpacked struct, the
+ // trailing field is aligned, so its `SplitAt` alignment guarantee applies.
+ // For a packed struct, wrapping its `Elem` in `Unalign` preserves those
+ // representation properties and gives alignment 1. Thus the trailing
+ // slice can be accessed as `[#elem]` even if packing misaligns its field.
+ // Packing does not change the field's internal layout [1], so its element
+ // size and byte offsets are preserved in either case.
+ //
+ // [1] Per https://doc.rust-lang.org/1.93.0/reference/type-layout.html#layout.repr.alignment.intro:
+ //
+ // `packed` may also alter the padding between fields (although it will
+ // not alter the padding inside of any field).
Ok(ImplBlockBuilder::new(ctx, &ctx.ast.data, Trait::SplitAt, FieldBounds::TRAILING_SELF)
.inner_extras(quote! {
- type Elem = <#trailing_field as #zerocopy_crate::SplitAt>::Elem;
+ type Elem = #elem;
})
.build())
}
diff --git a/rust/zerocopy-derive/derive/project.rs b/rust/zerocopy-derive/derive/project.rs
new file mode 100644
index 000000000000..1cfde70624a7
--- /dev/null
+++ b/rust/zerocopy-derive/derive/project.rs
@@ -0,0 +1,569 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+//
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use proc_macro2::TokenStream;
+use quote::quote;
+use syn::{
+ parse_quote, spanned::Spanned as _, Data, DataEnum, DeriveInput, Error, Expr, Fields, Ident,
+ Index, Type,
+};
+
+use crate::{
+ repr::{EnumRepr, StructUnionRepr},
+ util::{
+ const_block, generate_tag_enum, Client, Ctx, DataExt, FieldBounds, FieldProjection,
+ ImplBlockBuilder, Trait,
+ },
+};
+
+pub(crate) fn tag_ident(variant: &Ident) -> Ident {
+ ident!(("___ZEROCOPY_TAG_{}", variant), variant.span())
+}
+
+pub(crate) fn variant_struct_ident(variant: &Ident) -> Ident {
+ ident!(("___ZerocopyVariantStruct_{}", variant), variant.span())
+}
+
+pub(crate) fn variants_union_field_ident(variant: &Ident) -> Ident {
+ ident!(("__field_{}", variant), variant.span())
+}
+
+/// Generates a constant for the tag associated with each variant of the enum.
+/// When we match on the enum's tag, each arm matches one of these constants. We
+/// have to use constants here because:
+///
+/// - The type that we're matching on is not the type of the tag, it's an
+/// integer of the same size as the tag type and with the same bit patterns.
+/// - We can't read the enum tag as an enum because the bytes may not represent
+/// a valid variant.
+/// - Patterns do not currently support const expressions, so we have to assign
+/// these constants to names rather than use them inline in the `match`
+/// statement.
+pub(crate) fn generate_tag_consts(data: &DataEnum) -> TokenStream {
+ let tags = data.variants.iter().map(|variant| {
+ let variant_ident = &variant.ident;
+ let tag = tag_ident(variant_ident);
+
+ quote! {
+ // This casts the enum variant to its discriminant, and then
+ // converts the discriminant to the target integral type via a
+ // numeric cast [1].
+ //
+ // Because these are the same size, this is defined to be a no-op
+ // and therefore is a lossless conversion [2].
+ //
+ // [1] Per https://doc.rust-lang.org/1.81.0/reference/expressions/operator-expr.html#enum-cast:
+ //
+ // Casts an enum to its discriminant.
+ //
+ // [2] Per https://doc.rust-lang.org/1.81.0/reference/expressions/operator-expr.html#numeric-cast:
+ //
+ // Casting between two integers of the same size (e.g. i32 -> u32)
+ // is a no-op.
+ const #tag: ___ZerocopyTagPrimitive =
+ ___ZerocopyTag::#variant_ident as ___ZerocopyTagPrimitive;
+ }
+ });
+
+ quote! {
+ #(#tags)*
+ }
+}
+
+#[derive(Copy, Clone, Eq, PartialEq)]
+enum Validity {
+ Uninit,
+ Initialized,
+ Safe,
+}
+
+impl Validity {
+ fn as_type(self, ctx: &Ctx) -> Type {
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ match self {
+ Validity::Uninit => parse_quote!(#zerocopy_crate::invariant::Uninit),
+ Validity::Initialized => {
+ parse_quote!(#zerocopy_crate::invariant::Initialized)
+ }
+ Validity::Safe => parse_quote!(#zerocopy_crate::invariant::Safe),
+ }
+ }
+
+ /// Computes the output validity of projecting a struct or union field, or
+ /// returns `None` if that projection must not be generated.
+ ///
+ /// Struct fields preserve all three validity invariants. Union fields
+ /// preserve `Uninit` and `Initialized`. No projection is generated from a
+ /// `Safe` union.
+ fn output_validity_for_struct_or_union(self, data: &Data) -> Option<Self> {
+ match (data, self) {
+ (Data::Struct(_), validity) => Some(validity),
+ (Data::Union(_), validity @ (Validity::Uninit | Validity::Initialized)) => {
+ Some(validity)
+ }
+ (Data::Union(_), Validity::Safe) => None,
+ (Data::Enum(_), _) => unreachable!(),
+ }
+ }
+}
+
+/// Returns the projection variant ID for the struct or union described by
+/// `ctx`.
+///
+/// This intentionally does not handle enums: an enum field's variant ID
+/// identifies its source variant rather than the representation of its
+/// containing type.
+pub(crate) fn struct_union_variant_id(ctx: &Ctx) -> Box<Expr> {
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ match &ctx.ast.data {
+ Data::Struct(_) => parse_quote!({ #zerocopy_crate::STRUCT_VARIANT_ID }),
+ Data::Union(_) => {
+ if StructUnionRepr::from_attrs(&ctx.ast.attrs).map(|repr| repr.is_c()).unwrap() {
+ parse_quote!({ #zerocopy_crate::REPR_C_UNION_VARIANT_ID })
+ } else {
+ parse_quote!({ #zerocopy_crate::UNION_VARIANT_ID })
+ }
+ }
+ Data::Enum(_) => unreachable!("struct_union_variant_id called for enum"),
+ }
+}
+
+/// Generates field projection implementations for a struct or union.
+///
+/// `repr(C)` unions use `REPR_C_UNION_VARIANT_ID`, so their projections
+/// additionally implement `pointer::cast::Cast`. Other unions use
+/// `UNION_VARIANT_ID`.
+pub(crate) fn derive_projection_struct_union(
+ ctx: &Ctx,
+ data: &dyn DataExt,
+ client: Client,
+) -> TokenStream {
+ let fields = ctx.ast.data.fields();
+ if fields.is_empty() {
+ return quote! {};
+ }
+
+ let field_tokens = fields.iter().map(|(vis, ident, _)| {
+ let ident = ident!(("__z{}", ident), ident.span());
+ quote! {
+ #vis enum #ident {}
+ }
+ });
+
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let variant_id = struct_union_variant_id(ctx);
+
+ let core = ctx.core_path();
+ let has_tag = ImplBlockBuilder::new(ctx, data, Trait::HasTag { client }, FieldBounds::None)
+ .inner_extras(quote! {
+ type Tag = ();
+ type ProjectToTag = #zerocopy_crate::pointer::cast::CastToUnit;
+ })
+ .build();
+
+ let has_fields = fields.iter().map(move |(_, ident, ty)| {
+ let field_token = ident!(("__z{}", ident), ident.span());
+ let projection = FieldProjection {
+ variant_id: variant_id.clone(),
+ field: parse_quote!(#field_token),
+ field_id: parse_quote!({ #zerocopy_crate::ident_id!(#ident) }),
+ };
+ let has_field_trait = Trait::HasField { client, projection: projection.clone() };
+ let has_field_path = has_field_trait.crate_path(ctx);
+ let has_field = ImplBlockBuilder::new(ctx, data, has_field_trait, FieldBounds::None)
+ .inner_extras(quote! {
+ type Type = #ty;
+
+ #[inline(always)]
+ fn project(
+ slf: #zerocopy_crate::pointer::PtrInner<'_, Self>,
+ ) -> *mut <Self as #has_field_path>::Type {
+ let slf = slf.as_ptr();
+ // SAFETY: By invariant on `PtrInner`, `slf` is a non-null
+ // pointer whose referent is zero-sized or lives in a valid
+ // allocation. Since `#ident` is a struct or union field of
+ // `Self`, this projection preserves or shrinks the referent
+ // size, and so the resulting referent also fits in the same
+ // allocation.
+ unsafe { #core::ptr::addr_of_mut!((*slf).#ident) }
+ }
+ })
+ .build();
+
+ let derive_project_field = |input_validity: Validity| {
+ let output_validity =
+ match input_validity.output_validity_for_struct_or_union(&ctx.ast.data) {
+ Some(output_validity) => output_validity.as_type(ctx),
+ None => return TokenStream::new(),
+ };
+ let input_validity = input_validity.as_type(ctx);
+ let projection_preserves_alignment = StructUnionRepr::from_attrs(&ctx.ast.attrs)
+ .map(|repr| repr.get_packed().is_none())
+ .unwrap();
+ let output_alignment = if projection_preserves_alignment {
+ quote! { ___ZcAlignment }
+ } else {
+ quote! { #zerocopy_crate::invariant::Unaligned }
+ };
+ ImplBlockBuilder::new(
+ ctx,
+ data,
+ Trait::ProjectField {
+ client,
+ projection: projection.clone(),
+ invariants: parse_quote!((___ZcAliasing, ___ZcAlignment, #input_validity)),
+ },
+ FieldBounds::None,
+ )
+ .param_extras(vec![
+ parse_quote!(___ZcAliasing: #zerocopy_crate::invariant::Aliasing),
+ parse_quote!(___ZcAlignment: #zerocopy_crate::invariant::Alignment),
+ ])
+ .inner_extras(quote! {
+ // SAFETY: Struct and union projections do not depend on the value of
+ // the referent, and are therefore infallible.
+ type Error = #zerocopy_crate::util::macro_util::core_reexport::convert::Infallible;
+
+ // SAFETY: Projection preserves aliasing. It also preserves
+ // alignment unless the containing type is packed. `Validity`
+ // determines whether a projection is emitted and, if so,
+ // computes its output validity: struct fields preserve validity;
+ // union fields preserve `Uninit` and `Initialized`, and no
+ // projection is emitted for `Safe` union input.
+ type Invariants = (___ZcAliasing, #output_alignment, #output_validity);
+ })
+ .build()
+ };
+
+ let project_uninit = derive_project_field(Validity::Uninit);
+ let project_initialized = derive_project_field(Validity::Initialized);
+ let project_safe = derive_project_field(Validity::Safe);
+
+ quote! {
+ #has_field
+ #project_uninit
+ #project_initialized
+ #project_safe
+ }
+ });
+
+ const_block(field_tokens.into_iter().chain(Some(has_tag)).chain(has_fields).map(Some))
+}
+
+pub(crate) fn derive_enum(
+ ctx: &Ctx,
+ data: &DataEnum,
+ client: Client,
+) -> Result<TokenStream, Error> {
+ // With no fields, there are no `HasField` or `ProjectField` impls to emit,
+ // and therefore no representation requirement to enforce.
+ if data.fields().is_empty() {
+ return Ok(TokenStream::new());
+ }
+
+ let repr = EnumRepr::from_attrs(&ctx.ast.attrs)?;
+ let (outer_tag_type, inner_tag_type) = if repr.is_c() {
+ (quote! { ___ZerocopyTag }, quote! { () })
+ } else if repr.is_primitive() {
+ (quote! { () }, quote! { ___ZerocopyTag })
+ } else {
+ return Err(Error::new(
+ ctx.ast.span(),
+ "must have #[repr(C)] or #[repr(Int)] attribute in order to guarantee this type's memory layout",
+ ));
+ };
+
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ let core = &ctx.core_path();
+ let client_path = &client.crate_path(ctx);
+ let has_tag_trait = Trait::HasTag { client };
+ let has_tag_path = &has_tag_trait.crate_path(ctx);
+ let infallible =
+ quote! { #zerocopy_crate::util::macro_util::core_reexport::convert::Infallible };
+ let tag_enum = generate_tag_enum(ctx, &repr, data);
+ let tag_consts = generate_tag_consts(data);
+
+ let generics = &ctx.ast.generics;
+ let (impl_generics, ref ty_generics, where_clause) = generics.split_for_impl();
+
+ // Synthesize a `repr(C)` struct for each fieldful variant. We must do this
+ // because Rust does not provide an `addr_of!` that works on enums.
+ let variant_structs = data.variants.iter().filter_map(move |variant| {
+ if matches!(variant.fields, Fields::Unit) {
+ return None;
+ }
+
+ let ident = variant_struct_ident(&variant.ident);
+ let field_types = variant.fields.iter().map(|field| &field.ty);
+ let enum_name = &ctx.ast.ident;
+ let variant_struct: DeriveInput = parse_quote! {
+ #[repr(C)]
+ struct #ident #impl_generics (
+ #core::mem::MaybeUninit<___ZerocopyInnerTag>,
+ #(#field_types,)*
+ #core::marker::PhantomData<#enum_name #ty_generics>,
+ ) #where_clause;
+ };
+
+ let projections = derive_projection_struct_union(
+ &ctx.with_input(&variant_struct),
+ &variant_struct.data,
+ client,
+ );
+
+ Some(quote! {
+ #variant_struct
+ #projections
+ })
+ });
+
+ // Synthesize a `repr(C)` union encompassing all synthesized variants.
+ let variants_union = {
+ let fields = data.variants.iter().filter_map(|variant| {
+ if matches!(variant.fields, Fields::Unit) {
+ return None;
+ }
+ let field_name = variants_union_field_ident(&variant.ident);
+ let variant_struct = variant_struct_ident(&variant.ident);
+ Some(quote! {
+ #field_name: #core::mem::ManuallyDrop<#variant_struct #ty_generics>,
+ })
+ });
+
+ let variants_union: DeriveInput = parse_quote! {
+ #[repr(C)]
+ union ___ZerocopyVariants #generics {
+ #(#fields)*
+ // A fieldless enum produces no variant structs, but a union must
+ // have at least one field. This unit does not affect `repr(C)`
+ // layout.
+ __nonempty: (),
+ }
+ };
+ let projections = derive_projection_struct_union(
+ &ctx.with_input(&variants_union),
+ &variants_union.data,
+ client,
+ );
+
+ quote! {
+ #variants_union
+ #projections
+ }
+ };
+
+ // Synthesize a `repr(C)` struct that pairs the the tag with the synthesized
+ // variants union. In practice, the actual tag storage may be here, or
+ // inside the synthesized variant structs, depending on the `repr` of the
+ // enum.
+ let raw_enum: DeriveInput = parse_quote! {
+ #[repr(C)]
+ struct ___ZerocopyRawEnum #generics {
+ tag: ___ZerocopyOuterTag,
+ variants: ___ZerocopyVariants #ty_generics,
+ }
+ };
+
+ let raw_projections =
+ derive_projection_struct_union(&ctx.with_input(&raw_enum), &raw_enum.data, client);
+
+ let has_tag = ImplBlockBuilder::new(ctx, data, has_tag_trait, FieldBounds::None)
+ .inner_extras(quote! {
+ type Tag = ___ZerocopyTag;
+ type ProjectToTag = #zerocopy_crate::pointer::cast::CastSized;
+ })
+ .build();
+
+ let aliasing_param: &syn::GenericParam =
+ &parse_quote!(___ZcAliasing: #zerocopy_crate::invariant::Aliasing);
+ let reference_param: &syn::GenericParam =
+ &parse_quote!(___ZcAliasing: #zerocopy_crate::invariant::Reference);
+ let alignment_param: &syn::GenericParam =
+ &parse_quote!(___ZcAlignment: #zerocopy_crate::invariant::Alignment);
+
+ let has_fields = data.variants().into_iter().flat_map(|(variant, fields)| {
+ let variant_ident = &variant.unwrap().ident;
+ let variants_union_field = variants_union_field_ident(variant_ident);
+ let infallible = &infallible;
+
+ fields.into_iter().enumerate().map(move |(idx, (vis, ident, ty))| {
+ // Rust does not presently support explicit visibility modifiers on
+ // enum fields. Keep this assertion so that a future language
+ // change cannot silently invalidate the visibility invariant.
+ assert!(matches!(vis, syn::Visibility::Inherited));
+ let projection = FieldProjection {
+ variant_id: parse_quote!({ #zerocopy_crate::ident_id!(#variant_ident) }),
+ field: parse_quote!(()),
+ field_id: parse_quote!({ #zerocopy_crate::ident_id!(#ident) }),
+ };
+ // The generated variant struct's first field is its inner tag.
+ let variant_struct_field_index = Index::from(idx + 1);
+ let has_field_trait = Trait::HasField {
+ client,
+ projection: projection.clone(),
+ };
+ let has_field_path = has_field_trait.crate_path(ctx);
+ let has_field = ImplBlockBuilder::new(ctx, data, has_field_trait, FieldBounds::None)
+ .inner_extras(quote! {
+ type Type = #ty;
+
+ #[inline(always)]
+ fn project(
+ slf: #zerocopy_crate::pointer::PtrInner<'_, Self>,
+ ) -> *mut <Self as #has_field_path>::Type {
+ use #zerocopy_crate::pointer::cast::{CastSized, Projection};
+
+ slf.project::<___ZerocopyRawEnum #ty_generics, CastSized>()
+ .project::<_, Projection<#client_path, _, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(variants) }>>()
+ .project::<_, Projection<#client_path, _, { #zerocopy_crate::REPR_C_UNION_VARIANT_ID }, { #zerocopy_crate::ident_id!(#variants_union_field) }>>()
+ .project::<_, Projection<#client_path, _, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(value) }>>()
+ .project::<_, Projection<#client_path, _, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(#variant_struct_field_index) }>>()
+ .as_ptr()
+ }
+ })
+ .build();
+
+ // SAFETY: `projection` is constructed from the current field and
+ // `variant_ident`, and that same `variant_ident` selects the
+ // variants union field through which `has_field` projects. Thus the
+ // `HasField` impl, `ProjectField` impls, and valid projection's tag
+ // check all identify the same enum field. `idx` is the current
+ // field's zero-based position, and the generated variant struct
+ // prepends its inner tag before reproducing the enum fields in
+ // source order, so `idx + 1` selects that same field. The assertion
+ // above enforces that `repr` is `repr(C)` or a primitive
+ // representation; `___ZerocopyRawEnum` and its nested types model
+ // exactly the representations specified by [1] and [2]. Those
+ // representations use `repr(C)` structs for variant fields, so field
+ // projection preserves alignment. Each `ProjectField` impl preserves
+ // the input validity. The `Uninit` and `Initialized` projections are
+ // infallible because those validity invariants do not depend on the
+ // enum's tag. The `Safe` projection first checks that the tag
+ // selects `variant_ident`; its `Reference` aliasing bound prevents
+ // mutation of the tag between that check and field projection. The
+ // corresponding `HasTag` impl projects to the raw representation's
+ // tag using `CastSized`.
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/reference/type-layout.html#reprc-enums-with-fields:
+ //
+ // The representation of a `repr(C)` enum with fields is a
+ // `repr(C)` struct with two fields [...]: a `repr(C)` version of
+ // the enum with all fields removed ("the tag") [and] a `repr(C)`
+ // union of `repr(C)` structs for the fields of each variant [...].
+ //
+ // [2] Per https://doc.rust-lang.org/1.56.0/reference/type-layout.html#primitive-representation-of-enums-with-fields:
+ //
+ // The representation of a primitive representation enum is a
+ // `repr(C)` union of `repr(C)` structs for each variant with a
+ // field. The first field of each struct in the union is [...] the
+ // tag and the remaining fields are the fields of that variant.
+ let derive_enum_project_field = |validity| {
+ let (error, is_projectable, aliasing_bound) = if validity == Validity::Safe {
+ assert!(!matches!(validity, Validity::Uninit | Validity::Initialized));
+ (
+ quote! { () },
+ quote! {
+ #[inline(always)]
+ fn is_projectable(
+ tag: #zerocopy_crate::pointer::Ptr<
+ '_,
+ <Self as #has_tag_path>::Tag,
+ (
+ #zerocopy_crate::invariant::Shared,
+ ___ZcAlignment,
+ #zerocopy_crate::invariant::Safe,
+ ),
+ >,
+ ) -> #core::result::Result<(), ()> {
+ let tag = tag.read::<#zerocopy_crate::BecauseImmutable>();
+ if tag == ___ZerocopyTag::#variant_ident {
+ #core::result::Result::Ok(())
+ } else {
+ #core::result::Result::Err(())
+ }
+ }
+ },
+ reference_param.clone(),
+ )
+ } else {
+ (infallible.clone(), quote! {}, aliasing_param.clone())
+ };
+
+ let validity = validity.as_type(ctx);
+ let invariants: Box<Type> =
+ parse_quote!((___ZcAliasing, ___ZcAlignment, #validity));
+ let project_field_trait = Trait::ProjectField {
+ client,
+ projection: projection.clone(),
+ invariants: invariants.clone(),
+ };
+
+ ImplBlockBuilder::new(ctx, data, project_field_trait, FieldBounds::None)
+ .param_extras(vec![aliasing_bound.clone(), alignment_param.clone()])
+ .inner_extras(quote! {
+ type Error = #error;
+ type Invariants = #invariants;
+ #is_projectable
+ })
+ .build()
+ };
+
+ let project_uninit = derive_enum_project_field(Validity::Uninit);
+ let project_initialized = derive_enum_project_field(Validity::Initialized);
+ let project_valid = derive_enum_project_field(Validity::Safe);
+
+ quote! {
+ #has_field
+ #project_uninit
+ #project_initialized
+ #project_valid
+ }
+ })
+ });
+
+ Ok(quote! {
+ #tag_enum
+
+ type ___ZerocopyTagPrimitive = #zerocopy_crate::util::macro_util::SizeToTag<
+ { #core::mem::size_of::<___ZerocopyTag>() },
+ >;
+
+ #tag_consts
+
+ type ___ZerocopyOuterTag = #outer_tag_type;
+ type ___ZerocopyInnerTag = #inner_tag_type;
+
+ #(#variant_structs)*
+ #variants_union
+
+ #raw_enum
+ #raw_projections
+
+ #has_tag
+ #(#has_fields)*
+ })
+}
+
+pub(crate) fn derive(ctx: &Ctx, _top_level: Trait) -> Result<TokenStream, Error> {
+ match &ctx.ast.data {
+ Data::Struct(strct) => {
+ Ok(derive_projection_struct_union(ctx, strct, Client::ProjectDerive))
+ }
+ Data::Union(unn) => Ok(derive_projection_struct_union(ctx, unn, Client::ProjectDerive)),
+ Data::Enum(enm) => {
+ let projections = derive_enum(ctx, enm, Client::ProjectDerive)?;
+ if projections.is_empty() {
+ Ok(projections)
+ } else {
+ Ok(const_block([Some(projections)]))
+ }
+ }
+ }
+}
diff --git a/rust/zerocopy-derive/derive/try_from_bytes.rs b/rust/zerocopy-derive/derive/try_from_bytes.rs
index 44f083328786..ae0c796b91b1 100644
--- a/rust/zerocopy-derive/derive/try_from_bytes.rs
+++ b/rust/zerocopy-derive/derive/try_from_bytes.rs
@@ -1,442 +1,224 @@
// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
//
-use proc_macro2::TokenStream;
-use quote::quote;
-use syn::{
- parse_quote, spanned::Spanned as _, Data, DataEnum, DataStruct, DataUnion, DeriveInput, Error,
- Expr, Fields, Ident, Index, Type,
-};
+use proc_macro2::{Ident, Span, TokenStream};
+use quote::{quote, ToTokens as _};
+use syn::{spanned::Spanned as _, Data, DataEnum, DataStruct, DataUnion, Error, Field};
use crate::{
- repr::{EnumRepr, StructUnionRepr},
+ derive::project::{
+ derive_enum, derive_projection_struct_union, generate_tag_consts, struct_union_variant_id,
+ tag_ident,
+ },
+ repr::EnumRepr,
util::{
- const_block, enum_size_from_repr, generate_tag_enum, Ctx, DataExt, FieldBounds,
+ enum_size_from_repr, generate_tag_enum, Client, Ctx, DataExt, FieldBounds,
ImplBlockBuilder, Trait, TraitBound,
},
};
-fn tag_ident(variant_ident: &Ident) -> Ident {
- ident!(("___ZEROCOPY_TAG_{}", variant_ident), variant_ident.span())
-}
-
-/// Generates a constant for the tag associated with each variant of the enum.
-/// When we match on the enum's tag, each arm matches one of these constants. We
-/// have to use constants here because:
-///
-/// - The type that we're matching on is not the type of the tag, it's an
-/// integer of the same size as the tag type and with the same bit patterns.
-/// - We can't read the enum tag as an enum because the bytes may not represent
-/// a valid variant.
-/// - Patterns do not currently support const expressions, so we have to assign
-/// these constants to names rather than use them inline in the `match`
-/// statement.
-fn generate_tag_consts(data: &DataEnum) -> TokenStream {
- let tags = data.variants.iter().map(|v| {
- let variant_ident = &v.ident;
- let tag_ident = tag_ident(variant_ident);
-
- quote! {
- // This casts the enum variant to its discriminant, and then
- // converts the discriminant to the target integral type via a
- // numeric cast [1].
- //
- // Because these are the same size, this is defined to be a no-op
- // and therefore is a lossless conversion [2].
- //
- // [1] Per https://doc.rust-lang.org/1.81.0/reference/expressions/operator-expr.html#enum-cast:
- //
- // Casts an enum to its discriminant.
- //
- // [2] Per https://doc.rust-lang.org/1.81.0/reference/expressions/operator-expr.html#numeric-cast:
- //
- // Casting between two integers of the same size (e.g. i32 -> u32)
- // is a no-op.
- const #tag_ident: ___ZerocopyTagPrimitive =
- ___ZerocopyTag::#variant_ident as ___ZerocopyTagPrimitive;
- }
- });
- quote! {
- #(#tags)*
+fn candidate_ident(ctx: &Ctx) -> Ident {
+ // Choose a source-pointer name distinct from the field names. Mixed-site
+ // hygiene keeps it out of the user's invariant expressions.
+ let fields = ctx.ast.data.fields();
+ let mut name = "candidate".to_owned();
+ while fields.iter().any(|(_, field, _)| field.to_string().trim_start_matches("r#") == name) {
+ name.push('_');
}
+ Ident::new(&name, Span::mixed_site())
}
-fn variant_struct_ident(variant_ident: &Ident) -> Ident {
- ident!(("___ZerocopyVariantStruct_{}", variant_ident), variant_ident.span())
-}
-
-/// Generates variant structs for the given enum variant.
-///
-/// These are structs associated with each variant of an enum. They are
-/// `repr(C)` tuple structs with the same fields as the variant after a
-/// `MaybeUninit<___ZerocopyInnerTag>`.
-///
-/// In order to unify the generated types for `repr(C)` and `repr(int)` enums,
-/// we use a "fused" representation with fields for both an inner tag and an
-/// outer tag. Depending on the repr, we will set one of these tags to the tag
-/// type and the other to `()`. This lets us generate the same code but put the
-/// tags in different locations.
-fn generate_variant_structs(ctx: &Ctx, data: &DataEnum) -> TokenStream {
- let (impl_generics, ty_generics, where_clause) = ctx.ast.generics.split_for_impl();
-
- let enum_name = &ctx.ast.ident;
+/// Generates validation of every field of a struct or enum variant, leaving
+/// the validated pointers in scope for subsequent field invariants.
+/// A union is checked by invoking this once per field.
+fn derive_variant_is_safe<'a>(
+ ctx: &Ctx,
+ variant_id: &TokenStream,
+ fields: impl IntoIterator<Item = &'a Field>,
+) -> Result<TokenStream, Error> {
+ let fields = fields.into_iter().collect::<Vec<_>>();
+ if fields.is_empty() {
+ return Ok(quote!(true));
+ }
- // All variant structs have a `PhantomData<MyEnum<...>>` field because we
- // don't know which generic parameters each variant will use, and unused
- // generic parameters are a compile error.
+ let zerocopy_crate = &ctx.zerocopy_crate;
let core = ctx.core_path();
- let phantom_ty = quote! {
- #core::marker::PhantomData<#enum_name #ty_generics>
- };
-
- let variant_structs = data.variants.iter().filter_map(|variant| {
- // We don't generate variant structs for unit variants because we only
- // need to check the tag. This helps cut down our generated code a bit.
- if matches!(variant.fields, Fields::Unit) {
- return None;
- }
-
- let variant_struct_ident = variant_struct_ident(&variant.ident);
- let field_types = variant.fields.iter().map(|f| &f.ty);
-
- let variant_struct = parse_quote! {
- #[repr(C)]
- struct #variant_struct_ident #impl_generics (
- #core::mem::MaybeUninit<___ZerocopyInnerTag>,
- #(#field_types,)*
- #phantom_ty,
- ) #where_clause;
- };
-
- // We do this rather than emitting `#[derive(::zerocopy::TryFromBytes)]`
- // because that is not hygienic, and this is also more performant.
- let try_from_bytes_impl =
- derive_try_from_bytes(&ctx.with_input(&variant_struct), Trait::TryFromBytes)
- .expect("derive_try_from_bytes should not fail on synthesized type");
-
- Some(quote! {
- #variant_struct
- #try_from_bytes_impl
- })
+ let candidate = candidate_ident(ctx);
+ let field_names = fields.iter().enumerate().map(|(idx, field)| {
+ field
+ .ident
+ .as_ref()
+ .map(|name| name.to_token_stream())
+ .unwrap_or_else(|| syn::Index::from(idx).to_token_stream())
});
-
- quote! {
- #(#variant_structs)*
+ let field_checks = field_names
+ .map(|name| {
+ quote! {
+ #zerocopy_crate::into_inner!(
+ #candidate.project::<
+ #zerocopy_crate::project_clients::TryFromBytesDerive,
+ _,
+ { #variant_id },
+ { #zerocopy_crate::ident_id!(#name) },
+ >()
+ ).try_into_safe::<_, #zerocopy_crate::BecauseImmutable>()
+ }
+ })
+ .collect::<Vec<_>>();
+ // Without invariants, no expression needs the validated pointers. Avoid
+ // introducing bindings that could collide with caller-defined items.
+ if ctx.invariant_span.is_none() {
+ return Ok(quote! { true #(&& #field_checks.is_ok())* });
}
-}
-
-fn variants_union_field_ident(ident: &Ident) -> Ident {
- // Field names are prefixed with `__field_` to prevent name collision
- // with the `__nonempty` field.
- ident!(("__field_{}", ident), ident.span())
-}
-
-fn generate_variants_union(ctx: &Ctx, data: &DataEnum) -> TokenStream {
- let generics = &ctx.ast.generics;
- let (_, ty_generics, _) = generics.split_for_impl();
-
- let fields = data.variants.iter().filter_map(|variant| {
- // We don't generate variant structs for unit variants because we only
- // need to check the tag. This helps cut down our generated code a bit.
- if matches!(variant.fields, Fields::Unit) {
- return None;
- }
- let field_name = variants_union_field_ident(&variant.ident);
- let variant_struct_ident = variant_struct_ident(&variant.ident);
-
- let core = ctx.core_path();
- Some(quote! {
- #field_name: #core::mem::ManuallyDrop<#variant_struct_ident #ty_generics>,
+ let field_bindings = fields
+ .iter()
+ .enumerate()
+ .map(|(idx, _)| ctx.fresh_ident(&format!("field_{}", idx)))
+ .collect::<Vec<_>>();
+ let bound_fields =
+ fields.iter().copied().zip(field_bindings.iter().cloned()).collect::<Vec<_>>();
+ let field_validations = fields
+ .iter()
+ .enumerate()
+ .map(|(idx, field)| {
+ let invariants = crate::invariant::parse(&field.attrs)?;
+ let invariants = invariants
+ .iter()
+ .map(|expression| crate::invariant::bind_fields(&bound_fields[..=idx], expression));
+ Ok(quote! {
+ #(
+ // Keep `return` in an invariant from bypassing later fields.
+ if !{ #[inline(always)] || -> #core::primitive::bool { #invariants } }() {
+ return false;
+ }
+ )*
+ })
})
- });
-
- let variants_union = parse_quote! {
- #[repr(C)]
- union ___ZerocopyVariants #generics {
- #(#fields)*
- // Enums can have variants with no fields, but unions must
- // have at least one field. So we just add a trailing unit
- // to ensure that this union always has at least one field.
- // Because this union is `repr(C)`, this unit type does not
- // affect the layout.
- __nonempty: (),
+ .collect::<Result<Vec<_>, Error>>()?;
+ Ok(quote! {
+ {
+ #(
+ // Retain each shared pointer for subsequent field invariants.
+ #[allow(unused_variables)]
+ let #field_bindings = match #field_checks {
+ #core::result::Result::Ok(#field_bindings) => #field_bindings,
+ #core::result::Result::Err(_) => return false,
+ };
+ #field_validations
+ )*
+ true
}
- };
-
- let has_field =
- derive_has_field_struct_union(&ctx.with_input(&variants_union), &variants_union.data);
-
- quote! {
- #variants_union
- #has_field
- }
+ })
}
-/// Generates an implementation of `is_bit_valid` for an arbitrary enum.
-///
-/// The general process is:
-///
-/// 1. Generate a tag enum. This is an enum with the same repr, variants, and
-/// corresponding discriminants as the original enum, but without any fields
-/// on the variants. This gives us access to an enum where the variants have
-/// the same discriminants as the one we're writing `is_bit_valid` for.
-/// 2. Make constants from the variants of the tag enum. We need these because
-/// we can't put const exprs in match arms.
-/// 3. Generate variant structs. These are structs which have the same fields as
-/// each variant of the enum, and are `#[repr(C)]` with an optional "inner
-/// tag".
-/// 4. Generate a variants union, with one field for each variant struct type.
-/// 5. And finally, our raw enum is a `#[repr(C)]` struct of an "outer tag" and
-/// the variants union.
+/// Generates an implementation of `is_safe` for an arbitrary enum.
///
-/// See these reference links for fully-worked example decompositions.
-///
-/// - `repr(C)`: <https://doc.rust-lang.org/reference/type-layout.html#reprc-enums-with-fields>
-/// - `repr(int)`: <https://doc.rust-lang.org/reference/type-layout.html#primitive-representation-of-enums-with-fields>
-/// - `repr(C, int)`: <https://doc.rust-lang.org/reference/type-layout.html#combining-primitive-representations-of-enums-with-fields-and-reprc>
-pub(crate) fn derive_is_bit_valid(
+/// For an enum with fields, [`derive_enum`] generates the representation model
+/// and projection impls. This function reads the tag, matches it against the
+/// enum's discriminants, and validates each field of the selected variant
+/// through those projections. A fieldless enum needs only the generated tag
+/// enum and discriminant constants.
+pub(crate) fn derive_is_safe(
ctx: &Ctx,
data: &DataEnum,
repr: &EnumRepr,
) -> Result<TokenStream, Error> {
- let trait_path = Trait::TryFromBytes.crate_path(ctx);
- let tag_enum = generate_tag_enum(ctx, repr, data);
- let tag_consts = generate_tag_consts(data);
-
- let (outer_tag_type, inner_tag_type) = if repr.is_c() {
- (quote! { ___ZerocopyTag }, quote! { () })
- } else if repr.is_primitive() {
- (quote! { () }, quote! { ___ZerocopyTag })
- } else {
+ if !(repr.is_c() || repr.is_primitive()) {
return Err(Error::new(
ctx.ast.span(),
"must have #[repr(C)] or #[repr(Int)] attribute in order to guarantee this type's memory layout",
));
- };
-
- let variant_structs = generate_variant_structs(ctx, data);
- let variants_union = generate_variants_union(ctx, data);
-
- let (impl_generics, ty_generics, where_clause) = ctx.ast.generics.split_for_impl();
+ }
let zerocopy_crate = &ctx.zerocopy_crate;
- let has_tag = ImplBlockBuilder::new(ctx, data, Trait::HasTag, FieldBounds::None)
- .inner_extras(quote! {
- type Tag = ___ZerocopyTag;
- type ProjectToTag = #zerocopy_crate::pointer::cast::CastSized;
- })
- .build();
- let has_fields = data.variants().into_iter().flat_map(|(variant, fields)| {
- let variant_ident = &variant.unwrap().ident;
- let variants_union_field_ident = variants_union_field_ident(variant_ident);
- let field: Box<syn::Type> = parse_quote!(());
- fields.into_iter().enumerate().map(move |(idx, (vis, ident, ty))| {
- // Rust does not presently support explicit visibility modifiers on
- // enum fields, but we guard against the possibility to ensure this
- // derive remains sound.
- assert!(matches!(vis, syn::Visibility::Inherited));
- let variant_struct_field_index = Index::from(idx + 1);
- let (_, ty_generics, _) = ctx.ast.generics.split_for_impl();
- let has_field_trait = Trait::HasField {
- variant_id: parse_quote!({ #zerocopy_crate::ident_id!(#variant_ident) }),
- // Since Rust does not presently support explicit visibility
- // modifiers on enum fields, any public type is suitable here;
- // we use `()`.
- field: field.clone(),
- field_id: parse_quote!({ #zerocopy_crate::ident_id!(#ident) }),
- };
- let has_field_path = has_field_trait.crate_path(ctx);
- let has_field = ImplBlockBuilder::new(
- ctx,
- data,
- has_field_trait,
- FieldBounds::None,
- )
- .inner_extras(quote! {
- type Type = #ty;
+ let core = ctx.core_path();
+ let candidate = candidate_ident(ctx);
+ let alignment = ctx.fresh_ident("___ZcAlignment");
+ let tag = Ident::new("tag", Span::mixed_site());
+ let projections = if data.fields().is_empty() {
+ let tag_enum = generate_tag_enum(ctx, repr, data);
+ let tag_consts = generate_tag_consts(data);
+ quote! {
+ #tag_enum
- #[inline(always)]
- fn project(slf: #zerocopy_crate::pointer::PtrInner<'_, Self>) -> *mut <Self as #has_field_path>::Type {
- use #zerocopy_crate::pointer::cast::{CastSized, Projection};
+ type ___ZerocopyTagPrimitive = #zerocopy_crate::util::macro_util::SizeToTag<
+ { #core::mem::size_of::<___ZerocopyTag>() },
+ >;
- slf.project::<___ZerocopyRawEnum #ty_generics, CastSized>()
- .project::<_, Projection<_, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(variants) }>>()
- .project::<_, Projection<_, { #zerocopy_crate::REPR_C_UNION_VARIANT_ID }, { #zerocopy_crate::ident_id!(#variants_union_field_ident) }>>()
- .project::<_, Projection<_, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(value) }>>()
- .project::<_, Projection<_, { #zerocopy_crate::STRUCT_VARIANT_ID }, { #zerocopy_crate::ident_id!(#variant_struct_field_index) }>>()
- .as_ptr()
- }
- })
- .build();
+ #tag_consts
+ }
+ } else {
+ derive_enum(ctx, data, Client::TryFromBytesDerive)?
+ };
- let project = ImplBlockBuilder::new(
- ctx,
- data,
- Trait::ProjectField {
- variant_id: parse_quote!({ #zerocopy_crate::ident_id!(#variant_ident) }),
- // Since Rust does not presently support explicit visibility
- // modifiers on enum fields, any public type is suitable
- // here; we use `()`.
- field: field.clone(),
- field_id: parse_quote!({ #zerocopy_crate::ident_id!(#ident) }),
- invariants: parse_quote!((Aliasing, Alignment, #zerocopy_crate::invariant::Initialized)),
- },
- FieldBounds::None,
- )
- .param_extras(vec![
- parse_quote!(Aliasing: #zerocopy_crate::invariant::Aliasing),
- parse_quote!(Alignment: #zerocopy_crate::invariant::Alignment),
- ])
- .inner_extras(quote! {
- type Error = #zerocopy_crate::util::macro_util::core_reexport::convert::Infallible;
- type Invariants = (Aliasing, Alignment, #zerocopy_crate::invariant::Initialized);
+ let match_arms = data
+ .variants
+ .iter()
+ .enumerate()
+ .map(|(idx, variant)| {
+ let name = &variant.ident;
+ let variant_id = quote! { #zerocopy_crate::ident_id!(#name) };
+ let fields_is_safe = derive_variant_is_safe(ctx, &variant_id, &variant.fields)?;
+ let pattern = if ctx.invariant_span.is_some() {
+ quote! { #core::option::Option::Some(#idx) }
+ } else {
+ tag_ident(name).to_token_stream()
+ };
+ Ok(quote! {
+ #pattern => { #fields_is_safe }
})
- .build();
-
- quote! {
- #has_field
- #project
- }
})
- });
+ .collect::<Result<Vec<_>, Error>>()?;
- let core = ctx.core_path();
- let match_arms = data.variants.iter().map(|variant| {
- let tag_ident = tag_ident(&variant.ident);
- let variant_struct_ident = variant_struct_ident(&variant.ident);
- let variants_union_field_ident = variants_union_field_ident(&variant.ident);
-
- if matches!(variant.fields, Fields::Unit) {
- // Unit variants don't need any further validation beyond checking
- // the tag.
- quote! {
- #tag_ident => true
- }
- } else {
- quote! {
- #tag_ident => {
- // SAFETY: Since we know that the tag is `#tag_ident`, we
- // know that no other `&`s exist which refer to this enum
- // as any other variant.
- let variant_md = variants.cast::<
- _,
- #zerocopy_crate::pointer::cast::Projection<
- // #zerocopy_crate::ReadOnly<_>,
- _,
- { #zerocopy_crate::REPR_C_UNION_VARIANT_ID },
- { #zerocopy_crate::ident_id!(#variants_union_field_ident) }
- >,
- _
- >();
- let variant = variant_md.cast::<
- #zerocopy_crate::ReadOnly<#variant_struct_ident #ty_generics>,
- #zerocopy_crate::pointer::cast::CastSized,
- (#zerocopy_crate::pointer::BecauseRead, _)
- >();
- <
- #variant_struct_ident #ty_generics as #trait_path
- >::is_bit_valid(variant)
+ let read_tag = quote! {
+ let #tag = #candidate
+ .reborrow()
+ .cast::<
+ ___ZerocopyTagPrimitive,
+ #zerocopy_crate::pointer::cast::CastSized,
+ (#zerocopy_crate::pointer::BecauseRead, _),
+ >()
+ .recall_validity::<_, (_, (_, _))>()
+ .read::<#zerocopy_crate::BecauseImmutable>();
+ };
+ let tag_init = if ctx.invariant_span.is_some() {
+ let allow = crate::util::allow_generated_code();
+ let tag_arms = data.variants.iter().enumerate().map(|(idx, variant)| {
+ let tag = tag_ident(&variant.ident);
+ quote! { #tag => #core::option::Option::Some(#idx) }
+ });
+ quote! {
+ // Keep implementation-only items and their lint allowances out
+ // of the scopes containing caller-authored invariant expressions.
+ #allow
+ let #tag = {
+ #projections
+ #read_tag
+ match #tag {
+ #(#tag_arms,)*
+ _ => #core::option::Option::None,
}
- }
- }
- });
-
- let generics = &ctx.ast.generics;
- let raw_enum: DeriveInput = parse_quote! {
- #[repr(C)]
- struct ___ZerocopyRawEnum #generics {
- tag: ___ZerocopyOuterTag,
- variants: ___ZerocopyVariants #ty_generics,
+ };
}
- };
-
- let self_ident = &ctx.ast.ident;
- let invariants_eq_impl = quote! {
- // SAFETY: `___ZerocopyRawEnum` is designed to have the same layout,
- // validity, and invariants as `Self`.
- unsafe impl #impl_generics #zerocopy_crate::pointer::InvariantsEq<___ZerocopyRawEnum #ty_generics> for #self_ident #ty_generics #where_clause {}
- };
-
- let raw_enum_projections =
- derive_has_field_struct_union(&ctx.with_input(&raw_enum), &raw_enum.data);
-
- let raw_enum = quote! {
- #raw_enum
- #invariants_eq_impl
- #raw_enum_projections
+ } else {
+ quote! { #projections #read_tag }
};
Ok(quote! {
- // SAFETY: We use `is_bit_valid` to validate that the bit pattern of the
+ // SAFETY: We use `is_safe` to validate that the bit pattern of the
// enum's tag corresponds to one of the enum's discriminants. Then, we
// check the bit validity of each field of the corresponding variant.
- // Thus, this is a sound implementation of `is_bit_valid`.
+ // Thus, this is a sound implementation of `is_safe`.
#[inline]
- fn is_bit_valid<___ZcAlignment>(
- mut candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>,
+ fn is_safe<#alignment>(
+ mut #candidate: #zerocopy_crate::Maybe<'_, Self, #alignment>,
) -> #core::primitive::bool
where
- ___ZcAlignment: #zerocopy_crate::invariant::Alignment,
+ #alignment: #zerocopy_crate::invariant::Alignment,
{
- #tag_enum
-
- type ___ZerocopyTagPrimitive = #zerocopy_crate::util::macro_util::SizeToTag<
- { #core::mem::size_of::<___ZerocopyTag>() },
- >;
-
- #tag_consts
-
- type ___ZerocopyOuterTag = #outer_tag_type;
- type ___ZerocopyInnerTag = #inner_tag_type;
+ #tag_init
- #variant_structs
-
- #variants_union
-
- #raw_enum
-
- #has_tag
-
- #(#has_fields)*
-
- let tag = {
- // SAFETY:
- // - The provided cast addresses a subset of the bytes addressed
- // by `candidate` because it addresses the starting tag of the
- // enum.
- // - Because the pointer is cast from `candidate`, it has the
- // same provenance as it.
- // - There are no `UnsafeCell`s in the tag because it is a
- // primitive integer.
- // - `tag_ptr` is casted from `candidate`, whose referent is
- // `Initialized`. Since we have not written uninitialized
- // bytes into the referent, `tag_ptr` is also `Initialized`.
- //
- // FIXME(#2874): Revise this to a `cast` once `candidate`
- // references a `ReadOnly<Self>`.
- let tag_ptr = unsafe {
- candidate.reborrow().project_transmute_unchecked::<
- _,
- #zerocopy_crate::invariant::Initialized,
- #zerocopy_crate::pointer::cast::CastSized
- >()
- };
- tag_ptr.recall_validity::<_, (_, (_, _))>().read::<#zerocopy_crate::BecauseImmutable>()
- };
-
- let mut raw_enum = candidate.cast::<
- #zerocopy_crate::ReadOnly<___ZerocopyRawEnum #ty_generics>,
- #zerocopy_crate::pointer::cast::CastSized,
- (#zerocopy_crate::pointer::BecauseRead, _)
- >();
-
- let variants = #zerocopy_crate::into_inner!(raw_enum.project::<
- _,
- { #zerocopy_crate::STRUCT_VARIANT_ID },
- { #zerocopy_crate::ident_id!(variants) }
- >());
-
- match tag {
+ match #tag {
#(#match_arms,)*
_ => false,
}
@@ -447,219 +229,88 @@ pub(crate) fn derive_try_from_bytes(ctx: &Ctx, top_level: Trait) -> Result<Token
match &ctx.ast.data {
Data::Struct(strct) => derive_try_from_bytes_struct(ctx, strct, top_level),
Data::Enum(enm) => derive_try_from_bytes_enum(ctx, enm, top_level),
- Data::Union(unn) => Ok(derive_try_from_bytes_union(ctx, unn, top_level)),
- }
-}
-fn derive_has_field_struct_union(ctx: &Ctx, data: &dyn DataExt) -> TokenStream {
- let fields = ctx.ast.data.fields();
- if fields.is_empty() {
- return quote! {};
+ Data::Union(unn) => derive_try_from_bytes_union(ctx, unn, top_level),
}
-
- let field_tokens = fields.iter().map(|(vis, ident, _)| {
- let ident = ident!(("__z{}", ident), ident.span());
- quote!(
- #vis enum #ident {}
- )
- });
-
- let zerocopy_crate = &ctx.zerocopy_crate;
- let variant_id: Box<Expr> = match &ctx.ast.data {
- Data::Struct(_) => parse_quote!({ #zerocopy_crate::STRUCT_VARIANT_ID }),
- Data::Union(_) => {
- let is_repr_c = StructUnionRepr::from_attrs(&ctx.ast.attrs)
- .map(|repr| repr.is_c())
- .unwrap_or(false);
- if is_repr_c {
- parse_quote!({ #zerocopy_crate::REPR_C_UNION_VARIANT_ID })
- } else {
- parse_quote!({ #zerocopy_crate::UNION_VARIANT_ID })
- }
- }
- _ => unreachable!(),
- };
-
- let core = ctx.core_path();
- let has_tag = ImplBlockBuilder::new(ctx, data, Trait::HasTag, FieldBounds::None)
- .inner_extras(quote! {
- type Tag = ();
- type ProjectToTag = #zerocopy_crate::pointer::cast::CastToUnit;
- })
- .build();
- let has_fields = fields.iter().map(move |(_, ident, ty)| {
- let field_token = ident!(("__z{}", ident), ident.span());
- let field: Box<Type> = parse_quote!(#field_token);
- let field_id: Box<Expr> = parse_quote!({ #zerocopy_crate::ident_id!(#ident) });
- let has_field_trait = Trait::HasField {
- variant_id: variant_id.clone(),
- field: field.clone(),
- field_id: field_id.clone(),
- };
- let has_field_path = has_field_trait.crate_path(ctx);
- ImplBlockBuilder::new(
- ctx,
- data,
- has_field_trait,
- FieldBounds::None,
- )
- .inner_extras(quote! {
- type Type = #ty;
-
- #[inline(always)]
- fn project(slf: #zerocopy_crate::pointer::PtrInner<'_, Self>) -> *mut <Self as #has_field_path>::Type {
- let slf = slf.as_ptr();
- // SAFETY: By invariant on `PtrInner`, `slf` is a non-null
- // pointer whose referent is zero-sized or lives in a valid
- // allocation. Since `#ident` is a struct or union field of
- // `Self`, this projection preserves or shrinks the referent
- // size, and so the resulting referent also fits in the same
- // allocation.
- unsafe { #core::ptr::addr_of_mut!((*slf).#ident) }
- }
- }).outer_extras(if matches!(&ctx.ast.data, Data::Struct(..)) {
- let fields_preserve_alignment = StructUnionRepr::from_attrs(&ctx.ast.attrs)
- .map(|repr| repr.get_packed().is_none())
- .unwrap();
- let alignment = if fields_preserve_alignment {
- quote! { Alignment }
- } else {
- quote! { #zerocopy_crate::invariant::Unaligned }
- };
- // SAFETY: See comments on items.
- ImplBlockBuilder::new(
- ctx,
- data,
- Trait::ProjectField {
- variant_id: variant_id.clone(),
- field,
- field_id,
- invariants: parse_quote!((Aliasing, Alignment, #zerocopy_crate::invariant::Initialized)),
- },
- FieldBounds::None,
- )
- .param_extras(vec![
- parse_quote!(Aliasing: #zerocopy_crate::invariant::Aliasing),
- parse_quote!(Alignment: #zerocopy_crate::invariant::Alignment),
- ])
- .inner_extras(quote! {
- // SAFETY: Projection into structs is always infallible.
- type Error = #zerocopy_crate::util::macro_util::core_reexport::convert::Infallible;
- // SAFETY: The alignment of the projected `Ptr` is `Unaligned`
- // if the structure is packed; otherwise inherited from the
- // outer `Ptr`. If the validity of the outer pointer is
- // `Initialized`, so too is the validity of its fields.
- type Invariants = (Aliasing, #alignment, #zerocopy_crate::invariant::Initialized);
- })
- .build()
- } else {
- quote! {}
- })
- .build()
- });
-
- const_block(field_tokens.into_iter().chain(Some(has_tag)).chain(has_fields).map(Some))
}
fn derive_try_from_bytes_struct(
ctx: &Ctx,
strct: &DataStruct,
top_level: Trait,
) -> Result<TokenStream, Error> {
- let extras = try_gen_trivial_is_bit_valid(ctx, top_level).unwrap_or_else(|| {
+ let extras = if let Some(extras) = try_gen_trivial_is_safe(ctx, top_level) {
+ extras
+ } else {
let zerocopy_crate = &ctx.zerocopy_crate;
- let fields = strct.fields();
- let field_names = fields.iter().map(|(_vis, name, _ty)| name);
- let field_tys = fields.iter().map(|(_vis, _name, ty)| ty);
+ let variant_id = quote! { #zerocopy_crate::STRUCT_VARIANT_ID };
+ let fields_is_safe = derive_variant_is_safe(ctx, &variant_id, &strct.fields)?;
let core = ctx.core_path();
+ let candidate = candidate_ident(ctx);
+ let alignment = ctx.fresh_ident("___ZcAlignment");
quote!(
- // SAFETY: We use `is_bit_valid` to validate that each field is
- // bit-valid, and only return `true` if all of them are. The bit
- // validity of a struct is just the composition of the bit
- // validities of its fields, so this is a sound implementation
- // of `is_bit_valid`.
+ // SAFETY: We use `is_safe` to validate that each field is bit-valid,
+ // and only return `true` if all of them are. The bit validity of a
+ // struct is just the composition of the bit validities of its
+ // fields, so this is a sound implementation of `is_safe`.
#[inline]
- fn is_bit_valid<___ZcAlignment>(
- mut candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>,
+ fn is_safe<#alignment>(
+ mut #candidate: #zerocopy_crate::Maybe<'_, Self, #alignment>,
) -> #core::primitive::bool
where
- ___ZcAlignment: #zerocopy_crate::invariant::Alignment,
+ #alignment: #zerocopy_crate::invariant::Alignment,
{
- true #(&& {
- let field_candidate = #zerocopy_crate::into_inner!(candidate.reborrow().project::<
- _,
- { #zerocopy_crate::STRUCT_VARIANT_ID },
- { #zerocopy_crate::ident_id!(#field_names) }
- >());
- <#field_tys as #zerocopy_crate::TryFromBytes>::is_bit_valid(field_candidate)
- })*
+ #fields_is_safe
}
)
- });
+ };
Ok(ImplBlockBuilder::new(ctx, strct, Trait::TryFromBytes, FieldBounds::ALL_SELF)
.inner_extras(extras)
- .outer_extras(derive_has_field_struct_union(ctx, strct))
+ .outer_extras(derive_projection_struct_union(ctx, strct, Client::TryFromBytesDerive))
.build())
}
-fn derive_try_from_bytes_union(ctx: &Ctx, unn: &DataUnion, top_level: Trait) -> TokenStream {
+fn derive_try_from_bytes_union(
+ ctx: &Ctx,
+ unn: &DataUnion,
+ top_level: Trait,
+) -> Result<TokenStream, Error> {
let field_type_trait_bounds = FieldBounds::All(&[TraitBound::Slf]);
let zerocopy_crate = &ctx.zerocopy_crate;
- let variant_id: Box<Expr> = {
- let is_repr_c =
- StructUnionRepr::from_attrs(&ctx.ast.attrs).map(|repr| repr.is_c()).unwrap_or(false);
- if is_repr_c {
- parse_quote!({ #zerocopy_crate::REPR_C_UNION_VARIANT_ID })
- } else {
- parse_quote!({ #zerocopy_crate::UNION_VARIANT_ID })
- }
- };
-
- let extras = try_gen_trivial_is_bit_valid(ctx, top_level).unwrap_or_else(|| {
- let fields = unn.fields();
- let field_names = fields.iter().map(|(_vis, name, _ty)| name);
- let field_tys = fields.iter().map(|(_vis, _name, ty)| ty);
+ let union_variant_id = struct_union_variant_id(ctx).to_token_stream();
+ let extras = if let Some(extras) = try_gen_trivial_is_safe(ctx, top_level) {
+ extras
+ } else {
+ let fields_is_safe = unn
+ .fields
+ .named
+ .iter()
+ .map(|field| derive_variant_is_safe(ctx, &union_variant_id, [field]))
+ .collect::<Result<Vec<_>, _>>()?;
let core = ctx.core_path();
+ let candidate = candidate_ident(ctx);
+ let alignment = ctx.fresh_ident("___ZcAlignment");
quote!(
- // SAFETY: We use `is_bit_valid` to validate that any field is
- // bit-valid; we only return `true` if at least one of them is.
- // The bit validity of a union is not yet well defined in Rust,
- // but it is guaranteed to be no more strict than this
- // definition. See #696 for a more in-depth discussion.
+ // SAFETY: We use `is_safe` to validate that any field is bit-valid;
+ // we only return `true` if at least one of them is and its
+ // invariants also hold. The bit validity of a union is not yet
+ // well defined in Rust, but it is guaranteed to be no more strict
+ // than this definition. See #696 for a more in-depth discussion.
#[inline]
- fn is_bit_valid<___ZcAlignment>(
- mut candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>,
+ fn is_safe<#alignment>(
+ mut #candidate: #zerocopy_crate::Maybe<'_, Self, #alignment>,
) -> #core::primitive::bool
where
- ___ZcAlignment: #zerocopy_crate::invariant::Alignment,
+ #alignment: #zerocopy_crate::invariant::Alignment,
{
- false #(|| {
- // SAFETY:
- // - Since `ReadOnly<Self>: Immutable` unconditionally,
- // neither `*slf` nor the returned pointer's referent
- // permit interior mutation.
- // - Both source and destination validity are
- // `Initialized`, which is always a sound
- // transmutation.
- let field_candidate = unsafe {
- candidate.reborrow().project_transmute_unchecked::<
- _,
- _,
- #zerocopy_crate::pointer::cast::Projection<
- _,
- #variant_id,
- { #zerocopy_crate::ident_id!(#field_names) }
- >
- >()
- };
-
- <#field_tys as #zerocopy_crate::TryFromBytes>::is_bit_valid(field_candidate)
- })*
+ // Keep each field's bindings and early returns local to that
+ // field, so a failure lets validation try the next field.
+ false #(|| { #[inline(always)] || { #fields_is_safe } }())*
}
)
- });
- ImplBlockBuilder::new(ctx, unn, Trait::TryFromBytes, field_type_trait_bounds)
+ };
+ Ok(ImplBlockBuilder::new(ctx, unn, Trait::TryFromBytes, field_type_trait_bounds)
.inner_extras(extras)
- .outer_extras(derive_has_field_struct_union(ctx, unn))
- .build()
+ .outer_extras(derive_projection_struct_union(ctx, unn, Client::TryFromBytesDerive))
+ .build())
}
fn derive_try_from_bytes_enum(
ctx: &Ctx,
@@ -677,16 +328,15 @@ fn derive_try_from_bytes_enum(
.map(|size| enm.fields().is_empty() && enm.variants.len() == 1usize << size)
.unwrap_or(false);
- let trivial_is_bit_valid = try_gen_trivial_is_bit_valid(ctx, top_level);
- let extra = match (trivial_is_bit_valid, could_be_from_bytes) {
- (Some(is_bit_valid), _) => is_bit_valid,
+ let trivial_is_safe = try_gen_trivial_is_safe(ctx, top_level);
+ let extra = match (trivial_is_safe, could_be_from_bytes) {
+ (Some(is_safe), _) => is_safe,
// SAFETY: It would be sound for the enum to implement `FromBytes`, as
- // required by `gen_trivial_is_bit_valid_unchecked`.
- (None, true) => unsafe { gen_trivial_is_bit_valid_unchecked(ctx) },
- (None, false) => match derive_is_bit_valid(ctx, enm, &repr) {
+ // required by `gen_trivial_is_safe_unchecked`.
+ (None, true) => unsafe { gen_trivial_is_safe_unchecked(ctx) },
+ (None, false) => match derive_is_safe(ctx, enm, &repr) {
Ok(extra) => extra,
- Err(_) if ctx.skip_on_error => return Ok(TokenStream::new()),
- Err(e) => return Err(e),
+ Err(e) => return ctx.error_or_skip(e),
},
};
@@ -694,15 +344,14 @@ fn derive_try_from_bytes_enum(
.inner_extras(extra)
.build())
}
-fn try_gen_trivial_is_bit_valid(ctx: &Ctx, top_level: Trait) -> Option<proc_macro2::TokenStream> {
+fn try_gen_trivial_is_safe(ctx: &Ctx, top_level: Trait) -> Option<proc_macro2::TokenStream> {
// If the top-level trait is `FromBytes` and `Self` has no type parameters,
// then the `FromBytes` derive will fail compilation if `Self` is not
- // actually soundly `FromBytes`, and so we can rely on that for our
- // `is_bit_valid` impl. It's plausible that we could make changes - or Rust
- // could make changes (such as the "trivial bounds" language feature) - that
- // make this no longer true. To hedge against these, we include an explicit
- // `Self: FromBytes` check in the generated `is_bit_valid`, which is
- // bulletproof.
+ // actually soundly `FromBytes`, and so we can rely on that for our `is_safe`
+ // impl. It's plausible that we could make changes - or Rust could make
+ // changes (such as the "trivial bounds" language feature) - that make this
+ // no longer true. To hedge against these, we include an explicit `Self:
+ // FromBytes` check in the generated `is_safe`, which is bulletproof.
//
// If `ctx.skip_on_error` is true, we can't rely on the `FromBytes` derive
// to fail compilation if `Self` is not actually soundly `FromBytes`.
@@ -715,7 +364,7 @@ fn try_gen_trivial_is_bit_valid(ctx: &Ctx, top_level: Trait) -> Option<proc_macr
Some(quote!(
// SAFETY: See inline.
#[inline(always)]
- fn is_bit_valid<___ZcAlignment>(
+ fn is_safe<___ZcAlignment>(
_candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>,
) -> #core::primitive::bool
where
@@ -746,14 +395,14 @@ fn assert_is_from_bytes<T>()
/// # Safety
///
/// All initialized bit patterns must be valid for `Self`.
-unsafe fn gen_trivial_is_bit_valid_unchecked(ctx: &Ctx) -> proc_macro2::TokenStream {
+unsafe fn gen_trivial_is_safe_unchecked(ctx: &Ctx) -> proc_macro2::TokenStream {
let zerocopy_crate = &ctx.zerocopy_crate;
let core = ctx.core_path();
quote!(
- // SAFETY: The caller of `gen_trivial_is_bit_valid_unchecked` has
- // promised that all initialized bit patterns are valid for `Self`.
+ // SAFETY: The caller of `gen_trivial_is_safe_unchecked` has promised
+ // that all initialized bit patterns are valid for `Self`.
#[inline(always)]
- fn is_bit_valid<___ZcAlignment>(
+ fn is_safe<___ZcAlignment>(
_candidate: #zerocopy_crate::Maybe<'_, Self, ___ZcAlignment>,
) -> #core::primitive::bool
where
@@ -763,3 +412,43 @@ fn is_bit_valid<___ZcAlignment>(
}
)
}
+
+#[cfg(test)]
+mod tests {
+ use syn::{parse_quote, DeriveInput};
+
+ use super::*;
+
+ fn derive_error(input: DeriveInput) -> String {
+ let ctx = Ctx::try_from_derive_input(input).unwrap();
+ match derive_try_from_bytes(&ctx, Trait::TryFromBytes) {
+ Ok(_) => panic!("expected derive to reject conflicting representations"),
+ Err(error) => error.to_string(),
+ }
+ }
+
+ #[test]
+ fn raw_repr_conflict_matches_ordinary_repr() {
+ let ordinary = derive_error(parse_quote! {
+ #[repr(C, align(8))]
+ #[repr(u8)]
+ enum Packet {
+ Value(core::num::NonZeroU8),
+ End,
+ }
+ });
+ let raw = derive_error(parse_quote! {
+ #[repr(C, align(8))]
+ #[r#repr(r#u8)]
+ enum Packet {
+ Value(core::num::NonZeroU8),
+ End,
+ }
+ });
+
+ // Rejecting both spellings prevents generating a validator whose
+ // synthetic enum layout disagrees with the compiler's actual layout.
+ assert_eq!(ordinary, "this conflicts with another representation hint");
+ assert_eq!(ordinary, raw);
+ }
+}
diff --git a/rust/zerocopy-derive/invariant.rs b/rust/zerocopy-derive/invariant.rs
new file mode 100644
index 000000000000..f7a1a617e378
--- /dev/null
+++ b/rust/zerocopy-derive/invariant.rs
@@ -0,0 +1,116 @@
+// SPDX-License-Identifier: (BSD-2-Clause OR Apache-2.0) OR MIT
+//
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use proc_macro2::{Ident, Span, TokenStream, TokenTree};
+use quote::{quote, ToTokens as _};
+use syn::{
+ punctuated::Punctuated, spanned::Spanned as _, Attribute, Data, DeriveInput, Error, Expr,
+ Field, MacroDelimiter, Meta, Token,
+};
+
+use crate::util::path_is_ident;
+
+/// Parses field invariants in attribute order.
+pub(crate) fn parse(attrs: &[Attribute]) -> Result<Vec<Expr>, Error> {
+ let mut invariants = Vec::new();
+ for attr in attrs.iter().filter(|attr| path_is_ident(attr.path(), "zerocopy")) {
+ let options = attr.parse_args_with(Punctuated::<Meta, Token![,]>::parse_terminated)?;
+ for option in options {
+ if !path_is_ident(option.path(), "invariant") {
+ return Err(Error::new_spanned(option, "expected `invariant(...)`"));
+ }
+ match option {
+ Meta::List(list) if matches!(list.delimiter, MacroDelimiter::Paren(_)) => {
+ invariants.push(list.parse_args::<Expr>()?)
+ }
+ other => return Err(Error::new_spanned(other, "expected `invariant(...)`")),
+ }
+ }
+ }
+ Ok(invariants)
+}
+
+/// Validates invariant syntax and placement, returning the first invariant's
+/// span so derives which cannot honor invariants can reject them.
+pub(crate) fn validate(ast: &DeriveInput) -> Result<Option<Span>, Error> {
+ let fields: Vec<&Field> = match &ast.data {
+ Data::Struct(data) => data.fields.iter().collect(),
+ Data::Enum(data) => {
+ for variant in &data.variants {
+ if let Some(invariant) = parse(&variant.attrs)?.first() {
+ return Err(Error::new(
+ invariant.span(),
+ "invariants are only supported on named fields",
+ ));
+ }
+ }
+ data.variants.iter().flat_map(|variant| &variant.fields).collect()
+ }
+ Data::Union(data) => data.fields.named.iter().collect(),
+ };
+ let mut first = None;
+ for field in fields {
+ if let Some(invariant) = parse(&field.attrs)?.first() {
+ if field.ident.is_none() {
+ return Err(Error::new(
+ invariant.span(),
+ "invariants are only supported on named fields",
+ ));
+ }
+ first.get_or_insert(invariant.span());
+ }
+ }
+ Ok(first)
+}
+
+/// Visits identifiers, including those inside macro arguments.
+pub(crate) fn idents(tokens: TokenStream) -> Vec<Ident> {
+ tokens
+ .into_iter()
+ .flat_map(|token| match token {
+ TokenTree::Ident(ident) => vec![ident],
+ TokenTree::Group(group) => idents(group.stream()),
+ _ => Vec::new(),
+ })
+ .collect()
+}
+
+/// Binds semantic field names in every syntax context used by the expression.
+/// Preserve the expression's tokens so its other names retain their hygiene.
+pub(crate) fn bind_fields(fields: &[(&Field, Ident)], expression: &Expr) -> TokenStream {
+ let mut body = expression.to_token_stream();
+ let references = idents(body.clone());
+ for (field, binding) in fields {
+ let name = match &field.ident {
+ Some(name) => name,
+ None => continue,
+ };
+ let spelling = name.to_string();
+ let spelling = spelling.trim_start_matches("r#");
+ // Also support references produced by a macro invoked by the expression.
+ let mut call_site = name.clone();
+ call_site.set_span(Span::call_site());
+ for alias in [name.clone(), call_site].into_iter().chain(
+ references
+ .iter()
+ .filter(|ident| ident.to_string().trim_start_matches("r#") == spelling)
+ .cloned(),
+ ) {
+ body = quote! {{
+ // Require a binding; reject names Rust would resolve as constant
+ // patterns. Item aliases could capture helper calls in macros.
+ #[allow(unused_variables, non_snake_case, clippy::redundant_pattern)]
+ let #alias @ _ = #binding;
+ #body
+ }};
+ }
+ }
+ body
+}
diff --git a/rust/zerocopy-derive/lib.rs b/rust/zerocopy-derive/lib.rs
index d387de368367..c650b59eaa82 100644
--- a/rust/zerocopy-derive/lib.rs
+++ b/rust/zerocopy-derive/lib.rs
@@ -28,6 +28,10 @@
#![allow(clippy::type_complexity)]
// Inlining format args isn't supported on our MSRV.
#![allow(clippy::uninlined_format_args)]
+// `cargo-zerocopy` supplies this cfg for pinned-nightly tests. During UI tests,
+// `testutil::UiTestRunner` explicitly supplies it to the host-built proc macro;
+// ordinary `RUSTFLAGS` are not sufficient when Cargo receives `--target`.
+#![cfg_attr(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS, feature(proc_macro_def_site))]
#![deny(
rustdoc::bare_urls,
rustdoc::broken_intra_doc_links,
@@ -46,6 +50,7 @@ macro_rules! ident {
}
mod derive;
+mod invariant;
#[cfg(test)]
mod output_tests;
mod repr;
@@ -79,7 +84,8 @@ macro_rules! ident {
/// are currently required to live at the crate root, and so the caller must
/// specify the name in order to avoid name collisions.
macro_rules! derive {
- ($trait:ident => $outer:ident => $inner:path) => {
+ ($(#[$attr:meta])* $trait:ident => $outer:ident => $inner:path) => {
+ $(#[$attr])*
#[proc_macro_derive($trait, attributes(zerocopy))]
pub fn $outer(ts: proc_macro::TokenStream) -> proc_macro::TokenStream {
let ast = syn::parse_macro_input!(ts as DeriveInput);
@@ -88,10 +94,13 @@ pub fn $outer(ts: proc_macro::TokenStream) -> proc_macro::TokenStream {
Err(e) => return e.into_compile_error().into(),
};
let ts = $inner(&ctx, Trait::$trait).into_ts();
- // We wrap in `const_block` as a backstop in case any derive fails
- // to wrap its output in `const_block` (and thus fails to annotate)
- // with the full set of `#[allow(...)]` attributes).
- let ts = const_block([Some(ts)]);
+ // Apply generated-code lint allowances as a backstop, except
+ // around caller-authored invariant expressions.
+ let ts = if matches!(Trait::$trait, Trait::TryFromBytes) {
+ ctx.const_block([Some(ts)])
+ } else {
+ const_block([Some(ts)])
+ };
#[cfg(test)]
crate::util::testutil::check_hygiene(ts.clone());
ts.into()
@@ -120,6 +129,7 @@ fn into_ts(self) -> proc_macro2::TokenStream {
derive!(KnownLayout => derive_known_layout => crate::derive::known_layout::derive);
derive!(Immutable => derive_immutable => crate::derive::derive_immutable);
+derive!(#[doc(hidden)] Project => derive_project => crate::derive::project::derive);
derive!(TryFromBytes => derive_try_from_bytes => crate::derive::try_from_bytes::derive_try_from_bytes);
derive!(FromZeros => derive_from_zeros => crate::derive::from_bytes::derive_from_zeros);
derive!(FromBytes => derive_from_bytes => crate::derive::from_bytes::derive_from_bytes);
@@ -129,6 +139,89 @@ fn into_ts(self) -> proc_macro2::TokenStream {
derive!(ByteEq => derive_eq => crate::derive::derive_eq);
derive!(SplitAt => derive_split_at => crate::derive::derive_split_at);
+/// Generates a struct whose two identically-printed field types resolve to
+/// different types. This is used to test that derives preserve type identity
+/// across macro hygiene contexts.
+#[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)]
+#[doc(hidden)]
+#[proc_macro]
+pub fn __test_hygienically_mixed_into_bytes(
+ _input: proc_macro::TokenStream,
+) -> proc_macro::TokenStream {
+ use proc_macro::{Group, Ident, Span, TokenStream, TokenTree};
+
+ fn rewrite(input: TokenStream) -> TokenStream {
+ input
+ .into_iter()
+ .map(|token| match token {
+ TokenTree::Ident(ident) if ident.to_string() == "CallT" => {
+ TokenTree::Ident(Ident::new("T", Span::call_site()))
+ }
+ TokenTree::Ident(ident) if ident.to_string() == "DefT" => {
+ TokenTree::Ident(Ident::new("T", Span::def_site()))
+ }
+ TokenTree::Group(group) => {
+ let mut rewritten = Group::new(group.delimiter(), rewrite(group.stream()));
+ rewritten.set_span(group.span());
+ TokenTree::Group(rewritten)
+ }
+ token => token,
+ })
+ .collect()
+ }
+
+ rewrite(
+ "#[derive(zerocopy_renamed::IntoBytes)] \
+ #[zerocopy(crate = \"zerocopy_renamed\")] \
+ #[repr(C)] \
+ struct IntoBytes15<DefT>(CallT, DefT);"
+ .parse()
+ .expect("test input must parse"),
+ )
+}
+
+/// Constructs an invariant whose field declaration and reference have different
+/// syntax contexts, while preserving all other caller tokens.
+#[doc(hidden)]
+#[proc_macro]
+pub fn __test_hygienically_mixed_invariant(
+ input: proc_macro::TokenStream,
+) -> proc_macro::TokenStream {
+ // Cross-compilation does not necessarily pass the nightly test cfg to
+ // host proc macros. Omit the fixture and its test together in that case.
+ #[cfg(not(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS))]
+ {
+ let _ = input;
+ proc_macro::TokenStream::new()
+ }
+ #[cfg(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS)]
+ {
+ use proc_macro::{Group, Ident, Span, TokenStream, TokenTree};
+
+ fn rewrite(input: TokenStream) -> TokenStream {
+ input
+ .into_iter()
+ .map(|token| match token {
+ TokenTree::Ident(ident) if ident.to_string() == "DefField" => {
+ TokenTree::Ident(Ident::new("field", Span::def_site()))
+ }
+ TokenTree::Ident(ident) if ident.to_string() == "CallField" => {
+ TokenTree::Ident(Ident::new("field", Span::call_site()))
+ }
+ TokenTree::Group(group) => {
+ let mut rewritten = Group::new(group.delimiter(), rewrite(group.stream()));
+ rewritten.set_span(group.span());
+ TokenTree::Group(rewritten)
+ }
+ token => token,
+ })
+ .collect()
+ }
+
+ rewrite(input)
+ }
+}
+
#[cfg_attr(not(zerocopy_unstable_linux), doc(hidden))]
#[proc_macro_derive(most_traits, attributes(zerocopy))]
pub fn most_traits(ts: proc_macro::TokenStream) -> proc_macro::TokenStream {
@@ -153,11 +246,18 @@ pub fn most_traits(ts: proc_macro::TokenStream) -> proc_macro::TokenStream {
for (derive, t) in derives {
tokens.extend(derive(&ctx, t))
}
+ // Invariants prevent `FromBytes` from generating its usual supertrait
+ // impls, but still permit checked conversions through `TryFromBytes`.
+ if ctx.invariant_span.is_some() {
+ tokens.extend(crate::derive::try_from_bytes::derive_try_from_bytes(
+ &ctx,
+ Trait::TryFromBytes,
+ ));
+ }
- // We wrap in `const_block` as a backstop in case any derive fails
- // to wrap its output in `const_block` (and thus fails to annotate)
- // with the full set of `#[allow(...)]` attributes).
- let ts = const_block([Some(tokens)]);
+ // Apply generated-code lint allowances as a backstop, except around
+ // caller-authored invariant expressions.
+ let ts = ctx.const_block([Some(tokens)]);
#[cfg(test)]
crate::util::testutil::check_hygiene(ts.clone());
ts.into()
diff --git a/rust/zerocopy-derive/repr.rs b/rust/zerocopy-derive/repr.rs
index 1525e94302d1..a1a877c0f47c 100644
--- a/rust/zerocopy-derive/repr.rs
+++ b/rust/zerocopy-derive/repr.rs
@@ -20,6 +20,8 @@
MetaList,
};
+use crate::util::{path_is_ident, to_ident_str};
+
/// The computed representation of a type.
///
/// This is the result of processing all `#[repr(...)]` attributes on a type, if
@@ -524,11 +526,11 @@ fn from_attrs(
let mut reprs = Vec::new();
for attr in attrs {
// Ignore documentation attributes.
- if attr.path().is_ident("doc") {
+ if path_is_ident(attr.path(), "doc") {
continue;
}
if let Meta::List(ref meta_list) = attr.meta {
- if meta_list.path.is_ident("repr") {
+ if path_is_ident(&meta_list.path, "repr") {
let parsed: Punctuated<Meta, Comma> =
match meta_list.parse_args_with(Punctuated::parse_terminated) {
Ok(parsed) => parsed,
@@ -564,6 +566,8 @@ fn from_meta(meta: &Meta) -> Result<RawRepr, UnrecognizedReprError> {
let ident = path.get_ident().ok_or(UnrecognizedReprError)?;
// Only returns `Ok` for non-zero power-of-two values.
+ // `NonZeroU32::is_power_of_two` is unsupported on our MSRV.
+ #[allow(clippy::needless_nonzero_get)]
let parse_nzu64 = |list: &MetaList| {
list.parse_args::<LitInt>()
.and_then(|int| int.base10_parse::<NonZeroU32>())
@@ -578,7 +582,8 @@ fn from_meta(meta: &Meta) -> Result<RawRepr, UnrecognizedReprError> {
};
use RawRepr::*;
- Ok(match (ident.to_string().as_str(), list) {
+ let ident = to_ident_str(ident);
+ Ok(match (ident.as_str(), list) {
("u8", None) => U8,
("u16", None) => U16,
("u32", None) => U32,
@@ -743,7 +748,40 @@ macro_rules! test {
use PrimitiveRepr::*;
let nz = |n: u32| NonZeroU32::new(n).unwrap();
+ // Rust treats raw identifiers and their ordinary spellings as the
+ // same representation hints. Exercise every spelling recognized by
+ // `RawRepr::from_meta` so that adding a hint cannot accidentally
+ // reintroduce raw-identifier-sensitive parsing.
+ macro_rules! test_raw_repr {
+ ($ordinary:meta, $raw:meta => $expected:expr) => {{
+ let ordinary: Meta = parse_quote!($ordinary);
+ let raw: Meta = parse_quote!($raw);
+ assert_eq!(RawRepr::from_meta(&ordinary).ok().unwrap(), $expected);
+ assert_eq!(RawRepr::from_meta(&raw).ok().unwrap(), $expected);
+ }};
+ }
+
+ test_raw_repr!(transparent, r#transparent => RawRepr::Transparent);
+ test_raw_repr!(C, r#C => RawRepr::C);
+ test_raw_repr!(Rust, r#Rust => RawRepr::Rust);
+ test_raw_repr!(u8, r#u8 => RawRepr::U8);
+ test_raw_repr!(u16, r#u16 => RawRepr::U16);
+ test_raw_repr!(u32, r#u32 => RawRepr::U32);
+ test_raw_repr!(u64, r#u64 => RawRepr::U64);
+ test_raw_repr!(u128, r#u128 => RawRepr::U128);
+ test_raw_repr!(usize, r#usize => RawRepr::Usize);
+ test_raw_repr!(i8, r#i8 => RawRepr::I8);
+ test_raw_repr!(i16, r#i16 => RawRepr::I16);
+ test_raw_repr!(i32, r#i32 => RawRepr::I32);
+ test_raw_repr!(i64, r#i64 => RawRepr::I64);
+ test_raw_repr!(i128, r#i128 => RawRepr::I128);
+ test_raw_repr!(isize, r#isize => RawRepr::Isize);
+ test_raw_repr!(align(2), r#align(2) => RawRepr::Align(nz(2)));
+ test_raw_repr!(packed, r#packed => RawRepr::Packed);
+ test_raw_repr!(packed(2), r#packed(2) => RawRepr::PackedN(nz(2)));
+
test!(#[repr(transparent)] => StructUnionRepr::Transparent(s()));
+ test!(#[r#repr(r#transparent)] => StructUnionRepr::Transparent(s()));
test!(#[repr()] => StructUnionRepr::Compound(Rust.into(), None));
test!(#[repr(packed)] => StructUnionRepr::Compound(Rust.into(), Some(Packed(nz(1)).into())));
test!(#[repr(packed(2))] => StructUnionRepr::Compound(Rust.into(), Some(Packed(nz(2)).into())));
@@ -754,6 +792,8 @@ macro_rules! test {
test!(#[repr(C, packed(2))] => StructUnionRepr::Compound(C.into(), Some(Packed(nz(2)).into())));
test!(#[repr(C, align(1))] => StructUnionRepr::Compound(C.into(), Some(Align(nz(1)).into())));
test!(#[repr(C, align(2))] => StructUnionRepr::Compound(C.into(), Some(Align(nz(2)).into())));
+ test!(#[r#repr(r#C, r#align(2))] => StructUnionRepr::Compound(C.into(), Some(Align(nz(2)).into())));
+ test!(#[r#repr(r#C, r#packed(2))] => StructUnionRepr::Compound(C.into(), Some(Packed(nz(2)).into())));
test!(#[repr(transparent)] => EnumRepr::Transparent(s()));
test!(#[repr()] => EnumRepr::Compound(Rust.into(), None));
diff --git a/rust/zerocopy-derive/util.rs b/rust/zerocopy-derive/util.rs
index 5c5e9d3bdcb8..717e068dd81c 100644
--- a/rust/zerocopy-derive/util.rs
+++ b/rust/zerocopy-derive/util.rs
@@ -13,9 +13,9 @@
use proc_macro2::{Span, TokenStream};
use quote::{quote, quote_spanned, ToTokens};
use syn::{
- parse_quote, spanned::Spanned as _, Data, DataEnum, DataStruct, DataUnion, DeriveInput, Error,
- Expr, ExprLit, Field, GenericParam, Ident, Index, Lit, LitStr, Meta, Path, Type, Variant,
- Visibility, WherePredicate,
+ parse::ParseBuffer, parse_quote, spanned::Spanned as _, token::PathSep, Data, DataEnum,
+ DataStruct, DataUnion, DeriveInput, Error, Expr, ExprLit, Field, GenericParam, Ident, Index,
+ Lit, LitStr, Meta, Path, Type, Variant, Visibility, WherePredicate,
};
use crate::repr::{CompoundRepr, EnumRepr, PrimitiveRepr, Repr, Spanned};
@@ -30,6 +30,59 @@ pub(crate) struct Ctx {
// The span of the last `#[zerocopy(on_error = ...)]` attribute, if any.
pub(crate) on_error_span: Option<proc_macro2::Span>,
+
+ /// The first field invariant on the source type, if any.
+ pub(crate) invariant_span: Option<Span>,
+}
+
+#[derive(Eq, PartialEq)]
+enum CratePath {
+ External,
+ CrateRelative,
+ ModuleRelative,
+}
+
+fn validate_crate_path(path: &Path) -> Result<CratePath, ()> {
+ if path.segments.is_empty() {
+ return Err(());
+ }
+
+ enum ModuleRelative {
+ Yes,
+ No,
+ }
+ let first = path.segments[0].ident.to_string();
+ let (mut prev_segment, path_type) = match first.as_str() {
+ "Self" => return Err(()),
+ "crate" => {
+ if path.leading_colon.is_some() {
+ return Err(());
+ }
+ (ModuleRelative::No, CratePath::CrateRelative)
+ }
+ "self" | "super" => {
+ if path.leading_colon.is_some() {
+ return Err(());
+ }
+ (ModuleRelative::Yes, CratePath::ModuleRelative)
+ }
+ _ => (ModuleRelative::No, CratePath::External),
+ };
+
+ for seg in path.segments.iter().skip(1) {
+ let ident = seg.ident.to_string();
+ match ident.as_str() {
+ "Self" | "crate" | "self" => return Err(()),
+ "super" => match prev_segment {
+ ModuleRelative::Yes => {}
+ ModuleRelative::No => return Err(()),
+ },
+ _ => {
+ prev_segment = ModuleRelative::No;
+ }
+ }
+ }
+ Ok(path_type)
}
impl Ctx {
@@ -42,24 +95,40 @@ pub(crate) fn try_from_derive_input(ast: DeriveInput) -> Result<Self, Error> {
for attr in &ast.attrs {
if let Meta::List(ref meta_list) = attr.meta {
- if meta_list.path.is_ident("zerocopy") {
+ if path_is_ident(&meta_list.path, "zerocopy") {
attr.parse_nested_meta(|meta| {
- if meta.path.is_ident("crate") {
- let expr = meta.value().and_then(|value| value.parse());
+ if path_is_ident(&meta.path, "crate") {
+ let expr = meta.value().and_then(ParseBuffer::parse);
if let Ok(Expr::Lit(ExprLit { lit: Lit::Str(lit), .. })) = expr {
- if let Ok(path_lit) = lit.parse::<Ident>() {
- path = parse_quote!(::#path_lit);
- return Ok(());
+ if let Ok(mut path_lit) = lit.parse_with(Path::parse_mod_style) {
+ if let Ok(crate_path) = validate_crate_path(&path_lit) {
+ // If not expressly relative, absolutize.
+ if path_lit.leading_colon.is_none() && crate_path == CratePath::External {
+ path_lit.leading_colon = Some(PathSep::default());
+ }
+ path = path_lit;
+ return Ok(());
+ }
+
+ return Err(Error::new(
+ lit.span(),
+ "`crate` attribute requires a valid module path",
+ ));
}
+
+ return Err(Error::new(
+ lit.span(),
+ "`crate` attribute requires a path as the value",
+ ));
}
return Err(Error::new(
- Span::call_site(),
+ meta.path.span(),
"`crate` attribute requires a path as the value",
));
}
- if meta.path.is_ident("on_error") {
+ if path_is_ident(&meta.path, "on_error") {
on_error_span = Some(meta.path.span());
let value = meta.value()?;
let s: LitStr = value.parse()?;
@@ -86,7 +155,8 @@ pub(crate) fn try_from_derive_input(ast: DeriveInput) -> Result<Self, Error> {
}
}
- Ok(Self { ast, zerocopy_crate: path, skip_on_error, on_error_span })
+ let invariant_span = crate::invariant::validate(&ast)?;
+ Ok(Self { ast, zerocopy_crate: path, skip_on_error, on_error_span, invariant_span })
}
pub(crate) fn with_input(&self, input: &DeriveInput) -> Self {
@@ -95,6 +165,7 @@ pub(crate) fn with_input(&self, input: &DeriveInput) -> Self {
zerocopy_crate: self.zerocopy_crate.clone(),
skip_on_error: self.skip_on_error,
on_error_span: self.on_error_span,
+ invariant_span: self.invariant_span,
}
}
@@ -108,15 +179,41 @@ pub(crate) fn core_path(&self) -> TokenStream {
quote!(#zerocopy_crate::util::macro_util::core_reexport)
}
+ /// Choose an implementation name absent from the caller's tokens. This is
+ /// best-effort; names introduced by nested macros are not considered.
+ /// Unlike local variables, type parameters are not hidden by mixed-site
+ /// hygiene.
+ pub(crate) fn fresh_ident(&self, name: &str) -> Ident {
+ let idents = crate::invariant::idents(self.ast.to_token_stream());
+ let mut name = name.to_owned();
+ while idents.iter().any(|ident| ident.to_string().trim_start_matches("r#") == name) {
+ name.push('_');
+ }
+ Ident::new(&name, Span::mixed_site())
+ }
+
+ /// Caller-authored invariant expressions must inherit the caller's lint
+ /// policy, rather than the blanket allowances for generated glue.
+ pub(crate) fn const_block(
+ &self,
+ items: impl IntoIterator<Item = Option<TokenStream>>,
+ ) -> TokenStream {
+ if self.invariant_span.is_none() {
+ const_block(items)
+ } else {
+ let items = items.into_iter().flatten();
+ quote! { const _: () = { #(#items)* }; }
+ }
+ }
+
pub(crate) fn cfg_compile_error(&self) -> TokenStream {
// By checking both during the compilation of the proc macro *and* in
- // the generated code, we ensure that `--cfg
- // zerocopy_unstable_linux` need only be passed *either* when
+ // the generated code, we ensure that each cfg need only be passed when
// compiling this crate *or* when compiling the user's crate. The former
// is preferable, but in some situations (such as when cross-compiling
// using `cargo build --target`), it doesn't get propagated to this
// crate's build by default.
- if cfg!(zerocopy_unstable_linux) {
+ let on_error = if cfg!(zerocopy_unstable_linux) {
quote!()
} else if let Some(span) = self.on_error_span {
let core = self.core_path();
@@ -131,7 +228,24 @@ pub(crate) fn cfg_compile_error(&self) -> TokenStream {
}
} else {
quote!()
- }
+ };
+ let invariant = if cfg!(zerocopy_unstable_ptr) {
+ quote!()
+ } else if let Some(span) = self.invariant_span {
+ let core = self.core_path();
+ let error_message =
+ "`invariant` is experimental; pass '--cfg zerocopy_unstable_ptr' to enable";
+ quote::quote_spanned! {span=>
+ #[allow(unused_attributes, unexpected_cfgs)]
+ const _: () = {
+ #[cfg(not(zerocopy_unstable_ptr))]
+ #core::compile_error!(#error_message);
+ };
+ }
+ } else {
+ quote!()
+ };
+ quote!(#on_error #invariant)
}
pub(crate) fn error_or_skip<E>(&self, error: E) -> Result<TokenStream, E> {
@@ -151,7 +265,7 @@ pub(crate) trait DataExt {
/// FIXME: Extracting field names for enums doesn't really make sense. Types
/// makes sense because we don't care about where they live - we just care
/// about transitive ownership. But for field names, we'd only use them when
- /// generating is_bit_valid, which cares about where they live.
+ /// generating is_safe, which cares about where they live.
fn fields(&self) -> Vec<(&Visibility, TokenStream, &Type)>;
fn variants(&self) -> Vec<(Option<&Variant>, Vec<(&Visibility, TokenStream, &Type)>)>;
@@ -255,6 +369,18 @@ pub(crate) fn to_ident_str(t: &impl ToString) -> String {
}
}
+/// Does `path` consist solely of the identifier `expected`?
+///
+/// Unlike [`Path::is_ident`], this treats a raw identifier and its ordinary
+/// spelling as the same identifier. Rust applies that same normalization when
+/// interpreting attribute names and arguments.
+pub(crate) fn path_is_ident(path: &Path, expected: &str) -> bool {
+ match path.get_ident() {
+ Some(ident) => to_ident_str(ident) == expected,
+ None => false,
+ }
+}
+
/// This enum describes what kind of padding check needs to be generated for the
/// associated impl.
pub(crate) enum PaddingCheck {
@@ -298,22 +424,46 @@ pub(crate) fn validator_macro_context(&self) -> Option<&TokenStream> {
}
}
+#[derive(Copy, Clone)]
+pub(crate) enum Client {
+ ProjectDerive,
+ TryFromBytesDerive,
+}
+
+impl ToTokens for Client {
+ fn to_tokens(&self, tokens: &mut TokenStream) {
+ let s = match self {
+ Client::ProjectDerive => "ProjectDerive",
+ Client::TryFromBytesDerive => "TryFromBytesDerive",
+ };
+ let ident = Ident::new(s, Span::call_site());
+ tokens.extend(quote!(project_clients::#ident));
+ }
+}
+
+impl Client {
+ pub(crate) fn crate_path(&self, ctx: &Ctx) -> Path {
+ let zerocopy_crate = &ctx.zerocopy_crate;
+ parse_quote!(#zerocopy_crate::#self)
+ }
+}
+
+/// Bundles the trait parameters for `HasField` and `ProjectField`.
+#[derive(Clone)]
+pub(crate) struct FieldProjection {
+ pub(crate) variant_id: Box<Expr>,
+ pub(crate) field: Box<Type>,
+ pub(crate) field_id: Box<Expr>,
+}
+
#[derive(Clone)]
pub(crate) enum Trait {
KnownLayout,
- HasTag,
- HasField {
- variant_id: Box<Expr>,
- field: Box<Type>,
- field_id: Box<Expr>,
- },
- ProjectField {
- variant_id: Box<Expr>,
- field: Box<Type>,
- field_id: Box<Expr>,
- invariants: Box<Type>,
- },
+ HasTag { client: Client },
+ HasField { client: Client, projection: FieldProjection },
+ ProjectField { client: Client, projection: FieldProjection, invariants: Box<Type> },
Immutable,
+ Project,
TryFromBytes,
FromZeros,
FromBytes,
@@ -340,8 +490,9 @@ fn to_tokens(&self, tokens: &mut TokenStream) {
Trait::HasField { .. } => "HasField",
Trait::ProjectField { .. } => "ProjectField",
Trait::KnownLayout => "KnownLayout",
- Trait::HasTag => "HasTag",
+ Trait::HasTag { .. } => "HasTag",
Trait::Immutable => "Immutable",
+ Trait::Project => "Project",
Trait::TryFromBytes => "TryFromBytes",
Trait::FromZeros => "FromZeros",
Trait::FromBytes => "FromBytes",
@@ -354,15 +505,18 @@ fn to_tokens(&self, tokens: &mut TokenStream) {
};
let ident = Ident::new(s, Span::call_site());
let arguments: Option<syn::AngleBracketedGenericArguments> = match self {
- Trait::HasField { variant_id, field, field_id } => {
- Some(parse_quote!(<#field, #variant_id, #field_id>))
+ Trait::HasTag { client } => Some(parse_quote!(<#client>)),
+ Trait::HasField { client, projection } => {
+ let FieldProjection { variant_id, field, field_id } = projection;
+ Some(parse_quote!(<#client, #field, #variant_id, #field_id>))
}
- Trait::ProjectField { variant_id, field, field_id, invariants } => {
- Some(parse_quote!(<#field, #invariants, #variant_id, #field_id>))
+ Trait::ProjectField { client, projection, invariants } => {
+ let FieldProjection { variant_id, field, field_id } = projection;
+ Some(parse_quote!(<#client, #field, #invariants, #variant_id, #field_id>))
}
Trait::KnownLayout
- | Trait::HasTag
| Trait::Immutable
+ | Trait::Project
| Trait::TryFromBytes
| Trait::FromZeros
| Trait::FromBytes
@@ -383,6 +537,28 @@ pub(crate) fn crate_path(&self, ctx: &Ctx) -> Path {
let core = ctx.core_path();
match self {
Self::Sized => parse_quote!(#core::marker::#self),
+ Self::HasTag { client } => {
+ let client = client.crate_path(ctx);
+ parse_quote!(#zerocopy_crate::HasTag<#client>)
+ }
+ Self::HasField { client, projection } => {
+ let FieldProjection { variant_id, field, field_id } = projection;
+ let client = client.crate_path(ctx);
+ parse_quote!(#zerocopy_crate::HasField<#client, #field, #variant_id, #field_id>)
+ }
+ Self::ProjectField { client, projection, invariants } => {
+ let FieldProjection { variant_id, field, field_id } = projection;
+ let client = client.crate_path(ctx);
+ parse_quote!(
+ #zerocopy_crate::ProjectField<
+ #client,
+ #field,
+ #invariants,
+ #variant_id,
+ #field_id
+ >
+ )
+ }
_ => parse_quote!(#zerocopy_crate::#self),
}
}
@@ -704,7 +880,12 @@ fn only_derive_is_allowed_to_implement_this_trait() {}
let outer_extras = self.outer_extras.filter(|e| !e.is_empty());
let cfg_compile_error = self.ctx.cfg_compile_error();
- const_block([Some(cfg_compile_error), Some(impl_tokens), outer_extras])
+ let items = [Some(cfg_compile_error), Some(impl_tokens), outer_extras];
+ if matches!(self.trt, Trait::TryFromBytes) {
+ self.ctx.const_block(items)
+ } else {
+ const_block(items)
+ }
}
}
@@ -730,8 +911,7 @@ fn then_some<T>(self, t: T) -> Option<T> {
}
}
-pub(crate) fn const_block(items: impl IntoIterator<Item = Option<TokenStream>>) -> TokenStream {
- let items = items.into_iter().flatten();
+pub(crate) fn allow_generated_code() -> TokenStream {
quote! {
#[allow(
// FIXME(#553): Add a test that generates a warning when
@@ -748,6 +928,14 @@ pub(crate) fn const_block(items: impl IntoIterator<Item = Option<TokenStream>>)
non_ascii_idents,
clippy::missing_inline_in_public_items,
)]
+ }
+}
+
+pub(crate) fn const_block(items: impl IntoIterator<Item = Option<TokenStream>>) -> TokenStream {
+ let items = items.into_iter().flatten();
+ let allow = allow_generated_code();
+ quote! {
+ #allow
#[deny(ambiguous_associated_items)]
// While there are not currently any warnings that this suppresses
// (that we're aware of), it's good future-proofing hygiene.
@@ -778,7 +966,8 @@ pub(crate) fn generate_tag_enum(ctx: &Ctx, repr: &EnumRepr, data: &DataEnum) ->
quote! {
#repr
- #[allow(dead_code)]
+ #[allow(dead_code, clippy::derive_partial_eq_without_eq)]
+ #[derive(Copy, Clone, PartialEq)]
pub enum ___ZerocopyTag {
#(#variants,)*
}
@@ -824,18 +1013,40 @@ pub(crate) mod testutil {
pub(crate) fn check_hygiene(ts: TokenStream) {
struct AmbiguousItemVisitor;
+ fn panic_ambiguous_path(path: &impl quote::ToTokens) -> ! {
+ panic!(
+ "Found ambiguous path `{}` in generated output. \
+ All associated item access must be fully qualified (e.g., `<Self as Trait>::Item`) \
+ to prevent hygiene issues.",
+ quote::quote!(#path)
+ );
+ }
+
impl<'ast> Visit<'ast> for AmbiguousItemVisitor {
+ fn visit_expr_path(&mut self, i: &'ast syn::ExprPath) {
+ if let Some(qself) = &i.qself {
+ if qself.position == 0 {
+ panic_ambiguous_path(i);
+ }
+ }
+ visit::visit_expr_path(self, i);
+ }
+
fn visit_path(&mut self, i: &'ast syn::Path) {
if i.segments.len() > 1 && i.segments.first().unwrap().ident == "Self" {
- panic!(
- "Found ambiguous path `{}` in generated output. \
- All associated item access must be fully qualified (e.g., `<Self as Trait>::Item`) \
- to prevent hygiene issues.",
- quote::quote!(#i)
- );
+ panic_ambiguous_path(i);
}
visit::visit_path(self, i);
}
+
+ fn visit_type_path(&mut self, i: &'ast syn::TypePath) {
+ if let Some(qself) = &i.qself {
+ if qself.position == 0 {
+ panic_ambiguous_path(i);
+ }
+ }
+ visit::visit_type_path(self, i);
+ }
}
let file = syn::parse2::<syn::File>(ts).expect("failed to parse generated output as File");
@@ -860,4 +1071,117 @@ fn foo() {
}
});
}
+
+ #[test]
+ #[should_panic(expected = "Found ambiguous path `< T > :: Ambiguous`")]
+ fn test_check_hygiene_type_relative_expr_failure() {
+ check_hygiene(quote::quote! {
+ fn foo<T>() {
+ let _ = <T>::Ambiguous;
+ }
+ });
+ }
+
+ #[test]
+ #[should_panic(expected = "Found ambiguous path `< T > :: Ambiguous`")]
+ fn test_check_hygiene_type_relative_type_failure() {
+ check_hygiene(quote::quote! {
+ fn foo<T>() {
+ let _: <T>::Ambiguous;
+ }
+ });
+ }
+
+ #[test]
+ fn test_validate_crate_path() {
+ use syn::parse_str;
+
+ let valid = [
+ "zerocopy",
+ "crate",
+ "crate::foo::bar",
+ "self",
+ "self::foo",
+ "self::super::foo",
+ "super",
+ "super::foo",
+ "super::super::foo",
+ "super::super::super",
+ "foo::bar::baz",
+ "::foo::bar",
+ ];
+
+ for path_str in valid {
+ let path = parse_str::<syn::Path>(path_str).unwrap();
+ assert!(
+ super::validate_crate_path(&path).is_ok(),
+ "expected valid path for `{}`",
+ path_str
+ );
+ }
+
+ let invalid = [
+ "::crate::foo",
+ "::self::foo",
+ "::super::foo",
+ "Self",
+ "Self::foo",
+ "foo::Self::bar",
+ "foo::crate::bar",
+ "foo::super::bar",
+ "foo::self",
+ "super::foo::super",
+ "super::crate::foo",
+ "crate::super::foo",
+ "self::self::foo",
+ ];
+
+ for path_str in invalid {
+ let path = parse_str::<syn::Path>(path_str).unwrap();
+ assert!(
+ super::validate_crate_path(&path).is_err(),
+ "expected invalid path for `{}`",
+ path_str
+ );
+ }
+ }
+
+ #[test]
+ fn test_path_is_ident() {
+ use syn::parse_str;
+
+ for (expected, ordinary, raw) in [
+ ("doc", "doc", "r#doc"),
+ ("repr", "repr", "r#repr"),
+ ("zerocopy", "zerocopy", "r#zerocopy"),
+ ("on_error", "on_error", "r#on_error"),
+ ] {
+ let ordinary = parse_str::<syn::Path>(ordinary).unwrap();
+ let raw = parse_str::<syn::Path>(raw).unwrap();
+ assert!(super::path_is_ident(&ordinary, expected));
+ assert!(super::path_is_ident(&raw, expected));
+ }
+
+ // `r#crate` is forbidden by Rust's raw identifier grammar.
+ let crate_path = parse_str::<syn::Path>("crate").unwrap();
+ assert!(super::path_is_ident(&crate_path, "crate"));
+
+ for path in ["::zerocopy", "module::zerocopy"] {
+ let path = parse_str::<syn::Path>(path).unwrap();
+ assert!(!super::path_is_ident(&path, "zerocopy"));
+ }
+ }
+
+ #[test]
+ fn test_raw_zerocopy_attributes() {
+ let ast: syn::DeriveInput = syn::parse_quote! {
+ #[r#zerocopy(crate = "renamed", r#on_error = "skip")]
+ struct Foo;
+ };
+
+ let ctx = super::Ctx::try_from_derive_input(ast).unwrap();
+ let path = &ctx.zerocopy_crate;
+ assert_eq!(quote::quote!(#path).to_string(), ":: renamed");
+ assert!(ctx.skip_on_error);
+ }
}
diff --git a/rust/zerocopy/README.md b/rust/zerocopy/README.md
index 8443f6166008..5cdea261e45f 100644
--- a/rust/zerocopy/README.md
+++ b/rust/zerocopy/README.md
@@ -1,13 +1,13 @@
# `zerocopy`
-These source files come from the Rust `zerocopy` crate, version v0.8.54
-(released 2026-07-08), hosted in the <https://github.com/google/zerocopy>
+These source files come from the Rust `zerocopy` crate, version v0.8.62
+(released 2026-10-08), hosted in the <https://github.com/google/zerocopy>
repository, licensed under "BSD-2-Clause OR Apache-2.0 OR MIT" and only
modified to tweak the SPDX license identifiers.
For copyright details, please see:
- https://github.com/google/zerocopy/blob/v0.8.54/zerocopy/README.md?plain=1
- https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-BSD
- https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-APACHE
- https://github.com/google/zerocopy/blob/v0.8.54/LICENSE-MIT
+ https://github.com/google/zerocopy/blob/v0.8.62/zerocopy/README.md?plain=1
+ https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-BSD
+ https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-APACHE
+ https://github.com/google/zerocopy/blob/v0.8.62/LICENSE-MIT
diff --git a/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64
index f68bad612695..12ab2e3c4b37 100644
--- a/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64
+++ b/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64
@@ -1,5 +1,4 @@
bench_as_bytes_dynamic_size:
mov rax, rdi
- lea rdx, [2*rsi + 5]
- and rdx, -2
+ lea rdx, [2*rsi + 4]
ret
diff --git a/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64.mca
index c3b92a9a95fe..d661f093fea6 100644
--- a/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64.mca
+++ b/rust/zerocopy/benches/as_bytes_dynamic_size.x86-64.mca
@@ -1,11 +1,11 @@
Iterations: 100
-Instructions: 400
-Total Cycles: 137
-Total uOps: 400
+Instructions: 300
+Total Cycles: 104
+Total uOps: 300
Dispatch Width: 4
-uOps Per Cycle: 2.92
-IPC: 2.92
+uOps Per Cycle: 2.88
+IPC: 2.88
Block RThroughput: 1.0
@@ -19,8 +19,7 @@ Instruction Info:
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 mov rax, rdi
- 1 1 0.50 lea rdx, [2*rsi + 5]
- 1 1 0.33 and rdx, -2
+ 1 1 0.50 lea rdx, [2*rsi + 4]
1 1 1.00 U ret
@@ -37,11 +36,10 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 1.33 1.33 - 1.34 - -
+ - - 0.99 1.00 - 1.01 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - - 0.66 - 0.34 - - mov rax, rdi
- - - 0.33 0.67 - - - - lea rdx, [2*rsi + 5]
- - - 1.00 - - - - - and rdx, -2
+ - - 0.99 - - 0.01 - - mov rax, rdi
+ - - - 1.00 - - - - lea rdx, [2*rsi + 4]
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/extend_vec_zeroed.x86-64 b/rust/zerocopy/benches/extend_vec_zeroed.x86-64
index 831b2a075fec..0790ce20273d 100644
--- a/rust/zerocopy/benches/extend_vec_zeroed.x86-64
+++ b/rust/zerocopy/benches/extend_vec_zeroed.x86-64
@@ -41,8 +41,8 @@ bench_extend_vec_zeroed:
.LBB6_6:
mov r13, rsi
lea rcx, [rax + rax]
- cmp r15, rcx
- cmova rcx, r15
+ cmp rcx, r15
+ cmovbe rcx, r15
cmp rcx, 5
mov r14d, 4
cmovae r14, rcx
diff --git a/rust/zerocopy/benches/extend_vec_zeroed.x86-64.mca b/rust/zerocopy/benches/extend_vec_zeroed.x86-64.mca
index cfab1eea8f56..cf3c5a5ad5e3 100644
--- a/rust/zerocopy/benches/extend_vec_zeroed.x86-64.mca
+++ b/rust/zerocopy/benches/extend_vec_zeroed.x86-64.mca
@@ -55,8 +55,8 @@ Instruction Info:
1 1 1.00 U ret
1 1 0.33 mov r13, rsi
1 1 0.50 lea rcx, [rax + rax]
- 1 1 0.33 cmp r15, rcx
- 3 3 1.00 cmova rcx, r15
+ 1 1 0.33 cmp rcx, r15
+ 3 3 1.00 cmovbe rcx, r15
1 1 0.33 cmp rcx, 5
1 1 0.33 mov r14d, 4
2 2 0.67 cmovae r14, rcx
@@ -128,8 +128,8 @@ Resource pressure by instruction:
- - - - - 1.00 - - ret
- - 1.00 - - - - - mov r13, rsi
- - 0.01 0.99 - - - - lea rcx, [rax + rax]
- - - 0.99 0.01 - - - - cmp r15, rcx
- - - 2.00 0.01 - 0.99 - - cmova rcx, r15
+ - - 0.99 0.01 - - - - cmp rcx, r15
+ - - 2.00 0.01 - 0.99 - - cmovbe rcx, r15
- - 0.01 0.99 - - - - cmp rcx, 5
- - 0.01 0.99 - - - - mov r14d, 4
- - 1.00 0.01 - 0.99 - - cmovae r14, rcx
diff --git a/rust/zerocopy/benches/insert_vec_zeroed.x86-64 b/rust/zerocopy/benches/insert_vec_zeroed.x86-64
index 9db87403cb55..b5426c3f4a3e 100644
--- a/rust/zerocopy/benches/insert_vec_zeroed.x86-64
+++ b/rust/zerocopy/benches/insert_vec_zeroed.x86-64
@@ -54,8 +54,8 @@ bench_insert_vec_zeroed:
mov r15, rsi
mov rbp, rdx
lea rcx, [rax + rax]
- cmp r12, rcx
- cmova rcx, r12
+ cmp rcx, r12
+ cmovbe rcx, r12
cmp rcx, 5
mov r14d, 4
cmovae r14, rcx
diff --git a/rust/zerocopy/benches/insert_vec_zeroed.x86-64.mca b/rust/zerocopy/benches/insert_vec_zeroed.x86-64.mca
index 665240667844..a6c17e236823 100644
--- a/rust/zerocopy/benches/insert_vec_zeroed.x86-64.mca
+++ b/rust/zerocopy/benches/insert_vec_zeroed.x86-64.mca
@@ -68,8 +68,8 @@ Instruction Info:
1 1 0.33 mov r15, rsi
1 1 0.33 mov rbp, rdx
1 1 0.50 lea rcx, [rax + rax]
- 1 1 0.33 cmp r12, rcx
- 3 3 1.00 cmova rcx, r12
+ 1 1 0.33 cmp rcx, r12
+ 3 3 1.00 cmovbe rcx, r12
1 1 0.33 cmp rcx, 5
1 1 0.33 mov r14d, 4
2 2 0.67 cmovae r14, rcx
@@ -159,8 +159,8 @@ Resource pressure by instruction:
- - 0.49 0.51 - - - - mov r15, rsi
- - 0.51 0.49 - - - - mov rbp, rdx
- - 0.49 0.51 - - - - lea rcx, [rax + rax]
- - - 0.49 0.50 - 0.01 - - cmp r12, rcx
- - - 1.04 0.50 - 1.46 - - cmova rcx, r12
+ - - 0.49 0.50 - 0.01 - - cmp rcx, r12
+ - - 1.04 0.50 - 1.46 - - cmovbe rcx, r12
- - 0.49 0.49 - 0.02 - - cmp rcx, 5
- - 0.50 - - 0.50 - - mov r14d, 4
- - 0.50 0.51 - 0.99 - - cmovae r14, rcx
diff --git a/rust/zerocopy/benches/new_vec_zeroed.x86-64 b/rust/zerocopy/benches/new_vec_zeroed.x86-64
index b5c083aa0d36..edf421e3192c 100644
--- a/rust/zerocopy/benches/new_vec_zeroed.x86-64
+++ b/rust/zerocopy/benches/new_vec_zeroed.x86-64
@@ -29,8 +29,7 @@ bench_new_vec_zeroed:
mov qword ptr [rax + 16], rsi
ret
.LBB5_5:
- movabs rcx, -9223372036854775808
- mov qword ptr [rax], rcx
+ mov qword ptr [rax], -1
ret
.LBB5_2:
mov ecx, 2
diff --git a/rust/zerocopy/benches/new_vec_zeroed.x86-64.mca b/rust/zerocopy/benches/new_vec_zeroed.x86-64.mca
index b4fb4544ec39..f0e47d004ccc 100644
--- a/rust/zerocopy/benches/new_vec_zeroed.x86-64.mca
+++ b/rust/zerocopy/benches/new_vec_zeroed.x86-64.mca
@@ -1,11 +1,11 @@
Iterations: 100
-Instructions: 3700
+Instructions: 3600
Total Cycles: 3486
-Total uOps: 4600
+Total uOps: 4500
Dispatch Width: 4
-uOps Per Cycle: 1.32
-IPC: 1.06
+uOps Per Cycle: 1.29
+IPC: 1.03
Block RThroughput: 12.0
@@ -47,8 +47,7 @@ Instruction Info:
1 1 1.00 * mov qword ptr [rax + 8], rcx
1 1 1.00 * mov qword ptr [rax + 16], rsi
1 1 1.00 U ret
- 1 1 0.33 movabs rcx, -9223372036854775808
- 1 1 1.00 * mov qword ptr [rax], rcx
+ 1 1 1.00 * mov qword ptr [rax], -1
1 1 1.00 U ret
1 1 0.33 mov ecx, 2
1 1 1.00 * mov qword ptr [rax], rsi
@@ -70,31 +69,31 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 6.99 6.99 12.00 10.02 8.00 9.00
+ - - 5.99 6.01 12.00 11.00 8.00 9.00
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.01 0.98 - 0.01 - - mov rax, rdi
- - - 0.98 0.02 - - - - movabs rcx, 1537228672809129301
- - - 0.02 0.98 - - - - cmp rsi, rcx
+ - - 0.97 0.02 - 0.01 - - mov rax, rdi
+ - - 0.02 0.98 - - - - movabs rcx, 1537228672809129301
+ - - 0.97 0.03 - - - - cmp rsi, rcx
- - - - - 1.00 - - ja .LBB5_5
- - - 0.98 - - 0.02 - - test rsi, rsi
+ - - 0.03 0.96 - 0.01 - - test rsi, rsi
- - - - - 1.00 - - je .LBB5_2
- - - - 1.00 - - 1.00 push r15
- - - - 1.00 - 1.00 - push r14
- - - - 1.00 - - 1.00 push rbx
- - - - 1.00 - - - - lea rcx, [rsi + rsi]
- - - - 1.00 - - - - lea rbx, [rcx + 2*rcx]
- - - 1.00 - - - - - mov r14, rax
- - - 1.00 - - - - - mov r15, rsi
+ - - 0.99 0.01 - - - - lea rcx, [rsi + rsi]
+ - - 0.99 0.01 - - - - lea rbx, [rcx + 2*rcx]
+ - - 0.01 0.99 - - - - mov r14, rax
+ - - 0.01 0.01 - 0.98 - - mov r15, rsi
- - - - 1.00 1.00 2.00 - call qword ptr [rip + __rustc::__rust_no_alloc_shim_is_unstable_v2@GOTPCREL]
- - - - 0.01 - 0.99 - - mov esi, 2
- - - 0.01 0.99 - - - - mov rdi, rbx
+ - - - 0.98 - 0.02 - - mov esi, 2
+ - - 0.96 0.02 - 0.02 - - mov rdi, rbx
- - - - 1.00 1.00 - 2.00 call qword ptr [rip + __rustc::__rust_alloc_zeroed@GOTPCREL]
- - - 0.01 - - 0.99 - - mov rsi, r15
- - - 0.99 0.01 - - - - mov rcx, rax
- - - - 0.99 - 0.01 - - mov rax, r14
- - - 0.99 0.01 - - - - test rcx, rcx
+ - - 0.02 0.96 - 0.02 - - mov rsi, r15
+ - - 0.02 0.02 - 0.96 - - mov rcx, rax
+ - - 0.96 0.02 - 0.02 - - mov rax, r14
+ - - 0.02 0.02 - 0.96 - - test rcx, rcx
- - - - - - - 1.00 pop rbx
- - - - - - 1.00 - pop r14
- - - - - - - 1.00 pop r15
@@ -103,10 +102,9 @@ Resource pressure by instruction:
- - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rcx
- - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rsi
- - - - - 1.00 - - ret
- - - 0.01 0.99 - - - - movabs rcx, -9223372036854775808
- - - - - 1.00 - - 1.00 mov qword ptr [rax], rcx
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], -1
- - - - - 1.00 - - ret
- - - 0.99 0.01 - - - - mov ecx, 2
+ - - 0.02 0.98 - - - - mov ecx, 2
- - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
- - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rcx
- - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rsi
diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64
index e844a4608fac..7f695f942e67 100644
--- a/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64
+++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64
@@ -1,22 +1,21 @@
bench_ref_from_bytes_dynamic_padding:
- test dil, 3
- jne .LBB5_3
- movabs rax, 9223372036854775804
- and rax, rsi
- cmp rax, 9
- jb .LBB5_3
- add rax, -9
+ cmp rsi, 12
+ setae al
+ mov ecx, esi
+ or ecx, edi
+ test cl, 3
+ sete cl
+ and cl, al
+ cmp cl, 1
+ je .LBB5_2
+ xor edi, edi
+ mov rax, rdi
+ ret
+.LBB5_2:
+ add rsi, -9
movabs rcx, -6148914691236517205
+ mov rax, rsi
mul rcx
shr rdx
- lea rax, [rdx + 2*rdx]
- or rax, 3
- add rax, 9
- cmp rsi, rax
- je .LBB5_4
-.LBB5_3:
- xor edi, edi
- mov rdx, rsi
-.LBB5_4:
mov rax, rdi
ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64.mca
index 423ed38ba28d..5eb4fd1a113d 100644
--- a/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64.mca
+++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_padding.x86-64.mca
@@ -1,11 +1,11 @@
Iterations: 100
Instructions: 1900
-Total Cycles: 645
+Total Cycles: 641
Total uOps: 2000
Dispatch Width: 4
-uOps Per Cycle: 3.10
-IPC: 2.95
+uOps Per Cycle: 3.12
+IPC: 2.96
Block RThroughput: 5.0
@@ -18,23 +18,23 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
- 1 1 0.33 test dil, 3
- 1 1 1.00 jne .LBB5_3
- 1 1 0.33 movabs rax, 9223372036854775804
- 1 1 0.33 and rax, rsi
- 1 1 0.33 cmp rax, 9
- 1 1 1.00 jb .LBB5_3
- 1 1 0.33 add rax, -9
+ 1 1 0.33 cmp rsi, 12
+ 1 1 0.50 setae al
+ 1 1 0.33 mov ecx, esi
+ 1 1 0.33 or ecx, edi
+ 1 1 0.33 test cl, 3
+ 1 1 0.50 sete cl
+ 1 1 0.33 and cl, al
+ 1 1 0.33 cmp cl, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor edi, edi
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+ 1 1 0.33 add rsi, -9
1 1 0.33 movabs rcx, -6148914691236517205
+ 1 1 0.33 mov rax, rsi
2 4 1.00 mul rcx
1 1 0.50 shr rdx
- 1 1 0.50 lea rax, [rdx + 2*rdx]
- 1 1 0.33 or rax, 3
- 1 1 0.33 add rax, 9
- 1 1 0.33 cmp rsi, rax
- 1 1 1.00 je .LBB5_4
- 1 0 0.25 xor edi, edi
- 1 1 0.33 mov rdx, rsi
1 1 0.33 mov rax, rdi
1 1 1.00 U ret
@@ -52,26 +52,26 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 6.32 6.33 - 6.35 - -
+ - - 6.33 6.33 - 6.34 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.64 0.35 - 0.01 - - test dil, 3
- - - - - - 1.00 - - jne .LBB5_3
- - - 0.34 0.65 - 0.01 - - movabs rax, 9223372036854775804
- - - 0.35 0.65 - - - - and rax, rsi
- - - 0.33 0.34 - 0.33 - - cmp rax, 9
- - - - - - 1.00 - - jb .LBB5_3
- - - 0.35 - - 0.65 - - add rax, -9
- - - 0.97 0.01 - 0.02 - - movabs rcx, -6148914691236517205
- - - 1.00 1.00 - - - - mul rcx
- - - 0.99 - - 0.01 - - shr rdx
- - - 0.33 0.67 - - - - lea rax, [rdx + 2*rdx]
- - - 0.34 0.66 - - - - or rax, 3
- - - 0.33 0.66 - 0.01 - - add rax, 9
- - - 0.01 0.99 - - - - cmp rsi, rax
- - - - - - 1.00 - - je .LBB5_4
+ - - 0.99 - - 0.01 - - cmp rsi, 12
+ - - 1.00 - - - - - setae al
+ - - - 0.34 - 0.66 - - mov ecx, esi
+ - - - 0.99 - 0.01 - - or ecx, edi
+ - - - 1.00 - - - - test cl, 3
+ - - 0.67 - - 0.33 - - sete cl
+ - - - 1.00 - - - - and cl, al
+ - - - 1.00 - - - - cmp cl, 1
+ - - - - - 1.00 - - je .LBB5_2
- - - - - - - - xor edi, edi
- - - 0.32 0.01 - 0.67 - - mov rdx, rsi
- - - 0.02 0.34 - 0.64 - - mov rax, rdi
+ - - 0.66 - - 0.34 - - mov rax, rdi
+ - - - - - 1.00 - - ret
+ - - 0.35 0.01 - 0.64 - - add rsi, -9
+ - - 0.65 0.33 - 0.02 - - movabs rcx, -6148914691236517205
+ - - 0.01 0.33 - 0.66 - - mov rax, rsi
+ - - 1.00 1.00 - - - - mul rcx
+ - - 1.00 - - - - - shr rdx
+ - - - 0.33 - 0.67 - - mov rax, rdi
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64
index cc905b76c06f..43cadf352783 100644
--- a/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64
+++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64
@@ -1,20 +1,11 @@
bench_ref_from_bytes_dynamic_size:
mov rdx, rsi
- cmp rsi, 4
- setb al
- or al, dil
- test al, 1
- je .LBB5_2
+ mov ecx, edx
+ or ecx, edi
xor eax, eax
- ret
-.LBB5_2:
- lea rcx, [rdx - 4]
- mov rsi, rcx
- and rsi, -2
- add rsi, 4
- shr rcx
- xor eax, eax
- cmp rdx, rsi
- cmove rdx, rcx
+ sub rdx, 4
+ cmovb rdi, rax
+ shr rdx
+ test cl, 1
cmove rax, rdi
ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64.mca
index 68aea583e401..444f4fad4ff3 100644
--- a/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64.mca
+++ b/rust/zerocopy/benches/ref_from_bytes_dynamic_size.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 1800
-Total Cycles: 704
-Total uOps: 2000
+Instructions: 1000
+Total Cycles: 373
+Total uOps: 1200
Dispatch Width: 4
-uOps Per Cycle: 2.84
-IPC: 2.56
-Block RThroughput: 5.0
+uOps Per Cycle: 3.22
+IPC: 2.68
+Block RThroughput: 3.0
Instruction Info:
@@ -19,21 +19,13 @@ Instruction Info:
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 mov rdx, rsi
- 1 1 0.33 cmp rsi, 4
- 1 1 0.50 setb al
- 1 1 0.33 or al, dil
- 1 1 0.33 test al, 1
- 1 1 1.00 je .LBB5_2
+ 1 1 0.33 mov ecx, edx
+ 1 1 0.33 or ecx, edi
1 0 0.25 xor eax, eax
- 1 1 1.00 U ret
- 1 1 0.50 lea rcx, [rdx - 4]
- 1 1 0.33 mov rsi, rcx
- 1 1 0.33 and rsi, -2
- 1 1 0.33 add rsi, 4
- 1 1 0.50 shr rcx
- 1 0 0.25 xor eax, eax
- 1 1 0.33 cmp rdx, rsi
- 2 2 0.67 cmove rdx, rcx
+ 1 1 0.33 sub rdx, 4
+ 2 2 0.67 cmovb rdi, rax
+ 1 1 0.50 shr rdx
+ 1 1 0.33 test cl, 1
2 2 0.67 cmove rax, rdi
1 1 1.00 U ret
@@ -51,25 +43,17 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 5.97 5.98 - 6.05 - -
+ - - 3.66 3.67 - 3.67 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.97 0.01 - 0.02 - - mov rdx, rsi
- - - 0.01 0.02 - 0.97 - - cmp rsi, 4
- - - 0.03 - - 0.97 - - setb al
- - - 0.01 0.02 - 0.97 - - or al, dil
- - - - 0.98 - 0.02 - - test al, 1
- - - - - - 1.00 - - je .LBB5_2
- - - - - - - - - xor eax, eax
- - - - - - 1.00 - - ret
- - - 0.98 0.02 - - - - lea rcx, [rdx - 4]
- - - 0.01 0.99 - - - - mov rsi, rcx
- - - - 0.98 - 0.02 - - and rsi, -2
- - - 0.98 0.01 - 0.01 - - add rsi, 4
- - - 0.99 - - 0.01 - - shr rcx
+ - - 0.32 0.01 - 0.67 - - mov rdx, rsi
+ - - 0.33 0.35 - 0.32 - - mov ecx, edx
+ - - 0.01 0.34 - 0.65 - - or ecx, edi
- - - - - - - - xor eax, eax
- - - 0.02 0.97 - 0.01 - - cmp rdx, rsi
- - - 0.99 0.99 - 0.02 - - cmove rdx, rcx
- - - 0.98 0.99 - 0.03 - - cmove rax, rdi
+ - - 0.67 0.33 - - - - sub rdx, 4
+ - - 0.99 0.98 - 0.03 - - cmovb rdi, rax
+ - - 0.03 - - 0.97 - - shr rdx
+ - - 0.32 0.67 - 0.01 - - test cl, 1
+ - - 0.99 0.99 - 0.02 - - cmove rax, rdi
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64 b/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64
index 2c8da68c8b5b..0e16b103344f 100644
--- a/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64
+++ b/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64
@@ -1,8 +1,7 @@
bench_ref_from_bytes_static_size:
- mov ecx, edi
- and ecx, 1
- xor rsi, 6
xor eax, eax
- or rsi, rcx
+ test dil, 1
+ cmovne rdi, rax
+ cmp rsi, 6
cmove rax, rdi
ret
diff --git a/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64.mca b/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64.mca
index 832697801ee2..f1e118abf09a 100644
--- a/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64.mca
+++ b/rust/zerocopy/benches/ref_from_bytes_static_size.x86-64.mca
@@ -1,11 +1,11 @@
Iterations: 100
-Instructions: 700
-Total Cycles: 240
+Instructions: 600
+Total Cycles: 305
Total uOps: 800
Dispatch Width: 4
-uOps Per Cycle: 3.33
-IPC: 2.92
+uOps Per Cycle: 2.62
+IPC: 1.97
Block RThroughput: 2.0
@@ -18,11 +18,10 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
- 1 1 0.33 mov ecx, edi
- 1 1 0.33 and ecx, 1
- 1 1 0.33 xor rsi, 6
1 0 0.25 xor eax, eax
- 1 1 0.33 or rsi, rcx
+ 1 1 0.33 test dil, 1
+ 2 2 0.67 cmovne rdi, rax
+ 1 1 0.33 cmp rsi, 6
2 2 0.67 cmove rax, rdi
1 1 1.00 U ret
@@ -40,14 +39,13 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 2.33 2.33 - 2.34 - -
+ - - 1.95 2.28 - 2.77 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.01 0.98 - 0.01 - - mov ecx, edi
- - - 0.02 0.66 - 0.32 - - and ecx, 1
- - - 0.33 0.66 - 0.01 - - xor rsi, 6
- - - - - - - - xor eax, eax
- - - 0.98 0.02 - - - - or rsi, rcx
- - - 0.99 0.01 - 1.00 - - cmove rax, rdi
+ - - 0.05 0.06 - 0.89 - - test dil, 1
+ - - 0.95 0.94 - 0.11 - - cmovne rdi, rax
+ - - - 0.34 - 0.66 - - cmp rsi, 6
+ - - 0.95 0.94 - 0.11 - - cmove rax, rdi
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64
index a58592a24503..e7f7820a69d1 100644
--- a/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64
+++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64
@@ -1,19 +1,19 @@
bench_ref_from_prefix_dynamic_padding:
+ test dil, 3
+ sete al
+ cmp rsi, 12
+ setae cl
+ test al, cl
+ jne .LBB5_2
xor edx, edx
- mov eax, 0
test dil, 3
- je .LBB5_1
- ret
-.LBB5_1:
- movabs rax, 9223372036854775804
- and rsi, rax
- cmp rsi, 9
- jae .LBB5_3
- mov edx, 1
- xor eax, eax
+ sete dl
+ xor edi, edi
+ mov rax, rdi
ret
-.LBB5_3:
- add rsi, -9
+.LBB5_2:
+ or rsi, 3
+ add rsi, -12
movabs rcx, -6148914691236517205
mov rax, rsi
mul rcx
diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64.mca
index 62ea4babaf28..a3d68f37a8be 100644
--- a/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64.mca
+++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_padding.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 1900
-Total Cycles: 608
-Total uOps: 2000
+Instructions: 2000
+Total Cycles: 640
+Total uOps: 2100
Dispatch Width: 4
-uOps Per Cycle: 3.29
+uOps Per Cycle: 3.28
IPC: 3.13
-Block RThroughput: 5.0
+Block RThroughput: 5.3
Instruction Info:
@@ -18,19 +18,20 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 test dil, 3
+ 1 1 0.50 sete al
+ 1 1 0.33 cmp rsi, 12
+ 1 1 0.50 setae cl
+ 1 1 0.33 test al, cl
+ 1 1 1.00 jne .LBB5_2
1 0 0.25 xor edx, edx
- 1 1 0.33 mov eax, 0
1 1 0.33 test dil, 3
- 1 1 1.00 je .LBB5_1
- 1 1 1.00 U ret
- 1 1 0.33 movabs rax, 9223372036854775804
- 1 1 0.33 and rsi, rax
- 1 1 0.33 cmp rsi, 9
- 1 1 1.00 jae .LBB5_3
- 1 1 0.33 mov edx, 1
- 1 0 0.25 xor eax, eax
+ 1 1 0.50 sete dl
+ 1 0 0.25 xor edi, edi
+ 1 1 0.33 mov rax, rdi
1 1 1.00 U ret
- 1 1 0.33 add rsi, -9
+ 1 1 0.33 or rsi, 3
+ 1 1 0.33 add rsi, -12
1 1 0.33 movabs rcx, -6148914691236517205
1 1 0.33 mov rax, rsi
2 4 1.00 mul rcx
@@ -52,26 +53,27 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 6.00 6.00 - 6.00 - -
+ - - 6.33 6.33 - 6.34 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.65 0.34 - 0.01 - - test dil, 3
+ - - 0.99 - - 0.01 - - sete al
+ - - 0.01 0.33 - 0.66 - - cmp rsi, 12
+ - - 0.34 - - 0.66 - - setae cl
+ - - 1.00 - - - - - test al, cl
+ - - - - - 1.00 - - jne .LBB5_2
- - - - - - - - xor edx, edx
- - - 0.01 0.98 - 0.01 - - mov eax, 0
- - - 0.98 0.01 - 0.01 - - test dil, 3
- - - - - - 1.00 - - je .LBB5_1
- - - - - - 1.00 - - ret
- - - 0.01 0.99 - - - - movabs rax, 9223372036854775804
- - - - 1.00 - - - - and rsi, rax
- - - - 1.00 - - - - cmp rsi, 9
- - - - - - 1.00 - - jae .LBB5_3
- - - 1.00 - - - - - mov edx, 1
- - - - - - - - - xor eax, eax
+ - - - 0.99 - 0.01 - - test dil, 3
+ - - - - - 1.00 - - sete dl
+ - - - - - - - - xor edi, edi
+ - - - 1.00 - - - - mov rax, rdi
- - - - - 1.00 - - ret
- - - 0.02 0.02 - 0.96 - - add rsi, -9
- - - 0.99 0.01 - - - - movabs rcx, -6148914691236517205
- - - 0.01 0.99 - - - - mov rax, rsi
+ - - 0.01 0.99 - - - - or rsi, 3
+ - - 0.33 0.67 - - - - add rsi, -12
+ - - 0.66 0.34 - - - - movabs rcx, -6148914691236517205
+ - - 0.34 0.33 - 0.33 - - mov rax, rsi
- - 1.00 1.00 - - - - mul rcx
- - 1.00 - - - - - shr rdx
- - - 0.98 - - 0.02 - - mov rax, rdi
+ - - - 0.34 - 0.66 - - mov rax, rdi
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64
index fe6332c9100c..3f2e87467897 100644
--- a/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64
+++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64
@@ -1,17 +1,10 @@
bench_ref_from_prefix_dynamic_size:
- xor edx, edx
- mov eax, 0
- test dil, 1
- jne .LBB5_4
- cmp rsi, 4
- jae .LBB5_3
- mov edx, 1
- xor eax, eax
- ret
-.LBB5_3:
- add rsi, -4
- shr rsi
mov rdx, rsi
- mov rax, rdi
-.LBB5_4:
+ xor eax, eax
+ sub rdx, 4
+ mov rcx, rdi
+ cmovb rcx, rax
+ shr rdx
+ test dil, 1
+ cmove rax, rcx
ret
diff --git a/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64.mca
index 3900a5946138..7ee51e0b8a40 100644
--- a/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64.mca
+++ b/rust/zerocopy/benches/ref_from_prefix_dynamic_size.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 1400
-Total Cycles: 405
-Total uOps: 1400
+Instructions: 900
+Total Cycles: 339
+Total uOps: 1100
Dispatch Width: 4
-uOps Per Cycle: 3.46
-IPC: 3.46
-Block RThroughput: 4.0
+uOps Per Cycle: 3.24
+IPC: 2.65
+Block RThroughput: 2.8
Instruction Info:
@@ -18,19 +18,14 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
- 1 0 0.25 xor edx, edx
- 1 1 0.33 mov eax, 0
- 1 1 0.33 test dil, 1
- 1 1 1.00 jne .LBB5_4
- 1 1 0.33 cmp rsi, 4
- 1 1 1.00 jae .LBB5_3
- 1 1 0.33 mov edx, 1
- 1 0 0.25 xor eax, eax
- 1 1 1.00 U ret
- 1 1 0.33 add rsi, -4
- 1 1 0.50 shr rsi
1 1 0.33 mov rdx, rsi
- 1 1 0.33 mov rax, rdi
+ 1 0 0.25 xor eax, eax
+ 1 1 0.33 sub rdx, 4
+ 1 1 0.33 mov rcx, rdi
+ 2 2 0.67 cmovb rcx, rax
+ 1 1 0.50 shr rdx
+ 1 1 0.33 test dil, 1
+ 2 2 0.67 cmove rax, rcx
1 1 1.00 U ret
@@ -47,21 +42,16 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 3.99 3.99 - 4.02 - -
+ - - 3.33 3.33 - 3.34 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - - - - - - - xor edx, edx
- - - 0.01 0.98 - 0.01 - - mov eax, 0
- - - 0.98 0.02 - - - - test dil, 1
- - - - - - 1.00 - - jne .LBB5_4
- - - 0.02 0.98 - - - - cmp rsi, 4
- - - - - - 1.00 - - jae .LBB5_3
- - - 0.98 0.01 - 0.01 - - mov edx, 1
+ - - 0.33 0.66 - 0.01 - - mov rdx, rsi
- - - - - - - - xor eax, eax
- - - - - - 1.00 - - ret
- - - 0.01 0.99 - - - - add rsi, -4
- - - 1.00 - - - - - shr rsi
- - - - 1.00 - - - - mov rdx, rsi
- - - 0.99 0.01 - - - - mov rax, rdi
+ - - 0.01 - - 0.99 - - sub rdx, 4
+ - - 0.66 0.34 - - - - mov rcx, rdi
+ - - 1.00 1.00 - - - - cmovb rcx, rax
+ - - 0.33 - - 0.67 - - shr rdx
+ - - - 0.33 - 0.67 - - test dil, 1
+ - - 1.00 1.00 - - - - cmove rax, rcx
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64
index 3e05f6023f38..68f21fe8018a 100644
--- a/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64
+++ b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64
@@ -1,23 +1,22 @@
bench_ref_from_suffix_dynamic_padding:
lea eax, [rsi + rdi]
test al, 3
- jne .LBB5_1
- movabs rax, 9223372036854775804
- and rax, rsi
- cmp rax, 9
- jae .LBB5_3
-.LBB5_1:
- xor eax, eax
+ setne al
+ cmp rsi, 12
+ setb cl
+ or cl, al
+ je .LBB5_2
+ xor edi, edi
+ mov rax, rdi
ret
-.LBB5_3:
- add rax, -9
+.LBB5_2:
+ mov rax, rsi
+ or rax, 3
+ add rax, -12
movabs rcx, -6148914691236517205
mul rcx
shr rdx
- lea rax, [rdx + 2*rdx]
- sub rsi, rax
- or rax, -4
- add rsi, rdi
- add rax, rsi
- add rax, -8
+ and esi, 3
+ add rdi, rsi
+ mov rax, rdi
ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64.mca
index 73599d5b6aab..c31e098aed4f 100644
--- a/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64.mca
+++ b/rust/zerocopy/benches/ref_from_suffix_dynamic_padding.x86-64.mca
@@ -1,11 +1,11 @@
Iterations: 100
Instructions: 2000
-Total Cycles: 682
+Total Cycles: 673
Total uOps: 2100
Dispatch Width: 4
-uOps Per Cycle: 3.08
-IPC: 2.93
+uOps Per Cycle: 3.12
+IPC: 2.97
Block RThroughput: 5.3
@@ -20,23 +20,23 @@ Instruction Info:
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.50 lea eax, [rsi + rdi]
1 1 0.33 test al, 3
- 1 1 1.00 jne .LBB5_1
- 1 1 0.33 movabs rax, 9223372036854775804
- 1 1 0.33 and rax, rsi
- 1 1 0.33 cmp rax, 9
- 1 1 1.00 jae .LBB5_3
- 1 0 0.25 xor eax, eax
+ 1 1 0.50 setne al
+ 1 1 0.33 cmp rsi, 12
+ 1 1 0.50 setb cl
+ 1 1 0.33 or cl, al
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor edi, edi
+ 1 1 0.33 mov rax, rdi
1 1 1.00 U ret
- 1 1 0.33 add rax, -9
+ 1 1 0.33 mov rax, rsi
+ 1 1 0.33 or rax, 3
+ 1 1 0.33 add rax, -12
1 1 0.33 movabs rcx, -6148914691236517205
2 4 1.00 mul rcx
1 1 0.50 shr rdx
- 1 1 0.50 lea rax, [rdx + 2*rdx]
- 1 1 0.33 sub rsi, rax
- 1 1 0.33 or rax, -4
- 1 1 0.33 add rsi, rdi
- 1 1 0.33 add rax, rsi
- 1 1 0.33 add rax, -8
+ 1 1 0.33 and esi, 3
+ 1 1 0.33 add rdi, rsi
+ 1 1 0.33 mov rax, rdi
1 1 1.00 U ret
@@ -53,27 +53,27 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 6.65 6.67 - 6.68 - -
+ - - 6.67 6.66 - 6.67 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.90 0.10 - - - - lea eax, [rsi + rdi]
- - - 0.93 - - 0.07 - - test al, 3
- - - - - - 1.00 - - jne .LBB5_1
- - - 0.51 0.47 - 0.02 - - movabs rax, 9223372036854775804
- - - - - - 1.00 - - and rax, rsi
- - - - 0.09 - 0.91 - - cmp rax, 9
- - - - - - 1.00 - - jae .LBB5_3
- - - - - - - - - xor eax, eax
+ - - 0.35 0.65 - - - - lea eax, [rsi + rdi]
+ - - 0.01 0.99 - - - - test al, 3
+ - - 1.00 - - - - - setne al
+ - - 0.97 - - 0.03 - - cmp rsi, 12
+ - - 0.66 - - 0.34 - - setb cl
+ - - - 1.00 - - - - or cl, al
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor edi, edi
+ - - 0.01 0.01 - 0.98 - - mov rax, rdi
- - - - - 1.00 - - ret
- - - 0.43 0.47 - 0.10 - - add rax, -9
- - - 0.42 0.39 - 0.19 - - movabs rcx, -6148914691236517205
+ - - 0.01 0.99 - - - - mov rax, rsi
+ - - - 0.34 - 0.66 - - or rax, 3
+ - - - 1.00 - - - - add rax, -12
+ - - 0.66 0.31 - 0.03 - - movabs rcx, -6148914691236517205
- - 1.00 1.00 - - - - mul rcx
- - - 0.69 - - 0.31 - - shr rdx
- - - 0.54 0.46 - - - - lea rax, [rdx + 2*rdx]
- - - 0.07 0.91 - 0.02 - - sub rsi, rax
- - - 0.91 0.05 - 0.04 - - or rax, -4
- - - 0.08 0.90 - 0.02 - - add rsi, rdi
- - - 0.09 0.91 - - - - add rax, rsi
- - - 0.08 0.92 - - - - add rax, -8
+ - - 1.00 - - - - - shr rdx
+ - - 0.67 0.01 - 0.32 - - and esi, 3
+ - - 0.32 0.01 - 0.67 - - add rdi, rsi
+ - - 0.01 0.35 - 0.64 - - mov rax, rdi
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64
index c3d10b5fc685..7d11849dd5c5 100644
--- a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64
+++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64
@@ -1,27 +1,21 @@
bench_ref_from_suffix_with_elems_dynamic_padding:
- movabs rax, 3074457345618258598
- cmp rdx, rax
- ja .LBB5_1
- lea r8d, [rsi + rdi]
- xor ecx, ecx
- mov eax, 0
- test r8b, 3
- je .LBB5_3
- mov rdx, rcx
- ret
+ movabs rcx, 3074457345618258598
+ cmp rdx, rcx
+ ja .LBB5_3
+ mov rax, rdi
+ lea rcx, [rdx + 2*rdx]
+ or rcx, 3
+ add rcx, 9
+ add edi, esi
+ test dil, 3
+ setne dil
+ sub rsi, rcx
+ setb cl
+ or cl, dil
+ je .LBB5_4
.LBB5_3:
- lea rax, [rdx + 2*rdx]
- or rax, 3
- add rax, 9
- sub rsi, rax
- jae .LBB5_4
-.LBB5_1:
xor eax, eax
- mov edx, 1
ret
.LBB5_4:
- add rdi, rsi
- mov rcx, rdx
- mov rax, rdi
- mov rdx, rcx
+ add rax, rsi
ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64.mca
index 92e6280bb4cc..d7850d93bc10 100644
--- a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64.mca
+++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_padding.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 2300
-Total Cycles: 706
-Total uOps: 2300
+Instructions: 1800
+Total Cycles: 572
+Total uOps: 1800
Dispatch Width: 4
-uOps Per Cycle: 3.26
-IPC: 3.26
-Block RThroughput: 6.0
+uOps Per Cycle: 3.15
+IPC: 3.15
+Block RThroughput: 4.5
Instruction Info:
@@ -18,28 +18,23 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
- 1 1 0.33 movabs rax, 3074457345618258598
- 1 1 0.33 cmp rdx, rax
- 1 1 1.00 ja .LBB5_1
- 1 1 0.50 lea r8d, [rsi + rdi]
- 1 0 0.25 xor ecx, ecx
- 1 1 0.33 mov eax, 0
- 1 1 0.33 test r8b, 3
- 1 1 1.00 je .LBB5_3
- 1 1 0.33 mov rdx, rcx
- 1 1 1.00 U ret
- 1 1 0.50 lea rax, [rdx + 2*rdx]
- 1 1 0.33 or rax, 3
- 1 1 0.33 add rax, 9
- 1 1 0.33 sub rsi, rax
- 1 1 1.00 jae .LBB5_4
+ 1 1 0.33 movabs rcx, 3074457345618258598
+ 1 1 0.33 cmp rdx, rcx
+ 1 1 1.00 ja .LBB5_3
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.50 lea rcx, [rdx + 2*rdx]
+ 1 1 0.33 or rcx, 3
+ 1 1 0.33 add rcx, 9
+ 1 1 0.33 add edi, esi
+ 1 1 0.33 test dil, 3
+ 1 1 0.50 setne dil
+ 1 1 0.33 sub rsi, rcx
+ 1 1 0.50 setb cl
+ 1 1 0.33 or cl, dil
+ 1 1 1.00 je .LBB5_4
1 0 0.25 xor eax, eax
- 1 1 0.33 mov edx, 1
1 1 1.00 U ret
- 1 1 0.33 add rdi, rsi
- 1 1 0.33 mov rcx, rdx
- 1 1 0.33 mov rax, rdi
- 1 1 0.33 mov rdx, rcx
+ 1 1 0.33 add rax, rsi
1 1 1.00 U ret
@@ -56,30 +51,25 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 6.99 7.00 - 7.01 - -
+ - - 5.66 5.66 - 5.68 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - - 0.99 - 0.01 - - movabs rax, 3074457345618258598
- - - 0.01 0.50 - 0.49 - - cmp rdx, rax
- - - - - - 1.00 - - ja .LBB5_1
- - - - 1.00 - - - - lea r8d, [rsi + rdi]
- - - - - - - - - xor ecx, ecx
- - - 0.50 0.49 - 0.01 - - mov eax, 0
- - - 0.49 0.51 - - - - test r8b, 3
- - - - - - 1.00 - - je .LBB5_3
- - - 0.51 0.49 - - - - mov rdx, rcx
- - - - - - 1.00 - - ret
- - - 0.50 0.50 - - - - lea rax, [rdx + 2*rdx]
- - - 1.00 - - - - - or rax, 3
- - - 1.00 - - - - - add rax, 9
- - - 0.99 0.01 - - - - sub rsi, rax
- - - - - - 1.00 - - jae .LBB5_4
+ - - 0.67 - - 0.33 - - movabs rcx, 3074457345618258598
+ - - 0.34 - - 0.66 - - cmp rdx, rcx
+ - - - - - 1.00 - - ja .LBB5_3
+ - - 0.33 0.01 - 0.66 - - mov rax, rdi
+ - - 0.33 0.67 - - - - lea rcx, [rdx + 2*rdx]
+ - - 0.01 0.99 - - - - or rcx, 3
+ - - 0.01 0.99 - - - - add rcx, 9
+ - - 0.99 - - 0.01 - - add edi, esi
+ - - 0.99 0.01 - - - - test dil, 3
+ - - 0.99 - - 0.01 - - setne dil
+ - - - 1.00 - - - - sub rsi, rcx
+ - - 1.00 - - - - - setb cl
+ - - - 0.99 - 0.01 - - or cl, dil
+ - - - - - 1.00 - - je .LBB5_4
- - - - - - - - xor eax, eax
- - - - 1.00 - - - - mov edx, 1
- - - - - 1.00 - - ret
- - - 1.00 - - - - - add rdi, rsi
- - - - 1.00 - - - - mov rcx, rdx
- - - 0.99 0.01 - - - - mov rax, rdi
- - - - 0.50 - 0.50 - - mov rdx, rcx
+ - - - 1.00 - - - - add rax, rsi
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64
index bdca57192455..7ee0278dbf68 100644
--- a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64
+++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64
@@ -1,23 +1,18 @@
bench_ref_from_suffix_with_elems_dynamic_size:
- movabs rax, 4611686018427387901
- cmp rdx, rax
- ja .LBB5_1
- lea r8d, [rsi + rdi]
- xor ecx, ecx
- mov eax, 0
- test r8b, 1
- jne .LBB5_5
- lea rax, [2*rdx + 4]
- sub rsi, rax
- jae .LBB5_4
-.LBB5_1:
+ movabs rcx, 4611686018427387901
+ cmp rdx, rcx
+ ja .LBB5_3
+ mov rax, rdi
+ lea rcx, [2*rdx + 4]
+ add edi, esi
+ sub rsi, rcx
+ setb cl
+ or cl, dil
+ test cl, 1
+ je .LBB5_4
+.LBB5_3:
xor eax, eax
- mov edx, 1
ret
.LBB5_4:
- add rdi, rsi
- mov rcx, rdx
- mov rax, rdi
-.LBB5_5:
- mov rdx, rcx
+ add rax, rsi
ret
diff --git a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64.mca
index 6d9de0b3eb5c..882316de3d27 100644
--- a/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64.mca
+++ b/rust/zerocopy/benches/ref_from_suffix_with_elems_dynamic_size.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 1900
-Total Cycles: 571
-Total uOps: 1900
+Instructions: 1500
+Total Cycles: 472
+Total uOps: 1500
Dispatch Width: 4
-uOps Per Cycle: 3.33
-IPC: 3.33
-Block RThroughput: 5.0
+uOps Per Cycle: 3.18
+IPC: 3.18
+Block RThroughput: 4.0
Instruction Info:
@@ -18,24 +18,20 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
- 1 1 0.33 movabs rax, 4611686018427387901
- 1 1 0.33 cmp rdx, rax
- 1 1 1.00 ja .LBB5_1
- 1 1 0.50 lea r8d, [rsi + rdi]
- 1 0 0.25 xor ecx, ecx
- 1 1 0.33 mov eax, 0
- 1 1 0.33 test r8b, 1
- 1 1 1.00 jne .LBB5_5
- 1 1 0.50 lea rax, [2*rdx + 4]
- 1 1 0.33 sub rsi, rax
- 1 1 1.00 jae .LBB5_4
+ 1 1 0.33 movabs rcx, 4611686018427387901
+ 1 1 0.33 cmp rdx, rcx
+ 1 1 1.00 ja .LBB5_3
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.50 lea rcx, [2*rdx + 4]
+ 1 1 0.33 add edi, esi
+ 1 1 0.33 sub rsi, rcx
+ 1 1 0.50 setb cl
+ 1 1 0.33 or cl, dil
+ 1 1 0.33 test cl, 1
+ 1 1 1.00 je .LBB5_4
1 0 0.25 xor eax, eax
- 1 1 0.33 mov edx, 1
1 1 1.00 U ret
- 1 1 0.33 add rdi, rsi
- 1 1 0.33 mov rcx, rdx
- 1 1 0.33 mov rax, rdi
- 1 1 0.33 mov rdx, rcx
+ 1 1 0.33 add rax, rsi
1 1 1.00 U ret
@@ -52,26 +48,22 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 5.66 5.66 - 5.68 - -
+ - - 4.66 4.66 - 4.68 - -
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.66 0.33 - 0.01 - - movabs rax, 4611686018427387901
- - - 0.01 0.99 - - - - cmp rdx, rax
- - - - - - 1.00 - - ja .LBB5_1
- - - 0.99 0.01 - - - - lea r8d, [rsi + rdi]
- - - - - - - - - xor ecx, ecx
- - - 0.33 0.33 - 0.34 - - mov eax, 0
- - - 0.33 0.34 - 0.33 - - test r8b, 1
- - - - - - 1.00 - - jne .LBB5_5
- - - 0.34 0.66 - - - - lea rax, [2*rdx + 4]
- - - - 1.00 - - - - sub rsi, rax
- - - - - - 1.00 - - jae .LBB5_4
+ - - 0.99 - - 0.01 - - movabs rcx, 4611686018427387901
+ - - 1.00 - - - - - cmp rdx, rcx
+ - - - - - 1.00 - - ja .LBB5_3
+ - - 0.63 0.02 - 0.35 - - mov rax, rdi
+ - - 0.35 0.65 - - - - lea rcx, [2*rdx + 4]
+ - - 0.65 0.03 - 0.32 - - add edi, esi
+ - - 0.03 0.97 - - - - sub rsi, rcx
+ - - 1.00 - - - - - setb cl
+ - - - 1.00 - - - - or cl, dil
+ - - 0.01 0.99 - - - - test cl, 1
+ - - - - - 1.00 - - je .LBB5_4
- - - - - - - - xor eax, eax
- - - 1.00 - - - - - mov edx, 1
- - - - - 1.00 - - ret
- - - - 1.00 - - - - add rdi, rsi
- - - 1.00 - - - - - mov rcx, rdx
- - - 0.32 0.68 - - - - mov rax, rdi
- - - 0.68 0.32 - - - - mov rdx, rcx
+ - - - 1.00 - - - - add rax, rsi
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.rs
new file mode 100644
index 000000000000..8d4e26394596
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_bytes_dynamic_padding<'a>(
+ source: &'a Ref<&[u8], format::LocoPacket>,
+) -> &'a [u8] {
+ Ref::bytes(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64
new file mode 100644
index 000000000000..a44c1510fe02
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64
@@ -0,0 +1,4 @@
+bench_ref_type_bytes_dynamic_padding:
+ mov rax, qword ptr [rdi]
+ mov rdx, qword ptr [rdi + 8]
+ ret
diff --git a/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64.mca
new file mode 100644
index 000000000000..a7f8947c0413
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_dynamic_padding.x86-64.mca
@@ -0,0 +1,45 @@
+Iterations: 100
+Instructions: 300
+Total Cycles: 107
+Total uOps: 300
+
+Dispatch Width: 4
+uOps Per Cycle: 2.80
+IPC: 2.80
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov rdx, qword ptr [rdi + 8]
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - - - - 1.00 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - 1.00 mov rax, qword ptr [rdi]
+ - - - - - - 1.00 - mov rdx, qword ptr [rdi + 8]
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_bytes_dynamic_size.rs b/rust/zerocopy/benches/ref_type_bytes_dynamic_size.rs
new file mode 100644
index 000000000000..ef7dcf83989d
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_dynamic_size.rs
@@ -0,0 +1,17 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_bytes_dynamic_size<'a>(source: &'a Ref<&[u8], format::LocoPacket>) -> &'a [u8] {
+ Ref::bytes(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64
new file mode 100644
index 000000000000..6ac3ad2f1fef
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64
@@ -0,0 +1,4 @@
+bench_ref_type_bytes_dynamic_size:
+ mov rax, qword ptr [rdi]
+ mov rdx, qword ptr [rdi + 8]
+ ret
diff --git a/rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64.mca
new file mode 100644
index 000000000000..a7f8947c0413
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_dynamic_size.x86-64.mca
@@ -0,0 +1,45 @@
+Iterations: 100
+Instructions: 300
+Total Cycles: 107
+Total uOps: 300
+
+Dispatch Width: 4
+uOps Per Cycle: 2.80
+IPC: 2.80
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov rdx, qword ptr [rdi + 8]
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - - - - 1.00 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - 1.00 mov rax, qword ptr [rdi]
+ - - - - - - 1.00 - mov rdx, qword ptr [rdi + 8]
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.rs
new file mode 100644
index 000000000000..e99b6704bd9b
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_bytes_mut_dynamic_padding<'a>(
+ source: &'a mut Ref<&mut [u8], format::LocoPacket>,
+) -> &'a mut [u8] {
+ Ref::bytes_mut(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64
new file mode 100644
index 000000000000..ebc6ee04941e
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64
@@ -0,0 +1,4 @@
+bench_ref_type_bytes_mut_dynamic_padding:
+ mov rax, qword ptr [rdi]
+ mov rdx, qword ptr [rdi + 8]
+ ret
diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64.mca
new file mode 100644
index 000000000000..a7f8947c0413
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_padding.x86-64.mca
@@ -0,0 +1,45 @@
+Iterations: 100
+Instructions: 300
+Total Cycles: 107
+Total uOps: 300
+
+Dispatch Width: 4
+uOps Per Cycle: 2.80
+IPC: 2.80
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov rdx, qword ptr [rdi + 8]
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - - - - 1.00 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - 1.00 mov rax, qword ptr [rdi]
+ - - - - - - 1.00 - mov rdx, qword ptr [rdi + 8]
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.rs b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.rs
new file mode 100644
index 000000000000..badf9de19f06
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_bytes_mut_dynamic_size<'a>(
+ source: &'a mut Ref<&mut [u8], format::LocoPacket>,
+) -> &'a mut [u8] {
+ Ref::bytes_mut(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64
new file mode 100644
index 000000000000..5d97d01e51e2
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64
@@ -0,0 +1,4 @@
+bench_ref_type_bytes_mut_dynamic_size:
+ mov rax, qword ptr [rdi]
+ mov rdx, qword ptr [rdi + 8]
+ ret
diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64.mca
new file mode 100644
index 000000000000..a7f8947c0413
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_mut_dynamic_size.x86-64.mca
@@ -0,0 +1,45 @@
+Iterations: 100
+Instructions: 300
+Total Cycles: 107
+Total uOps: 300
+
+Dispatch Width: 4
+uOps Per Cycle: 2.80
+IPC: 2.80
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov rdx, qword ptr [rdi + 8]
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - - - - 1.00 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - 1.00 mov rax, qword ptr [rdi]
+ - - - - - - 1.00 - mov rdx, qword ptr [rdi + 8]
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_static_size.rs b/rust/zerocopy/benches/ref_type_bytes_mut_static_size.rs
new file mode 100644
index 000000000000..1a49c2492a41
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_mut_static_size.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_bytes_mut_static_size<'a>(
+ source: &'a mut Ref<&mut [u8], format::LocoPacket>,
+) -> &'a mut [u8] {
+ Ref::bytes_mut(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64 b/rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64
new file mode 100644
index 000000000000..3411bd45e96a
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64
@@ -0,0 +1,4 @@
+bench_ref_type_bytes_mut_static_size:
+ mov rax, qword ptr [rdi]
+ mov rdx, qword ptr [rdi + 8]
+ ret
diff --git a/rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64.mca
new file mode 100644
index 000000000000..a7f8947c0413
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_mut_static_size.x86-64.mca
@@ -0,0 +1,45 @@
+Iterations: 100
+Instructions: 300
+Total Cycles: 107
+Total uOps: 300
+
+Dispatch Width: 4
+uOps Per Cycle: 2.80
+IPC: 2.80
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov rdx, qword ptr [rdi + 8]
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - - - - 1.00 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - 1.00 mov rax, qword ptr [rdi]
+ - - - - - - 1.00 - mov rdx, qword ptr [rdi + 8]
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_bytes_static_size.rs b/rust/zerocopy/benches/ref_type_bytes_static_size.rs
new file mode 100644
index 000000000000..b9c3fe5b7bb5
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_static_size.rs
@@ -0,0 +1,17 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_bytes_static_size<'a>(source: &'a Ref<&[u8], format::LocoPacket>) -> &'a [u8] {
+ Ref::bytes(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_bytes_static_size.x86-64 b/rust/zerocopy/benches/ref_type_bytes_static_size.x86-64
new file mode 100644
index 000000000000..10a024acaafd
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_static_size.x86-64
@@ -0,0 +1,4 @@
+bench_ref_type_bytes_static_size:
+ mov rax, qword ptr [rdi]
+ mov rdx, qword ptr [rdi + 8]
+ ret
diff --git a/rust/zerocopy/benches/ref_type_bytes_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_bytes_static_size.x86-64.mca
new file mode 100644
index 000000000000..a7f8947c0413
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_bytes_static_size.x86-64.mca
@@ -0,0 +1,45 @@
+Iterations: 100
+Instructions: 300
+Total Cycles: 107
+Total uOps: 300
+
+Dispatch Width: 4
+uOps Per Cycle: 2.80
+IPC: 2.80
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov rdx, qword ptr [rdi + 8]
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - - - - 1.00 1.00 1.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - - 1.00 mov rax, qword ptr [rdi]
+ - - - - - - 1.00 - mov rdx, qword ptr [rdi + 8]
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_deref_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_deref_dynamic_padding.rs
new file mode 100644
index 000000000000..c0ebe56ccc75
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_dynamic_padding.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_deref_dynamic_padding<'a>(
+ source: &'a Ref<&[u8], format::LocoPacket>,
+) -> &'a format::LocoPacket {
+ core::ops::Deref::deref(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64
new file mode 100644
index 000000000000..765c5305d86d
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64
@@ -0,0 +1,29 @@
+bench_ref_type_deref_dynamic_padding:
+ mov rdx, qword ptr [rdi + 16]
+ mov rax, qword ptr [rdi]
+ mov rcx, qword ptr [rdi + 8]
+ mov esi, 1
+ movabs rdi, 3074457345618258599
+ cmp rdx, rdi
+ jae .LBB8_3
+ test al, 3
+ jne .LBB8_2
+ lea rdi, [rdx + 2*rdx]
+ or rdi, 3
+ add rdi, 9
+ cmp rcx, rdi
+ jne .LBB8_3
+ ret
+.LBB8_2:
+ xor esi, esi
+.LBB8_3:
+ sub rsp, 24
+ mov qword ptr [rsp], rsi
+ mov qword ptr [rsp + 8], rax
+ mov qword ptr [rsp + 16], rcx
+ lea rdi, [rip + .Lanon.HASH.7]
+ lea rcx, [rip + .Lanon.HASH.1]
+ lea r8, [rip + .Lanon.HASH.9]
+ mov rdx, rsp
+ mov esi, 58
+ call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
diff --git a/rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64.mca
new file mode 100644
index 000000000000..bea2c8a20ea7
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_dynamic_padding.x86-64.mca
@@ -0,0 +1,91 @@
+Iterations: 100
+Instructions: 2600
+Total Cycles: 1812
+Total uOps: 2900
+
+Dispatch Width: 4
+uOps Per Cycle: 1.60
+IPC: 1.43
+Block RThroughput: 7.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rdx, qword ptr [rdi + 16]
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov rcx, qword ptr [rdi + 8]
+ 1 1 0.33 mov esi, 1
+ 1 1 0.33 movabs rdi, 3074457345618258599
+ 1 1 0.33 cmp rdx, rdi
+ 1 1 1.00 jae .LBB8_3
+ 1 1 0.33 test al, 3
+ 1 1 1.00 jne .LBB8_2
+ 1 1 0.50 lea rdi, [rdx + 2*rdx]
+ 1 1 0.33 or rdi, 3
+ 1 1 0.33 add rdi, 9
+ 1 1 0.33 cmp rcx, rdi
+ 1 1 1.00 jne .LBB8_3
+ 1 1 1.00 U ret
+ 1 0 0.25 xor esi, esi
+ 1 1 0.33 sub rsp, 24
+ 1 1 1.00 * mov qword ptr [rsp], rsi
+ 1 1 1.00 * mov qword ptr [rsp + 8], rax
+ 1 1 1.00 * mov qword ptr [rsp + 16], rcx
+ 1 1 0.50 lea rdi, [rip + .Lanon.HASH.7]
+ 1 1 0.50 lea rcx, [rip + .Lanon.HASH.1]
+ 1 1 0.50 lea r8, [rip + .Lanon.HASH.9]
+ 1 1 0.33 mov rdx, rsp
+ 1 1 0.33 mov esi, 58
+ 4 7 1.00 * call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.41 6.42 4.00 6.17 3.99 4.01
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - 0.54 0.46 mov rdx, qword ptr [rdi + 16]
+ - - - - - - 0.50 0.50 mov rax, qword ptr [rdi]
+ - - - - - - 0.49 0.51 mov rcx, qword ptr [rdi + 8]
+ - - 0.52 0.47 - 0.01 - - mov esi, 1
+ - - 0.47 0.53 - - - - movabs rdi, 3074457345618258599
+ - - 0.47 0.53 - - - - cmp rdx, rdi
+ - - - - - 1.00 - - jae .LBB8_3
+ - - 0.54 0.44 - 0.02 - - test al, 3
+ - - - - - 1.00 - - jne .LBB8_2
+ - - 0.44 0.56 - - - - lea rdi, [rdx + 2*rdx]
+ - - 0.48 0.52 - - - - or rdi, 3
+ - - 0.48 0.48 - 0.04 - - add rdi, 9
+ - - 0.51 0.39 - 0.10 - - cmp rcx, rdi
+ - - - - - 1.00 - - jne .LBB8_3
+ - - - - - 1.00 - - ret
+ - - - - - - - - xor esi, esi
+ - - 0.53 0.47 - - - - sub rsp, 24
+ - - - - 1.00 - 0.52 0.48 mov qword ptr [rsp], rsi
+ - - - - 1.00 - 0.48 0.52 mov qword ptr [rsp + 8], rax
+ - - - - 1.00 - 0.52 0.48 mov qword ptr [rsp + 16], rcx
+ - - 0.49 0.51 - - - - lea rdi, [rip + .Lanon.HASH.7]
+ - - 0.46 0.54 - - - - lea rcx, [rip + .Lanon.HASH.1]
+ - - 0.54 0.46 - - - - lea r8, [rip + .Lanon.HASH.9]
+ - - 0.04 0.06 - 0.90 - - mov rdx, rsp
+ - - 0.44 0.46 - 0.10 - - mov esi, 58
+ - - - - 1.00 1.00 0.94 1.06 call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
diff --git a/rust/zerocopy/benches/ref_type_deref_dynamic_size.rs b/rust/zerocopy/benches/ref_type_deref_dynamic_size.rs
new file mode 100644
index 000000000000..adcf89c22508
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_dynamic_size.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_deref_dynamic_size<'a>(
+ source: &'a Ref<&[u8], format::LocoPacket>,
+) -> &'a format::LocoPacket {
+ core::ops::Deref::deref(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64
new file mode 100644
index 000000000000..60ca3b508a7e
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64
@@ -0,0 +1,27 @@
+bench_ref_type_deref_dynamic_size:
+ mov rdx, qword ptr [rdi + 16]
+ mov rax, qword ptr [rdi]
+ mov rcx, qword ptr [rdi + 8]
+ mov esi, 1
+ movabs rdi, 4611686018427387902
+ cmp rdx, rdi
+ jae .LBB8_3
+ test al, 1
+ jne .LBB8_2
+ lea rdi, [2*rdx + 4]
+ cmp rcx, rdi
+ jne .LBB8_3
+ ret
+.LBB8_2:
+ xor esi, esi
+.LBB8_3:
+ sub rsp, 24
+ mov qword ptr [rsp], rsi
+ mov qword ptr [rsp + 8], rax
+ mov qword ptr [rsp + 16], rcx
+ lea rdi, [rip + .Lanon.HASH.7]
+ lea rcx, [rip + .Lanon.HASH.1]
+ lea r8, [rip + .Lanon.HASH.9]
+ mov rdx, rsp
+ mov esi, 58
+ call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
diff --git a/rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64.mca
new file mode 100644
index 000000000000..66a8f8575acd
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_dynamic_size.x86-64.mca
@@ -0,0 +1,87 @@
+Iterations: 100
+Instructions: 2400
+Total Cycles: 1647
+Total uOps: 2700
+
+Dispatch Width: 4
+uOps Per Cycle: 1.64
+IPC: 1.46
+Block RThroughput: 6.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rdx, qword ptr [rdi + 16]
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov rcx, qword ptr [rdi + 8]
+ 1 1 0.33 mov esi, 1
+ 1 1 0.33 movabs rdi, 4611686018427387902
+ 1 1 0.33 cmp rdx, rdi
+ 1 1 1.00 jae .LBB8_3
+ 1 1 0.33 test al, 1
+ 1 1 1.00 jne .LBB8_2
+ 1 1 0.50 lea rdi, [2*rdx + 4]
+ 1 1 0.33 cmp rcx, rdi
+ 1 1 1.00 jne .LBB8_3
+ 1 1 1.00 U ret
+ 1 0 0.25 xor esi, esi
+ 1 1 0.33 sub rsp, 24
+ 1 1 1.00 * mov qword ptr [rsp], rsi
+ 1 1 1.00 * mov qword ptr [rsp + 8], rax
+ 1 1 1.00 * mov qword ptr [rsp + 16], rcx
+ 1 1 0.50 lea rdi, [rip + .Lanon.HASH.7]
+ 1 1 0.50 lea rcx, [rip + .Lanon.HASH.1]
+ 1 1 0.50 lea r8, [rip + .Lanon.HASH.9]
+ 1 1 0.33 mov rdx, rsp
+ 1 1 0.33 mov esi, 58
+ 4 7 1.00 * call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 5.15 5.74 4.00 6.11 3.98 4.02
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - 0.71 0.29 mov rdx, qword ptr [rdi + 16]
+ - - - - - - 0.72 0.28 mov rax, qword ptr [rdi]
+ - - - - - - 0.29 0.71 mov rcx, qword ptr [rdi + 8]
+ - - 0.83 0.16 - 0.01 - - mov esi, 1
+ - - 0.16 0.84 - - - - movabs rdi, 4611686018427387902
+ - - 0.80 0.04 - 0.16 - - cmp rdx, rdi
+ - - - - - 1.00 - - jae .LBB8_3
+ - - 0.13 0.74 - 0.13 - - test al, 1
+ - - - - - 1.00 - - jne .LBB8_2
+ - - 0.10 0.90 - - - - lea rdi, [2*rdx + 4]
+ - - 0.09 0.91 - - - - cmp rcx, rdi
+ - - - - - 1.00 - - jne .LBB8_3
+ - - - - - 1.00 - - ret
+ - - - - - - - - xor esi, esi
+ - - 0.29 0.13 - 0.58 - - sub rsp, 24
+ - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp], rsi
+ - - - - 1.00 - 0.28 0.72 mov qword ptr [rsp + 8], rax
+ - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp + 16], rcx
+ - - 0.76 0.24 - - - - lea rdi, [rip + .Lanon.HASH.7]
+ - - 0.82 0.18 - - - - lea rcx, [rip + .Lanon.HASH.1]
+ - - 0.92 0.08 - - - - lea r8, [rip + .Lanon.HASH.9]
+ - - 0.09 0.68 - 0.23 - - mov rdx, rsp
+ - - 0.16 0.84 - - - - mov esi, 58
+ - - - - 1.00 1.00 0.54 1.46 call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
diff --git a/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.rs b/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.rs
new file mode 100644
index 000000000000..263f2c575379
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_deref_mut_dynamic_size<'a>(
+ source: &'a mut Ref<&mut [u8], format::LocoPacket>,
+) -> &'a mut format::LocoPacket {
+ core::ops::DerefMut::deref_mut(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64
new file mode 100644
index 000000000000..5ae0ae69efad
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64
@@ -0,0 +1,27 @@
+bench_ref_type_deref_mut_dynamic_size:
+ mov rdx, qword ptr [rdi + 16]
+ mov rax, qword ptr [rdi]
+ mov rcx, qword ptr [rdi + 8]
+ mov esi, 1
+ movabs rdi, 4611686018427387902
+ cmp rdx, rdi
+ jae .LBB8_3
+ test al, 1
+ jne .LBB8_2
+ lea rdi, [2*rdx + 4]
+ cmp rcx, rdi
+ jne .LBB8_3
+ ret
+.LBB8_2:
+ xor esi, esi
+.LBB8_3:
+ sub rsp, 24
+ mov qword ptr [rsp], rsi
+ mov qword ptr [rsp + 8], rax
+ mov qword ptr [rsp + 16], rcx
+ lea rdi, [rip + .Lanon.HASH.7]
+ lea rcx, [rip + .Lanon.HASH.1]
+ lea r8, [rip + .Lanon.HASH.9]
+ mov rdx, rsp
+ mov esi, 65
+ call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
diff --git a/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64.mca
new file mode 100644
index 000000000000..ed3a2bc24701
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_mut_dynamic_size.x86-64.mca
@@ -0,0 +1,87 @@
+Iterations: 100
+Instructions: 2400
+Total Cycles: 1647
+Total uOps: 2700
+
+Dispatch Width: 4
+uOps Per Cycle: 1.64
+IPC: 1.46
+Block RThroughput: 6.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rdx, qword ptr [rdi + 16]
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov rcx, qword ptr [rdi + 8]
+ 1 1 0.33 mov esi, 1
+ 1 1 0.33 movabs rdi, 4611686018427387902
+ 1 1 0.33 cmp rdx, rdi
+ 1 1 1.00 jae .LBB8_3
+ 1 1 0.33 test al, 1
+ 1 1 1.00 jne .LBB8_2
+ 1 1 0.50 lea rdi, [2*rdx + 4]
+ 1 1 0.33 cmp rcx, rdi
+ 1 1 1.00 jne .LBB8_3
+ 1 1 1.00 U ret
+ 1 0 0.25 xor esi, esi
+ 1 1 0.33 sub rsp, 24
+ 1 1 1.00 * mov qword ptr [rsp], rsi
+ 1 1 1.00 * mov qword ptr [rsp + 8], rax
+ 1 1 1.00 * mov qword ptr [rsp + 16], rcx
+ 1 1 0.50 lea rdi, [rip + .Lanon.HASH.7]
+ 1 1 0.50 lea rcx, [rip + .Lanon.HASH.1]
+ 1 1 0.50 lea r8, [rip + .Lanon.HASH.9]
+ 1 1 0.33 mov rdx, rsp
+ 1 1 0.33 mov esi, 65
+ 4 7 1.00 * call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 5.15 5.74 4.00 6.11 3.98 4.02
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - 0.71 0.29 mov rdx, qword ptr [rdi + 16]
+ - - - - - - 0.72 0.28 mov rax, qword ptr [rdi]
+ - - - - - - 0.29 0.71 mov rcx, qword ptr [rdi + 8]
+ - - 0.83 0.16 - 0.01 - - mov esi, 1
+ - - 0.16 0.84 - - - - movabs rdi, 4611686018427387902
+ - - 0.80 0.04 - 0.16 - - cmp rdx, rdi
+ - - - - - 1.00 - - jae .LBB8_3
+ - - 0.13 0.74 - 0.13 - - test al, 1
+ - - - - - 1.00 - - jne .LBB8_2
+ - - 0.10 0.90 - - - - lea rdi, [2*rdx + 4]
+ - - 0.09 0.91 - - - - cmp rcx, rdi
+ - - - - - 1.00 - - jne .LBB8_3
+ - - - - - 1.00 - - ret
+ - - - - - - - - xor esi, esi
+ - - 0.29 0.13 - 0.58 - - sub rsp, 24
+ - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp], rsi
+ - - - - 1.00 - 0.28 0.72 mov qword ptr [rsp + 8], rax
+ - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp + 16], rcx
+ - - 0.76 0.24 - - - - lea rdi, [rip + .Lanon.HASH.7]
+ - - 0.82 0.18 - - - - lea rcx, [rip + .Lanon.HASH.1]
+ - - 0.92 0.08 - - - - lea r8, [rip + .Lanon.HASH.9]
+ - - 0.09 0.68 - 0.23 - - mov rdx, rsp
+ - - 0.16 0.84 - - - - mov esi, 65
+ - - - - 1.00 1.00 0.54 1.46 call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
diff --git a/rust/zerocopy/benches/ref_type_deref_mut_static_size.rs b/rust/zerocopy/benches/ref_type_deref_mut_static_size.rs
new file mode 100644
index 000000000000..4f85a031c845
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_mut_static_size.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_deref_mut_static_size<'a>(
+ source: &'a mut Ref<&mut [u8], format::LocoPacket>,
+) -> &'a mut format::LocoPacket {
+ core::ops::DerefMut::deref_mut(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64 b/rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64
new file mode 100644
index 000000000000..259c39646bb3
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64
@@ -0,0 +1,3 @@
+bench_ref_type_deref_mut_static_size:
+ mov rax, qword ptr [rdi]
+ ret
diff --git a/rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64.mca
new file mode 100644
index 000000000000..2de0e6f9791c
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_mut_static_size.x86-64.mca
@@ -0,0 +1,43 @@
+Iterations: 100
+Instructions: 200
+Total Cycles: 103
+Total uOps: 200
+
+Dispatch Width: 4
+uOps Per Cycle: 1.94
+IPC: 1.94
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - - - - 1.00 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - 0.50 0.50 mov rax, qword ptr [rdi]
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_deref_static_size.rs b/rust/zerocopy/benches/ref_type_deref_static_size.rs
new file mode 100644
index 000000000000..60397dc28580
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_static_size.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_deref_static_size<'a>(
+ source: &'a Ref<&[u8], format::LocoPacket>,
+) -> &'a format::LocoPacket {
+ core::ops::Deref::deref(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_deref_static_size.x86-64 b/rust/zerocopy/benches/ref_type_deref_static_size.x86-64
new file mode 100644
index 000000000000..2d4368dd61df
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_static_size.x86-64
@@ -0,0 +1,3 @@
+bench_ref_type_deref_static_size:
+ mov rax, qword ptr [rdi]
+ ret
diff --git a/rust/zerocopy/benches/ref_type_deref_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_deref_static_size.x86-64.mca
new file mode 100644
index 000000000000..2de0e6f9791c
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_deref_static_size.x86-64.mca
@@ -0,0 +1,43 @@
+Iterations: 100
+Instructions: 200
+Total Cycles: 103
+Total uOps: 200
+
+Dispatch Width: 4
+uOps Per Cycle: 1.94
+IPC: 1.94
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - - - - 1.00 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - 0.50 0.50 mov rax, qword ptr [rdi]
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.rs
new file mode 100644
index 000000000000..d11cae87a1df
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_bytes_dynamic_padding(
+ source: &[u8],
+) -> Option<Ref<&[u8], format::LocoPacket>> {
+ Ref::from_bytes(source).ok()
+}
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64
new file mode 100644
index 000000000000..9f2beef51011
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64
@@ -0,0 +1,25 @@
+bench_ref_type_from_bytes_dynamic_padding:
+ cmp rdx, 12
+ setae al
+ mov ecx, edx
+ or ecx, esi
+ test cl, 3
+ sete cl
+ and cl, al
+ cmp cl, 1
+ je .LBB5_2
+ xor esi, esi
+ mov qword ptr [rdi], rsi
+ mov rax, rdi
+ ret
+.LBB5_2:
+ lea rax, [rdx - 9]
+ movabs rcx, -6148914691236517205
+ mov r8, rdx
+ mul rcx
+ shr rdx
+ mov qword ptr [rdi + 8], r8
+ mov qword ptr [rdi + 16], rdx
+ mov qword ptr [rdi], rsi
+ mov rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64.mca
new file mode 100644
index 000000000000..e239f2fbe9e7
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_padding.x86-64.mca
@@ -0,0 +1,85 @@
+Iterations: 100
+Instructions: 2300
+Total Cycles: 909
+Total uOps: 2400
+
+Dispatch Width: 4
+uOps Per Cycle: 2.64
+IPC: 2.53
+Block RThroughput: 6.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 cmp rdx, 12
+ 1 1 0.50 setae al
+ 1 1 0.33 mov ecx, edx
+ 1 1 0.33 or ecx, esi
+ 1 1 0.33 test cl, 3
+ 1 1 0.50 sete cl
+ 1 1 0.33 and cl, al
+ 1 1 0.33 cmp cl, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor esi, esi
+ 1 1 1.00 * mov qword ptr [rdi], rsi
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+ 1 1 0.50 lea rax, [rdx - 9]
+ 1 1 0.33 movabs rcx, -6148914691236517205
+ 1 1 0.33 mov r8, rdx
+ 2 4 1.00 mul rcx
+ 1 1 0.50 shr rdx
+ 1 1 1.00 * mov qword ptr [rdi + 8], r8
+ 1 1 1.00 * mov qword ptr [rdi + 16], rdx
+ 1 1 1.00 * mov qword ptr [rdi], rsi
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.49 5.99 4.00 6.52 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - 0.99 - 0.01 - - cmp rdx, 12
+ - - 0.53 - - 0.47 - - setae al
+ - - 0.52 0.01 - 0.47 - - mov ecx, edx
+ - - - 0.99 - 0.01 - - or ecx, esi
+ - - - 0.51 - 0.49 - - test cl, 3
+ - - 0.52 - - 0.48 - - sete cl
+ - - 0.47 0.48 - 0.05 - - and cl, al
+ - - 0.48 0.48 - 0.04 - - cmp cl, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor esi, esi
+ - - - - 1.00 - - 1.00 mov qword ptr [rdi], rsi
+ - - 0.03 0.48 - 0.49 - - mov rax, rdi
+ - - - - - 1.00 - - ret
+ - - 0.97 0.03 - - - - lea rax, [rdx - 9]
+ - - 0.49 0.51 - - - - movabs rcx, -6148914691236517205
+ - - 0.47 0.01 - 0.52 - - mov r8, rdx
+ - - 1.00 1.00 - - - - mul rcx
+ - - 1.00 - - - - - shr rdx
+ - - - - 1.00 - 1.00 - mov qword ptr [rdi + 8], r8
+ - - - - 1.00 - - 1.00 mov qword ptr [rdi + 16], rdx
+ - - - - 1.00 - 1.00 - mov qword ptr [rdi], rsi
+ - - 0.01 0.50 - 0.49 - - mov rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.rs b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.rs
new file mode 100644
index 000000000000..cf0b278eaf09
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.rs
@@ -0,0 +1,17 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_bytes_dynamic_size(source: &[u8]) -> Option<Ref<&[u8], format::LocoPacket>> {
+ Ref::from_bytes(source).ok()
+}
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64
new file mode 100644
index 000000000000..c50dfc333bd6
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64
@@ -0,0 +1,21 @@
+bench_ref_type_from_bytes_dynamic_size:
+ mov rax, rdi
+ cmp rdx, 4
+ setae cl
+ mov edi, edx
+ or edi, esi
+ test dil, 1
+ sete dil
+ and dil, cl
+ cmp dil, 1
+ je .LBB5_2
+ xor esi, esi
+ mov qword ptr [rax], rsi
+ ret
+.LBB5_2:
+ lea rcx, [rdx - 4]
+ shr rcx
+ mov qword ptr [rax + 8], rdx
+ mov qword ptr [rax + 16], rcx
+ mov qword ptr [rax], rsi
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64.mca
new file mode 100644
index 000000000000..92d98788cc09
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_dynamic_size.x86-64.mca
@@ -0,0 +1,77 @@
+Iterations: 100
+Instructions: 1900
+Total Cycles: 482
+Total uOps: 1900
+
+Dispatch Width: 4
+uOps Per Cycle: 3.94
+IPC: 3.94
+Block RThroughput: 4.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 cmp rdx, 4
+ 1 1 0.50 setae cl
+ 1 1 0.33 mov edi, edx
+ 1 1 0.33 or edi, esi
+ 1 1 0.33 test dil, 1
+ 1 1 0.50 sete dil
+ 1 1 0.33 and dil, cl
+ 1 1 0.33 cmp dil, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor esi, esi
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+ 1 1 0.50 lea rcx, [rdx - 4]
+ 1 1 0.50 shr rcx
+ 1 1 1.00 * mov qword ptr [rax + 8], rdx
+ 1 1 1.00 * mov qword ptr [rax + 16], rcx
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 4.66 4.66 4.00 4.68 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.40 0.43 - 0.17 - - mov rax, rdi
+ - - 0.13 0.83 - 0.04 - - cmp rdx, 4
+ - - 0.67 - - 0.33 - - setae cl
+ - - 0.05 0.79 - 0.16 - - mov edi, edx
+ - - 0.36 0.52 - 0.12 - - or edi, esi
+ - - 0.39 0.52 - 0.09 - - test dil, 1
+ - - 0.98 - - 0.02 - - sete dil
+ - - 0.42 0.49 - 0.09 - - and dil, cl
+ - - 0.43 0.41 - 0.16 - - cmp dil, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor esi, esi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
+ - - 0.33 0.67 - - - - lea rcx, [rdx - 4]
+ - - 0.50 - - 0.50 - - shr rcx
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], rdx
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], rcx
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_static_size.rs b/rust/zerocopy/benches/ref_type_from_bytes_static_size.rs
new file mode 100644
index 000000000000..6a2ced8aad89
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_static_size.rs
@@ -0,0 +1,17 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_bytes_static_size(source: &[u8]) -> Option<Ref<&[u8], format::LocoPacket>> {
+ Ref::from_bytes(source).ok()
+}
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64 b/rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64
new file mode 100644
index 000000000000..59c3862eb569
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64
@@ -0,0 +1,15 @@
+bench_ref_type_from_bytes_static_size:
+ mov rax, rdi
+ cmp rdx, 6
+ setne cl
+ or cl, sil
+ test cl, 1
+ je .LBB5_2
+ xor esi, esi
+ mov qword ptr [rax], rsi
+ ret
+.LBB5_2:
+ mov qword ptr [rax + 8], 6
+ mov qword ptr [rax + 16], 0
+ mov qword ptr [rax], rsi
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64.mca
new file mode 100644
index 000000000000..286354a1a014
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_static_size.x86-64.mca
@@ -0,0 +1,65 @@
+Iterations: 100
+Instructions: 1300
+Total Cycles: 404
+Total uOps: 1300
+
+Dispatch Width: 4
+uOps Per Cycle: 3.22
+IPC: 3.22
+Block RThroughput: 4.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 cmp rdx, 6
+ 1 1 0.50 setne cl
+ 1 1 0.33 or cl, sil
+ 1 1 0.33 test cl, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor esi, esi
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+ 1 1 1.00 * mov qword ptr [rax + 8], 6
+ 1 1 1.00 * mov qword ptr [rax + 16], 0
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 2.55 2.34 4.00 3.11 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.81 0.18 - 0.01 - - mov rax, rdi
+ - - 0.56 0.43 - 0.01 - - cmp rdx, 6
+ - - 1.00 - - - - - setne cl
+ - - - 0.99 - 0.01 - - or cl, sil
+ - - 0.18 0.74 - 0.08 - - test cl, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor esi, esi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], 6
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], 0
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.rs
new file mode 100644
index 000000000000..5d3ee937c2b3
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.rs
@@ -0,0 +1,20 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_bytes_with_elems_dynamic_padding(
+ source: &[u8],
+ count: usize,
+) -> Option<Ref<&[u8], format::LocoPacket>> {
+ Ref::from_bytes_with_elems(source, count).ok()
+}
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64
new file mode 100644
index 000000000000..0eb8287a145e
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64
@@ -0,0 +1,26 @@
+bench_ref_type_from_bytes_with_elems_dynamic_padding:
+ mov rax, rdi
+ movabs rdi, 6148914691236517201
+ cmp rcx, rdi
+ ja .LBB5_3
+ lea rdi, [rcx + 2*rcx]
+ or rdi, 3
+ add rdi, 9
+ cmp rdx, rdi
+ sete dil
+ movabs r8, 3074457345618258599
+ cmp rcx, r8
+ setb r8b
+ and r8b, dil
+ cmp r8b, 1
+ jne .LBB5_3
+ test sil, 3
+ je .LBB5_4
+.LBB5_3:
+ mov qword ptr [rax], 0
+ ret
+.LBB5_4:
+ mov qword ptr [rax], rsi
+ mov qword ptr [rax + 8], rdx
+ mov qword ptr [rax + 16], rcx
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64.mca
new file mode 100644
index 000000000000..19928fdca6b8
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_padding.x86-64.mca
@@ -0,0 +1,85 @@
+Iterations: 100
+Instructions: 2300
+Total Cycles: 640
+Total uOps: 2300
+
+Dispatch Width: 4
+uOps Per Cycle: 3.59
+IPC: 3.59
+Block RThroughput: 5.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 movabs rdi, 6148914691236517201
+ 1 1 0.33 cmp rcx, rdi
+ 1 1 1.00 ja .LBB5_3
+ 1 1 0.50 lea rdi, [rcx + 2*rcx]
+ 1 1 0.33 or rdi, 3
+ 1 1 0.33 add rdi, 9
+ 1 1 0.33 cmp rdx, rdi
+ 1 1 0.50 sete dil
+ 1 1 0.33 movabs r8, 3074457345618258599
+ 1 1 0.33 cmp rcx, r8
+ 1 1 0.50 setb r8b
+ 1 1 0.33 and r8b, dil
+ 1 1 0.33 cmp r8b, 1
+ 1 1 1.00 jne .LBB5_3
+ 1 1 0.33 test sil, 3
+ 1 1 1.00 je .LBB5_4
+ 1 1 1.00 * mov qword ptr [rax], 0
+ 1 1 1.00 U ret
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 * mov qword ptr [rax + 8], rdx
+ 1 1 1.00 * mov qword ptr [rax + 16], rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.32 6.33 4.00 6.35 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.99 - - 0.01 - - mov rax, rdi
+ - - 0.66 0.34 - - - - movabs rdi, 6148914691236517201
+ - - 0.34 0.66 - - - - cmp rcx, rdi
+ - - - - - 1.00 - - ja .LBB5_3
+ - - 0.66 0.34 - - - - lea rdi, [rcx + 2*rcx]
+ - - 0.01 0.99 - - - - or rdi, 3
+ - - 0.01 0.99 - - - - add rdi, 9
+ - - 0.01 0.99 - - - - cmp rdx, rdi
+ - - 0.99 - - 0.01 - - sete dil
+ - - 0.35 0.01 - 0.64 - - movabs r8, 3074457345618258599
+ - - 0.99 - - 0.01 - - cmp rcx, r8
+ - - 0.99 - - 0.01 - - setb r8b
+ - - - 1.00 - - - - and r8b, dil
+ - - - 1.00 - - - - cmp r8b, 1
+ - - - - - 1.00 - - jne .LBB5_3
+ - - 0.32 0.01 - 0.67 - - test sil, 3
+ - - - - - 1.00 - - je .LBB5_4
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], 0
+ - - - - - 1.00 - - ret
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rdx
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.rs b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.rs
new file mode 100644
index 000000000000..0ea631ab0c48
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.rs
@@ -0,0 +1,20 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_bytes_with_elems_dynamic_size(
+ source: &[u8],
+ count: usize,
+) -> Option<Ref<&[u8], format::LocoPacket>> {
+ Ref::from_bytes_with_elems(source, count).ok()
+}
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64
new file mode 100644
index 000000000000..67cc5b58dd14
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64
@@ -0,0 +1,28 @@
+bench_ref_type_from_bytes_with_elems_dynamic_size:
+ mov rax, rdi
+ lea rdi, [rcx + rcx]
+ test rcx, rcx
+ setns r8b
+ cmp rdi, -5
+ setb dil
+ and dil, r8b
+ lea r8, [2*rcx + 4]
+ cmp rdx, r8
+ sete r8b
+ movabs r9, 4611686018427387902
+ cmp rcx, r9
+ setb r9b
+ and r9b, r8b
+ and r9b, dil
+ cmp r9b, 1
+ je .LBB5_1
+.LBB5_2:
+ mov qword ptr [rax], 0
+ ret
+.LBB5_1:
+ test sil, 1
+ jne .LBB5_2
+ mov qword ptr [rax], rsi
+ mov qword ptr [rax + 8], rdx
+ mov qword ptr [rax + 16], rcx
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64.mca
new file mode 100644
index 000000000000..cc99f6a4c6f3
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_bytes_with_elems_dynamic_size.x86-64.mca
@@ -0,0 +1,89 @@
+Iterations: 100
+Instructions: 2500
+Total Cycles: 706
+Total uOps: 2500
+
+Dispatch Width: 4
+uOps Per Cycle: 3.54
+IPC: 3.54
+Block RThroughput: 6.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.50 lea rdi, [rcx + rcx]
+ 1 1 0.33 test rcx, rcx
+ 1 1 0.50 setns r8b
+ 1 1 0.33 cmp rdi, -5
+ 1 1 0.50 setb dil
+ 1 1 0.33 and dil, r8b
+ 1 1 0.50 lea r8, [2*rcx + 4]
+ 1 1 0.33 cmp rdx, r8
+ 1 1 0.50 sete r8b
+ 1 1 0.33 movabs r9, 4611686018427387902
+ 1 1 0.33 cmp rcx, r9
+ 1 1 0.50 setb r9b
+ 1 1 0.33 and r9b, r8b
+ 1 1 0.33 and r9b, dil
+ 1 1 0.33 cmp r9b, 1
+ 1 1 1.00 je .LBB5_1
+ 1 1 1.00 * mov qword ptr [rax], 0
+ 1 1 1.00 U ret
+ 1 1 0.33 test sil, 1
+ 1 1 1.00 jne .LBB5_2
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 * mov qword ptr [rax + 8], rdx
+ 1 1 1.00 * mov qword ptr [rax + 16], rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.99 7.00 4.00 7.01 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.08 - - 0.92 - - mov rax, rdi
+ - - 0.99 0.01 - - - - lea rdi, [rcx + rcx]
+ - - 0.92 0.01 - 0.07 - - test rcx, rcx
+ - - 0.99 - - 0.01 - - setns r8b
+ - - 0.01 0.01 - 0.98 - - cmp rdi, -5
+ - - 0.98 - - 0.02 - - setb dil
+ - - - 0.99 - 0.01 - - and dil, r8b
+ - - 0.01 0.99 - - - - lea r8, [2*rcx + 4]
+ - - 0.99 0.01 - - - - cmp rdx, r8
+ - - 1.00 - - - - - sete r8b
+ - - 0.01 - - 0.99 - - movabs r9, 4611686018427387902
+ - - - 1.00 - - - - cmp rcx, r9
+ - - 0.99 - - 0.01 - - setb r9b
+ - - 0.01 0.99 - - - - and r9b, r8b
+ - - 0.01 0.99 - - - - and r9b, dil
+ - - - 1.00 - - - - cmp r9b, 1
+ - - - - - 1.00 - - je .LBB5_1
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], 0
+ - - - - - 1.00 - - ret
+ - - - 1.00 - - - - test sil, 1
+ - - - - - 1.00 - - jne .LBB5_2
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rdx
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.rs
new file mode 100644
index 000000000000..7826991fc546
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.rs
@@ -0,0 +1,22 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_prefix_dynamic_padding(
+ source: &[u8],
+) -> Option<Ref<&[u8], format::LocoPacket>> {
+ match Ref::from_prefix(source) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64
new file mode 100644
index 000000000000..a549e3337985
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64
@@ -0,0 +1,26 @@
+bench_ref_type_from_prefix_dynamic_padding:
+ test sil, 3
+ sete al
+ cmp rdx, 12
+ setae cl
+ and cl, al
+ cmp cl, 1
+ je .LBB5_2
+ xor esi, esi
+ mov qword ptr [rdi], rsi
+ mov rax, rdi
+ ret
+.LBB5_2:
+ movabs rcx, 9223372036854775804
+ and rcx, rdx
+ or rdx, 3
+ add rdx, -12
+ movabs r8, -6148914691236517205
+ mov rax, rdx
+ mul r8
+ shr rdx
+ mov qword ptr [rdi + 8], rcx
+ mov qword ptr [rdi + 16], rdx
+ mov qword ptr [rdi], rsi
+ mov rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64.mca
new file mode 100644
index 000000000000..d85fe71ca66f
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_padding.x86-64.mca
@@ -0,0 +1,87 @@
+Iterations: 100
+Instructions: 2400
+Total Cycles: 907
+Total uOps: 2500
+
+Dispatch Width: 4
+uOps Per Cycle: 2.76
+IPC: 2.65
+Block RThroughput: 6.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 test sil, 3
+ 1 1 0.50 sete al
+ 1 1 0.33 cmp rdx, 12
+ 1 1 0.50 setae cl
+ 1 1 0.33 and cl, al
+ 1 1 0.33 cmp cl, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor esi, esi
+ 1 1 1.00 * mov qword ptr [rdi], rsi
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+ 1 1 0.33 movabs rcx, 9223372036854775804
+ 1 1 0.33 and rcx, rdx
+ 1 1 0.33 or rdx, 3
+ 1 1 0.33 add rdx, -12
+ 1 1 0.33 movabs r8, -6148914691236517205
+ 1 1 0.33 mov rax, rdx
+ 2 4 1.00 mul r8
+ 1 1 0.50 shr rdx
+ 1 1 1.00 * mov qword ptr [rdi + 8], rcx
+ 1 1 1.00 * mov qword ptr [rdi + 16], rdx
+ 1 1 1.00 * mov qword ptr [rdi], rsi
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.96 6.07 4.00 6.97 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.92 0.07 - 0.01 - - test sil, 3
+ - - 0.98 - - 0.02 - - sete al
+ - - 0.03 0.97 - - - - cmp rdx, 12
+ - - 0.99 - - 0.01 - - setae cl
+ - - 0.03 0.94 - 0.03 - - and cl, al
+ - - 0.06 0.94 - - - - cmp cl, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor esi, esi
+ - - - - 1.00 - - 1.00 mov qword ptr [rdi], rsi
+ - - 0.01 0.05 - 0.94 - - mov rax, rdi
+ - - - - - 1.00 - - ret
+ - - 0.92 0.08 - - - - movabs rcx, 9223372036854775804
+ - - 0.96 0.01 - 0.03 - - and rcx, rdx
+ - - - 0.03 - 0.97 - - or rdx, 3
+ - - 0.01 0.96 - 0.03 - - add rdx, -12
+ - - 0.03 0.06 - 0.91 - - movabs r8, -6148914691236517205
+ - - 0.94 0.03 - 0.03 - - mov rax, rdx
+ - - 1.00 1.00 - - - - mul r8
+ - - 0.05 - - 0.95 - - shr rdx
+ - - - - 1.00 - 1.00 - mov qword ptr [rdi + 8], rcx
+ - - - - 1.00 - - 1.00 mov qword ptr [rdi + 16], rdx
+ - - - - 1.00 - 1.00 - mov qword ptr [rdi], rsi
+ - - 0.03 0.93 - 0.04 - - mov rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.rs b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.rs
new file mode 100644
index 000000000000..131a12103656
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.rs
@@ -0,0 +1,22 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_prefix_dynamic_size(
+ source: &[u8],
+) -> Option<Ref<&[u8], format::LocoPacket>> {
+ match Ref::from_prefix(source) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64
new file mode 100644
index 000000000000..fc050aac2637
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64
@@ -0,0 +1,19 @@
+bench_ref_type_from_prefix_dynamic_size:
+ mov rax, rdi
+ cmp rdx, 4
+ setb cl
+ or cl, sil
+ test cl, 1
+ je .LBB5_2
+ xor esi, esi
+ mov qword ptr [rax], rsi
+ ret
+.LBB5_2:
+ movabs rcx, 9223372036854775806
+ and rcx, rdx
+ add rdx, -4
+ shr rdx
+ mov qword ptr [rax + 8], rcx
+ mov qword ptr [rax + 16], rdx
+ mov qword ptr [rax], rsi
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64.mca
new file mode 100644
index 000000000000..a249f0305291
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_dynamic_size.x86-64.mca
@@ -0,0 +1,73 @@
+Iterations: 100
+Instructions: 1700
+Total Cycles: 431
+Total uOps: 1700
+
+Dispatch Width: 4
+uOps Per Cycle: 3.94
+IPC: 3.94
+Block RThroughput: 4.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 cmp rdx, 4
+ 1 1 0.50 setb cl
+ 1 1 0.33 or cl, sil
+ 1 1 0.33 test cl, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor esi, esi
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+ 1 1 0.33 movabs rcx, 9223372036854775806
+ 1 1 0.33 and rcx, rdx
+ 1 1 0.33 add rdx, -4
+ 1 1 0.50 shr rdx
+ 1 1 1.00 * mov qword ptr [rax + 8], rcx
+ 1 1 1.00 * mov qword ptr [rax + 16], rdx
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 3.99 3.75 4.00 4.26 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - 0.99 - 0.01 - - mov rax, rdi
+ - - 0.74 0.26 - - - - cmp rdx, 4
+ - - 1.00 - - - - - setb cl
+ - - 0.25 0.74 - 0.01 - - or cl, sil
+ - - 0.01 0.97 - 0.02 - - test cl, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor esi, esi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
+ - - 0.48 0.52 - - - - movabs rcx, 9223372036854775806
+ - - 0.73 0.27 - - - - and rcx, rdx
+ - - 0.02 - - 0.98 - - add rdx, -4
+ - - 0.76 - - 0.24 - - shr rdx
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], rcx
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], rdx
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_static_size.rs b/rust/zerocopy/benches/ref_type_from_prefix_static_size.rs
new file mode 100644
index 000000000000..c149fd5088ee
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_static_size.rs
@@ -0,0 +1,20 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_prefix_static_size(source: &[u8]) -> Option<Ref<&[u8], format::LocoPacket>> {
+ match Ref::from_prefix(source) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64 b/rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64
new file mode 100644
index 000000000000..420fd6f3b4bd
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64
@@ -0,0 +1,15 @@
+bench_ref_type_from_prefix_static_size:
+ mov rax, rdi
+ cmp rdx, 6
+ setb cl
+ or cl, sil
+ test cl, 1
+ je .LBB5_2
+ xor esi, esi
+ mov qword ptr [rax], rsi
+ ret
+.LBB5_2:
+ mov qword ptr [rax + 8], 6
+ mov qword ptr [rax + 16], 0
+ mov qword ptr [rax], rsi
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64.mca
new file mode 100644
index 000000000000..6b65f303a786
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_static_size.x86-64.mca
@@ -0,0 +1,65 @@
+Iterations: 100
+Instructions: 1300
+Total Cycles: 404
+Total uOps: 1300
+
+Dispatch Width: 4
+uOps Per Cycle: 3.22
+IPC: 3.22
+Block RThroughput: 4.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 cmp rdx, 6
+ 1 1 0.50 setb cl
+ 1 1 0.33 or cl, sil
+ 1 1 0.33 test cl, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor esi, esi
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+ 1 1 1.00 * mov qword ptr [rax + 8], 6
+ 1 1 1.00 * mov qword ptr [rax + 16], 0
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 2.55 2.34 4.00 3.11 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.81 0.18 - 0.01 - - mov rax, rdi
+ - - 0.56 0.43 - 0.01 - - cmp rdx, 6
+ - - 1.00 - - - - - setb cl
+ - - - 0.99 - 0.01 - - or cl, sil
+ - - 0.18 0.74 - 0.08 - - test cl, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor esi, esi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], 6
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], 0
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.rs
new file mode 100644
index 000000000000..a4fe5a6c86ca
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.rs
@@ -0,0 +1,23 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_prefix_with_elems_dynamic_padding(
+ source: &[u8],
+ count: usize,
+) -> Option<Ref<&[u8], format::LocoPacket>> {
+ match Ref::from_prefix_with_elems(source, count) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64
new file mode 100644
index 000000000000..fb9d6bfaac27
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64
@@ -0,0 +1,26 @@
+bench_ref_type_from_prefix_with_elems_dynamic_padding:
+ mov rax, rdi
+ movabs rdi, 6148914691236517201
+ cmp rcx, rdi
+ ja .LBB5_4
+ lea rdi, [rcx + 2*rcx]
+ or rdi, 3
+ add rdi, 9
+ cmp rdi, rdx
+ setbe dl
+ movabs r8, 3074457345618258599
+ cmp rcx, r8
+ setb r8b
+ and r8b, dl
+ cmp r8b, 1
+ jne .LBB5_4
+ test sil, 3
+ je .LBB5_3
+.LBB5_4:
+ mov qword ptr [rax], 0
+ ret
+.LBB5_3:
+ mov qword ptr [rax], rsi
+ mov qword ptr [rax + 8], rdi
+ mov qword ptr [rax + 16], rcx
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64.mca
new file mode 100644
index 000000000000..0ef4c84142b0
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_padding.x86-64.mca
@@ -0,0 +1,85 @@
+Iterations: 100
+Instructions: 2300
+Total Cycles: 674
+Total uOps: 2400
+
+Dispatch Width: 4
+uOps Per Cycle: 3.56
+IPC: 3.41
+Block RThroughput: 6.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 movabs rdi, 6148914691236517201
+ 1 1 0.33 cmp rcx, rdi
+ 1 1 1.00 ja .LBB5_4
+ 1 1 0.50 lea rdi, [rcx + 2*rcx]
+ 1 1 0.33 or rdi, 3
+ 1 1 0.33 add rdi, 9
+ 1 1 0.33 cmp rdi, rdx
+ 2 2 1.00 setbe dl
+ 1 1 0.33 movabs r8, 3074457345618258599
+ 1 1 0.33 cmp rcx, r8
+ 1 1 0.50 setb r8b
+ 1 1 0.33 and r8b, dl
+ 1 1 0.33 cmp r8b, 1
+ 1 1 1.00 jne .LBB5_4
+ 1 1 0.33 test sil, 3
+ 1 1 1.00 je .LBB5_3
+ 1 1 1.00 * mov qword ptr [rax], 0
+ 1 1 1.00 U ret
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 * mov qword ptr [rax + 8], rdi
+ 1 1 1.00 * mov qword ptr [rax + 16], rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.66 6.65 4.00 6.69 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.01 0.98 - 0.01 - - mov rax, rdi
+ - - 0.01 0.99 - - - - movabs rdi, 6148914691236517201
+ - - 0.02 0.98 - - - - cmp rcx, rdi
+ - - - - - 1.00 - - ja .LBB5_4
+ - - 0.02 0.98 - - - - lea rdi, [rcx + 2*rcx]
+ - - 0.68 0.32 - - - - or rdi, 3
+ - - 0.98 0.02 - - - - add rdi, 9
+ - - 0.97 0.03 - - - - cmp rdi, rdx
+ - - 1.96 - - 0.04 - - setbe dl
+ - - 0.02 0.98 - - - - movabs r8, 3074457345618258599
+ - - 0.02 0.34 - 0.64 - - cmp rcx, r8
+ - - 0.02 - - 0.98 - - setb r8b
+ - - 0.97 0.02 - 0.01 - - and r8b, dl
+ - - 0.97 0.02 - 0.01 - - cmp r8b, 1
+ - - - - - 1.00 - - jne .LBB5_4
+ - - 0.01 0.99 - - - - test sil, 3
+ - - - - - 1.00 - - je .LBB5_3
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], 0
+ - - - - - 1.00 - - ret
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rdi
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.rs b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.rs
new file mode 100644
index 000000000000..b0911e8fd8e7
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.rs
@@ -0,0 +1,23 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_prefix_with_elems_dynamic_size(
+ source: &[u8],
+ count: usize,
+) -> Option<Ref<&[u8], format::LocoPacket>> {
+ match Ref::from_prefix_with_elems(source, count) {
+ Ok((packet, _rest)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64
new file mode 100644
index 000000000000..20ba73bd552e
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64
@@ -0,0 +1,27 @@
+bench_ref_type_from_prefix_with_elems_dynamic_size:
+ mov rax, rdi
+ lea rdi, [rcx + rcx]
+ test rcx, rcx
+ sets r8b
+ cmp rdi, -5
+ setae r9b
+ or r9b, r8b
+ jne .LBB5_3
+ add rdi, 4
+ cmp rdi, rdx
+ ja .LBB5_3
+ movabs rdx, 4611686018427387902
+ cmp rcx, rdx
+ setb dl
+ test sil, 1
+ sete r8b
+ test dl, r8b
+ jne .LBB5_4
+.LBB5_3:
+ mov qword ptr [rax], 0
+ ret
+.LBB5_4:
+ mov qword ptr [rax], rsi
+ mov qword ptr [rax + 8], rdi
+ mov qword ptr [rax + 16], rcx
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64.mca
new file mode 100644
index 000000000000..5600d4ea6f69
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_prefix_with_elems_dynamic_size.x86-64.mca
@@ -0,0 +1,87 @@
+Iterations: 100
+Instructions: 2400
+Total Cycles: 672
+Total uOps: 2400
+
+Dispatch Width: 4
+uOps Per Cycle: 3.57
+IPC: 3.57
+Block RThroughput: 6.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.50 lea rdi, [rcx + rcx]
+ 1 1 0.33 test rcx, rcx
+ 1 1 0.50 sets r8b
+ 1 1 0.33 cmp rdi, -5
+ 1 1 0.50 setae r9b
+ 1 1 0.33 or r9b, r8b
+ 1 1 1.00 jne .LBB5_3
+ 1 1 0.33 add rdi, 4
+ 1 1 0.33 cmp rdi, rdx
+ 1 1 1.00 ja .LBB5_3
+ 1 1 0.33 movabs rdx, 4611686018427387902
+ 1 1 0.33 cmp rcx, rdx
+ 1 1 0.50 setb dl
+ 1 1 0.33 test sil, 1
+ 1 1 0.50 sete r8b
+ 1 1 0.33 test dl, r8b
+ 1 1 1.00 jne .LBB5_4
+ 1 1 1.00 * mov qword ptr [rax], 0
+ 1 1 1.00 U ret
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 * mov qword ptr [rax + 8], rdi
+ 1 1 1.00 * mov qword ptr [rax + 16], rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.66 6.65 4.00 6.69 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.03 0.96 - 0.01 - - mov rax, rdi
+ - - 0.64 0.36 - - - - lea rdi, [rcx + rcx]
+ - - 0.34 0.66 - - - - test rcx, rcx
+ - - 0.99 - - 0.01 - - sets r8b
+ - - 0.66 0.34 - - - - cmp rdi, -5
+ - - 0.67 - - 0.33 - - setae r9b
+ - - 0.01 0.99 - - - - or r9b, r8b
+ - - - - - 1.00 - - jne .LBB5_3
+ - - - 0.99 - 0.01 - - add rdi, 4
+ - - 0.66 0.33 - 0.01 - - cmp rdi, rdx
+ - - - - - 1.00 - - ja .LBB5_3
+ - - - 0.01 - 0.99 - - movabs rdx, 4611686018427387902
+ - - 0.33 0.34 - 0.33 - - cmp rcx, rdx
+ - - 1.00 - - - - - setb dl
+ - - 0.33 0.67 - - - - test sil, 1
+ - - 1.00 - - - - - sete r8b
+ - - - 1.00 - - - - test dl, r8b
+ - - - - - 1.00 - - jne .LBB5_4
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], 0
+ - - - - - 1.00 - - ret
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rdi
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.rs
new file mode 100644
index 000000000000..894b7d875b42
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.rs
@@ -0,0 +1,22 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_suffix_dynamic_padding(
+ source: &[u8],
+) -> Option<Ref<&[u8], format::LocoPacket>> {
+ match Ref::from_suffix(source) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64
new file mode 100644
index 000000000000..4a0f5e88612e
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64
@@ -0,0 +1,31 @@
+bench_ref_type_from_suffix_dynamic_padding:
+ lea eax, [rdx + rsi]
+ test al, 3
+ sete al
+ cmp rdx, 12
+ setae cl
+ and cl, al
+ cmp cl, 1
+ je .LBB5_2
+ xor esi, esi
+ mov qword ptr [rdi], rsi
+ mov rax, rdi
+ ret
+.LBB5_2:
+ mov rax, rdx
+ or rax, 3
+ add rax, -12
+ movabs rcx, -6148914691236517205
+ mov r8, rdx
+ mul rcx
+ shr rdx
+ mov eax, r8d
+ and eax, 3
+ movabs rcx, 9223372036854775804
+ and rcx, r8
+ add rsi, rax
+ mov qword ptr [rdi + 8], rcx
+ mov qword ptr [rdi + 16], rdx
+ mov qword ptr [rdi], rsi
+ mov rax, rdi
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64.mca
new file mode 100644
index 000000000000..ad884432de74
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_padding.x86-64.mca
@@ -0,0 +1,97 @@
+Iterations: 100
+Instructions: 2900
+Total Cycles: 942
+Total uOps: 3000
+
+Dispatch Width: 4
+uOps Per Cycle: 3.18
+IPC: 3.08
+Block RThroughput: 7.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.50 lea eax, [rdx + rsi]
+ 1 1 0.33 test al, 3
+ 1 1 0.50 sete al
+ 1 1 0.33 cmp rdx, 12
+ 1 1 0.50 setae cl
+ 1 1 0.33 and cl, al
+ 1 1 0.33 cmp cl, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor esi, esi
+ 1 1 1.00 * mov qword ptr [rdi], rsi
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+ 1 1 0.33 mov rax, rdx
+ 1 1 0.33 or rax, 3
+ 1 1 0.33 add rax, -12
+ 1 1 0.33 movabs rcx, -6148914691236517205
+ 1 1 0.33 mov r8, rdx
+ 2 4 1.00 mul rcx
+ 1 1 0.50 shr rdx
+ 1 1 0.33 mov eax, r8d
+ 1 1 0.33 and eax, 3
+ 1 1 0.33 movabs rcx, 9223372036854775804
+ 1 1 0.33 and rcx, r8
+ 1 1 0.33 add rsi, rax
+ 1 1 1.00 * mov qword ptr [rdi + 8], rcx
+ 1 1 1.00 * mov qword ptr [rdi + 16], rdx
+ 1 1 1.00 * mov qword ptr [rdi], rsi
+ 1 1 0.33 mov rax, rdi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 8.34 8.01 4.00 8.65 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.36 0.64 - - - - lea eax, [rdx + rsi]
+ - - 0.32 0.36 - 0.32 - - test al, 3
+ - - 0.69 - - 0.31 - - sete al
+ - - 0.32 0.04 - 0.64 - - cmp rdx, 12
+ - - 0.68 - - 0.32 - - setae cl
+ - - 0.31 0.37 - 0.32 - - and cl, al
+ - - 0.31 0.37 - 0.32 - - cmp cl, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor esi, esi
+ - - - - 1.00 - - 1.00 mov qword ptr [rdi], rsi
+ - - 0.33 0.36 - 0.31 - - mov rax, rdi
+ - - - - - 1.00 - - ret
+ - - 0.32 0.32 - 0.36 - - mov rax, rdx
+ - - 0.01 0.63 - 0.36 - - or rax, 3
+ - - - 0.68 - 0.32 - - add rax, -12
+ - - 0.34 0.63 - 0.03 - - movabs rcx, -6148914691236517205
+ - - 0.32 0.32 - 0.36 - - mov r8, rdx
+ - - 1.00 1.00 - - - - mul rcx
+ - - 0.69 - - 0.31 - - shr rdx
+ - - 0.37 0.32 - 0.31 - - mov eax, r8d
+ - - 0.35 0.33 - 0.32 - - and eax, 3
+ - - 0.63 0.35 - 0.02 - - movabs rcx, 9223372036854775804
+ - - 0.32 0.32 - 0.36 - - and rcx, r8
+ - - 0.33 0.35 - 0.32 - - add rsi, rax
+ - - - - 1.00 - 1.00 - mov qword ptr [rdi + 8], rcx
+ - - - - 1.00 - - 1.00 mov qword ptr [rdi + 16], rdx
+ - - - - 1.00 - 1.00 - mov qword ptr [rdi], rsi
+ - - 0.34 0.62 - 0.04 - - mov rax, rdi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.rs b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.rs
new file mode 100644
index 000000000000..2cdadc3316d6
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.rs
@@ -0,0 +1,22 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_suffix_dynamic_size(
+ source: &[u8],
+) -> Option<Ref<&[u8], format::LocoPacket>> {
+ match Ref::from_suffix(source) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64
new file mode 100644
index 000000000000..9dc329867dde
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64
@@ -0,0 +1,23 @@
+bench_ref_type_from_suffix_dynamic_size:
+ mov rax, rdi
+ lea ecx, [rdx + rsi]
+ cmp rdx, 4
+ setb dil
+ or dil, cl
+ test dil, 1
+ je .LBB5_2
+ xor esi, esi
+ mov qword ptr [rax], rsi
+ ret
+.LBB5_2:
+ lea rcx, [rdx - 4]
+ shr rcx
+ mov edi, edx
+ and edi, 1
+ movabs r8, 9223372036854775806
+ and r8, rdx
+ add rsi, rdi
+ mov qword ptr [rax + 8], r8
+ mov qword ptr [rax + 16], rcx
+ mov qword ptr [rax], rsi
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64.mca
new file mode 100644
index 000000000000..7639f237aa87
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_dynamic_size.x86-64.mca
@@ -0,0 +1,81 @@
+Iterations: 100
+Instructions: 2100
+Total Cycles: 540
+Total uOps: 2100
+
+Dispatch Width: 4
+uOps Per Cycle: 3.89
+IPC: 3.89
+Block RThroughput: 5.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.50 lea ecx, [rdx + rsi]
+ 1 1 0.33 cmp rdx, 4
+ 1 1 0.50 setb dil
+ 1 1 0.33 or dil, cl
+ 1 1 0.33 test dil, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor esi, esi
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+ 1 1 0.50 lea rcx, [rdx - 4]
+ 1 1 0.50 shr rcx
+ 1 1 0.33 mov edi, edx
+ 1 1 0.33 and edi, 1
+ 1 1 0.33 movabs r8, 9223372036854775806
+ 1 1 0.33 and r8, rdx
+ 1 1 0.33 add rsi, rdi
+ 1 1 1.00 * mov qword ptr [rax + 8], r8
+ 1 1 1.00 * mov qword ptr [rax + 16], rcx
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 5.32 5.33 4.00 5.35 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.33 0.66 - 0.01 - - mov rax, rdi
+ - - 0.99 0.01 - - - - lea ecx, [rdx + rsi]
+ - - 0.01 0.98 - 0.01 - - cmp rdx, 4
+ - - 0.99 - - 0.01 - - setb dil
+ - - - 1.00 - - - - or dil, cl
+ - - - 1.00 - - - - test dil, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor esi, esi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
+ - - 0.65 0.35 - - - - lea rcx, [rdx - 4]
+ - - 0.68 - - 0.32 - - shr rcx
+ - - - 0.64 - 0.36 - - mov edi, edx
+ - - 0.66 0.01 - 0.33 - - and edi, 1
+ - - 0.34 - - 0.66 - - movabs r8, 9223372036854775806
+ - - 0.32 0.68 - - - - and r8, rdx
+ - - 0.35 - - 0.65 - - add rsi, rdi
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], r8
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], rcx
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_static_size.rs b/rust/zerocopy/benches/ref_type_from_suffix_static_size.rs
new file mode 100644
index 000000000000..2ad2d2f5ee33
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_static_size.rs
@@ -0,0 +1,20 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_suffix_static_size(source: &[u8]) -> Option<Ref<&[u8], format::LocoPacket>> {
+ match Ref::from_suffix(source) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64 b/rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64
new file mode 100644
index 000000000000..37a3d72e7ca1
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64
@@ -0,0 +1,18 @@
+bench_ref_type_from_suffix_static_size:
+ mov rax, rdi
+ lea ecx, [rdx + rsi]
+ cmp rdx, 6
+ setb dil
+ or dil, cl
+ test dil, 1
+ je .LBB5_2
+ xor ecx, ecx
+ mov qword ptr [rax], rcx
+ ret
+.LBB5_2:
+ lea rcx, [rsi + rdx]
+ add rcx, -6
+ mov qword ptr [rax + 8], 6
+ mov qword ptr [rax + 16], 0
+ mov qword ptr [rax], rcx
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64.mca
new file mode 100644
index 000000000000..4dfa9bda8ceb
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_static_size.x86-64.mca
@@ -0,0 +1,71 @@
+Iterations: 100
+Instructions: 1600
+Total Cycles: 405
+Total uOps: 1600
+
+Dispatch Width: 4
+uOps Per Cycle: 3.95
+IPC: 3.95
+Block RThroughput: 4.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.50 lea ecx, [rdx + rsi]
+ 1 1 0.33 cmp rdx, 6
+ 1 1 0.50 setb dil
+ 1 1 0.33 or dil, cl
+ 1 1 0.33 test dil, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor ecx, ecx
+ 1 1 1.00 * mov qword ptr [rax], rcx
+ 1 1 1.00 U ret
+ 1 1 0.50 lea rcx, [rsi + rdx]
+ 1 1 0.33 add rcx, -6
+ 1 1 1.00 * mov qword ptr [rax + 8], 6
+ 1 1 1.00 * mov qword ptr [rax + 16], 0
+ 1 1 1.00 * mov qword ptr [rax], rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 3.66 3.66 4.00 3.68 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.66 0.33 - 0.01 - - mov rax, rdi
+ - - 0.33 0.67 - - - - lea ecx, [rdx + rsi]
+ - - 0.34 0.33 - 0.33 - - cmp rdx, 6
+ - - 0.99 - - 0.01 - - setb dil
+ - - 0.33 0.67 - - - - or dil, cl
+ - - - 1.00 - - - - test dil, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor ecx, ecx
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], rcx
+ - - - - - 1.00 - - ret
+ - - 0.34 0.66 - - - - lea rcx, [rsi + rdx]
+ - - 0.67 - - 0.33 - - add rcx, -6
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 8], 6
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 16], 0
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.rs
new file mode 100644
index 000000000000..105086dd4dc9
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.rs
@@ -0,0 +1,23 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_suffix_with_elems_dynamic_padding(
+ source: &[u8],
+ count: usize,
+) -> Option<Ref<&[u8], format::LocoPacket>> {
+ match Ref::from_suffix_with_elems(source, count) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64
new file mode 100644
index 000000000000..a52ec2c34dd9
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64
@@ -0,0 +1,27 @@
+bench_ref_type_from_suffix_with_elems_dynamic_padding:
+ mov rax, rdi
+ movabs rdi, 6148914691236517201
+ cmp rcx, rdi
+ ja .LBB5_3
+ lea rdi, [rcx + 2*rcx]
+ or rdi, 3
+ add rdi, 9
+ sub rdx, rdi
+ setae r8b
+ movabs r9, 3074457345618258599
+ cmp rcx, r9
+ setb r9b
+ and r9b, r8b
+ cmp r9b, 1
+ jne .LBB5_3
+ add rsi, rdx
+ test sil, 3
+ je .LBB5_4
+.LBB5_3:
+ mov qword ptr [rax], 0
+ ret
+.LBB5_4:
+ mov qword ptr [rax], rsi
+ mov qword ptr [rax + 8], rdi
+ mov qword ptr [rax + 16], rcx
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64.mca
new file mode 100644
index 000000000000..b8036059b98a
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_padding.x86-64.mca
@@ -0,0 +1,87 @@
+Iterations: 100
+Instructions: 2400
+Total Cycles: 673
+Total uOps: 2400
+
+Dispatch Width: 4
+uOps Per Cycle: 3.57
+IPC: 3.57
+Block RThroughput: 6.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.33 movabs rdi, 6148914691236517201
+ 1 1 0.33 cmp rcx, rdi
+ 1 1 1.00 ja .LBB5_3
+ 1 1 0.50 lea rdi, [rcx + 2*rcx]
+ 1 1 0.33 or rdi, 3
+ 1 1 0.33 add rdi, 9
+ 1 1 0.33 sub rdx, rdi
+ 1 1 0.50 setae r8b
+ 1 1 0.33 movabs r9, 3074457345618258599
+ 1 1 0.33 cmp rcx, r9
+ 1 1 0.50 setb r9b
+ 1 1 0.33 and r9b, r8b
+ 1 1 0.33 cmp r9b, 1
+ 1 1 1.00 jne .LBB5_3
+ 1 1 0.33 add rsi, rdx
+ 1 1 0.33 test sil, 3
+ 1 1 1.00 je .LBB5_4
+ 1 1 1.00 * mov qword ptr [rax], 0
+ 1 1 1.00 U ret
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 * mov qword ptr [rax + 8], rdi
+ 1 1 1.00 * mov qword ptr [rax + 16], rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.66 6.66 4.00 6.68 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.33 0.01 - 0.66 - - mov rax, rdi
+ - - 0.34 0.02 - 0.64 - - movabs rdi, 6148914691236517201
+ - - 0.34 0.34 - 0.32 - - cmp rcx, rdi
+ - - - - - 1.00 - - ja .LBB5_3
+ - - 0.66 0.34 - - - - lea rdi, [rcx + 2*rcx]
+ - - 0.02 0.97 - 0.01 - - or rdi, 3
+ - - 0.02 0.98 - - - - add rdi, 9
+ - - 0.01 0.99 - - - - sub rdx, rdi
+ - - 1.00 - - - - - setae r8b
+ - - 0.97 0.02 - 0.01 - - movabs r9, 3074457345618258599
+ - - 0.98 0.01 - 0.01 - - cmp rcx, r9
+ - - 0.98 - - 0.02 - - setb r9b
+ - - 0.01 0.98 - 0.01 - - and r9b, r8b
+ - - 0.02 0.98 - - - - cmp r9b, 1
+ - - - - - 1.00 - - jne .LBB5_3
+ - - - 1.00 - - - - add rsi, rdx
+ - - 0.98 0.02 - - - - test sil, 3
+ - - - - - 1.00 - - je .LBB5_4
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], 0
+ - - - - - 1.00 - - ret
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rdi
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.rs b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.rs
new file mode 100644
index 000000000000..926f3044b4bf
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.rs
@@ -0,0 +1,23 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_from_suffix_with_elems_dynamic_size(
+ source: &[u8],
+ count: usize,
+) -> Option<Ref<&[u8], format::LocoPacket>> {
+ match Ref::from_suffix_with_elems(source, count) {
+ Ok((_rest, packet)) => Some(packet),
+ _ => None,
+ }
+}
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64
new file mode 100644
index 000000000000..a6c84ec32242
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64
@@ -0,0 +1,27 @@
+bench_ref_type_from_suffix_with_elems_dynamic_size:
+ mov rax, rdi
+ lea rdi, [rcx + rcx]
+ test rcx, rcx
+ sets r8b
+ cmp rdi, -5
+ setae r9b
+ or r9b, r8b
+ jne .LBB5_4
+ add rdi, 4
+ sub rdx, rdi
+ jb .LBB5_4
+ movabs r8, 4611686018427387901
+ cmp rcx, r8
+ ja .LBB5_4
+ add rsi, rdx
+ mov edx, esi
+ and edx, 1
+ je .LBB5_5
+.LBB5_4:
+ mov qword ptr [rax], 0
+ ret
+.LBB5_5:
+ mov qword ptr [rax], rsi
+ mov qword ptr [rax + 8], rdi
+ mov qword ptr [rax + 16], rcx
+ ret
diff --git a/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64.mca
new file mode 100644
index 000000000000..113b75b4172d
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_from_suffix_with_elems_dynamic_size.x86-64.mca
@@ -0,0 +1,87 @@
+Iterations: 100
+Instructions: 2400
+Total Cycles: 673
+Total uOps: 2400
+
+Dispatch Width: 4
+uOps Per Cycle: 3.57
+IPC: 3.57
+Block RThroughput: 6.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 mov rax, rdi
+ 1 1 0.50 lea rdi, [rcx + rcx]
+ 1 1 0.33 test rcx, rcx
+ 1 1 0.50 sets r8b
+ 1 1 0.33 cmp rdi, -5
+ 1 1 0.50 setae r9b
+ 1 1 0.33 or r9b, r8b
+ 1 1 1.00 jne .LBB5_4
+ 1 1 0.33 add rdi, 4
+ 1 1 0.33 sub rdx, rdi
+ 1 1 1.00 jb .LBB5_4
+ 1 1 0.33 movabs r8, 4611686018427387901
+ 1 1 0.33 cmp rcx, r8
+ 1 1 1.00 ja .LBB5_4
+ 1 1 0.33 add rsi, rdx
+ 1 1 0.33 mov edx, esi
+ 1 1 0.33 and edx, 1
+ 1 1 1.00 je .LBB5_5
+ 1 1 1.00 * mov qword ptr [rax], 0
+ 1 1 1.00 U ret
+ 1 1 1.00 * mov qword ptr [rax], rsi
+ 1 1 1.00 * mov qword ptr [rax + 8], rdi
+ 1 1 1.00 * mov qword ptr [rax + 16], rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.65 6.65 4.00 6.70 2.00 2.00
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - 0.99 - 0.01 - - mov rax, rdi
+ - - 0.01 0.99 - - - - lea rdi, [rcx + rcx]
+ - - 0.67 0.33 - - - - test rcx, rcx
+ - - 0.99 - - 0.01 - - sets r8b
+ - - - 1.00 - - - - cmp rdi, -5
+ - - 1.00 - - - - - setae r9b
+ - - 1.00 - - - - - or r9b, r8b
+ - - - - - 1.00 - - jne .LBB5_4
+ - - - 0.99 - 0.01 - - add rdi, 4
+ - - - 0.33 - 0.67 - - sub rdx, rdi
+ - - - - - 1.00 - - jb .LBB5_4
+ - - - 1.00 - - - - movabs r8, 4611686018427387901
+ - - 0.66 0.34 - - - - cmp rcx, r8
+ - - - - - 1.00 - - ja .LBB5_4
+ - - 0.34 0.66 - - - - add rsi, rdx
+ - - 0.98 0.02 - - - - mov edx, esi
+ - - 1.00 - - - - - and edx, 1
+ - - - - - 1.00 - - je .LBB5_5
+ - - - - 1.00 - - 1.00 mov qword ptr [rax], 0
+ - - - - - 1.00 - - ret
+ - - - - 1.00 - 1.00 - mov qword ptr [rax], rsi
+ - - - - 1.00 - - 1.00 mov qword ptr [rax + 8], rdi
+ - - - - 1.00 - 1.00 - mov qword ptr [rax + 16], rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.rs b/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.rs
new file mode 100644
index 000000000000..04ac859a98a8
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_into_mut_dynamic_size(
+ source: Ref<&mut [u8], format::LocoPacket>,
+) -> &mut format::LocoPacket {
+ Ref::into_mut(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64
new file mode 100644
index 000000000000..b7568f6631ce
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64
@@ -0,0 +1,27 @@
+bench_ref_type_into_mut_dynamic_size:
+ mov rdx, qword ptr [rdi + 16]
+ mov rax, qword ptr [rdi]
+ mov rcx, qword ptr [rdi + 8]
+ mov esi, 1
+ movabs rdi, 4611686018427387902
+ cmp rdx, rdi
+ jae .LBB8_3
+ test al, 1
+ jne .LBB8_2
+ lea rdi, [2*rdx + 4]
+ cmp rcx, rdi
+ jne .LBB8_3
+ ret
+.LBB8_2:
+ xor esi, esi
+.LBB8_3:
+ sub rsp, 24
+ mov qword ptr [rsp], rsi
+ mov qword ptr [rsp + 8], rax
+ mov qword ptr [rsp + 16], rcx
+ lea rdi, [rip + .Lanon.HASH.2]
+ lea rcx, [rip + .Lanon.HASH.1]
+ lea r8, [rip + .Lanon.HASH.4]
+ mov rdx, rsp
+ mov esi, 54
+ call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
diff --git a/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64.mca
new file mode 100644
index 000000000000..03137756a8b5
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_mut_dynamic_size.x86-64.mca
@@ -0,0 +1,87 @@
+Iterations: 100
+Instructions: 2400
+Total Cycles: 1647
+Total uOps: 2700
+
+Dispatch Width: 4
+uOps Per Cycle: 1.64
+IPC: 1.46
+Block RThroughput: 6.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rdx, qword ptr [rdi + 16]
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov rcx, qword ptr [rdi + 8]
+ 1 1 0.33 mov esi, 1
+ 1 1 0.33 movabs rdi, 4611686018427387902
+ 1 1 0.33 cmp rdx, rdi
+ 1 1 1.00 jae .LBB8_3
+ 1 1 0.33 test al, 1
+ 1 1 1.00 jne .LBB8_2
+ 1 1 0.50 lea rdi, [2*rdx + 4]
+ 1 1 0.33 cmp rcx, rdi
+ 1 1 1.00 jne .LBB8_3
+ 1 1 1.00 U ret
+ 1 0 0.25 xor esi, esi
+ 1 1 0.33 sub rsp, 24
+ 1 1 1.00 * mov qword ptr [rsp], rsi
+ 1 1 1.00 * mov qword ptr [rsp + 8], rax
+ 1 1 1.00 * mov qword ptr [rsp + 16], rcx
+ 1 1 0.50 lea rdi, [rip + .Lanon.HASH.2]
+ 1 1 0.50 lea rcx, [rip + .Lanon.HASH.1]
+ 1 1 0.50 lea r8, [rip + .Lanon.HASH.4]
+ 1 1 0.33 mov rdx, rsp
+ 1 1 0.33 mov esi, 54
+ 4 7 1.00 * call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 5.15 5.74 4.00 6.11 3.98 4.02
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - 0.71 0.29 mov rdx, qword ptr [rdi + 16]
+ - - - - - - 0.72 0.28 mov rax, qword ptr [rdi]
+ - - - - - - 0.29 0.71 mov rcx, qword ptr [rdi + 8]
+ - - 0.83 0.16 - 0.01 - - mov esi, 1
+ - - 0.16 0.84 - - - - movabs rdi, 4611686018427387902
+ - - 0.80 0.04 - 0.16 - - cmp rdx, rdi
+ - - - - - 1.00 - - jae .LBB8_3
+ - - 0.13 0.74 - 0.13 - - test al, 1
+ - - - - - 1.00 - - jne .LBB8_2
+ - - 0.10 0.90 - - - - lea rdi, [2*rdx + 4]
+ - - 0.09 0.91 - - - - cmp rcx, rdi
+ - - - - - 1.00 - - jne .LBB8_3
+ - - - - - 1.00 - - ret
+ - - - - - - - - xor esi, esi
+ - - 0.29 0.13 - 0.58 - - sub rsp, 24
+ - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp], rsi
+ - - - - 1.00 - 0.28 0.72 mov qword ptr [rsp + 8], rax
+ - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp + 16], rcx
+ - - 0.76 0.24 - - - - lea rdi, [rip + .Lanon.HASH.2]
+ - - 0.82 0.18 - - - - lea rcx, [rip + .Lanon.HASH.1]
+ - - 0.92 0.08 - - - - lea r8, [rip + .Lanon.HASH.4]
+ - - 0.09 0.68 - 0.23 - - mov rdx, rsp
+ - - 0.16 0.84 - - - - mov esi, 54
+ - - - - 1.00 1.00 0.54 1.46 call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
diff --git a/rust/zerocopy/benches/ref_type_into_mut_static_size.rs b/rust/zerocopy/benches/ref_type_into_mut_static_size.rs
new file mode 100644
index 000000000000..edab9abe0aa8
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_mut_static_size.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_into_mut_static_size(
+ source: Ref<&mut [u8], format::LocoPacket>,
+) -> &mut format::LocoPacket {
+ Ref::into_mut(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64 b/rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64
new file mode 100644
index 000000000000..0412b33d6cc3
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64
@@ -0,0 +1,3 @@
+bench_ref_type_into_mut_static_size:
+ mov rax, qword ptr [rdi]
+ ret
diff --git a/rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64.mca
new file mode 100644
index 000000000000..2de0e6f9791c
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_mut_static_size.x86-64.mca
@@ -0,0 +1,43 @@
+Iterations: 100
+Instructions: 200
+Total Cycles: 103
+Total uOps: 200
+
+Dispatch Width: 4
+uOps Per Cycle: 1.94
+IPC: 1.94
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - - - - 1.00 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - 0.50 0.50 mov rax, qword ptr [rdi]
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.rs b/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.rs
new file mode 100644
index 000000000000..9f5f3b618c2f
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_padding.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_into_ref_dynamic_padding(
+ source: Ref<&[u8], format::LocoPacket>,
+) -> &format::LocoPacket {
+ Ref::into_ref(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64 b/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64
new file mode 100644
index 000000000000..ca16dbc03e8e
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64
@@ -0,0 +1,29 @@
+bench_ref_type_into_ref_dynamic_padding:
+ mov rdx, qword ptr [rdi + 16]
+ mov rax, qword ptr [rdi]
+ mov rcx, qword ptr [rdi + 8]
+ mov esi, 1
+ movabs rdi, 3074457345618258599
+ cmp rdx, rdi
+ jae .LBB8_3
+ test al, 3
+ jne .LBB8_2
+ lea rdi, [rdx + 2*rdx]
+ or rdi, 3
+ add rdi, 9
+ cmp rcx, rdi
+ jne .LBB8_3
+ ret
+.LBB8_2:
+ xor esi, esi
+.LBB8_3:
+ sub rsp, 24
+ mov qword ptr [rsp], rsi
+ mov qword ptr [rsp + 8], rax
+ mov qword ptr [rsp + 16], rcx
+ lea rdi, [rip + .Lanon.HASH.2]
+ lea rcx, [rip + .Lanon.HASH.1]
+ lea r8, [rip + .Lanon.HASH.4]
+ mov rdx, rsp
+ mov esi, 54
+ call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
diff --git a/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64.mca
new file mode 100644
index 000000000000..bf8eade92561
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_ref_dynamic_padding.x86-64.mca
@@ -0,0 +1,91 @@
+Iterations: 100
+Instructions: 2600
+Total Cycles: 1812
+Total uOps: 2900
+
+Dispatch Width: 4
+uOps Per Cycle: 1.60
+IPC: 1.43
+Block RThroughput: 7.3
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rdx, qword ptr [rdi + 16]
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov rcx, qword ptr [rdi + 8]
+ 1 1 0.33 mov esi, 1
+ 1 1 0.33 movabs rdi, 3074457345618258599
+ 1 1 0.33 cmp rdx, rdi
+ 1 1 1.00 jae .LBB8_3
+ 1 1 0.33 test al, 3
+ 1 1 1.00 jne .LBB8_2
+ 1 1 0.50 lea rdi, [rdx + 2*rdx]
+ 1 1 0.33 or rdi, 3
+ 1 1 0.33 add rdi, 9
+ 1 1 0.33 cmp rcx, rdi
+ 1 1 1.00 jne .LBB8_3
+ 1 1 1.00 U ret
+ 1 0 0.25 xor esi, esi
+ 1 1 0.33 sub rsp, 24
+ 1 1 1.00 * mov qword ptr [rsp], rsi
+ 1 1 1.00 * mov qword ptr [rsp + 8], rax
+ 1 1 1.00 * mov qword ptr [rsp + 16], rcx
+ 1 1 0.50 lea rdi, [rip + .Lanon.HASH.2]
+ 1 1 0.50 lea rcx, [rip + .Lanon.HASH.1]
+ 1 1 0.50 lea r8, [rip + .Lanon.HASH.4]
+ 1 1 0.33 mov rdx, rsp
+ 1 1 0.33 mov esi, 54
+ 4 7 1.00 * call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 6.41 6.42 4.00 6.17 3.99 4.01
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - 0.54 0.46 mov rdx, qword ptr [rdi + 16]
+ - - - - - - 0.50 0.50 mov rax, qword ptr [rdi]
+ - - - - - - 0.49 0.51 mov rcx, qword ptr [rdi + 8]
+ - - 0.52 0.47 - 0.01 - - mov esi, 1
+ - - 0.47 0.53 - - - - movabs rdi, 3074457345618258599
+ - - 0.47 0.53 - - - - cmp rdx, rdi
+ - - - - - 1.00 - - jae .LBB8_3
+ - - 0.54 0.44 - 0.02 - - test al, 3
+ - - - - - 1.00 - - jne .LBB8_2
+ - - 0.44 0.56 - - - - lea rdi, [rdx + 2*rdx]
+ - - 0.48 0.52 - - - - or rdi, 3
+ - - 0.48 0.48 - 0.04 - - add rdi, 9
+ - - 0.51 0.39 - 0.10 - - cmp rcx, rdi
+ - - - - - 1.00 - - jne .LBB8_3
+ - - - - - 1.00 - - ret
+ - - - - - - - - xor esi, esi
+ - - 0.53 0.47 - - - - sub rsp, 24
+ - - - - 1.00 - 0.52 0.48 mov qword ptr [rsp], rsi
+ - - - - 1.00 - 0.48 0.52 mov qword ptr [rsp + 8], rax
+ - - - - 1.00 - 0.52 0.48 mov qword ptr [rsp + 16], rcx
+ - - 0.49 0.51 - - - - lea rdi, [rip + .Lanon.HASH.2]
+ - - 0.46 0.54 - - - - lea rcx, [rip + .Lanon.HASH.1]
+ - - 0.54 0.46 - - - - lea r8, [rip + .Lanon.HASH.4]
+ - - 0.04 0.06 - 0.90 - - mov rdx, rsp
+ - - 0.44 0.46 - 0.10 - - mov esi, 54
+ - - - - 1.00 1.00 0.94 1.06 call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
diff --git a/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.rs b/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.rs
new file mode 100644
index 000000000000..535c67985200
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_dynamic_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_into_ref_dynamic_size(
+ source: Ref<&[u8], format::LocoPacket>,
+) -> &format::LocoPacket {
+ Ref::into_ref(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64 b/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64
new file mode 100644
index 000000000000..81624de69046
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64
@@ -0,0 +1,27 @@
+bench_ref_type_into_ref_dynamic_size:
+ mov rdx, qword ptr [rdi + 16]
+ mov rax, qword ptr [rdi]
+ mov rcx, qword ptr [rdi + 8]
+ mov esi, 1
+ movabs rdi, 4611686018427387902
+ cmp rdx, rdi
+ jae .LBB8_3
+ test al, 1
+ jne .LBB8_2
+ lea rdi, [2*rdx + 4]
+ cmp rcx, rdi
+ jne .LBB8_3
+ ret
+.LBB8_2:
+ xor esi, esi
+.LBB8_3:
+ sub rsp, 24
+ mov qword ptr [rsp], rsi
+ mov qword ptr [rsp + 8], rax
+ mov qword ptr [rsp + 16], rcx
+ lea rdi, [rip + .Lanon.HASH.2]
+ lea rcx, [rip + .Lanon.HASH.1]
+ lea r8, [rip + .Lanon.HASH.4]
+ mov rdx, rsp
+ mov esi, 54
+ call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
diff --git a/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64.mca b/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64.mca
new file mode 100644
index 000000000000..03137756a8b5
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_ref_dynamic_size.x86-64.mca
@@ -0,0 +1,87 @@
+Iterations: 100
+Instructions: 2400
+Total Cycles: 1647
+Total uOps: 2700
+
+Dispatch Width: 4
+uOps Per Cycle: 1.64
+IPC: 1.46
+Block RThroughput: 6.8
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rdx, qword ptr [rdi + 16]
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov rcx, qword ptr [rdi + 8]
+ 1 1 0.33 mov esi, 1
+ 1 1 0.33 movabs rdi, 4611686018427387902
+ 1 1 0.33 cmp rdx, rdi
+ 1 1 1.00 jae .LBB8_3
+ 1 1 0.33 test al, 1
+ 1 1 1.00 jne .LBB8_2
+ 1 1 0.50 lea rdi, [2*rdx + 4]
+ 1 1 0.33 cmp rcx, rdi
+ 1 1 1.00 jne .LBB8_3
+ 1 1 1.00 U ret
+ 1 0 0.25 xor esi, esi
+ 1 1 0.33 sub rsp, 24
+ 1 1 1.00 * mov qword ptr [rsp], rsi
+ 1 1 1.00 * mov qword ptr [rsp + 8], rax
+ 1 1 1.00 * mov qword ptr [rsp + 16], rcx
+ 1 1 0.50 lea rdi, [rip + .Lanon.HASH.2]
+ 1 1 0.50 lea rcx, [rip + .Lanon.HASH.1]
+ 1 1 0.50 lea r8, [rip + .Lanon.HASH.4]
+ 1 1 0.33 mov rdx, rsp
+ 1 1 0.33 mov esi, 54
+ 4 7 1.00 * call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 5.15 5.74 4.00 6.11 3.98 4.02
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - 0.71 0.29 mov rdx, qword ptr [rdi + 16]
+ - - - - - - 0.72 0.28 mov rax, qword ptr [rdi]
+ - - - - - - 0.29 0.71 mov rcx, qword ptr [rdi + 8]
+ - - 0.83 0.16 - 0.01 - - mov esi, 1
+ - - 0.16 0.84 - - - - movabs rdi, 4611686018427387902
+ - - 0.80 0.04 - 0.16 - - cmp rdx, rdi
+ - - - - - 1.00 - - jae .LBB8_3
+ - - 0.13 0.74 - 0.13 - - test al, 1
+ - - - - - 1.00 - - jne .LBB8_2
+ - - 0.10 0.90 - - - - lea rdi, [2*rdx + 4]
+ - - 0.09 0.91 - - - - cmp rcx, rdi
+ - - - - - 1.00 - - jne .LBB8_3
+ - - - - - 1.00 - - ret
+ - - - - - - - - xor esi, esi
+ - - 0.29 0.13 - 0.58 - - sub rsp, 24
+ - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp], rsi
+ - - - - 1.00 - 0.28 0.72 mov qword ptr [rsp + 8], rax
+ - - - - 1.00 - 0.72 0.28 mov qword ptr [rsp + 16], rcx
+ - - 0.76 0.24 - - - - lea rdi, [rip + .Lanon.HASH.2]
+ - - 0.82 0.18 - - - - lea rcx, [rip + .Lanon.HASH.1]
+ - - 0.92 0.08 - - - - lea r8, [rip + .Lanon.HASH.4]
+ - - 0.09 0.68 - 0.23 - - mov rdx, rsp
+ - - 0.16 0.84 - - - - mov esi, 54
+ - - - - 1.00 1.00 0.54 1.46 call qword ptr [rip + core::result::unwrap_failed@GOTPCREL]
diff --git a/rust/zerocopy/benches/ref_type_into_ref_static_size.rs b/rust/zerocopy/benches/ref_type_into_ref_static_size.rs
new file mode 100644
index 000000000000..a2bdf559a903
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_ref_static_size.rs
@@ -0,0 +1,19 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_into_ref_static_size(
+ source: Ref<&[u8], format::LocoPacket>,
+) -> &format::LocoPacket {
+ Ref::into_ref(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64 b/rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64
new file mode 100644
index 000000000000..173562c46561
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64
@@ -0,0 +1,3 @@
+bench_ref_type_into_ref_static_size:
+ mov rax, qword ptr [rdi]
+ ret
diff --git a/rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64.mca
new file mode 100644
index 000000000000..2de0e6f9791c
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_into_ref_static_size.x86-64.mca
@@ -0,0 +1,43 @@
+Iterations: 100
+Instructions: 200
+Total Cycles: 103
+Total uOps: 200
+
+Dispatch Width: 4
+uOps Per Cycle: 1.94
+IPC: 1.94
+Block RThroughput: 1.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - - - - 1.00 0.50 0.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - 0.50 0.50 mov rax, qword ptr [rdi]
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_read_static_size.rs b/rust/zerocopy/benches/ref_type_read_static_size.rs
new file mode 100644
index 000000000000..da7111261b80
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_read_static_size.rs
@@ -0,0 +1,17 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_read_static_size(source: &Ref<&[u8], format::LocoPacket>) -> format::LocoPacket {
+ Ref::read(source)
+}
diff --git a/rust/zerocopy/benches/ref_type_read_static_size.x86-64 b/rust/zerocopy/benches/ref_type_read_static_size.x86-64
new file mode 100644
index 000000000000..c5ebb527f419
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_read_static_size.x86-64
@@ -0,0 +1,7 @@
+bench_ref_type_read_static_size:
+ mov rax, qword ptr [rdi]
+ mov ecx, dword ptr [rax]
+ movzx eax, word ptr [rax + 4]
+ shl rax, 32
+ or rax, rcx
+ ret
diff --git a/rust/zerocopy/benches/ref_type_read_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_read_static_size.x86-64.mca
new file mode 100644
index 000000000000..37cfa368ce16
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_read_static_size.x86-64.mca
@@ -0,0 +1,51 @@
+Iterations: 100
+Instructions: 600
+Total Cycles: 169
+Total uOps: 600
+
+Dispatch Width: 4
+uOps Per Cycle: 3.55
+IPC: 3.55
+Block RThroughput: 1.5
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 5 0.50 * mov ecx, dword ptr [rax]
+ 1 5 0.50 * movzx eax, word ptr [rax + 4]
+ 1 1 0.50 shl rax, 32
+ 1 1 0.33 or rax, rcx
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 0.91 0.90 - 1.19 1.50 1.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - 0.50 0.50 mov rax, qword ptr [rdi]
+ - - - - - - 0.31 0.69 mov ecx, dword ptr [rax]
+ - - - - - - 0.69 0.31 movzx eax, word ptr [rax + 4]
+ - - 0.91 - - 0.09 - - shl rax, 32
+ - - - 0.90 - 0.10 - - or rax, rcx
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/ref_type_write_static_size.rs b/rust/zerocopy/benches/ref_type_write_static_size.rs
new file mode 100644
index 000000000000..6537cc4609e9
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_write_static_size.rs
@@ -0,0 +1,20 @@
+// Copyright 2026 The Fuchsia Authors
+//
+// Licensed under a BSD-style license <LICENSE-BSD>, Apache License, Version 2.0
+// <LICENSE-APACHE or https://www.apache.org/licenses/LICENSE-2.0>, or the MIT
+// license <LICENSE-MIT or https://opensource.org/licenses/MIT>, at your option.
+// This file may not be copied, modified, or distributed except according to
+// those terms.
+
+use zerocopy::Ref;
+
+#[path = "formats/coco_static_size.rs"]
+mod format;
+
+#[unsafe(no_mangle)]
+fn bench_ref_type_write_static_size(
+ destination: &mut Ref<&mut [u8], format::LocoPacket>,
+ value: format::LocoPacket,
+) {
+ Ref::write(destination, value)
+}
diff --git a/rust/zerocopy/benches/ref_type_write_static_size.x86-64 b/rust/zerocopy/benches/ref_type_write_static_size.x86-64
new file mode 100644
index 000000000000..1b919d01409a
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_write_static_size.x86-64
@@ -0,0 +1,6 @@
+bench_ref_type_write_static_size:
+ mov rax, qword ptr [rdi]
+ mov dword ptr [rax], esi
+ shr rsi, 32
+ mov word ptr [rax + 4], si
+ ret
diff --git a/rust/zerocopy/benches/ref_type_write_static_size.x86-64.mca b/rust/zerocopy/benches/ref_type_write_static_size.x86-64.mca
new file mode 100644
index 000000000000..a7089d3b1336
--- /dev/null
+++ b/rust/zerocopy/benches/ref_type_write_static_size.x86-64.mca
@@ -0,0 +1,49 @@
+Iterations: 100
+Instructions: 500
+Total Cycles: 208
+Total uOps: 500
+
+Dispatch Width: 4
+uOps Per Cycle: 2.40
+IPC: 2.40
+Block RThroughput: 2.0
+
+
+Instruction Info:
+[1]: #uOps
+[2]: Latency
+[3]: RThroughput
+[4]: MayLoad
+[5]: MayStore
+[6]: HasSideEffects (U)
+
+[1] [2] [3] [4] [5] [6] Instructions:
+ 1 5 0.50 * mov rax, qword ptr [rdi]
+ 1 1 1.00 * mov dword ptr [rax], esi
+ 1 1 0.50 shr rsi, 32
+ 1 1 1.00 * mov word ptr [rax + 4], si
+ 1 1 1.00 U ret
+
+
+Resources:
+[0] - SBDivider
+[1] - SBFPDivider
+[2] - SBPort0
+[3] - SBPort1
+[4] - SBPort4
+[5] - SBPort5
+[6.0] - SBPort23
+[6.1] - SBPort23
+
+
+Resource pressure per iteration:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1]
+ - - 0.99 - 2.00 1.01 1.50 1.50
+
+Resource pressure by instruction:
+[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - - - - - 0.50 0.50 mov rax, qword ptr [rdi]
+ - - - - 1.00 - 0.33 0.67 mov dword ptr [rax], esi
+ - - 0.99 - - 0.01 - - shr rsi, 32
+ - - - - 1.00 - 0.67 0.33 mov word ptr [rax + 4], si
+ - - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64 b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64
index 03684cb09f8f..0a8a4f501a07 100644
--- a/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64
+++ b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64
@@ -1,9 +1,8 @@
bench_split_via_runtime_check_dynamic_padding:
mov rax, rdi
mov rdx, qword ptr [rsi + 16]
- mov ecx, edx
- and ecx, 3
- cmp ecx, 1
+ lea ecx, [rdx - 1]
+ test cl, 3
jne .LBB5_1
mov rcx, qword ptr [rsi]
mov rsi, qword ptr [rsi + 8]
diff --git a/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64.mca
index 5034ab0583bd..7140b06e8842 100644
--- a/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64.mca
+++ b/rust/zerocopy/benches/split_via_runtime_check_dynamic_padding.x86-64.mca
@@ -1,11 +1,11 @@
Iterations: 100
-Instructions: 2000
+Instructions: 1900
Total Cycles: 708
-Total uOps: 2000
+Total uOps: 1900
Dispatch Width: 4
-uOps Per Cycle: 2.82
-IPC: 2.82
+uOps Per Cycle: 2.68
+IPC: 2.68
Block RThroughput: 5.0
@@ -20,9 +20,8 @@ Instruction Info:
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 mov rax, rdi
1 5 0.50 * mov rdx, qword ptr [rsi + 16]
- 1 1 0.33 mov ecx, edx
- 1 1 0.33 and ecx, 3
- 1 1 0.33 cmp ecx, 1
+ 1 1 0.50 lea ecx, [rdx - 1]
+ 1 1 0.33 test cl, 3
1 1 1.00 jne .LBB5_1
1 5 0.50 * mov rcx, qword ptr [rsi]
1 5 0.50 * mov rsi, qword ptr [rsi + 8]
@@ -53,22 +52,21 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 3.00 3.02 5.00 4.98 4.00 4.00
+ - - 2.99 3.00 5.00 4.01 4.00 4.00
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - - 0.99 - 0.01 - - mov rax, rdi
+ - - 0.99 - - 0.01 - - mov rax, rdi
- - - - - - - 1.00 mov rdx, qword ptr [rsi + 16]
- - - 0.99 0.01 - - - - mov ecx, edx
- - - 0.99 0.01 - - - - and ecx, 3
- - - 0.97 0.03 - - - - cmp ecx, 1
+ - - - 1.00 - - - - lea ecx, [rdx - 1]
+ - - - 0.98 - 0.02 - - test cl, 3
- - - - - 1.00 - - jne .LBB5_1
- - - - - - 1.00 - mov rcx, qword ptr [rsi]
- - - - - - 0.99 0.01 mov rsi, qword ptr [rsi + 8]
- - - 0.01 0.99 - - - - lea rdi, [rdx + 2*rdx]
- - - - 0.96 - 0.04 - - add rdi, rcx
- - - 0.03 - - 0.97 - - add rdi, 9
- - - 0.01 0.03 - 0.96 - - sub rsi, rdx
+ - - 1.00 - - - - - lea rdi, [rdx + 2*rdx]
+ - - 0.01 0.01 - 0.98 - - add rdi, rcx
+ - - - 1.00 - - - - add rdi, 9
+ - - 0.99 0.01 - - - - sub rsi, rdx
- - - - 1.00 - 1.00 - mov qword ptr [rax + 8], rdx
- - - - 1.00 - - 1.00 mov qword ptr [rax + 16], rdi
- - - - 1.00 - 0.01 0.99 mov qword ptr [rax + 24], rsi
diff --git a/rust/zerocopy/benches/try_read_from_bytes.x86-64 b/rust/zerocopy/benches/try_read_from_bytes.x86-64
index 08088a08fd85..2135facc2655 100644
--- a/rust/zerocopy/benches/try_read_from_bytes.x86-64
+++ b/rust/zerocopy/benches/try_read_from_bytes.x86-64
@@ -1,23 +1,11 @@
bench_try_read_from_bytes_static_size:
- mov ax, -16191
+ mov eax, 49345
cmp rsi, 6
- jne .LBB5_1
- mov ecx, dword ptr [rdi]
- movzx edx, cx
- cmp edx, 49344
- jne .LBB5_4
- movzx eax, word ptr [rdi + 4]
- shl rax, 32
- or rcx, rax
- shr rcx, 16
- mov ax, -16192
-.LBB5_4:
- shl rcx, 16
- movzx eax, ax
- or rax, rcx
- ret
-.LBB5_1:
- shl rcx, 16
- movzx eax, ax
- or rax, rcx
+ jne .LBB5_3
+ cmp word ptr [rdi], -16192
+ jne .LBB5_3
+ mov eax, dword ptr [rdi + 2]
+ shl rax, 16
+ or rax, 49344
+.LBB5_3:
ret
diff --git a/rust/zerocopy/benches/try_read_from_bytes.x86-64.mca b/rust/zerocopy/benches/try_read_from_bytes.x86-64.mca
index 385e6a480253..f5568fef5ae7 100644
--- a/rust/zerocopy/benches/try_read_from_bytes.x86-64.mca
+++ b/rust/zerocopy/benches/try_read_from_bytes.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 2000
-Total Cycles: 608
-Total uOps: 2000
+Instructions: 900
+Total Cycles: 305
+Total uOps: 1000
Dispatch Width: 4
-uOps Per Cycle: 3.29
-IPC: 3.29
-Block RThroughput: 5.0
+uOps Per Cycle: 3.28
+IPC: 2.95
+Block RThroughput: 3.0
Instruction Info:
@@ -18,25 +18,14 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
- 1 1 0.33 mov ax, -16191
+ 1 1 0.33 mov eax, 49345
1 1 0.33 cmp rsi, 6
- 1 1 1.00 jne .LBB5_1
- 1 5 0.50 * mov ecx, dword ptr [rdi]
- 1 1 0.33 movzx edx, cx
- 1 1 0.33 cmp edx, 49344
- 1 1 1.00 jne .LBB5_4
- 1 5 0.50 * movzx eax, word ptr [rdi + 4]
- 1 1 0.50 shl rax, 32
- 1 1 0.33 or rcx, rax
- 1 1 0.50 shr rcx, 16
- 1 1 0.33 mov ax, -16192
- 1 1 0.50 shl rcx, 16
- 1 1 0.33 movzx eax, ax
- 1 1 0.33 or rax, rcx
- 1 1 1.00 U ret
- 1 1 0.50 shl rcx, 16
- 1 1 0.33 movzx eax, ax
- 1 1 0.33 or rax, rcx
+ 1 1 1.00 jne .LBB5_3
+ 2 6 0.50 * cmp word ptr [rdi], -16192
+ 1 1 1.00 jne .LBB5_3
+ 1 5 0.50 * mov eax, dword ptr [rdi + 2]
+ 1 1 0.50 shl rax, 16
+ 1 1 0.33 or rax, 49344
1 1 1.00 U ret
@@ -53,27 +42,16 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 5.99 5.99 - 6.02 1.00 1.00
+ - - 2.50 2.49 - 3.01 1.00 1.00
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - - 0.99 - 0.01 - - mov ax, -16191
- - - - 0.01 - 0.99 - - cmp rsi, 6
- - - - - - 1.00 - - jne .LBB5_1
- - - - - - - - 1.00 mov ecx, dword ptr [rdi]
- - - 0.98 - - 0.02 - - movzx edx, cx
- - - 0.99 0.01 - - - - cmp edx, 49344
- - - - - - 1.00 - - jne .LBB5_4
- - - - - - - 1.00 - movzx eax, word ptr [rdi + 4]
- - - 0.01 - - 0.99 - - shl rax, 32
- - - 0.02 0.98 - - - - or rcx, rax
- - - 1.00 - - - - - shr rcx, 16
- - - 0.99 0.01 - - - - mov ax, -16192
- - - 1.00 - - - - - shl rcx, 16
- - - - 1.00 - - - - movzx eax, ax
- - - - 1.00 - - - - or rax, rcx
- - - - - - 1.00 - - ret
- - - 1.00 - - - - - shl rcx, 16
- - - - 1.00 - - - - movzx eax, ax
- - - - 0.99 - 0.01 - - or rax, rcx
+ - - 0.50 0.49 - 0.01 - - mov eax, 49345
+ - - 0.49 0.51 - - - - cmp rsi, 6
+ - - - - - 1.00 - - jne .LBB5_3
+ - - 0.47 0.53 - - 0.43 0.57 cmp word ptr [rdi], -16192
+ - - - - - 1.00 - - jne .LBB5_3
+ - - - - - - 0.57 0.43 mov eax, dword ptr [rdi + 2]
+ - - 1.00 - - - - - shl rax, 16
+ - - 0.04 0.96 - - - - or rax, 49344
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_read_from_prefix.x86-64 b/rust/zerocopy/benches/try_read_from_prefix.x86-64
index d3e1edc3ea73..0f820efff5f8 100644
--- a/rust/zerocopy/benches/try_read_from_prefix.x86-64
+++ b/rust/zerocopy/benches/try_read_from_prefix.x86-64
@@ -1,16 +1,11 @@
bench_try_read_from_prefix_static_size:
mov eax, 49345
cmp rsi, 6
- jb .LBB5_2
- mov eax, dword ptr [rdi]
- movzx ecx, word ptr [rdi + 4]
- shl rcx, 32
- or rcx, rax
- movzx eax, cx
- and rcx, -65536
- or rcx, 49344
- cmp eax, 49344
- mov eax, 49345
- cmove rax, rcx
-.LBB5_2:
+ jb .LBB5_3
+ cmp word ptr [rdi], -16192
+ jne .LBB5_3
+ mov eax, dword ptr [rdi + 2]
+ shl rax, 16
+ or rax, 49344
+.LBB5_3:
ret
diff --git a/rust/zerocopy/benches/try_read_from_prefix.x86-64.mca b/rust/zerocopy/benches/try_read_from_prefix.x86-64.mca
index 40401d89e83d..e0ba4c8320e3 100644
--- a/rust/zerocopy/benches/try_read_from_prefix.x86-64.mca
+++ b/rust/zerocopy/benches/try_read_from_prefix.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 1400
-Total Cycles: 442
-Total uOps: 1500
+Instructions: 900
+Total Cycles: 305
+Total uOps: 1000
Dispatch Width: 4
-uOps Per Cycle: 3.39
-IPC: 3.17
-Block RThroughput: 3.8
+uOps Per Cycle: 3.28
+IPC: 2.95
+Block RThroughput: 3.0
Instruction Info:
@@ -20,17 +20,12 @@ Instruction Info:
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 mov eax, 49345
1 1 0.33 cmp rsi, 6
- 1 1 1.00 jb .LBB5_2
- 1 5 0.50 * mov eax, dword ptr [rdi]
- 1 5 0.50 * movzx ecx, word ptr [rdi + 4]
- 1 1 0.50 shl rcx, 32
- 1 1 0.33 or rcx, rax
- 1 1 0.33 movzx eax, cx
- 1 1 0.33 and rcx, -65536
- 1 1 0.33 or rcx, 49344
- 1 1 0.33 cmp eax, 49344
- 1 1 0.33 mov eax, 49345
- 2 2 0.67 cmove rax, rcx
+ 1 1 1.00 jb .LBB5_3
+ 2 6 0.50 * cmp word ptr [rdi], -16192
+ 1 1 1.00 jne .LBB5_3
+ 1 5 0.50 * mov eax, dword ptr [rdi + 2]
+ 1 1 0.50 shl rax, 16
+ 1 1 0.33 or rax, 49344
1 1 1.00 U ret
@@ -47,21 +42,16 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 4.33 4.33 - 4.34 1.00 1.00
+ - - 2.50 2.49 - 3.01 1.00 1.00
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.65 0.01 - 0.34 - - mov eax, 49345
- - - 0.01 0.33 - 0.66 - - cmp rsi, 6
- - - - - - 1.00 - - jb .LBB5_2
- - - - - - - - 1.00 mov eax, dword ptr [rdi]
- - - - - - - 1.00 - movzx ecx, word ptr [rdi + 4]
- - - 0.65 - - 0.35 - - shl rcx, 32
- - - - 0.67 - 0.33 - - or rcx, rax
- - - 0.01 0.99 - - - - movzx eax, cx
- - - 0.99 0.01 - - - - and rcx, -65536
- - - 0.01 0.99 - - - - or rcx, 49344
- - - 0.99 0.01 - - - - cmp eax, 49344
- - - 0.02 0.33 - 0.65 - - mov eax, 49345
- - - 1.00 0.99 - 0.01 - - cmove rax, rcx
+ - - 0.50 0.49 - 0.01 - - mov eax, 49345
+ - - 0.49 0.51 - - - - cmp rsi, 6
+ - - - - - 1.00 - - jb .LBB5_3
+ - - 0.47 0.53 - - 0.43 0.57 cmp word ptr [rdi], -16192
+ - - - - - 1.00 - - jne .LBB5_3
+ - - - - - - 0.57 0.43 mov eax, dword ptr [rdi + 2]
+ - - 1.00 - - - - - shl rax, 16
+ - - 0.04 0.96 - - - - or rax, 49344
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_read_from_suffix.x86-64 b/rust/zerocopy/benches/try_read_from_suffix.x86-64
index 095e326f0467..a91c074e1806 100644
--- a/rust/zerocopy/benches/try_read_from_suffix.x86-64
+++ b/rust/zerocopy/benches/try_read_from_suffix.x86-64
@@ -1,18 +1,11 @@
bench_try_read_from_suffix_static_size:
mov eax, 49345
cmp rsi, 6
- jb .LBB5_2
- mov eax, dword ptr [rdi + rsi - 6]
- movzx ecx, word ptr [rdi + rsi - 2]
- shl rcx, 32
- or rcx, rax
- movzx edx, cx
- xor eax, eax
- cmp edx, 49344
- cmovne rcx, rsi
- sete al
- and rcx, -65536
- xor rax, 49345
- or rax, rcx
-.LBB5_2:
+ jb .LBB5_3
+ cmp word ptr [rdi + rsi - 6], -16192
+ jne .LBB5_3
+ mov eax, dword ptr [rdi + rsi - 4]
+ shl rax, 16
+ or rax, 49344
+.LBB5_3:
ret
diff --git a/rust/zerocopy/benches/try_read_from_suffix.x86-64.mca b/rust/zerocopy/benches/try_read_from_suffix.x86-64.mca
index d3eaadbb8a81..e42f6b93effb 100644
--- a/rust/zerocopy/benches/try_read_from_suffix.x86-64.mca
+++ b/rust/zerocopy/benches/try_read_from_suffix.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 1600
-Total Cycles: 478
-Total uOps: 1700
+Instructions: 900
+Total Cycles: 305
+Total uOps: 1000
Dispatch Width: 4
-uOps Per Cycle: 3.56
-IPC: 3.35
-Block RThroughput: 4.3
+uOps Per Cycle: 3.28
+IPC: 2.95
+Block RThroughput: 3.0
Instruction Info:
@@ -20,19 +20,12 @@ Instruction Info:
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 mov eax, 49345
1 1 0.33 cmp rsi, 6
- 1 1 1.00 jb .LBB5_2
- 1 5 0.50 * mov eax, dword ptr [rdi + rsi - 6]
- 1 5 0.50 * movzx ecx, word ptr [rdi + rsi - 2]
- 1 1 0.50 shl rcx, 32
- 1 1 0.33 or rcx, rax
- 1 1 0.33 movzx edx, cx
- 1 0 0.25 xor eax, eax
- 1 1 0.33 cmp edx, 49344
- 2 2 0.67 cmovne rcx, rsi
- 1 1 0.50 sete al
- 1 1 0.33 and rcx, -65536
- 1 1 0.33 xor rax, 49345
- 1 1 0.33 or rax, rcx
+ 1 1 1.00 jb .LBB5_3
+ 2 6 0.50 * cmp word ptr [rdi + rsi - 6], -16192
+ 1 1 1.00 jne .LBB5_3
+ 1 5 0.50 * mov eax, dword ptr [rdi + rsi - 4]
+ 1 1 0.50 shl rax, 16
+ 1 1 0.33 or rax, 49344
1 1 1.00 U ret
@@ -49,23 +42,16 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 4.66 4.66 - 4.68 1.00 1.00
+ - - 2.50 2.49 - 3.01 1.00 1.00
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.32 0.01 - 0.67 - - mov eax, 49345
- - - 0.62 0.02 - 0.36 - - cmp rsi, 6
- - - - - - 1.00 - - jb .LBB5_2
- - - - - - - - 1.00 mov eax, dword ptr [rdi + rsi - 6]
- - - - - - - 1.00 - movzx ecx, word ptr [rdi + rsi - 2]
- - - 0.37 - - 0.63 - - shl rcx, 32
- - - 0.99 0.01 - - - - or rcx, rax
- - - 1.00 - - - - - movzx edx, cx
- - - - - - - - - xor eax, eax
- - - 0.35 0.64 - 0.01 - - cmp edx, 49344
- - - 1.00 1.00 - - - - cmovne rcx, rsi
- - - - - - 1.00 - - sete al
- - - 0.01 0.99 - - - - and rcx, -65536
- - - - 1.00 - - - - xor rax, 49345
- - - - 0.99 - 0.01 - - or rax, rcx
+ - - 0.50 0.49 - 0.01 - - mov eax, 49345
+ - - 0.49 0.51 - - - - cmp rsi, 6
+ - - - - - 1.00 - - jb .LBB5_3
+ - - 0.47 0.53 - - 0.43 0.57 cmp word ptr [rdi + rsi - 6], -16192
+ - - - - - 1.00 - - jne .LBB5_3
+ - - - - - - 0.57 0.43 mov eax, dword ptr [rdi + rsi - 4]
+ - - 1.00 - - - - - shl rax, 16
+ - - 0.04 0.96 - - - - or rax, 49344
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64
index 217c5fc61796..5584e9054185 100644
--- a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64
+++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64
@@ -1,24 +1,22 @@
bench_try_ref_from_bytes_dynamic_padding:
- test dil, 3
- jne .LBB5_4
- movabs rax, 9223372036854775804
- and rax, rsi
- cmp rax, 9
- jb .LBB5_4
- add rax, -9
+ cmp rsi, 12
+ setae al
+ mov ecx, esi
+ or ecx, edi
+ test cl, 3
+ sete cl
+ and cl, al
+ cmp cl, 1
+ je .LBB5_2
+ xor eax, eax
+ ret
+.LBB5_2:
+ add rsi, -9
movabs rcx, -6148914691236517205
+ mov rax, rsi
mul rcx
shr rdx
- lea rax, [rdx + 2*rdx]
- or rax, 3
- add rax, 9
- cmp rsi, rax
- jne .LBB5_4
+ xor eax, eax
cmp word ptr [rdi], -16192
- je .LBB5_5
-.LBB5_4:
- xor edi, edi
- mov rdx, rsi
-.LBB5_5:
- mov rax, rdi
+ cmove rax, rdi
ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64.mca
index 95b993c7e0a8..ccd610c7ef41 100644
--- a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64.mca
+++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_padding.x86-64.mca
@@ -1,10 +1,10 @@
Iterations: 100
-Instructions: 2100
-Total Cycles: 709
+Instructions: 2000
+Total Cycles: 675
Total uOps: 2300
Dispatch Width: 4
-uOps Per Cycle: 3.24
+uOps Per Cycle: 3.41
IPC: 2.96
Block RThroughput: 5.8
@@ -18,26 +18,25 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
- 1 1 0.33 test dil, 3
- 1 1 1.00 jne .LBB5_4
- 1 1 0.33 movabs rax, 9223372036854775804
- 1 1 0.33 and rax, rsi
- 1 1 0.33 cmp rax, 9
- 1 1 1.00 jb .LBB5_4
- 1 1 0.33 add rax, -9
+ 1 1 0.33 cmp rsi, 12
+ 1 1 0.50 setae al
+ 1 1 0.33 mov ecx, esi
+ 1 1 0.33 or ecx, edi
+ 1 1 0.33 test cl, 3
+ 1 1 0.50 sete cl
+ 1 1 0.33 and cl, al
+ 1 1 0.33 cmp cl, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor eax, eax
+ 1 1 1.00 U ret
+ 1 1 0.33 add rsi, -9
1 1 0.33 movabs rcx, -6148914691236517205
+ 1 1 0.33 mov rax, rsi
2 4 1.00 mul rcx
1 1 0.50 shr rdx
- 1 1 0.50 lea rax, [rdx + 2*rdx]
- 1 1 0.33 or rax, 3
- 1 1 0.33 add rax, 9
- 1 1 0.33 cmp rsi, rax
- 1 1 1.00 jne .LBB5_4
+ 1 0 0.25 xor eax, eax
2 6 0.50 * cmp word ptr [rdi], -16192
- 1 1 1.00 je .LBB5_5
- 1 0 0.25 xor edi, edi
- 1 1 0.33 mov rdx, rsi
- 1 1 0.33 mov rax, rdi
+ 2 2 0.67 cmove rax, rdi
1 1 1.00 U ret
@@ -54,28 +53,27 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 6.98 6.99 - 7.03 0.50 0.50
+ - - 6.67 6.66 - 6.67 0.50 0.50
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.48 0.51 - 0.01 - - test dil, 3
- - - - - - 1.00 - - jne .LBB5_4
- - - 0.51 0.49 - - - - movabs rax, 9223372036854775804
- - - 0.01 0.99 - - - - and rax, rsi
- - - 0.51 0.49 - - - - cmp rax, 9
- - - - - - 1.00 - - jb .LBB5_4
- - - 0.98 - - 0.02 - - add rax, -9
- - - 0.98 0.02 - - - - movabs rcx, -6148914691236517205
+ - - 0.33 0.66 - 0.01 - - cmp rsi, 12
+ - - 1.00 - - - - - setae al
+ - - 0.66 0.34 - - - - mov ecx, esi
+ - - - 0.99 - 0.01 - - or ecx, edi
+ - - - 0.67 - 0.33 - - test cl, 3
+ - - 0.67 - - 0.33 - - sete cl
+ - - 0.33 0.34 - 0.33 - - and cl, al
+ - - 0.33 - - 0.67 - - cmp cl, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor eax, eax
+ - - - - - 1.00 - - ret
+ - - 0.67 0.33 - - - - add rsi, -9
+ - - 0.99 0.01 - - - - movabs rcx, -6148914691236517205
+ - - - 0.99 - 0.01 - - mov rax, rsi
- - 1.00 1.00 - - - - mul rcx
- - - 0.99 - - 0.01 - - shr rdx
- - - - 1.00 - - - - lea rax, [rdx + 2*rdx]
- - - - 0.51 - 0.49 - - or rax, 3
- - - 0.01 0.49 - 0.50 - - add rax, 9
- - - - 0.02 - 0.98 - - cmp rsi, rax
- - - - - - 1.00 - - jne .LBB5_4
- - - 0.51 0.49 - - 0.50 0.50 cmp word ptr [rdi], -16192
- - - - - - 1.00 - - je .LBB5_5
- - - - - - - - - xor edi, edi
- - - 0.50 0.50 - - - - mov rdx, rsi
- - - 0.50 0.48 - 0.02 - - mov rax, rdi
+ - - 0.01 - - 0.99 - - shr rdx
+ - - - - - - - - xor eax, eax
+ - - 0.67 0.33 - - 0.50 0.50 cmp word ptr [rdi], -16192
+ - - 0.01 1.00 - 0.99 - - cmove rax, rdi
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64 b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64
index cf67afd31ce0..39b12e14eec1 100644
--- a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64
+++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64
@@ -1,22 +1,21 @@
bench_try_ref_from_bytes_dynamic_size:
- mov rdx, rsi
- mov rax, rdi
cmp rsi, 4
- setb cl
- or cl, al
+ setae al
+ mov ecx, esi
+ or ecx, edi
test cl, 1
- jne .LBB5_4
- lea rcx, [rdx - 4]
- mov rsi, rcx
- and rsi, -2
- add rsi, 4
- cmp rdx, rsi
- jne .LBB5_4
- cmp word ptr [rax], -16192
- jne .LBB5_4
- shr rcx
- mov rdx, rcx
+ sete cl
+ and cl, al
+ cmp cl, 1
+ je .LBB5_2
+ xor eax, eax
ret
-.LBB5_4:
+.LBB5_2:
+ lea rdx, [rsi - 4]
+ shr rdx
+ movzx ecx, word ptr [rdi]
xor eax, eax
+ cmp ecx, 49344
+ cmovne rdx, rsi
+ cmove rax, rdi
ret
diff --git a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64.mca
index ecd7a18f6d6d..eecf3a1554cb 100644
--- a/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64.mca
+++ b/rust/zerocopy/benches/try_ref_from_bytes_dynamic_size.x86-64.mca
@@ -1,10 +1,10 @@
Iterations: 100
-Instructions: 2000
-Total Cycles: 639
+Instructions: 1900
+Total Cycles: 607
Total uOps: 2100
Dispatch Width: 4
-uOps Per Cycle: 3.29
+uOps Per Cycle: 3.46
IPC: 3.13
Block RThroughput: 5.3
@@ -18,25 +18,24 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
- 1 1 0.33 mov rdx, rsi
- 1 1 0.33 mov rax, rdi
1 1 0.33 cmp rsi, 4
- 1 1 0.50 setb cl
- 1 1 0.33 or cl, al
+ 1 1 0.50 setae al
+ 1 1 0.33 mov ecx, esi
+ 1 1 0.33 or ecx, edi
1 1 0.33 test cl, 1
- 1 1 1.00 jne .LBB5_4
- 1 1 0.50 lea rcx, [rdx - 4]
- 1 1 0.33 mov rsi, rcx
- 1 1 0.33 and rsi, -2
- 1 1 0.33 add rsi, 4
- 1 1 0.33 cmp rdx, rsi
- 1 1 1.00 jne .LBB5_4
- 2 6 0.50 * cmp word ptr [rax], -16192
- 1 1 1.00 jne .LBB5_4
- 1 1 0.50 shr rcx
- 1 1 0.33 mov rdx, rcx
+ 1 1 0.50 sete cl
+ 1 1 0.33 and cl, al
+ 1 1 0.33 cmp cl, 1
+ 1 1 1.00 je .LBB5_2
+ 1 0 0.25 xor eax, eax
1 1 1.00 U ret
+ 1 1 0.50 lea rdx, [rsi - 4]
+ 1 1 0.50 shr rdx
+ 1 5 0.50 * movzx ecx, word ptr [rdi]
1 0 0.25 xor eax, eax
+ 1 1 0.33 cmp ecx, 49344
+ 2 2 0.67 cmovne rdx, rsi
+ 2 2 0.67 cmove rax, rdi
1 1 1.00 U ret
@@ -53,27 +52,26 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 6.32 6.32 - 6.36 0.50 0.50
+ - - 5.99 5.99 - 6.02 0.50 0.50
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.33 0.66 - 0.01 - - mov rdx, rsi
- - - 0.66 0.34 - - - - mov rax, rdi
- - - 0.34 0.66 - - - - cmp rsi, 4
- - - 0.99 - - 0.01 - - setb cl
- - - 0.01 0.99 - - - - or cl, al
- - - - 1.00 - - - - test cl, 1
- - - - - - 1.00 - - jne .LBB5_4
- - - 0.66 0.34 - - - - lea rcx, [rdx - 4]
- - - 0.33 0.66 - 0.01 - - mov rsi, rcx
- - - 1.00 - - - - - and rsi, -2
- - - 0.66 0.34 - - - - add rsi, 4
- - - - 1.00 - - - - cmp rdx, rsi
- - - - - - 1.00 - - jne .LBB5_4
- - - - - - 1.00 0.50 0.50 cmp word ptr [rax], -16192
- - - - - - 1.00 - - jne .LBB5_4
- - - 0.67 - - 0.33 - - shr rcx
- - - 0.67 0.33 - - - - mov rdx, rcx
+ - - 0.02 0.97 - 0.01 - - cmp rsi, 4
+ - - 1.00 - - - - - setae al
+ - - - 1.00 - - - - mov ecx, esi
+ - - 0.93 0.05 - 0.02 - - or ecx, edi
+ - - 0.94 0.06 - - - - test cl, 1
+ - - 0.99 - - 0.01 - - sete cl
+ - - 0.98 0.02 - - - - and cl, al
+ - - 0.98 0.02 - - - - cmp cl, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - - - - - - - xor eax, eax
- - - - - 1.00 - - ret
+ - - 0.01 0.99 - - - - lea rdx, [rsi - 4]
+ - - - - - 1.00 - - shr rdx
+ - - - - - - 0.50 0.50 movzx ecx, word ptr [rdi]
- - - - - - - - xor eax, eax
+ - - 0.02 0.96 - 0.02 - - cmp ecx, 49344
+ - - 0.05 0.97 - 0.98 - - cmovne rdx, rsi
+ - - 0.07 0.95 - 0.98 - - cmove rax, rdi
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64
index d832cb7ecf7f..ff64bb659591 100644
--- a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64
+++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64
@@ -1,19 +1,18 @@
bench_try_ref_from_prefix_dynamic_padding:
+ test dil, 3
+ sete al
+ cmp rsi, 12
+ setae cl
+ test al, cl
+ jne .LBB5_2
xor edx, edx
- mov eax, 0
test dil, 3
- je .LBB5_1
- ret
-.LBB5_1:
- movabs rax, 9223372036854775804
- and rsi, rax
- cmp rsi, 9
- jae .LBB5_3
- mov edx, 1
+ sete dl
xor eax, eax
ret
-.LBB5_3:
- add rsi, -9
+.LBB5_2:
+ or rsi, 3
+ add rsi, -12
movabs rcx, -6148914691236517205
mov rax, rsi
mul rcx
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64.mca
index 482112a39b33..29225404a28f 100644
--- a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64.mca
+++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_padding.x86-64.mca
@@ -18,19 +18,19 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
+ 1 1 0.33 test dil, 3
+ 1 1 0.50 sete al
+ 1 1 0.33 cmp rsi, 12
+ 1 1 0.50 setae cl
+ 1 1 0.33 test al, cl
+ 1 1 1.00 jne .LBB5_2
1 0 0.25 xor edx, edx
- 1 1 0.33 mov eax, 0
1 1 0.33 test dil, 3
- 1 1 1.00 je .LBB5_1
- 1 1 1.00 U ret
- 1 1 0.33 movabs rax, 9223372036854775804
- 1 1 0.33 and rsi, rax
- 1 1 0.33 cmp rsi, 9
- 1 1 1.00 jae .LBB5_3
- 1 1 0.33 mov edx, 1
+ 1 1 0.50 sete dl
1 0 0.25 xor eax, eax
1 1 1.00 U ret
- 1 1 0.33 add rsi, -9
+ 1 1 0.33 or rsi, 3
+ 1 1 0.33 add rsi, -12
1 1 0.33 movabs rcx, -6148914691236517205
1 1 0.33 mov rax, rsi
2 4 1.00 mul rcx
@@ -63,29 +63,29 @@ Resource pressure per iteration:
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
+ - - 0.01 0.98 - 0.01 - - test dil, 3
+ - - 0.98 - - 0.02 - - sete al
+ - - - 1.00 - - - - cmp rsi, 12
+ - - - - - 1.00 - - setae cl
+ - - 0.01 0.03 - 0.96 - - test al, cl
+ - - - - - 1.00 - - jne .LBB5_2
- - - - - - - - xor edx, edx
- - - 0.32 0.34 - 0.34 - - mov eax, 0
- - - 0.34 0.33 - 0.33 - - test dil, 3
- - - - - - 1.00 - - je .LBB5_1
- - - - - - 1.00 - - ret
- - - 0.35 0.65 - - - - movabs rax, 9223372036854775804
- - - 0.96 0.03 - 0.01 - - and rsi, rax
- - - 0.01 0.97 - 0.02 - - cmp rsi, 9
- - - - - - 1.00 - - jae .LBB5_3
- - - 0.67 0.01 - 0.32 - - mov edx, 1
+ - - - 1.00 - - - - test dil, 3
+ - - 0.99 - - 0.01 - - sete dl
- - - - - - - - xor eax, eax
- - - - - 1.00 - - ret
- - - 0.02 0.34 - 0.64 - - add rsi, -9
- - - 0.33 0.66 - 0.01 - - movabs rcx, -6148914691236517205
- - - 0.66 0.34 - - - - mov rax, rsi
+ - - - 1.00 - - - - or rsi, 3
+ - - 0.01 0.99 - - - - add rsi, -12
+ - - 0.01 0.99 - - - - movabs rcx, -6148914691236517205
+ - - 0.34 0.66 - - - - mov rax, rsi
- - 1.00 1.00 - - - - mul rcx
- - - 0.01 0.99 - - - - mov rax, rdx
- - - 0.99 - - 0.01 - - shr rax
+ - - 0.99 - - 0.01 - - mov rax, rdx
+ - - 1.00 - - - - - shr rax
- - - - - - 0.50 0.50 movzx ecx, word ptr [rdi]
- - - 0.33 0.03 - 0.64 - - cmp cx, -16192
- - - 0.01 0.31 - 0.68 - - mov edx, 2
- - - 1.00 1.00 - - - - cmove rdx, rax
+ - - 0.34 0.32 - 0.34 - - cmp cx, -16192
+ - - 0.33 0.01 - 0.66 - - mov edx, 2
+ - - 0.99 0.01 - 1.00 - - cmove rdx, rax
- - - - - - - - xor eax, eax
- - 0.33 0.33 - 0.34 - - cmp ecx, 49344
- - - 1.00 1.00 - - - - cmove rax, rdi
+ - - 1.00 0.01 - 0.99 - - cmove rax, rdi
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64 b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64
index be7f34b9f8f5..de31f571f3c9 100644
--- a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64
+++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64
@@ -1,14 +1,15 @@
bench_try_ref_from_prefix_dynamic_size:
- xor edx, edx
- mov eax, 0
- test dil, 1
- jne .LBB5_4
cmp rsi, 4
- jae .LBB5_3
- mov edx, 1
+ setb al
+ or al, dil
+ test al, 1
+ je .LBB5_2
+ not edi
+ and edi, 1
xor eax, eax
+ mov rdx, rdi
ret
-.LBB5_3:
+.LBB5_2:
add rsi, -4
shr rsi
movzx ecx, word ptr [rdi]
@@ -18,5 +19,4 @@ bench_try_ref_from_prefix_dynamic_size:
xor eax, eax
cmp cx, -16192
cmove rax, rdi
-.LBB5_4:
ret
diff --git a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64.mca
index 11706defe11e..cd0919d04c08 100644
--- a/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64.mca
+++ b/rust/zerocopy/benches/try_ref_from_prefix_dynamic_size.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 1900
-Total Cycles: 573
-Total uOps: 2100
+Instructions: 2000
+Total Cycles: 643
+Total uOps: 2200
Dispatch Width: 4
-uOps Per Cycle: 3.66
-IPC: 3.32
-Block RThroughput: 5.3
+uOps Per Cycle: 3.42
+IPC: 3.11
+Block RThroughput: 5.5
Instruction Info:
@@ -18,14 +18,15 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
- 1 0 0.25 xor edx, edx
- 1 1 0.33 mov eax, 0
- 1 1 0.33 test dil, 1
- 1 1 1.00 jne .LBB5_4
1 1 0.33 cmp rsi, 4
- 1 1 1.00 jae .LBB5_3
- 1 1 0.33 mov edx, 1
+ 1 1 0.50 setb al
+ 1 1 0.33 or al, dil
+ 1 1 0.33 test al, 1
+ 1 1 1.00 je .LBB5_2
+ 1 1 0.33 not edi
+ 1 1 0.33 and edi, 1
1 0 0.25 xor eax, eax
+ 1 1 0.33 mov rdx, rdi
1 1 1.00 U ret
1 1 0.33 add rsi, -4
1 1 0.50 shr rsi
@@ -52,26 +53,27 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 5.66 5.67 - 5.67 0.50 0.50
+ - - 6.33 6.33 - 6.34 0.50 0.50
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - - - - - - - xor edx, edx
- - - 0.30 0.37 - 0.33 - - mov eax, 0
- - - 0.35 0.32 - 0.33 - - test dil, 1
- - - - - - 1.00 - - jne .LBB5_4
- - - 0.32 0.33 - 0.35 - - cmp rsi, 4
- - - - - - 1.00 - - jae .LBB5_3
- - - 0.33 0.35 - 0.32 - - mov edx, 1
+ - - 0.67 0.32 - 0.01 - - cmp rsi, 4
+ - - 0.99 - - 0.01 - - setb al
+ - - 0.19 0.48 - 0.33 - - or al, dil
+ - - 0.50 0.16 - 0.34 - - test al, 1
+ - - - - - 1.00 - - je .LBB5_2
+ - - 0.50 0.50 - - - - not edi
+ - - 0.49 0.34 - 0.17 - - and edi, 1
- - - - - - - - xor eax, eax
+ - - 0.16 0.84 - - - - mov rdx, rdi
- - - - - 1.00 - - ret
- - - 0.34 0.64 - 0.02 - - add rsi, -4
- - - 1.00 - - - - - shr rsi
+ - - 0.68 0.32 - - - - add rsi, -4
+ - - 0.85 - - 0.15 - - shr rsi
- - - - - - 0.50 0.50 movzx ecx, word ptr [rdi]
- - - 0.60 0.40 - - - - cmp ecx, 49344
- - - 0.05 0.95 - - - - mov edx, 2
- - - 1.00 1.00 - - - - cmove rdx, rsi
+ - - - 0.50 - 0.50 - - cmp ecx, 49344
+ - - 0.63 0.37 - - - - mov edx, 2
+ - - 0.17 1.00 - 0.83 - - cmove rdx, rsi
- - - - - - - - xor eax, eax
- - - 0.37 0.31 - 0.32 - - cmp cx, -16192
- - - 1.00 1.00 - - - - cmove rax, rdi
+ - - 0.50 0.50 - - - - cmp cx, -16192
+ - - - 1.00 - 1.00 - - cmove rax, rdi
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64
index b3e924442865..cee9a3179097 100644
--- a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64
+++ b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64
@@ -1,26 +1,23 @@
bench_try_ref_from_suffix_dynamic_padding:
lea eax, [rsi + rdi]
test al, 3
- jne .LBB5_1
- movabs rax, 9223372036854775804
- and rax, rsi
- cmp rax, 9
- jae .LBB5_3
-.LBB5_1:
+ setne al
+ cmp rsi, 12
+ setb cl
+ or cl, al
+ je .LBB5_2
xor eax, eax
ret
-.LBB5_3:
- add rax, -9
+.LBB5_2:
+ mov rax, rsi
+ or rax, 3
+ add rax, -12
movabs rcx, -6148914691236517205
mul rcx
shr rdx
- lea rcx, [rdx + 2*rdx]
- sub rsi, rcx
- or rcx, -4
- add rsi, rdi
- lea rdi, [rcx + rsi]
- add rdi, -8
+ and esi, 3
+ lea rcx, [rdi + rsi]
xor eax, eax
- cmp word ptr [rcx + rsi - 8], -16192
- cmove rax, rdi
+ cmp word ptr [rdi + rsi], -16192
+ cmove rax, rcx
ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64.mca
index d56ae56d854a..92df4484b823 100644
--- a/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64.mca
+++ b/rust/zerocopy/benches/try_ref_from_suffix_dynamic_padding.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 2300
-Total Cycles: 791
-Total uOps: 2600
+Instructions: 2100
+Total Cycles: 710
+Total uOps: 2400
Dispatch Width: 4
-uOps Per Cycle: 3.29
-IPC: 2.91
-Block RThroughput: 6.5
+uOps Per Cycle: 3.38
+IPC: 2.96
+Block RThroughput: 6.0
Instruction Info:
@@ -20,26 +20,24 @@ Instruction Info:
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.50 lea eax, [rsi + rdi]
1 1 0.33 test al, 3
- 1 1 1.00 jne .LBB5_1
- 1 1 0.33 movabs rax, 9223372036854775804
- 1 1 0.33 and rax, rsi
- 1 1 0.33 cmp rax, 9
- 1 1 1.00 jae .LBB5_3
+ 1 1 0.50 setne al
+ 1 1 0.33 cmp rsi, 12
+ 1 1 0.50 setb cl
+ 1 1 0.33 or cl, al
+ 1 1 1.00 je .LBB5_2
1 0 0.25 xor eax, eax
1 1 1.00 U ret
- 1 1 0.33 add rax, -9
+ 1 1 0.33 mov rax, rsi
+ 1 1 0.33 or rax, 3
+ 1 1 0.33 add rax, -12
1 1 0.33 movabs rcx, -6148914691236517205
2 4 1.00 mul rcx
1 1 0.50 shr rdx
- 1 1 0.50 lea rcx, [rdx + 2*rdx]
- 1 1 0.33 sub rsi, rcx
- 1 1 0.33 or rcx, -4
- 1 1 0.33 add rsi, rdi
- 1 1 0.50 lea rdi, [rcx + rsi]
- 1 1 0.33 add rdi, -8
+ 1 1 0.33 and esi, 3
+ 1 1 0.50 lea rcx, [rdi + rsi]
1 0 0.25 xor eax, eax
- 2 6 0.50 * cmp word ptr [rcx + rsi - 8], -16192
- 2 2 0.67 cmove rax, rdi
+ 2 6 0.50 * cmp word ptr [rdi + rsi], -16192
+ 2 2 0.67 cmove rax, rcx
1 1 1.00 U ret
@@ -56,30 +54,28 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 7.70 7.58 - 7.72 0.50 0.50
+ - - 7.00 7.00 - 7.00 0.50 0.50
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.26 0.74 - - - - lea eax, [rsi + rdi]
- - - 0.19 0.28 - 0.53 - - test al, 3
- - - - - - 1.00 - - jne .LBB5_1
- - - 0.93 0.06 - 0.01 - - movabs rax, 9223372036854775804
- - - 0.81 0.14 - 0.05 - - and rax, rsi
- - - 0.55 0.43 - 0.02 - - cmp rax, 9
- - - - - - 1.00 - - jae .LBB5_3
+ - - 0.99 0.01 - - - - lea eax, [rsi + rdi]
+ - - 0.01 0.99 - - - - test al, 3
+ - - 0.98 - - 0.02 - - setne al
+ - - 0.99 - - 0.01 - - cmp rsi, 12
+ - - 0.02 - - 0.98 - - setb cl
+ - - - 0.03 - 0.97 - - or cl, al
+ - - - - - 1.00 - - je .LBB5_2
- - - - - - - - xor eax, eax
- - - - - 1.00 - - ret
- - - 0.42 0.56 - 0.02 - - add rax, -9
- - - 0.67 0.30 - 0.03 - - movabs rcx, -6148914691236517205
+ - - - 1.00 - - - - mov rax, rsi
+ - - 0.03 0.97 - - - - or rax, 3
+ - - - 1.00 - - - - add rax, -12
+ - - 0.99 - - 0.01 - - movabs rcx, -6148914691236517205
- - 1.00 1.00 - - - - mul rcx
- - - 0.71 - - 0.29 - - shr rdx
- - - 0.32 0.68 - - - - lea rcx, [rdx + 2*rdx]
- - - 0.57 0.04 - 0.39 - - sub rsi, rcx
- - - 0.28 0.67 - 0.05 - - or rcx, -4
- - - 0.29 0.29 - 0.42 - - add rsi, rdi
- - - 0.02 0.98 - - - - lea rdi, [rcx + rsi]
- - - 0.02 0.41 - 0.57 - - add rdi, -8
+ - - 0.01 - - 0.99 - - shr rdx
+ - - 1.00 - - - - - and esi, 3
+ - - 0.97 0.03 - - - - lea rcx, [rdi + rsi]
- - - - - - - - xor eax, eax
- - - 0.57 0.01 - 0.42 0.50 0.50 cmp word ptr [rcx + rsi - 8], -16192
- - - 0.09 0.99 - 0.92 - - cmove rax, rdi
+ - - - 0.97 - 0.03 0.50 0.50 cmp word ptr [rdi + rsi], -16192
+ - - 0.01 1.00 - 0.99 - - cmove rax, rcx
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64 b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64
index c7530d8b6815..ca483dee63ba 100644
--- a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64
+++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64
@@ -1,32 +1,23 @@
bench_try_ref_from_suffix_with_elems_dynamic_padding:
movabs rax, 3074457345618258598
cmp rdx, rax
- ja .LBB5_1
- lea r8d, [rsi + rdi]
- xor ecx, ecx
- mov eax, 0
- test r8b, 3
- je .LBB5_3
- mov rdx, rcx
- ret
-.LBB5_3:
+ ja .LBB5_3
lea rax, [rdx + 2*rdx]
or rax, 3
add rax, 9
+ lea ecx, [rsi + rdi]
+ test cl, 3
+ setne cl
sub rsi, rax
- jae .LBB5_4
-.LBB5_1:
+ setb al
+ or al, cl
+ je .LBB5_4
+.LBB5_3:
xor eax, eax
- mov edx, 1
ret
.LBB5_4:
- lea r8, [rdi + rsi]
- movzx esi, word ptr [rdi + rsi]
- cmp si, -16192
- mov ecx, 2
- cmove rcx, rdx
+ lea rcx, [rdi + rsi]
xor eax, eax
- cmp esi, 49344
- cmove rax, r8
- mov rdx, rcx
+ cmp word ptr [rdi + rsi], -16192
+ cmove rax, rcx
ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64.mca b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64.mca
index be736c00c250..c37a42c160f7 100644
--- a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64.mca
+++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_padding.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 2800
-Total Cycles: 878
-Total uOps: 3000
+Instructions: 2000
+Total Cycles: 643
+Total uOps: 2200
Dispatch Width: 4
uOps Per Cycle: 3.42
-IPC: 3.19
-Block RThroughput: 7.5
+IPC: 3.11
+Block RThroughput: 5.5
Instruction Info:
@@ -20,31 +20,23 @@ Instruction Info:
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 movabs rax, 3074457345618258598
1 1 0.33 cmp rdx, rax
- 1 1 1.00 ja .LBB5_1
- 1 1 0.50 lea r8d, [rsi + rdi]
- 1 0 0.25 xor ecx, ecx
- 1 1 0.33 mov eax, 0
- 1 1 0.33 test r8b, 3
- 1 1 1.00 je .LBB5_3
- 1 1 0.33 mov rdx, rcx
- 1 1 1.00 U ret
+ 1 1 1.00 ja .LBB5_3
1 1 0.50 lea rax, [rdx + 2*rdx]
1 1 0.33 or rax, 3
1 1 0.33 add rax, 9
+ 1 1 0.50 lea ecx, [rsi + rdi]
+ 1 1 0.33 test cl, 3
+ 1 1 0.50 setne cl
1 1 0.33 sub rsi, rax
- 1 1 1.00 jae .LBB5_4
+ 1 1 0.50 setb al
+ 1 1 0.33 or al, cl
+ 1 1 1.00 je .LBB5_4
1 0 0.25 xor eax, eax
- 1 1 0.33 mov edx, 1
1 1 1.00 U ret
- 1 1 0.50 lea r8, [rdi + rsi]
- 1 5 0.50 * movzx esi, word ptr [rdi + rsi]
- 1 1 0.33 cmp si, -16192
- 1 1 0.33 mov ecx, 2
- 2 2 0.67 cmove rcx, rdx
+ 1 1 0.50 lea rcx, [rdi + rsi]
1 0 0.25 xor eax, eax
- 1 1 0.33 cmp esi, 49344
- 2 2 0.67 cmove rax, r8
- 1 1 0.33 mov rdx, rcx
+ 2 6 0.50 * cmp word ptr [rdi + rsi], -16192
+ 2 2 0.67 cmove rax, rcx
1 1 1.00 U ret
@@ -61,35 +53,27 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 8.65 8.65 - 8.70 0.50 0.50
+ - - 6.32 6.33 - 6.35 0.50 0.50
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.67 0.30 - 0.03 - - movabs rax, 3074457345618258598
- - - 0.01 0.99 - - - - cmp rdx, rax
- - - - - - 1.00 - - ja .LBB5_1
- - - 0.99 0.01 - - - - lea r8d, [rsi + rdi]
- - - - - - - - - xor ecx, ecx
- - - 0.35 0.62 - 0.03 - - mov eax, 0
- - - 0.99 0.01 - - - - test r8b, 3
- - - - - - 1.00 - - je .LBB5_3
- - - 0.68 0.30 - 0.02 - - mov rdx, rcx
- - - - - - 1.00 - - ret
- - - 0.07 0.93 - - - - lea rax, [rdx + 2*rdx]
- - - 0.06 0.35 - 0.59 - - or rax, 3
- - - 0.02 0.07 - 0.91 - - add rax, 9
- - - 0.01 0.04 - 0.95 - - sub rsi, rax
- - - - - - 1.00 - - jae .LBB5_4
+ - - - 0.99 - 0.01 - - movabs rax, 3074457345618258598
+ - - 0.02 0.98 - - - - cmp rdx, rax
+ - - - - - 1.00 - - ja .LBB5_3
+ - - 0.66 0.34 - - - - lea rax, [rdx + 2*rdx]
+ - - 0.99 - - 0.01 - - or rax, 3
+ - - 0.99 0.01 - - - - add rax, 9
+ - - 0.01 0.99 - - - - lea ecx, [rsi + rdi]
+ - - - 0.99 - 0.01 - - test cl, 3
+ - - 0.02 - - 0.98 - - setne cl
+ - - 0.97 0.02 - 0.01 - - sub rsi, rax
+ - - 0.99 - - 0.01 - - setb al
+ - - 0.32 0.67 - 0.01 - - or al, cl
+ - - - - - 1.00 - - je .LBB5_4
- - - - - - - - xor eax, eax
- - - 0.92 0.01 - 0.07 - - mov edx, 1
- - - - - 1.00 - - ret
- - - - 1.00 - - - - lea r8, [rdi + rsi]
- - - - - - - 0.50 0.50 movzx esi, word ptr [rdi + rsi]
- - - 0.01 0.99 - - - - cmp si, -16192
- - - 0.88 0.04 - 0.08 - - mov ecx, 2
- - - 1.00 0.99 - 0.01 - - cmove rcx, rdx
+ - - - 1.00 - - - - lea rcx, [rdi + rsi]
- - - - - - - - xor eax, eax
- - - 0.99 0.01 - - - - cmp esi, 49344
- - - 1.00 1.00 - - - - cmove rax, r8
- - - - 0.99 - 0.01 - - mov rdx, rcx
+ - - 0.35 0.32 - 0.33 0.50 0.50 cmp word ptr [rdi + rsi], -16192
+ - - 1.00 0.02 - 0.98 - - cmove rax, rcx
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64 b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64
index 952eb12de8d6..748eb1166e13 100644
--- a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64
+++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64
@@ -1,28 +1,20 @@
bench_try_ref_from_suffix_with_elems_dynamic_size:
movabs rax, 4611686018427387901
cmp rdx, rax
- ja .LBB5_1
- lea r8d, [rsi + rdi]
- xor ecx, ecx
- mov eax, 0
- test r8b, 1
- jne .LBB5_5
+ ja .LBB5_3
lea rax, [2*rdx + 4]
+ lea ecx, [rsi + rdi]
sub rsi, rax
- jae .LBB5_4
-.LBB5_1:
+ setb al
+ or al, cl
+ test al, 1
+ je .LBB5_4
+.LBB5_3:
xor eax, eax
- mov edx, 1
ret
.LBB5_4:
- lea r8, [rdi + rsi]
- movzx esi, word ptr [rdi + rsi]
- cmp si, -16192
- mov ecx, 2
- cmove rcx, rdx
+ lea rcx, [rdi + rsi]
xor eax, eax
- cmp esi, 49344
- cmove rax, r8
-.LBB5_5:
- mov rdx, rcx
+ cmp word ptr [rdi + rsi], -16192
+ cmove rax, rcx
ret
diff --git a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64.mca b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64.mca
index d4f78f67a252..53ff06192696 100644
--- a/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64.mca
+++ b/rust/zerocopy/benches/try_ref_from_suffix_with_elems_dynamic_size.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 2400
-Total Cycles: 1107
-Total uOps: 2600
+Instructions: 1700
+Total Cycles: 544
+Total uOps: 1900
Dispatch Width: 4
-uOps Per Cycle: 2.35
-IPC: 2.17
-Block RThroughput: 6.5
+uOps Per Cycle: 3.49
+IPC: 3.13
+Block RThroughput: 4.8
Instruction Info:
@@ -20,27 +20,20 @@ Instruction Info:
[1] [2] [3] [4] [5] [6] Instructions:
1 1 0.33 movabs rax, 4611686018427387901
1 1 0.33 cmp rdx, rax
- 1 1 1.00 ja .LBB5_1
- 1 1 0.50 lea r8d, [rsi + rdi]
- 1 0 0.25 xor ecx, ecx
- 1 1 0.33 mov eax, 0
- 1 1 0.33 test r8b, 1
- 1 1 1.00 jne .LBB5_5
+ 1 1 1.00 ja .LBB5_3
1 1 0.50 lea rax, [2*rdx + 4]
+ 1 1 0.50 lea ecx, [rsi + rdi]
1 1 0.33 sub rsi, rax
- 1 1 1.00 jae .LBB5_4
+ 1 1 0.50 setb al
+ 1 1 0.33 or al, cl
+ 1 1 0.33 test al, 1
+ 1 1 1.00 je .LBB5_4
1 0 0.25 xor eax, eax
- 1 1 0.33 mov edx, 1
1 1 1.00 U ret
- 1 1 0.50 lea r8, [rdi + rsi]
- 1 5 0.50 * movzx esi, word ptr [rdi + rsi]
- 1 1 0.33 cmp si, -16192
- 1 1 0.33 mov ecx, 2
- 2 2 0.67 cmove rcx, rdx
+ 1 1 0.50 lea rcx, [rdi + rsi]
1 0 0.25 xor eax, eax
- 1 1 0.33 cmp esi, 49344
- 2 2 0.67 cmove rax, r8
- 1 1 0.33 mov rdx, rcx
+ 2 6 0.50 * cmp word ptr [rdi + rsi], -16192
+ 2 2 0.67 cmove rax, rcx
1 1 1.00 U ret
@@ -57,31 +50,24 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 6.99 7.00 - 8.01 0.50 0.50
+ - - 5.33 5.33 - 5.34 0.50 0.50
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.02 0.95 - 0.03 - - movabs rax, 4611686018427387901
- - - 0.93 0.04 - 0.03 - - cmp rdx, rax
- - - - - - 1.00 - - ja .LBB5_1
- - - 0.96 0.04 - - - - lea r8d, [rsi + rdi]
- - - - - - - - - xor ecx, ecx
- - - 0.95 0.02 - 0.03 - - mov eax, 0
- - - 0.95 0.05 - - - - test r8b, 1
- - - - - - 1.00 - - jne .LBB5_5
- - - 0.06 0.94 - - - - lea rax, [2*rdx + 4]
- - - 0.93 0.07 - - - - sub rsi, rax
- - - - - - 1.00 - - jae .LBB5_4
+ - - 0.66 0.33 - 0.01 - - movabs rax, 4611686018427387901
+ - - 0.02 0.98 - - - - cmp rdx, rax
+ - - - - - 1.00 - - ja .LBB5_3
+ - - 0.98 0.02 - - - - lea rax, [2*rdx + 4]
+ - - 0.66 0.34 - - - - lea ecx, [rsi + rdi]
+ - - 0.33 0.65 - 0.02 - - sub rsi, rax
+ - - 0.99 - - 0.01 - - setb al
+ - - 0.01 0.36 - 0.63 - - or al, cl
+ - - 0.01 0.96 - 0.03 - - test al, 1
+ - - - - - 1.00 - - je .LBB5_4
- - - - - - - - xor eax, eax
- - - 0.03 0.95 - 0.02 - - mov edx, 1
- - - - - 1.00 - - ret
- - - 0.97 0.03 - - - - lea r8, [rdi + rsi]
- - - - - - - 0.50 0.50 movzx esi, word ptr [rdi + rsi]
- - - 0.03 0.97 - - - - cmp si, -16192
- - - 0.05 0.94 - 0.01 - - mov ecx, 2
- - - 0.06 0.98 - 0.96 - - cmove rcx, rdx
+ - - 0.32 0.68 - - - - lea rcx, [rdi + rsi]
- - - - - - - - xor eax, eax
- - - 0.97 0.03 - - - - cmp esi, 49344
- - - 0.06 0.96 - 0.98 - - cmove rax, r8
- - - 0.02 0.03 - 0.95 - - mov rdx, rcx
+ - - 0.36 0.02 - 0.62 0.50 0.50 cmp word ptr [rdi + rsi], -16192
+ - - 0.99 0.99 - 0.02 - - cmove rax, rcx
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/write_to_dynamic_size.x86-64 b/rust/zerocopy/benches/write_to_dynamic_size.x86-64
index c5abb17f7eba..714b68c51734 100644
--- a/rust/zerocopy/benches/write_to_dynamic_size.x86-64
+++ b/rust/zerocopy/benches/write_to_dynamic_size.x86-64
@@ -3,8 +3,7 @@ bench_write_to_dynamic_size:
push rbx
push rax
mov rbx, rcx
- lea r14, [2*rsi + 5]
- and r14, -2
+ lea r14, [2*rsi + 4]
cmp rcx, r14
jne .LBB5_2
mov rax, rdi
diff --git a/rust/zerocopy/benches/write_to_dynamic_size.x86-64.mca b/rust/zerocopy/benches/write_to_dynamic_size.x86-64.mca
index 5b2c08a31a29..2cc0a58c70fc 100644
--- a/rust/zerocopy/benches/write_to_dynamic_size.x86-64.mca
+++ b/rust/zerocopy/benches/write_to_dynamic_size.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 1900
+Instructions: 1800
Total Cycles: 2890
-Total uOps: 2500
+Total uOps: 2400
Dispatch Width: 4
-uOps Per Cycle: 0.87
-IPC: 0.66
-Block RThroughput: 6.3
+uOps Per Cycle: 0.83
+IPC: 0.62
+Block RThroughput: 6.0
Instruction Info:
@@ -22,8 +22,7 @@ Instruction Info:
2 5 1.00 * push rbx
2 5 1.00 * push rax
1 1 0.33 mov rbx, rcx
- 1 1 0.50 lea r14, [2*rsi + 5]
- 1 1 0.33 and r14, -2
+ 1 1 0.50 lea r14, [2*rsi + 4]
1 1 0.33 cmp rcx, r14
1 1 1.00 jne .LBB5_2
1 1 0.33 mov rax, rdi
@@ -52,25 +51,24 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 4.66 4.64 4.00 4.70 4.00 3.00
+ - - 4.47 4.01 4.00 4.52 4.00 3.00
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - - 1.00 - - 1.00 push r14
- - - - 1.00 - 1.00 - push rbx
- - - - 1.00 - - 1.00 push rax
- - - 0.02 0.97 - 0.01 - - mov rbx, rcx
- - - 0.97 0.03 - - - - lea r14, [2*rsi + 5]
- - - 0.63 0.35 - 0.02 - - and r14, -2
- - - 0.31 0.34 - 0.35 - - cmp rcx, r14
+ - - - 0.99 - 0.01 - - mov rbx, rcx
+ - - 0.99 0.01 - - - - lea r14, [2*rsi + 4]
+ - - 0.02 0.95 - 0.03 - - cmp rcx, r14
- - - - - 1.00 - - jne .LBB5_2
- - - 0.33 0.33 - 0.34 - - mov rax, rdi
- - - 0.36 0.31 - 0.33 - - mov rdi, rdx
- - - 0.33 0.35 - 0.32 - - mov rsi, rax
- - - 0.35 0.63 - 0.02 - - mov rdx, rbx
+ - - 0.48 0.04 - 0.48 - - mov rax, rdi
+ - - 0.51 0.01 - 0.48 - - mov rdi, rdx
+ - - 0.01 0.98 - 0.01 - - mov rsi, rax
+ - - 0.98 0.02 - - - - mov rdx, rbx
- - - - 1.00 1.00 2.00 - call qword ptr [rip + memcpy@GOTPCREL]
- - - 0.65 0.35 - - - - cmp rbx, r14
- - - 0.69 - - 0.31 - - sete al
+ - - 0.94 0.03 - 0.03 - - cmp rbx, r14
+ - - 0.52 - - 0.48 - - sete al
- - 0.02 0.98 - - - - add rsp, 8
- - - - - - - 1.00 pop rbx
- - - - - - 1.00 - pop r14
diff --git a/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64 b/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64
index d7779c6c9178..63e5d5200962 100644
--- a/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64
+++ b/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64
@@ -3,8 +3,7 @@ bench_write_to_prefix_dynamic_size:
push rbx
push rax
mov rbx, rcx
- lea r14, [2*rsi + 5]
- and r14, -2
+ lea r14, [2*rsi + 4]
cmp r14, rcx
ja .LBB5_2
mov rax, rdi
diff --git a/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64.mca
index 4cebe24d4f3b..e623ab3f302c 100644
--- a/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64.mca
+++ b/rust/zerocopy/benches/write_to_prefix_dynamic_size.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 1900
+Instructions: 1800
Total Cycles: 2890
-Total uOps: 2600
+Total uOps: 2500
Dispatch Width: 4
-uOps Per Cycle: 0.90
-IPC: 0.66
-Block RThroughput: 6.5
+uOps Per Cycle: 0.87
+IPC: 0.62
+Block RThroughput: 6.3
Instruction Info:
@@ -22,8 +22,7 @@ Instruction Info:
2 5 1.00 * push rbx
2 5 1.00 * push rax
1 1 0.33 mov rbx, rcx
- 1 1 0.50 lea r14, [2*rsi + 5]
- 1 1 0.33 and r14, -2
+ 1 1 0.50 lea r14, [2*rsi + 4]
1 1 0.33 cmp r14, rcx
1 1 1.00 ja .LBB5_2
1 1 0.33 mov rax, rdi
@@ -52,26 +51,25 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 5.47 4.49 4.00 5.04 4.00 3.00
+ - - 4.09 4.00 4.00 5.91 4.00 3.00
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - - 1.00 - - 1.00 push r14
- - - - 1.00 - 1.00 - push rbx
- - - - 1.00 - - 1.00 push rax
- - - 0.48 0.51 - 0.01 - - mov rbx, rcx
- - - 0.51 0.49 - - - - lea r14, [2*rsi + 5]
- - - 0.48 0.05 - 0.47 - - and r14, -2
- - - 0.48 0.49 - 0.03 - - cmp r14, rcx
+ - - 0.02 0.97 - 0.01 - - mov rbx, rcx
+ - - 0.04 0.96 - - - - lea r14, [2*rsi + 4]
+ - - 0.02 0.04 - 0.94 - - cmp r14, rcx
- - - - - 1.00 - - ja .LBB5_2
- - - 0.04 0.47 - 0.49 - - mov rax, rdi
- - - 0.49 0.03 - 0.48 - - mov rdi, rdx
- - - 0.03 0.48 - 0.49 - - mov rsi, rax
- - - 0.48 0.51 - 0.01 - - mov rdx, r14
+ - - 0.96 0.01 - 0.03 - - mov rax, rdi
+ - - 0.03 0.93 - 0.04 - - mov rdi, rdx
+ - - 0.94 0.06 - - - - mov rsi, rax
+ - - 0.06 0.93 - 0.01 - - mov rdx, r14
- - - - 1.00 1.00 2.00 - call qword ptr [rip + memcpy@GOTPCREL]
- - - 0.51 0.49 - - - - cmp r14, rbx
- - - 1.94 - - 0.06 - - setbe al
- - - 0.03 0.97 - - - - add rsp, 8
+ - - 0.93 0.05 - 0.02 - - cmp r14, rbx
+ - - 0.14 - - 1.86 - - setbe al
+ - - 0.95 0.05 - - - - add rsp, 8
- - - - - - - 1.00 pop rbx
- - - - - - 1.00 - pop r14
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64 b/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64
index 75f349562db6..d13a9bb3b83c 100644
--- a/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64
+++ b/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64
@@ -3,8 +3,7 @@ bench_write_to_suffix_dynamic_size:
push rbx
push rax
mov rbx, rcx
- lea r14, [2*rsi + 5]
- and r14, -2
+ lea r14, [2*rsi + 4]
sub rcx, r14
jb .LBB5_2
mov rax, rdi
diff --git a/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64.mca b/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64.mca
index 95cb9dfe2eb7..fb8fce317e0c 100644
--- a/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64.mca
+++ b/rust/zerocopy/benches/write_to_suffix_dynamic_size.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 2000
+Instructions: 1900
Total Cycles: 2890
-Total uOps: 2600
+Total uOps: 2500
Dispatch Width: 4
-uOps Per Cycle: 0.90
-IPC: 0.69
-Block RThroughput: 6.5
+uOps Per Cycle: 0.87
+IPC: 0.66
+Block RThroughput: 6.3
Instruction Info:
@@ -22,8 +22,7 @@ Instruction Info:
2 5 1.00 * push rbx
2 5 1.00 * push rax
1 1 0.33 mov rbx, rcx
- 1 1 0.50 lea r14, [2*rsi + 5]
- 1 1 0.33 and r14, -2
+ 1 1 0.50 lea r14, [2*rsi + 4]
1 1 0.33 sub rcx, r14
1 1 1.00 jb .LBB5_2
1 1 0.33 mov rax, rdi
@@ -53,27 +52,26 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 4.98 4.98 4.00 5.04 4.00 3.00
+ - - 4.65 4.64 4.00 4.71 4.00 3.00
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - - 1.00 - - 1.00 push r14
- - - - 1.00 - 1.00 - push rbx
- - - - 1.00 - - 1.00 push rax
- - - 0.94 0.05 - 0.01 - - mov rbx, rcx
- - - 0.06 0.94 - - - - lea r14, [2*rsi + 5]
- - - 0.93 0.02 - 0.05 - - and r14, -2
- - - 0.05 0.94 - 0.01 - - sub rcx, r14
+ - - 0.02 0.97 - 0.01 - - mov rbx, rcx
+ - - 0.97 0.03 - - - - lea r14, [2*rsi + 4]
+ - - 0.64 0.35 - 0.01 - - sub rcx, r14
- - - - - 1.00 - - jb .LBB5_2
- - - 0.02 0.04 - 0.94 - - mov rax, rdi
- - - 0.03 0.97 - - - - add rdx, rcx
- - - 0.95 0.05 - - - - mov rdi, rdx
- - - 0.94 0.03 - 0.03 - - mov rsi, rax
- - - 0.01 0.03 - 0.96 - - mov rdx, r14
+ - - 0.35 0.31 - 0.34 - - mov rax, rdi
+ - - 0.32 0.68 - - - - add rdx, rcx
+ - - 0.32 0.34 - 0.34 - - mov rdi, rdx
+ - - 0.67 0.32 - 0.01 - - mov rsi, rax
+ - - 0.34 0.35 - 0.31 - - mov rdx, r14
- - - - 1.00 1.00 2.00 - call qword ptr [rip + memcpy@GOTPCREL]
- - - 0.05 0.94 - 0.01 - - cmp rbx, r14
- - - 0.97 - - 0.03 - - setae al
- - - 0.03 0.97 - - - - add rsp, 8
+ - - 0.35 0.31 - 0.34 - - cmp rbx, r14
+ - - 0.65 - - 0.35 - - setae al
+ - - 0.02 0.98 - - - - add rsp, 8
- - - - - - - 1.00 pop rbx
- - - - - - 1.00 - pop r14
- - - - - 1.00 - - ret
diff --git a/rust/zerocopy/benches/zero_dynamic_size.x86-64 b/rust/zerocopy/benches/zero_dynamic_size.x86-64
index 2b31ed644eaf..92e2748b1353 100644
--- a/rust/zerocopy/benches/zero_dynamic_size.x86-64
+++ b/rust/zerocopy/benches/zero_dynamic_size.x86-64
@@ -1,5 +1,4 @@
bench_zero_dynamic_size:
- lea rdx, [2*rsi + 5]
- and rdx, -2
+ lea rdx, [2*rsi + 4]
xor esi, esi
jmp qword ptr [rip + memset@GOTPCREL]
diff --git a/rust/zerocopy/benches/zero_dynamic_size.x86-64.mca b/rust/zerocopy/benches/zero_dynamic_size.x86-64.mca
index 0b086a29b0ed..7dee3b1de141 100644
--- a/rust/zerocopy/benches/zero_dynamic_size.x86-64.mca
+++ b/rust/zerocopy/benches/zero_dynamic_size.x86-64.mca
@@ -1,12 +1,12 @@
Iterations: 100
-Instructions: 400
-Total Cycles: 142
-Total uOps: 500
+Instructions: 300
+Total Cycles: 108
+Total uOps: 400
Dispatch Width: 4
-uOps Per Cycle: 3.52
-IPC: 2.82
-Block RThroughput: 1.3
+uOps Per Cycle: 3.70
+IPC: 2.78
+Block RThroughput: 1.0
Instruction Info:
@@ -18,8 +18,7 @@ Instruction Info:
[6]: HasSideEffects (U)
[1] [2] [3] [4] [5] [6] Instructions:
- 1 1 0.50 lea rdx, [2*rsi + 5]
- 1 1 0.33 and rdx, -2
+ 1 1 0.50 lea rdx, [2*rsi + 4]
1 0 0.25 xor esi, esi
2 6 1.00 * jmp qword ptr [rip + memset@GOTPCREL]
@@ -37,11 +36,10 @@ Resources:
Resource pressure per iteration:
[0] [1] [2] [3] [4] [5] [6.0] [6.1]
- - - 0.99 1.00 - 1.01 0.50 0.50
+ - - 0.50 0.50 - 1.00 0.50 0.50
Resource pressure by instruction:
[0] [1] [2] [3] [4] [5] [6.0] [6.1] Instructions:
- - - 0.99 0.01 - - - - lea rdx, [2*rsi + 5]
- - - - 0.99 - 0.01 - - and rdx, -2
+ - - 0.50 0.50 - - - - lea rdx, [2*rsi + 4]
- - - - - - - - xor esi, esi
- - - - - 1.00 0.50 0.50 jmp qword ptr [rip + memset@GOTPCREL]
diff --git a/rust/zerocopy/src/byte_slice.rs b/rust/zerocopy/src/byte_slice.rs
index b7f85098dbc4..26fcc088aecf 100644
--- a/rust/zerocopy/src/byte_slice.rs
+++ b/rust/zerocopy/src/byte_slice.rs
@@ -427,8 +427,12 @@ fn test_ref_mut_split_at_unchecked() {
let borrow_mut = cell.borrow_mut();
let slice_ref_mut: cell::RefMut<'_, [u8]> = cell::RefMut::map(borrow_mut, |a| &mut a[..]);
// SAFETY: 2 is within bounds of [1, 2, 3, 4]
- let (l, r) = unsafe { slice_ref_mut.split_at_unchecked(2) };
+ let (mut l, mut r) = unsafe { slice_ref_mut.split_at_unchecked(2) };
assert_eq!(*l, [1, 2]);
assert_eq!(*r, [3, 4]);
+ l[0] = 5;
+ r[0] = 6;
+ drop((l, r));
+ assert_eq!(*cell.borrow(), [5, 2, 6, 4]);
}
}
diff --git a/rust/zerocopy/src/byteorder.rs b/rust/zerocopy/src/byteorder.rs
index c761d5728320..6189a6e7f77c 100644
--- a/rust/zerocopy/src/byteorder.rs
+++ b/rust/zerocopy/src/byteorder.rs
@@ -87,7 +87,7 @@
pub trait ByteOrder:
Copy + Clone + Debug + Display + Eq + PartialEq + Ord + PartialOrd + Hash + private::Sealed
{
- #[doc(hidden)]
+ /// A value-level representation of byte order.
const ORDER: Order;
}
@@ -98,11 +98,12 @@ impl Sealed for super::BigEndian {}
impl Sealed for super::LittleEndian {}
}
-#[allow(missing_copy_implementations, missing_debug_implementations)]
-#[doc(hidden)]
-#[derive(PartialEq)]
+/// A value-level representation of [`ByteOrder`].
+#[derive(Copy, Clone, Debug, Eq, PartialEq, Ord, PartialOrd, Hash)]
pub enum Order {
+ /// A value-level representation of [`BigEndian`].
BigEndian,
+ /// A value-level representation of [`LittleEndian`].
LittleEndian,
}
@@ -1235,7 +1236,7 @@ fn is_nan(self) -> bool { self.is_nan() }
impl_traits!(I32, i32, signed);
impl_traits!(I64, i64, signed);
impl_traits!(I128, i128, signed);
- impl_traits!(Isize, isize, unsigned);
+ impl_traits!(Isize, isize, signed);
impl_traits!(F32, f32, signed, @float);
impl_traits!(F64, f64, signed, @float);
@@ -1383,6 +1384,18 @@ fn test_non_native<T: ByteOrderType>() {
call_for_all_types!(test_non_native, NonNativeEndian);
}
+ #[test]
+ fn test_float_bit_patterns() {
+ for bits in [0u32, 0x8000_0000, 0x7F80_0000, 0xFF80_0000, 0x7FC0_1234] {
+ let be = F32::<BE>::new(f32::from_bits(bits));
+ let le = F32::<LE>::new(f32::from_bits(bits));
+ assert_eq!(be.to_bytes(), bits.to_be_bytes());
+ assert_eq!(le.to_bytes(), bits.to_le_bytes());
+ assert_eq!(be.get().to_bits(), bits);
+ assert_eq!(le.get().to_bits(), bits);
+ }
+ }
+
#[test]
fn test_ops_impls() {
// Test implementations of traits in `core::ops`. Some of these are
@@ -1551,6 +1564,17 @@ fn t<T>()
test!(@unary Not, not, call_for_signed_types, call_for_unsigned_types);
test!(@unary Neg, neg, call_for_signed_types, call_for_float_types);
+
+ for shift in [0u64, 1, 31, 63] {
+ let n = 0x0123_4567_89AB_CDEFu64;
+ let shift_u32: u32 = shift.try_into().unwrap();
+ let shl = n.checked_shl(shift_u32).unwrap();
+ let shr = n.checked_shr(shift_u32).unwrap();
+ assert_eq!(core::ops::Shl::shl(U64::<NativeEndian>::new(n), shift).get(), shl);
+ assert_eq!(core::ops::Shr::shr(U64::<NativeEndian>::new(n), shift).get(), shr);
+ assert_eq!(core::ops::Shl::shl(U64::<NonNativeEndian>::new(n), shift).get(), shl);
+ assert_eq!(core::ops::Shr::shr(U64::<NonNativeEndian>::new(n), shift).get(), shr);
+ }
}
#[test]
@@ -1579,6 +1603,13 @@ fn test_byteorder_traits_coverage() {
assert!(val_be <= val_be);
assert_eq!(val_be.cmp(&val_be), core::cmp::Ordering::Equal);
+ let low = U16::<LE>::new(255);
+ let high = U16::<LE>::new(256);
+ assert!(low < high);
+ assert!(low < 256u16);
+ assert_eq!(low.cmp(&high), core::cmp::Ordering::Less);
+ assert!(I16::<LE>::new(-1) < I16::<LE>::new(0));
+
// PartialOrd with native
assert!(val_be == 1u16);
assert!(val_be >= 1u16);
diff --git a/rust/zerocopy/src/error.rs b/rust/zerocopy/src/error.rs
index 5eb30de934f3..98399bab7e97 100644
--- a/rust/zerocopy/src/error.rs
+++ b/rust/zerocopy/src/error.rs
@@ -199,10 +199,8 @@ impl<Src, Dst: ?Sized + Unaligned, S, V> From<ConvertError<AlignmentError<Src, D
#[inline]
fn from(err: ConvertError<AlignmentError<Src, Dst>, S, V>) -> ConvertError<Infallible, S, V> {
match err {
- ConvertError::Alignment(e) => {
- #[allow(unreachable_code)]
- return ConvertError::Alignment(Infallible::from(e));
- }
+ #[allow(unreachable_code)]
+ ConvertError::Alignment(e) => match Infallible::from(e) {},
ConvertError::Size(e) => ConvertError::Size(e),
ConvertError::Validity(e) => ConvertError::Validity(e),
}
@@ -237,7 +235,7 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
}
#[cfg(any(not(no_zerocopy_core_error_1_81_0), feature = "std", test))]
-#[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.81.0", feature = "std"))))]
+#[cfg_attr(doc_cfg, doc(cfg(any(rust = "1.81.0", feature = "std"))))]
impl<A, S, V> Error for ConvertError<A, S, V>
where
A: fmt::Display + fmt::Debug,
@@ -410,7 +408,7 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
}
#[cfg(any(not(no_zerocopy_core_error_1_81_0), feature = "std", test))]
-#[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.81.0", feature = "std"))))]
+#[cfg_attr(doc_cfg, doc(cfg(any(rust = "1.81.0", feature = "std"))))]
impl<Src, Dst: ?Sized> Error for AlignmentError<Src, Dst>
where
Src: Deref,
@@ -571,7 +569,7 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
}
#[cfg(any(not(no_zerocopy_core_error_1_81_0), feature = "std", test))]
-#[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.81.0", feature = "std"))))]
+#[cfg_attr(doc_cfg, doc(cfg(any(rust = "1.81.0", feature = "std"))))]
impl<Src, Dst: ?Sized> Error for SizeError<Src, Dst>
where
Src: Deref,
@@ -707,7 +705,7 @@ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
}
#[cfg(any(not(no_zerocopy_core_error_1_81_0), feature = "std", test))]
-#[cfg_attr(doc_cfg, doc(cfg(all(rust = "1.81.0", feature = "std"))))]
+#[cfg_attr(doc_cfg, doc(cfg(any(rust = "1.81.0", feature = "std"))))]
impl<Src, Dst: ?Sized> Error for ValidityError<Src, Dst> where Dst: KnownLayout + TryFromBytes {}
impl<Src, Dst: ?Sized + TryFromBytes, A, S> From<ValidityError<Src, Dst>>
diff --git a/rust/zerocopy/src/impls.rs b/rust/zerocopy/src/impls.rs
index 62e234c5202b..01b509b98264 100644
--- a/rust/zerocopy/src/impls.rs
+++ b/rust/zerocopy/src/impls.rs
@@ -111,7 +111,7 @@
// pattern 0x01.
const _: () = unsafe {
unsafe_impl!(=> TryFromBytes for bool; |byte| {
- let byte = byte.transmute_with::<u8, invariant::Valid, CastSizedExact, BecauseImmutable>();
+ let byte = byte.transmute_with::<u8, invariant::Safe, CastSizedExact, BecauseImmutable>();
*byte.unaligned_as_ref() < 2
})
};
@@ -140,7 +140,7 @@
// `char`.
const _: () = unsafe {
unsafe_impl!(=> TryFromBytes for char; |c| {
- let c = c.transmute_with::<Unalign<u32>, invariant::Valid, CastSizedExact, BecauseImmutable>();
+ let c = c.transmute_with::<Unalign<u32>, invariant::Safe, CastSizedExact, BecauseImmutable>();
let c = c.read().into_inner();
char::from_u32(c).is_some()
});
@@ -173,7 +173,7 @@
// Returns `Err` if the slice is not UTF-8.
const _: () = unsafe {
unsafe_impl!(=> TryFromBytes for str; |c| {
- let c = c.transmute_with::<[u8], invariant::Valid, CastUnsized, BecauseImmutable>();
+ let c = c.transmute_with::<[u8], invariant::Safe, CastUnsized, BecauseImmutable>();
let c = c.unaligned_as_ref();
core::str::from_utf8(c).is_ok()
})
@@ -183,7 +183,7 @@ macro_rules! unsafe_impl_try_from_bytes_for_nonzero {
($($nonzero:ident[$prim:ty]),*) => {
$(
unsafe_impl!(=> TryFromBytes for $nonzero; |n| {
- let n = n.transmute_with::<Unalign<$prim>, invariant::Valid, CastSizedExact, BecauseImmutable>();
+ let n = n.transmute_with::<Unalign<$prim>, invariant::Safe, CastSizedExact, BecauseImmutable>();
$nonzero::new(n.read().into_inner()).is_some()
});
)*
@@ -421,15 +421,15 @@ macro_rules! unsafe_impl_transmute_from_for_atomic {
($($($tyvar:ident)? => $atomic:ty [$prim:ty]),*) => {{
crate::util::macros::__unsafe();
- use crate::pointer::{SizeEq, TransmuteFrom, invariant::Valid};
+ use crate::pointer::{SizeEq, TransmuteFrom, invariant::Safe};
$(
// SAFETY: The caller promised that `$atomic` and `$prim` have
// the same size and bit validity.
- unsafe impl<$($tyvar)?> TransmuteFrom<$atomic, Valid, Valid> for $prim {}
+ unsafe impl<$($tyvar)?> TransmuteFrom<$atomic, Safe, Safe> for $prim {}
// SAFETY: The caller promised that `$atomic` and `$prim` have
// the same size and bit validity.
- unsafe impl<$($tyvar)?> TransmuteFrom<$prim, Valid, Valid> for $atomic {}
+ unsafe impl<$($tyvar)?> TransmuteFrom<$prim, Safe, Safe> for $atomic {}
impl<$($tyvar)?> SizeEq<ReadOnly<$atomic>> for ReadOnly<$prim> {
type CastFrom = $crate::pointer::cast::CastSizedExact;
@@ -444,9 +444,9 @@ impl<$($tyvar)?> SizeEq<ReadOnly<$atomic>> for ReadOnly<$prim> {
// `UnsafeCell<T>` has the same in-memory representation as
// its inner type `T`. A consequence of this guarantee is that
// it is possible to convert between `T` and `UnsafeCell<T>`.
- unsafe impl<$($tyvar)?> TransmuteFrom<$atomic, Valid, Valid> for core::cell::UnsafeCell<$prim> {}
+ unsafe impl<$($tyvar)?> TransmuteFrom<$atomic, Safe, Safe> for core::cell::UnsafeCell<$prim> {}
// SAFETY: See previous safety comment.
- unsafe impl<$($tyvar)?> TransmuteFrom<core::cell::UnsafeCell<$prim>, Valid, Valid> for $atomic {}
+ unsafe impl<$($tyvar)?> TransmuteFrom<core::cell::UnsafeCell<$prim>, Safe, Safe> for $atomic {}
)*
}};
}
@@ -758,7 +758,7 @@ mod atomic_ptr {
pub enum value {}
// SAFETY: See safety comment on `ProjectToTag`.
- unsafe impl<T: ?Sized> HasTag for ManuallyDrop<T> {
+ unsafe impl<T: ?Sized, Client> HasTag<Client> for ManuallyDrop<T> {
#[inline]
fn only_derive_is_allowed_to_implement_this_trait()
where
@@ -796,8 +796,8 @@ fn only_derive_is_allowed_to_implement_this_trait()
// private field, and because it is the name it is referred to in the public
// documentation of `ManuallyDrop::new`, `ManuallyDrop::into_inner`,
// `ManuallyDrop::take` and `ManuallyDrop::drop`.
- unsafe impl<T: ?Sized>
- HasField<value, { crate::STRUCT_VARIANT_ID }, { crate::ident_id!(value) }>
+ unsafe impl<T: ?Sized, Client>
+ HasField<Client, value, { crate::STRUCT_VARIANT_ID }, { crate::ident_id!(value) }>
for ManuallyDrop<T>
{
#[inline]
@@ -849,7 +849,7 @@ fn project(slf: PtrInner<'_, Self>) -> *mut T {
const _: () = unsafe { unsafe_impl!(T: ?Sized + Unaligned => Unaligned for UnsafeCell<T>) };
assert_unaligned!(UnsafeCell<()>, UnsafeCell<u8>);
-// SAFETY: See safety comment in `is_bit_valid` impl.
+// SAFETY: See safety comment in `is_safe` impl.
unsafe impl<T: TryFromBytes + ?Sized> TryFromBytes for UnsafeCell<T> {
#[allow(clippy::missing_inline_in_public_items)]
fn only_derive_is_allowed_to_implement_this_trait()
@@ -859,11 +859,11 @@ fn only_derive_is_allowed_to_implement_this_trait()
}
#[inline(always)]
- fn is_bit_valid<A>(candidate: Maybe<'_, Self, A>) -> bool
+ fn is_safe<A>(candidate: Maybe<'_, Self, A>) -> bool
where
A: invariant::Alignment,
{
- T::is_bit_valid(candidate.transmute::<_, _, BecauseImmutable>())
+ T::is_safe(candidate.transmute::<_, _, BecauseImmutable>())
}
}
@@ -897,10 +897,10 @@ fn is_bit_valid<A>(candidate: Maybe<'_, Self, A>) -> bool
let c: Ptr<'_, ReadOnly<[T]>, _> = c.cast::<_, crate::pointer::cast::CastUnsized, _>();
// Note that this call may panic, but it would still be sound even if it
- // did. `is_bit_valid` does not promise that it will not panic (in fact,
- // it explicitly warns that it's a possibility), and we have not
- // violated any safety invariants that we must fix before returning.
- <[T] as TryFromBytes>::is_bit_valid(c)
+ // did. `is_safe` does not promise that it will not panic (in fact, it
+ // explicitly warns that it's a possibility), and we have not violated
+ // any safety invariants that we must fix before returning.
+ <[T] as TryFromBytes>::is_safe(c)
});
unsafe_impl!(const N: usize, T: FromZeros => FromZeros for [T; N]);
unsafe_impl!(const N: usize, T: FromBytes => FromBytes for [T; N]);
@@ -924,14 +924,14 @@ fn is_bit_valid<A>(candidate: Maybe<'_, Self, A>) -> bool
//
// In other words, the layout of a `[T] is a sequence of `T`s laid out
// back-to-back with no bytes in between. If all elements in `candidate`
- // are `is_bit_valid`, so too is `candidate`.
+ // are `is_safe`, so too is `candidate`.
//
// Note that any of the below calls may panic, but it would still be
- // sound even if it did. `is_bit_valid` does not promise that it will
- // not panic (in fact, it explicitly warns that it's a possibility), and
- // we have not violated any safety invariants that we must fix before
+ // sound even if it did. `is_safe` does not promise that it will not
+ // panic (in fact, it explicitly warns that it's a possibility), and we
+ // have not violated any safety invariants that we must fix before
// returning.
- c.iter().all(<T as TryFromBytes>::is_bit_valid)
+ c.iter().all(<T as TryFromBytes>::is_safe)
});
unsafe_impl!(T: FromZeros => FromZeros for [T]);
unsafe_impl!(T: FromBytes => FromBytes for [T]);
@@ -946,7 +946,7 @@ fn is_bit_valid<A>(candidate: Maybe<'_, Self, A>) -> bool
// null pointers, so this is not a footgun.
// - `TryFromBytes`: By the same reasoning as for `FromZeroes`, we can implement
// `TryFromBytes` for thin pointers provided that
-// [`TryFromByte::is_bit_valid`] only produces `true` for zeroed bytes.
+// [`TryFromBytes::is_safe`] only produces `true` for zeroed bytes.
//
// NOTE(#170): Implementing `FromBytes` and `IntoBytes` for raw pointers would
// be sound, but carries provenance footguns. We want to support `FromBytes` and
@@ -1012,11 +1012,11 @@ macro_rules! impl_tuple {
// SAFETY: If all fields of the tuple `Self` are `Immutable`, so too is `Self`.
unsafe_impl!($($head_T: Immutable,)* $next_T: Immutable => Immutable for ($($head_T,)* $next_T,));
- // SAFETY: If all fields in `c` are `is_bit_valid`, so too is `c`.
+ // SAFETY: If all fields in `c` are `is_safe`, so too is `c`.
unsafe_impl!($($head_T: TryFromBytes,)* $next_T: TryFromBytes => TryFromBytes for ($($head_T,)* $next_T,); |c| {
let mut c = c;
- $(TryFromBytes::is_bit_valid(into_inner!(c.reborrow().project::<_, { crate::STRUCT_VARIANT_ID }, { crate::ident_id!($head_I) }>())) &&)*
- TryFromBytes::is_bit_valid(into_inner!(c.reborrow().project::<_, { crate::STRUCT_VARIANT_ID }, { crate::ident_id!($next_I) }>()))
+ $(TryFromBytes::is_safe(into_inner!(c.reborrow().project::<crate::project_clients::TryFromBytesDerive, _, { crate::STRUCT_VARIANT_ID }, { crate::ident_id!($head_I) }>())) &&)*
+ TryFromBytes::is_safe(into_inner!(c.reborrow().project::<crate::project_clients::TryFromBytesDerive, _, { crate::STRUCT_VARIANT_ID }, { crate::ident_id!($next_I) }>()))
});
// SAFETY: If all fields in `Self` are `FromZeros`, so too is `Self`.
@@ -1026,7 +1026,9 @@ macro_rules! impl_tuple {
unsafe_impl!($($head_T: FromBytes,)* $next_T: FromBytes => FromBytes for ($($head_T,)* $next_T,));
// SAFETY: See safety comment on `ProjectToTag`.
- unsafe impl<$($head_T,)* $next_T> crate::HasTag for ($($head_T,)* $next_T,) {
+ unsafe impl<Client, $($head_T,)* $next_T> crate::HasTag<Client>
+ for ($($head_T,)* $next_T,)
+ {
#[inline]
fn only_derive_is_allowed_to_implement_this_trait()
where
@@ -1077,7 +1079,8 @@ fn only_derive_is_allowed_to_implement_this_trait()
// - `()` has the same visibility as the `.$CurrI` field (ie, `.0`,
// `.1`, etc)
// - `Type` has the same type as `$CurrI`; i.e., `$CurrT`.
- unsafe impl<$($AllT),+> crate::HasField<
+ unsafe impl<Client, $($AllT),+> crate::HasField<
+ Client,
(),
{ crate::STRUCT_VARIANT_ID },
{ crate::ident_id!($CurrI)}
@@ -1103,7 +1106,8 @@ fn project(slf: crate::PtrInner<'_, Self>) -> *mut Self::Type {
}
// SAFETY: See comments on items.
- unsafe impl<Aliasing, Alignment, $($AllT),+> crate::ProjectField<
+ unsafe impl<Client, Aliasing, Alignment, $($AllT),+> crate::ProjectField<
+ Client,
(),
(Aliasing, Alignment, crate::invariant::Uninit),
{ crate::STRUCT_VARIANT_ID },
@@ -1129,7 +1133,8 @@ fn only_derive_is_allowed_to_implement_this_trait()
}
// SAFETY: See comments on items.
- unsafe impl<Aliasing, Alignment, $($AllT),+> crate::ProjectField<
+ unsafe impl<Client, Aliasing, Alignment, $($AllT),+> crate::ProjectField<
+ Client,
(),
(Aliasing, Alignment, crate::invariant::Initialized),
{ crate::STRUCT_VARIANT_ID },
@@ -1155,9 +1160,10 @@ fn only_derive_is_allowed_to_implement_this_trait()
}
// SAFETY: See comments on items.
- unsafe impl<Aliasing, Alignment, $($AllT),+> crate::ProjectField<
+ unsafe impl<Client, Aliasing, Alignment, $($AllT),+> crate::ProjectField<
+ Client,
(),
- (Aliasing, Alignment, crate::invariant::Valid),
+ (Aliasing, Alignment, crate::invariant::Safe),
{ crate::STRUCT_VARIANT_ID },
{ crate::ident_id!($CurrI)}
> for ($($AllT,)+)
@@ -1174,7 +1180,7 @@ fn only_derive_is_allowed_to_implement_this_trait()
// SAFETY: Tuples are product types whose fields are
// well-aligned, so projection preserves both the alignment and
// validity invariants of the outer pointer.
- type Invariants = (Aliasing, Alignment, crate::invariant::Valid);
+ type Invariants = (Aliasing, Alignment, crate::invariant::Safe);
// SAFETY: Tuples are product types and so projection is infallible;
type Error = core::convert::Infallible;
@@ -1389,10 +1395,9 @@ mod tests {
#[test]
fn test_impls() {
- // A type that can supply test cases for testing
- // `TryFromBytes::is_bit_valid`. All types passed to `assert_impls!`
- // must implement this trait; that macro uses it to generate runtime
- // tests for `TryFromBytes` impls.
+ // A type that can supply test cases for testing `TryFromBytes::is_safe`.
+ // All types passed to `assert_impls!` must implement this trait; that
+ // macro uses it to generate runtime tests for `TryFromBytes` impls.
//
// All `T: FromBytes` types are provided with a blanket impl. Other
// types must implement `TryFromBytesTestable` directly (ie using
@@ -1574,19 +1579,15 @@ mod autoref_trick {
pub(super) struct AutorefWrapper<T: ?Sized>(pub(super) PhantomData<T>);
- pub(super) trait TestIsBitValidShared<T: ?Sized> {
+ pub(super) trait TestIsSafeShared<T: ?Sized> {
#[allow(clippy::needless_lifetimes)]
- fn test_is_bit_valid_shared<'ptr>(&self, candidate: Maybe<'ptr, T>)
- -> Option<bool>;
+ fn test_is_safe_shared<'ptr>(&self, candidate: Maybe<'ptr, T>) -> Option<bool>;
}
- impl<T: TryFromBytes + Immutable + ?Sized> TestIsBitValidShared<T> for AutorefWrapper<T> {
+ impl<T: TryFromBytes + Immutable + ?Sized> TestIsSafeShared<T> for AutorefWrapper<T> {
#[allow(clippy::needless_lifetimes)]
- fn test_is_bit_valid_shared<'ptr>(
- &self,
- candidate: Maybe<'ptr, T>,
- ) -> Option<bool> {
- Some(T::is_bit_valid(candidate))
+ fn test_is_safe_shared<'ptr>(&self, candidate: Maybe<'ptr, T>) -> Option<bool> {
+ Some(T::is_safe(candidate))
}
}
@@ -1690,12 +1691,12 @@ macro_rules! assert_impls {
#[allow(unused, non_local_definitions)]
impl AutorefWrapper<$ty> {
#[allow(clippy::needless_lifetimes)]
- fn test_is_bit_valid_shared<'ptr>(
+ fn test_is_safe_shared<'ptr>(
&mut self,
candidate: Maybe<'ptr, $ty>,
) -> Option<bool> {
assert_on_allowlist!(
- test_is_bit_valid_shared($ty):
+ test_is_safe_shared($ty):
ManuallyDrop<UnsafeCell<()>>,
ManuallyDrop<[UnsafeCell<u8>]>,
ManuallyDrop<[UnsafeCell<bool>]>,
@@ -1799,9 +1800,9 @@ fn test_as_bytes(&mut self, _t: &ReadOnly<$ty>) -> Option<&[u8]> {
// necessarily `IntoBytes`, but that's the corner we've
// backed ourselves into by using `Ptr::from_ref`.
let c = unsafe { c.assume_initialized() };
- let res = w.test_is_bit_valid_shared(c);
+ let res = w.test_is_safe_shared(c);
if let Some(res) = res {
- assert!(res, "{}::is_bit_valid (shared `Ptr`): got false, expected true", stringify!($ty));
+ assert!(res, "{}::is_safe (shared `Ptr`): got false, expected true", stringify!($ty));
}
let c = Ptr::from_mut(&mut *val);
@@ -1810,8 +1811,8 @@ fn test_as_bytes(&mut self, _t: &ReadOnly<$ty>) -> Option<&[u8]> {
// necessarily `IntoBytes`, but that's the corner we've
// backed ourselves into by using `Ptr::from_ref`.
let mut c = unsafe { c.assume_initialized() };
- let res = <$ty as TryFromBytes>::is_bit_valid(c.reborrow_shared());
- assert!(res, "{}::is_bit_valid (exclusive `Ptr`): got false, expected true", stringify!($ty));
+ let res = <$ty as TryFromBytes>::is_safe(c.reborrow_shared());
+ assert!(res, "{}::is_safe (exclusive `Ptr`): got false, expected true", stringify!($ty));
// `bytes` is `Some(val.as_bytes())` if `$ty: IntoBytes +
// Immutable` and `None` otherwise.
@@ -2237,11 +2238,11 @@ fn test_as_bytes(&mut self, _t: &ReadOnly<$ty>) -> Option<&[u8]> {
assert_impls!(ManuallyDrop<u8>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
// This test is important because it allows us to test our hand-rolled
- // implementation of `<ManuallyDrop<T> as TryFromBytes>::is_bit_valid`.
+ // implementation of `<ManuallyDrop<T> as TryFromBytes>::is_safe`.
assert_impls!(ManuallyDrop<bool>: KnownLayout, Immutable, TryFromBytes, FromZeros, IntoBytes, Unaligned, !FromBytes);
assert_impls!(ManuallyDrop<[u8]>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
// This test is important because it allows us to test our hand-rolled
- // implementation of `<ManuallyDrop<T> as TryFromBytes>::is_bit_valid`.
+ // implementation of `<ManuallyDrop<T> as TryFromBytes>::is_safe`.
assert_impls!(ManuallyDrop<[bool]>: KnownLayout, Immutable, TryFromBytes, FromZeros, IntoBytes, Unaligned, !FromBytes);
assert_impls!(ManuallyDrop<NotZerocopy>: !Immutable, !TryFromBytes, !KnownLayout, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
assert_impls!(ManuallyDrop<[NotZerocopy]>: KnownLayout, !Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
@@ -2255,14 +2256,14 @@ fn test_as_bytes(&mut self, _t: &ReadOnly<$ty>) -> Option<&[u8]> {
assert_impls!(Wrapping<u8>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
// This test is important because it allows us to test our hand-rolled
- // implementation of `<Wrapping<T> as TryFromBytes>::is_bit_valid`.
+ // implementation of `<Wrapping<T> as TryFromBytes>::is_safe`.
assert_impls!(Wrapping<bool>: KnownLayout, Immutable, TryFromBytes, FromZeros, IntoBytes, Unaligned, !FromBytes);
assert_impls!(Wrapping<NotZerocopy>: KnownLayout, !Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes, !Unaligned);
assert_impls!(Wrapping<UnsafeCell<()>>: KnownLayout, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned, !Immutable);
assert_impls!(Unalign<u8>: KnownLayout, Immutable, TryFromBytes, FromZeros, FromBytes, IntoBytes, Unaligned);
// This test is important because it allows us to test our hand-rolled
- // implementation of `<Unalign<T> as TryFromBytes>::is_bit_valid`.
+ // implementation of `<Unalign<T> as TryFromBytes>::is_safe`.
assert_impls!(Unalign<bool>: KnownLayout, Immutable, TryFromBytes, FromZeros, IntoBytes, Unaligned, !FromBytes);
assert_impls!(Unalign<NotZerocopy>: KnownLayout, Unaligned, !Immutable, !TryFromBytes, !FromZeros, !FromBytes, !IntoBytes);
diff --git a/rust/zerocopy/src/layout.rs b/rust/zerocopy/src/layout.rs
index b1fa0cd436db..5d5598cd0cfb 100644
--- a/rust/zerocopy/src/layout.rs
+++ b/rust/zerocopy/src/layout.rs
@@ -30,6 +30,7 @@
#[allow(missing_debug_implementations, missing_copy_implementations)]
#[cfg_attr(any(kani, test), derive(Debug, PartialEq, Eq))]
#[derive(Copy, Clone)]
+#[cfg_attr(kani, derive(kani::Arbitrary))]
pub struct DstLayout {
pub(crate) align: NonZeroUsize,
pub(crate) size_info: SizeInfo,
@@ -42,29 +43,680 @@ pub struct DstLayout {
#[cfg_attr(any(kani, test), derive(Debug, PartialEq, Eq))]
#[derive(Copy, Clone)]
+#[cfg_attr(kani, derive(kani::Arbitrary))]
pub(crate) enum SizeInfo<E = usize> {
Sized { size: usize },
SliceDst(TrailingSliceLayout<E>),
}
+/// The alignment and phase of a rounded size calculation.
+///
+/// If `encoded` is the stored non-zero integer, then its highest set bit is a
+/// power-of-two alignment `align`, and its remaining lower bits are a `phase`
+/// in `0..align`. Thus every non-zero bit pattern is a valid encoding, and:
+///
+/// ```text
+/// encoded = align + phase
+/// ```
+#[cfg_attr(any(kani, test), derive(Debug, PartialEq, Eq))]
+#[repr(transparent)]
+#[derive(Copy, Clone)]
+#[cfg_attr(kani, derive(kani::Arbitrary))]
+pub(crate) struct RoundingAlignAndPhase(NonZeroUsize);
+
+impl RoundingAlignAndPhase {
+ /// Encodes `align` and `phase`.
+ ///
+ /// # Panics
+ ///
+ /// Panics if `align` is not a power of two and `phase` is not less than
+ /// `align`.
+ #[inline(always)]
+ #[cfg_attr(kani, kani::requires(align.is_power_of_two() && phase < align.get()))]
+ #[cfg_attr(kani, kani::ensures(|result| result.0.get() == align.get() + phase))]
+ pub(crate) const fn new(align: NonZeroUsize, phase: usize) -> Self {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(RoundingAlignAndPhase::new)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ RoundingAlignAndPhase::new(kani::any(), kani::any());
+ }
+
+ const_assert!(align.get().is_power_of_two());
+ const_assert!(phase < align.get());
+
+ // Since `align` is a power of two and `phase < align`, their set bits
+ // are disjoint. The result is therefore non-zero and losslessly stores
+ // both components.
+ let encoded = align.get() | phase;
+ match NonZeroUsize::new(encoded) {
+ Some(encoded) => Self(encoded),
+ None => const_unreachable!(),
+ }
+ }
+
+ /// Decodes the alignment and phase.
+ #[inline(always)]
+ #[cfg_attr(kani, kani::ensures(|&(align, phase)| {
+ align.is_power_of_two() && phase < align.get()
+ && align.get().checked_add(phase) == Some(self.0.get())
+ }))]
+ pub(crate) const fn components(self) -> (NonZeroUsize, usize) {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(RoundingAlignAndPhase::components)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ kani::any::<RoundingAlignAndPhase>().components();
+ }
+ // `leading_zeros <= POINTER_WIDTH_BITS - 1` because the encoded value
+ // is non-zero. Converting `leading_zeros` to `usize` cannot truncate
+ // because it is at most the number of bits in a `usize`.
+ #[allow(
+ clippy::arithmetic_side_effects,
+ clippy::as_conversions,
+ clippy::needless_nonzero_get
+ )]
+ let shift = (POINTER_WIDTH_BITS - 1) - (self.0.get().leading_zeros() as usize);
+ #[allow(clippy::arithmetic_side_effects)]
+ let align = 1usize << shift;
+ let align = match NonZeroUsize::new(align) {
+ Some(align) => align,
+ None => const_unreachable!(),
+ };
+ // `align` is the highest set bit, so XOR removes exactly that bit and
+ // leaves a value in `0..align`.
+ let phase = self.0.get() ^ align.get();
+ (align, phase)
+ }
+
+ /// Decodes the alignment, which is the highest set bit of `self`.
+ #[inline(always)]
+ #[cfg_attr(kani, kani::ensures(|align| {
+ align.is_power_of_two() && align.get() <= self.0.get()
+ && self.0.get() - align.get() < align.get()
+ }))]
+ pub(crate) const fn align(self) -> NonZeroUsize {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(RoundingAlignAndPhase::align)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ kani::any::<RoundingAlignAndPhase>().align();
+ }
+
+ self.components().0
+ }
+}
+
#[cfg_attr(any(kani, test), derive(Debug, PartialEq, Eq))]
#[derive(Copy, Clone)]
+#[cfg_attr(kani, derive(kani::Arbitrary))]
pub(crate) struct TrailingSliceLayout<E = usize> {
- // The offset of the first byte of the trailing slice field. Note that this
- // is NOT the same as the minimum size of the type. For example, consider
- // the following type:
- //
- // struct Foo {
- // a: u16,
- // b: u8,
- // c: [u8],
- // }
- //
- // In `Foo`, `c` is at byte offset 3. When `c.len() == 0`, `c` is followed
- // by a padding byte.
+ /// The offset of the first byte of the trailing slice field. Note that this
+ /// is NOT the same as the minimum size of the type. For example, consider
+ /// the following type:
+ ///
+ /// ```ignore
+ /// #[repr(C, align(2))]
+ /// struct Foo {
+ /// header: u16,
+ /// tag: u8,
+ /// tail: [u8],
+ /// }
+ /// ```
+ ///
+ /// In `Foo`, `tail` is at byte offset 3. When `tail.len() == 0`, `tail` is
+ /// followed by a padding byte.
pub(crate) offset: usize,
- // The size of the element type of the trailing slice field.
+ /// The size of the element type of the trailing slice field.
pub(crate) elem_size: E,
+ /// The trailing slice's byte offset plus fixed trailing padding, minus the
+ /// rounding phase.
+ ///
+ /// Fixed trailing padding excludes padding added by the final alignment
+ /// rounding. The rounding alignment and phase are stored in
+ /// `size_rounding_align_and_phase`. This value is independent of the slice
+ /// length and can be smaller than, equal to, or larger than `offset`.
+ ///
+ /// For a completed layout, this value is a multiple of `DstLayout::align`.
+ /// The minimum object size equals this value when the phase is zero;
+ /// otherwise, it exceeds this value by one rounding-alignment unit.
+ pub(crate) size_base: usize,
+ /// The alignment and byte phase of the rounding applied to the trailing
+ /// slice's contribution to object size.
+ ///
+ /// The alignment `size_align` is a nonzero power of two. The phase
+ /// `size_phase` is the fixed byte contribution inside that rounding, in
+ /// `0..size_align`. Both are independent of the trailing slice's length.
+ /// For `elems` trailing elements, the enclosing object's size is:
+ ///
+ /// ```text
+ /// size_base + round_up(size_phase + elems * elem_size, size_align)
+ /// ```
+ ///
+ /// For a completed layout, `size_align` is at least `DstLayout::align`.
+ /// Packing can reduce the enclosing type's alignment and the alignment
+ /// used to place its trailing field without removing padding required by
+ /// a nested DST's own layout. For example, a `repr(packed(2))` wrapper
+ /// around a DST with alignment 4 can have an object alignment of 2 and a
+ /// rounding alignment of 4.
+ ///
+ /// The rounding parameters account for padding preserved within nested
+ /// fields. Their rounding origin can differ from the physical start of
+ /// the trailing slice, so the phase need not equal `offset % size_align`.
+ pub(crate) size_rounding_align_and_phase: RoundingAlignAndPhase,
+}
+
+impl<E> TrailingSliceLayout<E> {
+ /// Produces `self.size_base` rounded down to a multiple of `size_align`,
+ /// plus `size_phase`, where both components come from
+ /// `self.size_rounding_align_and_phase`.
+ ///
+ /// This value need not equal `self.offset`, the trailing slice's byte
+ /// offset.
+ #[inline(always)]
+ #[cfg_attr(kani, kani::ensures(|&offset| {
+ let (align, phase) = self.size_rounding_align_and_phase.components();
+ offset == util::round_down_to_next_multiple_of_alignment(self.size_base, align) + phase
+ }))]
+ const fn size_offset(&self) -> usize {
+ #[cfg(all(kani, kani_slow))]
+ #[kani::proof_for_contract(TrailingSliceLayout::<usize>::size_offset)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ kani::any::<TrailingSliceLayout>().size_offset();
+ }
+
+ let (size_align, size_phase) = self.size_rounding_align_and_phase.components();
+ let aligned_base =
+ util::round_down_to_next_multiple_of_alignment(self.size_base, size_align);
+ // The aligned base and phase have disjoint set bits.
+ aligned_base | size_phase
+ }
+
+ /// Produces the largest trailing-slice byte capacity whose complete, padded
+ /// object fits in `available_bytes`, or `None` if the object does not fit
+ /// even with an empty trailing slice.
+ ///
+ /// This capacity is independent of `elem_size`: it need not represent a
+ /// whole number of elements, even for a valid Rust layout. For example, a
+ /// slice of `[u8; 3]` elements has three-byte elements and alignment 1.
+ /// With `available_bytes = 4`, this method produces `Some(4)`, although
+ /// only one whole element, occupying three bytes, fits.
+ #[inline(always)]
+ #[cfg_attr(kani, kani::ensures(|&result| {
+ let (align, phase) = self.size_rounding_align_and_phase.components();
+ result == available_bytes.checked_sub(self.size_base)
+ .map(|bytes| util::round_down_to_next_multiple_of_alignment(bytes, align))
+ .and_then(|rounded| rounded.checked_sub(phase))
+ }))]
+ const fn max_trailing_bytes(&self, available_bytes: usize) -> Option<usize> {
+ #[cfg(all(kani, kani_slow))]
+ #[kani::proof_for_contract(TrailingSliceLayout::<usize>::max_trailing_bytes)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ kani::any::<TrailingSliceLayout>().max_trailing_bytes(kani::any());
+ }
+
+ #[cfg(all(kani, kani_slow))]
+ #[kani::proof_for_contract(TrailingSliceLayout::<NonZeroUsize>::max_trailing_bytes)]
+ #[kani::solver(kissat)]
+ fn proof_nonzero() {
+ kani::any::<TrailingSliceLayout<NonZeroUsize>>().max_trailing_bytes(kani::any());
+ }
+
+ let (size_align, size_phase) = self.size_rounding_align_and_phase.components();
+
+ // Interpret the normalized size formula over the nonnegative integers:
+ //
+ // object_size(trailing_bytes)
+ // = self.size_base
+ // + round_up(size_phase + trailing_bytes, size_align).
+ //
+ // We seek the largest byte count with
+ // `object_size(trailing_bytes) <= available_bytes`.
+ //
+ // At zero trailing bytes, the input to rounding is just `size_phase`.
+ // Since `0 <= size_phase < size_align`, the least alignment multiple at
+ // or above it is zero for a zero phase, and `size_align` otherwise.
+ // Thus `rounded_phase = round_up(size_phase, size_align)` is exactly
+ // the rounded contribution of an empty tail.
+ let rounded_phase = if size_phase == 0 { 0 } else { size_align.get() };
+
+ // Adding the fixed base gives `minimum_size = object_size(0)`. The size
+ // formula is nondecreasing in the trailing byte count, so this is the
+ // minimum object size. If this sum exceeds `usize::MAX`, it also
+ // exceeds `available_bytes`, and no tail can fit.
+ let minimum_size = match self.size_base.checked_add(rounded_phase) {
+ Some(size) => size,
+ None => return None,
+ };
+
+ // Subtracting the minimum size gives the budget for increasing the
+ // rounded contribution. Failure means even the minimum size exceeds the
+ // available space. Success establishes `minimum_size + extra_bytes =
+ // available_bytes`.
+ let extra_bytes = match available_bytes.checked_sub(minimum_size) {
+ Some(bytes) => bytes,
+ None => return None,
+ };
+
+ // The rounded contribution can increase only by multiples of
+ // `size_align`. Its largest permitted increase is therefore
+ // `aligned_bytes = round_down(extra_bytes, size_align)`, with:
+ //
+ // aligned_bytes <= extra_bytes < aligned_bytes + size_align.
+ //
+ // Hence `rounded_phase + aligned_bytes` is the largest alignment
+ // multiple that fits after the base: adding the base gives at most
+ // `minimum_size + extra_bytes = available_bytes`, while the next
+ // alignment multiple would exceed that budget.
+ let aligned_bytes = util::round_down_to_next_multiple_of_alignment(extra_bytes, size_align);
+
+ // Rounding an empty tail adds `rounded_phase - size_phase` bytes. This
+ // is `initial_padding`: the tail can use these bytes without increasing
+ // the rounded contribution. The two cases for `rounded_phase` give `0
+ // <= initial_padding < size_align`, so the subtraction is
+ // representable.
+ #[allow(clippy::arithmetic_side_effects)]
+ let initial_padding = rounded_phase - size_phase;
+
+ // For any nonnegative candidate byte count, the aligned limit above
+ // gives the following equivalences:
+ //
+ // object_size(candidate_bytes) <= available_bytes
+ // iff round_up(size_phase + candidate_bytes, size_align)
+ // <= rounded_phase + aligned_bytes
+ // iff size_phase + candidate_bytes
+ // <= rounded_phase + aligned_bytes
+ // iff candidate_bytes <= aligned_bytes + initial_padding.
+ //
+ // Removing `round_up` is valid because its upper bound is an alignment
+ // multiple. Thus `aligned_bytes + initial_padding` is exactly the
+ // largest tail that fits. Since `aligned_bytes` is a multiple of the
+ // power-of-two `size_align` and `initial_padding < size_align`, their
+ // set bits are disjoint. Bitwise OR therefore equals their sum, with no
+ // carry or overflow.
+ let trailing_bytes = aligned_bytes | initial_padding;
+ Some(trailing_bytes)
+ }
+}
+
+impl TrailingSliceLayout {
+ /// Returns the difference between the complete object size and the
+ /// trailing-slice end for `elems` elements, modulo `usize::MAX + 1`.
+ ///
+ /// If the object size fits in `usize` and contains the trailing slice,
+ /// this is the exact trailing padding. Other inputs need not describe a
+ /// Rust type; their result is still the modular difference.
+ #[inline(always)]
+ #[cfg_attr(kani, kani::ensures(|&padding| {
+ let (align, phase) = self.size_rounding_align_and_phase.components();
+ let trailing_bytes = elems.wrapping_mul(self.elem_size);
+ let mask = align.get() - 1;
+ let rounded = phase.wrapping_add(trailing_bytes).wrapping_add(mask) & !mask;
+ let object_size = self.size_base.wrapping_add(rounded);
+ padding == object_size.wrapping_sub(self.offset.wrapping_add(trailing_bytes))
+ }))]
+ pub(crate) const fn padding_for_elems(self, elems: usize) -> usize {
+ #[cfg(all(kani, kani_slow))]
+ #[kani::proof_for_contract(TrailingSliceLayout::padding_for_elems)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ kani::any::<TrailingSliceLayout>().padding_for_elems(kani::any());
+ }
+
+ let (size_align, size_phase) = self.size_rounding_align_and_phase.components();
+
+ // Since `size_align` is a nonzero power of two, subtracting one gives
+ // a mask selecting precisely the remainder modulo `size_align`.
+ #[allow(clippy::arithmetic_side_effects)]
+ let size_mask = size_align.get() - 1;
+
+ // Removing whole alignment multiples from each element's size
+ // preserves the trailing-byte count modulo `size_align`. If elements
+ // occupy whole alignment units, this is zero, making the padding's
+ // independence from the element count explicit to the optimizer.
+ let elem_remainder = self.elem_size & size_mask;
+
+ // The product is congruent to `elems * self.elem_size` modulo
+ // `size_align`. Wrapping preserves that congruence because every
+ // power-of-two `size_align` divides `usize::MAX + 1`. Masking then
+ // gives the exact trailing-byte remainder in `0..size_align`.
+ let trailing_remainder = elems.wrapping_mul(elem_remainder) & size_mask;
+
+ // Adding `size_phase` gives the same remainder as the mathematical
+ // `size_phase + elems * self.elem_size`. Reducing before adding the
+ // phase also exposes this periodicity to the optimizer.
+ let rounding_input = size_phase.wrapping_add(trailing_remainder);
+
+ // `rounding_padding` is the unique value in `0..size_align` that
+ // rounds `rounding_input` up to an alignment multiple. Since only
+ // the remainder determines this value, it is also the padding added
+ // by `round_up(size_phase + elems * self.elem_size, size_align)`.
+ let rounding_padding = util::padding_needed_for(rounding_input, size_align);
+
+ // Write `trailing_bytes = elems * self.elem_size` over mathematical
+ // integers. Subtracting the trailing-slice end from the complete
+ // object size cancels `trailing_bytes`:
+ //
+ // size_base + round_up(size_phase + trailing_bytes, size_align)
+ // - (offset + trailing_bytes)
+ // = size_base + size_phase - offset + rounding_padding.
+ //
+ // Wrapping computes this difference modulo `usize::MAX + 1`, even
+ // if an intermediate value or the difference is unrepresentable.
+ // If the object size fits in `usize` and contains the trailing slice,
+ // the difference is in `0..=usize::MAX`; its residue is then the
+ // exact number of padding bytes after the trailing slice.
+ self.size_base
+ .wrapping_add(size_phase)
+ .wrapping_sub(self.offset)
+ .wrapping_add(rounding_padding)
+ }
+
+ /// Computes the size described by this layout for `elems` trailing slice
+ /// elements.
+ ///
+ /// Returns `None` if the element-byte count or complete rounded size
+ /// cannot be represented as a `usize`.
+ #[inline(always)]
+ #[cfg_attr(kani, kani::ensures(|&size| {
+ size == elems.checked_mul(self.elem_size)
+ .and_then(|bytes| proofs::slice_dst_size_for_trailing_bytes(self, bytes))
+ }))]
+ pub(crate) const fn size_for_elems(self, elems: usize) -> Option<usize> {
+ #[cfg(all(kani, kani_slow))]
+ #[kani::proof_for_contract(TrailingSliceLayout::size_for_elems)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ kani::any::<TrailingSliceLayout>().size_for_elems(kani::any());
+ }
+
+ // Let `size_align` and `size_phase` denote the components stored in
+ // `size_rounding_align_and_phase`. The normalized layout formula,
+ // evaluated over the nonnegative integers, is:
+ //
+ // object_size(trailing_bytes)
+ // = self.size_base
+ // + round_up(size_phase + trailing_bytes, size_align)
+ //
+ // Here, `round_up` gives the least multiple of `size_align` greater
+ // than or equal to its input. Thus `object_size` is nondecreasing.
+ // By the capacity helper's contract, `max_trailing_bytes` is the
+ // largest byte count for which `object_size` fits in a `usize`.
+ // Monotonicity then gives the equivalence:
+ //
+ // object_size(trailing_bytes) <= usize::MAX
+ // iff trailing_bytes <= max_trailing_bytes.
+ //
+ // If the helper returns `None`, even `object_size(0)` exceeds the
+ // limit, so no element count can produce a representable size.
+ let max_trailing_bytes = match self.max_trailing_bytes(usize::MAX) {
+ Some(bytes) => bytes,
+ None => return None,
+ };
+
+ // Each of the `elems` elements contributes `self.elem_size` bytes,
+ // so substitution into the formula requires their product. Checked
+ // multiplication yields that exact product, including zero for
+ // zero-sized elements. If it overflows, the complete size cannot fit
+ // either: `object_size(trailing_bytes) >= trailing_bytes` because
+ // the base, phase, and rounding increment are all nonnegative.
+ let trailing_size = match self.elem_size.checked_mul(elems) {
+ Some(bytes) => bytes,
+ None => return None,
+ };
+
+ // The equivalence above rejects exactly the remaining cases whose
+ // complete size exceeds `usize::MAX`. After this check we have
+ // `object_size(trailing_size) <= usize::MAX`.
+ if trailing_size > max_trailing_bytes {
+ return None;
+ }
+
+ // The trailing slice starts at `self.offset` and occupies
+ // `trailing_size` bytes. Thus `trailing_end` is its end offset modulo
+ // `usize::MAX + 1`. This sum need not fit: a raw layout need not
+ // describe a Rust type, so its trailing slice need not lie within
+ // the complete object size bounded above.
+ let trailing_end = self.offset.wrapping_add(trailing_size);
+
+ // `padding_for_elems` returns `object_size(trailing_size)` minus the
+ // trailing-slice end, modulo `usize::MAX + 1`. Adding the end back
+ // therefore gives `object_size(trailing_size)` modulo the same
+ // modulus. The bound above establishes that this size fits in
+ // `usize`, so its residue is the exact size specified by the layout.
+ let size = trailing_end.wrapping_add(self.padding_for_elems(elems));
+ Some(size)
+ }
+
+ /// Returns `true` only when `self` and `other` describe the same size for
+ /// every trailing slice length.
+ ///
+ /// This recognizes sufficient conditions for equality. A `false` result
+ /// also includes equivalent size sequences that this test cannot recognize.
+ #[inline(always)]
+ #[cfg_attr(kani, kani::ensures(|&same| {
+ let (self_align, self_phase) = self.size_rounding_align_and_phase.components();
+ let (other_align, other_phase) = other.size_rounding_align_and_phase.components();
+ let initial_size = proofs::slice_dst_size_for_trailing_bytes(self, 0);
+ same == (self.elem_size == other.elem_size
+ && initial_size.is_some()
+ && initial_size == proofs::slice_dst_size_for_trailing_bytes(other, 0)
+ && ((self.elem_size % self_align.get() == 0
+ && self.elem_size % other_align.get() == 0)
+ || (self_align == other_align && self_phase == other_phase)))
+ }))]
+ pub(crate) const fn has_same_size_sequence(self, other: Self) -> bool {
+ #[cfg(all(kani, kani_slow))]
+ #[kani::proof_for_contract(TrailingSliceLayout::has_same_size_sequence)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ kani::any::<TrailingSliceLayout>().has_same_size_sequence(kani::any());
+ }
+
+ // For either layout, interpret its normalized size formula over the
+ // nonnegative integers:
+ //
+ // object_size(elems)
+ // = size_base
+ // + round_up(size_phase + elems * elem_size, size_align).
+ //
+ // In two cases below, we establish equality of these mathematical sizes
+ // for every `elems`. Equal sizes also have the same `usize`
+ // representability bound.
+ //
+ // Both cases require a common byte contribution at each element count.
+ // Equal element sizes give that shared term, `elems * elem_size`.
+ if self.elem_size != other.elem_size {
+ return false;
+ }
+
+ // At `elems == 0`, the element contribution vanishes. Require equal,
+ // representable initial sizes; each case below then shows that
+ // increasing `elems` adds the same amount to both initial sizes.
+ match (self.size_for_elems(0), other.size_for_elems(0)) {
+ (Some(self_size), Some(other_size)) if self_size == other_size => {}
+ _ => return false,
+ }
+
+ let (self_align, self_phase) = self.size_rounding_align_and_phase.components();
+ let (other_align, other_phase) = other.size_rounding_align_and_phase.components();
+ let self_align = self_align.get();
+ let other_align = other_align.get();
+
+ // Case 1: The element size is a multiple of both alignments.
+ //
+ // For two power-of-two alignments, the larger is divisible by the
+ // smaller. Thus `max_align` is their least common multiple, and a byte
+ // count is divisible by both alignments exactly when it is divisible by
+ // `max_align`.
+ let max_align = if self_align > other_align { self_align } else { other_align };
+
+ // If the element size is divisible by both alignments, so is `elems *
+ // elem_size`. Adding an alignment multiple commutes with rounding,
+ // giving the following identity for each layout:
+ //
+ // round_up(size_phase + elems * elem_size, size_align)
+ // = round_up(size_phase, size_align) + elems * elem_size.
+ //
+ // Consequently, `object_size(elems) = object_size(0) + elems *
+ // elem_size`. The initial sizes and element sizes already agree, so the
+ // complete size sequences agree. This also covers zero-sized elements,
+ // whose contribution is zero for every element count.
+ #[allow(clippy::arithmetic_side_effects)]
+ if self.elem_size % max_align == 0 {
+ return true;
+ }
+
+ // Case 2: The rounding alignments and phases match.
+ //
+ // Require a common rounding alignment before comparing phases. If the
+ // alignments differ, neither case establishes equality.
+ if self_align != other_align {
+ return false;
+ }
+
+ // If the phases also agree, both `round_up` operations have the same
+ // input and alignment at every element count. In particular, their
+ // rounded terms agree at zero. Subtracting that common term from the
+ // equal initial sizes proves `self.size_base == other.size_base`. Both
+ // the fixed and rounded contributions therefore agree for every
+ // `elems`, which establishes equality of the complete size formulas.
+ self_phase == other_phase
+ }
+
+ /// Advances the size formula by `bytes` while replacing its element size.
+ ///
+ /// Let `(size_align, size_phase)` be decoded from
+ /// `self.size_rounding_align_and_phase`. For a trailing slice length
+ /// `elems`, the returned layout describes:
+ ///
+ /// ```text
+ /// self.size_base
+ /// + round_up(size_phase + bytes + elems * elem_size, size_align)
+ /// ```
+ ///
+ /// Here, `elem_size` is the replacement element size passed to this
+ /// method. The returned layout preserves the physical slice offset
+ /// `self.offset` and the rounding alignment; advancing changes the size
+ /// calculation.
+ ///
+ /// Returns `None` if advancing cannot produce representable normalized
+ /// components. Even when those components fit, evaluating the rounded
+ /// object size may overflow for some or all trailing slice lengths.
+ #[inline(always)]
+ #[cfg_attr(kani, kani::ensures(|result| {
+ let expected_offset = self.size_offset().checked_add(bytes);
+ match result {
+ Some(advanced) => {
+ let align = self.size_rounding_align_and_phase.align();
+ expected_offset == Some(advanced.size_offset())
+ && advanced.offset == self.offset
+ && advanced.elem_size == elem_size
+ && advanced.size_rounding_align_and_phase.align() == align
+ && (advanced.size_base & (align.get() - 1)) == (self.size_base & (align.get() - 1))
+ }
+ None => expected_offset.is_none(),
+ }
+ }))]
+ const fn advance(self, bytes: usize, elem_size: usize) -> Option<Self> {
+ #[cfg(all(kani, kani_slow))]
+ #[kani::proof_for_contract(TrailingSliceLayout::advance)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ kani::any::<TrailingSliceLayout>().advance(kani::any(), kani::any());
+ }
+
+ let (size_align, size_phase) = self.size_rounding_align_and_phase.components();
+
+ // Advancing replaces the phase inside `round_up` with `size_phase +
+ // bytes`. To normalize this sum, separate it into whole alignment units
+ // and a remainder. The whole units can move outside `round_up` into the
+ // base; the remainder becomes the new phase.
+ //
+ // Since `size_align` is a nonzero power of two, `align_mask` selects
+ // exactly the bits representing a remainder modulo `size_align`.
+ // Nonzero `size_align` also makes this subtraction representable.
+ #[allow(clippy::arithmetic_side_effects)]
+ let align_mask = size_align.get() - 1;
+
+ // The base can increase by at most `usize::MAX - self.size_base`. Since
+ // only whole alignment units move into the base, the largest
+ // transferable contribution is:
+ //
+ // whole_capacity
+ // = round_down(usize::MAX - self.size_base, size_align).
+ //
+ // A remainder up to `align_mask` stays in the phase and consumes no
+ // base capacity. Thus the largest advanced phase that can be split into
+ // representable normalized components is `phase_capacity =
+ // whole_capacity + align_mask`. Setting the low bits of the remaining
+ // base capacity computes exactly that sum. Its two terms have disjoint
+ // set bits, so it cannot overflow. Any larger sum would transfer more
+ // whole units than the base can hold.
+ #[allow(clippy::arithmetic_side_effects)]
+ let phase_capacity = (usize::MAX - self.size_base) | align_mask;
+
+ // The existing phase already occupies `size_phase` bytes of this
+ // capacity. Subtracting it gives the greatest permitted increment:
+ //
+ // bytes <= max_advance
+ // iff size_phase + bytes <= phase_capacity.
+ //
+ // Since `phase_capacity >= align_mask >= size_phase`, the subtraction
+ // cannot underflow.
+ #[allow(clippy::arithmetic_side_effects)]
+ let max_advance = phase_capacity - size_phase;
+ if bytes > max_advance {
+ return None;
+ }
+
+ // This is the fixed term inside rounding after the requested
+ // advancement, before normalization. The check above proves `size_phase
+ // + bytes <= phase_capacity <= usize::MAX`, so the addition is
+ // representable.
+ #[allow(clippy::arithmetic_side_effects)]
+ let advanced_phase = size_phase + bytes;
+
+ // Taking the remainder retains precisely the part that cannot move
+ // outside rounding as whole alignment units. In particular, `0 <=
+ // normalized_phase < size_align`, as required for the new rounding
+ // parameters.
+ let normalized_phase = advanced_phase & align_mask;
+
+ // Rounding down removes precisely the remainder retained in the
+ // phase, so `whole_bytes = advanced_phase - normalized_phase`.
+ let whole_bytes =
+ util::round_down_to_next_multiple_of_alignment(advanced_phase, size_align);
+
+ // Because `advanced_phase <= phase_capacity`, we have
+ // `whole_bytes <= whole_capacity <= usize::MAX - self.size_base`.
+ // Adding this transferred contribution to the base therefore fits.
+ #[allow(clippy::arithmetic_side_effects)]
+ let size_base = self.size_base + whole_bytes;
+
+ // To verify the returned formula, let `trailing_size = elems *
+ // elem_size` and evaluate the following over the nonnegative integers.
+ // Since `whole_bytes` is a multiple of `size_align`, it can move back
+ // inside `round_up` without changing the result:
+ //
+ // size_base + round_up(normalized_phase + trailing_size, size_align)
+ // = self.size_base
+ // + round_up(whole_bytes + normalized_phase + trailing_size, size_align)
+ // = self.size_base
+ // + round_up(size_phase + bytes + trailing_size, size_align).
+ //
+ // This is the requested advanced formula. The replacement element size
+ // supplies its new coefficient of `elems`; `offset` retains the
+ // caller's field-placement description.
+ Some(Self {
+ offset: self.offset,
+ elem_size,
+ size_base,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(size_align, normalized_phase),
+ })
+ }
}
impl SizeInfo {
@@ -73,12 +725,42 @@ impl SizeInfo {
#[allow(unused)]
#[cfg_attr(not(zerocopy_inline_always), inline)]
#[cfg_attr(zerocopy_inline_always, inline(always))]
+ #[cfg_attr(kani, kani::ensures(|result| match (*self, *result) {
+ (SizeInfo::Sized { size }, Some(SizeInfo::Sized { size: converted })) => {
+ size == converted
+ }
+ (SizeInfo::SliceDst(original), Some(SizeInfo::SliceDst(converted))) => {
+ converted.elem_size.get() == original.elem_size
+ && converted.offset == original.offset
+ && converted.size_base == original.size_base
+ && converted.size_rounding_align_and_phase == original.size_rounding_align_and_phase
+ }
+ (SizeInfo::SliceDst(original), None) => original.elem_size == 0,
+ _ => false,
+ }))]
const fn try_to_nonzero_elem_size(&self) -> Option<SizeInfo<NonZeroUsize>> {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(SizeInfo::try_to_nonzero_elem_size)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ kani::any::<SizeInfo>().try_to_nonzero_elem_size();
+ }
+
Some(match *self {
SizeInfo::Sized { size } => SizeInfo::Sized { size },
- SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => {
+ SizeInfo::SliceDst(TrailingSliceLayout {
+ offset,
+ elem_size,
+ size_base,
+ size_rounding_align_and_phase,
+ }) => {
if let Some(elem_size) = NonZeroUsize::new(elem_size) {
- SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size })
+ SizeInfo::SliceDst(TrailingSliceLayout {
+ offset,
+ elem_size,
+ size_base,
+ size_rounding_align_and_phase,
+ })
} else {
return None;
}
@@ -87,6 +769,40 @@ const fn try_to_nonzero_elem_size(&self) -> Option<SizeInfo<NonZeroUsize>> {
}
}
+/// Returns the largest number of `elem_size`-byte elements which fit in
+/// `bytes`, along with the number of bytes those elements occupy.
+#[cfg_attr(
+ kani,
+ kani::ensures(|&(elems, used)| {
+ (match elems.checked_mul(elem_size.get()) {
+ Some(product) => {
+ product == used && used <= bytes && bytes - used < elem_size.get()
+ }
+ None => false,
+ }) && used
+ .checked_add(elem_size.get())
+ .map(|next_used| next_used > bytes)
+ .unwrap_or(true)
+ })
+)]
+#[inline(always)]
+const fn max_elems_for_bytes(bytes: usize, elem_size: NonZeroUsize) -> (usize, usize) {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(max_elems_for_bytes)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ max_elems_for_bytes(kani::any(), kani::any());
+ }
+
+ #[allow(clippy::arithmetic_side_effects)]
+ let elems = bytes / elem_size.get();
+ let used = match elems.checked_mul(elem_size.get()) {
+ Some(used) => used,
+ None => const_unreachable!(),
+ };
+ (elems, used)
+}
+
#[doc(hidden)]
#[derive(Copy, Clone)]
#[cfg_attr(test, derive(Debug))]
@@ -97,6 +813,7 @@ pub enum CastType {
}
#[cfg_attr(test, derive(Debug))]
+#[cfg_attr(kani, derive(kani::Arbitrary))]
pub(crate) enum MetadataCastError {
Alignment,
Size,
@@ -128,7 +845,7 @@ impl DstLayout {
/// 2<sup>29</sup>.
#[cfg(not(kani))]
#[cfg(not(target_pointer_width = "16"))]
- pub(crate) const CURRENT_MAX_ALIGN: NonZeroUsize = match NonZeroUsize::new(1 << 28) {
+ pub(crate) const CURRENT_MAX_ALIGN: NonZeroUsize = match NonZeroUsize::new(1 << 29) {
Some(max_align) => max_align,
None => const_unreachable!(),
};
@@ -164,7 +881,18 @@ impl DstLayout {
/// shallow padding, unsafe code may assume that the result of this method
/// accurately reflects the size, alignment, and lack of static shallow
/// padding of that type.
+ #[cfg_attr(kani, kani::ensures(|result| {
+ result.align == self.align && result.size_info == self.size_info
+ && result.statically_shallow_unpadded
+ }))]
const fn assume_shallow_unpadded(self) -> Self {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(DstLayout::assume_shallow_unpadded)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ kani::any::<DstLayout>().assume_shallow_unpadded();
+ }
+
Self { statically_shallow_unpadded: true, ..self }
}
@@ -182,7 +910,20 @@ const fn assume_shallow_unpadded(self) -> Self {
#[doc(hidden)]
#[must_use]
#[inline]
+ #[cfg_attr(kani, kani::requires(repr_align.map_or(true, |align| align.is_power_of_two())))]
+ #[cfg_attr(kani, kani::ensures(|result| {
+ result.align == repr_align.unwrap_or(Self::MIN_ALIGN)
+ && result.size_info == SizeInfo::Sized { size: 0 }
+ && result.statically_shallow_unpadded
+ }))]
pub const fn new_zst(repr_align: Option<NonZeroUsize>) -> DstLayout {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(DstLayout::new_zst)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ let _ = DstLayout::new_zst(kani::any());
+ }
+
let align = match repr_align {
Some(align) => align,
None => Self::MIN_ALIGN,
@@ -206,7 +947,24 @@ pub const fn new_zst(repr_align: Option<NonZeroUsize>) -> DstLayout {
#[doc(hidden)]
#[must_use]
#[inline]
+ #[cfg_attr(kani, kani::ensures(|result| {
+ result.align.get() == mem::align_of::<T>()
+ && result.size_info == SizeInfo::Sized { size: mem::size_of::<T>() }
+ && result.statically_shallow_unpadded == false
+ }))]
pub const fn for_type<T>() -> DstLayout {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(DstLayout::for_type::<()>)]
+ fn proof_zst() {
+ let _ = DstLayout::for_type::<()>();
+ }
+
+ #[cfg(kani)]
+ #[kani::proof_for_contract(DstLayout::for_type::<(u8, u64)>)]
+ fn proof_padded() {
+ let _ = DstLayout::for_type::<(u8, u64)>();
+ }
+
// SAFETY: `align` is correct by construction. `T: Sized`, and so it is
// sound to initialize `size_info` to `SizeInfo::Sized { size }`; the
// `size` field is also correct by construction. `unpadded` can safely
@@ -221,16 +979,39 @@ pub const fn for_type<T>() -> DstLayout {
}
}
- /// Constructs a `DstLayout` which describes a `T` that does not contain
- /// padding.
+ /// Produces `T`'s size and alignment with static shallow padding assumed
+ /// absent.
+ ///
+ /// This method records the assumption without checking `T` for padding.
+ /// It can be used to ignore padding within a field when computing padding
+ /// introduced by its enclosing struct.
///
/// # Safety
///
- /// Unsafe code may assume that `DstLayout` is the correct layout for `T`.
+ /// Unsafe code may rely on the result's size and alignment being correct
+ /// for `T`. Relying on its recorded absence of static shallow padding
+ /// requires an independent justification that `T` lacks such padding.
#[doc(hidden)]
#[must_use]
#[inline]
+ #[cfg_attr(kani, kani::ensures(|result| {
+ result.align.get() == mem::align_of::<T>()
+ && result.size_info == SizeInfo::Sized { size: mem::size_of::<T>() }
+ && result.statically_shallow_unpadded == true
+ }))]
pub const fn for_unpadded_type<T>() -> DstLayout {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(DstLayout::for_unpadded_type::<()>)]
+ fn proof_zst() {
+ let _ = DstLayout::for_unpadded_type::<()>();
+ }
+
+ #[cfg(kani)]
+ #[kani::proof_for_contract(DstLayout::for_unpadded_type::<(u8, u64)>)]
+ fn proof_padded() {
+ let _ = DstLayout::for_unpadded_type::<(u8, u64)>();
+ }
+
Self::for_type::<T>().assume_shallow_unpadded()
}
@@ -239,7 +1020,31 @@ pub const fn for_unpadded_type<T>() -> DstLayout {
/// # Safety
///
/// Unsafe code may assume that `DstLayout` is the correct layout for `[T]`.
+ #[cfg_attr(kani, kani::ensures(|result| {
+ result.align.get() == mem::align_of::<T>() && result.statically_shallow_unpadded
+ && matches!(result.size_info, SizeInfo::SliceDst(trailing)
+ if trailing.offset == 0 && trailing.size_base == 0
+ && trailing.elem_size == mem::size_of::<T>()
+ && trailing.size_rounding_align_and_phase.components() == (result.align, 0))
+ }))]
pub(crate) const fn for_slice<T>() -> DstLayout {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(DstLayout::for_slice::<()>)]
+ fn proof_zst() {
+ let _ = DstLayout::for_slice::<()>();
+ }
+
+ #[cfg(kani)]
+ #[kani::proof_for_contract(DstLayout::for_slice::<(u8, u64)>)]
+ fn proof_padded() {
+ let _ = DstLayout::for_slice::<(u8, u64)>();
+ }
+
+ let align = match NonZeroUsize::new(mem::align_of::<T>()) {
+ Some(align) => align,
+ None => const_unreachable!(),
+ };
+
// SAFETY: The alignment of a slice is equal to the alignment of its
// element type, and so `align` is initialized correctly.
//
@@ -249,13 +1054,12 @@ pub(crate) const fn for_slice<T>() -> DstLayout {
// construction. Since `[T]` is a (degenerate case of a) slice DST, it
// is correct to initialize `size_info` to `SizeInfo::SliceDst`.
DstLayout {
- align: match NonZeroUsize::new(mem::align_of::<T>()) {
- Some(align) => align,
- None => const_unreachable!(),
- },
+ align,
size_info: SizeInfo::SliceDst(TrailingSliceLayout {
offset: 0,
elem_size: mem::size_of::<T>(),
+ size_base: 0,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(align, 0),
}),
statically_shallow_unpadded: true,
}
@@ -273,15 +1077,42 @@ pub(crate) const fn for_slice<T>() -> DstLayout {
/// alignment modifiers and fields corresponding to that definition, the
/// resulting `DstLayout` will correctly encode the layout of that struct.
///
+ /// This guarantee uses one explicit compatibility premise for generic
+ /// indirection: while Rust accepts an instantiation which transitively
+ /// places a `repr(align)` type inside a `repr(packed)` type, that
+ /// instantiation follows the otherwise-applicable `repr(C)`,
+ /// `repr(packed)`, and inner-type layout rules. The Reference prohibits the
+ /// direct form, and Rust's continued acceptance and layout behavior are
+ /// outside the algebra proved by this implementation. Re-audit if the
+ /// compiler rejects such an instantiation or assigns it a different
+ /// layout.
+ ///
/// We make no guarantees to the behavior of this method when it is invoked
/// with arguments that cannot correspond to a valid `repr(C)` struct.
#[must_use]
#[inline]
+ #[cfg_attr(kani, kani::requires(proofs::repr_c_layout(repr_align, repr_packed, fields).is_some()))]
+ #[cfg_attr(kani, kani::ensures(|&result| {
+ Some(result) == proofs::repr_c_layout(repr_align, repr_packed, fields)
+ }))]
pub const fn for_repr_c_struct(
repr_align: Option<NonZeroUsize>,
repr_packed: Option<NonZeroUsize>,
fields: &[DstLayout],
) -> DstLayout {
+ // This harness covers up to three fields, including an optional final
+ // DST. Arbitrary-length composition is not established by this bound.
+ #[cfg(all(kani, kani_slow))]
+ #[kani::proof_for_contract(DstLayout::for_repr_c_struct)]
+ #[kani::solver(kissat)]
+ #[kani::unwind(4)]
+ fn proof() {
+ let fields: [DstLayout; 3] = kani::any();
+ let len: usize = kani::any();
+ kani::assume(len <= fields.len());
+ let _ = DstLayout::for_repr_c_struct(kani::any(), kani::any(), &fields[..len]);
+ }
+
let mut layout = DstLayout::new_zst(repr_align);
let mut i = 0;
@@ -338,7 +1169,18 @@ pub const fn for_repr_c_struct(
#[doc(hidden)]
#[must_use]
#[inline]
+ #[cfg_attr(kani, kani::requires(proofs::extended_layout(self, field, repr_packed).is_some()))]
+ #[cfg_attr(kani, kani::ensures(|&result| {
+ Some(result) == proofs::extended_layout(self, field, repr_packed)
+ }))]
pub const fn extend(self, field: DstLayout, repr_packed: Option<NonZeroUsize>) -> Self {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(DstLayout::extend)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ let _ = kani::any::<DstLayout>().extend(kani::any(), kani::any());
+ }
+
use util::{max, min, padding_needed_for};
// If `repr_packed` is `None`, there are no alignment constraints, and
@@ -427,6 +1269,8 @@ pub const fn extend(self, field: DstLayout, repr_packed: Option<NonZeroUsize>) -
SizeInfo::SliceDst(TrailingSliceLayout {
offset: trailing_offset,
elem_size,
+ size_base,
+ size_rounding_align_and_phase,
}) => {
// If the trailing field is dynamically sized, so too
// will the resulting layout. The offset of the trailing
@@ -440,11 +1284,30 @@ pub const fn extend(self, field: DstLayout, repr_packed: Option<NonZeroUsize>) -
// panic otherwise (e.g., combining or aligning the
// components would create a size exceeding
// `usize::MAX`).
- let offset = match offset.checked_add(trailing_offset) {
+ let trailing_offset = match offset.checked_add(trailing_offset) {
Some(offset) => offset,
None => const_panic!("`field` cannot be appended without the total size overflowing `usize`"),
};
- SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size })
+
+ // Before normalization, the composite's size is:
+ //
+ // offset + size_base
+ // + round_up(size_phase + elems * elem_size, size_align)
+ //
+ // `offset` does not participate in the rounding,
+ // so it can be added directly to the normalized
+ // base.
+ let size_base = match offset.checked_add(size_base) {
+ Some(base) => base,
+ None => const_panic!("`field` cannot be appended without the total size overflowing `usize`"),
+ };
+
+ SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: trailing_offset,
+ elem_size,
+ size_base,
+ size_rounding_align_and_phase,
+ })
}
},
)
@@ -459,18 +1322,11 @@ pub const fn extend(self, field: DstLayout, repr_packed: Option<NonZeroUsize>) -
}
/// Like `Layout::pad_to_align`, this routine rounds the size of this layout
- /// up to the nearest multiple of this type's alignment or `repr_packed`
- /// (whichever is less). This method leaves DST layouts unchanged, since the
- /// trailing padding of DSTs is computed at runtime.
- ///
- /// The accompanying boolean is `true` if the resulting composition of
- /// fields necessitated static (as opposed to dynamic) padding; otherwise
- /// `false`.
+ /// up to the nearest multiple of this type's alignment. For DST layouts,
+ /// this updates the runtime size formula to account for trailing padding.
///
/// In order to match the layout of a `#[repr(C)]` struct, this method
- /// should be invoked after the invocations of [`DstLayout::extend`]. If
- /// `self` corresponds to a type marked with `repr(packed(N))`, then
- /// `repr_packed` should be set to `Some(N)`, otherwise `None`.
+ /// should be invoked after the invocations of [`DstLayout::extend`].
///
/// This method cannot be used to match the layout of a record with the
/// default representation, as that representation is mostly unspecified.
@@ -480,7 +1336,7 @@ pub const fn extend(self, field: DstLayout, repr_packed: Option<NonZeroUsize>) -
/// If a (potentially hypothetical) valid `repr(C)` type begins with fields
/// whose layout are `self` followed only by zero or more bytes of trailing
/// padding (not included in `self`), then unsafe code may rely on
- /// `self.pad_to_align(repr_packed)` producing a layout that correctly
+ /// `self.pad_to_align()` producing a layout that correctly
/// encapsulates the layout of that type.
///
/// We make no guarantees to the behavior of this method if `self` cannot
@@ -489,7 +1345,16 @@ pub const fn extend(self, field: DstLayout, repr_packed: Option<NonZeroUsize>) -
#[doc(hidden)]
#[must_use]
#[inline]
+ #[cfg_attr(kani, kani::requires(proofs::padded_layout(self).is_some()))]
+ #[cfg_attr(kani, kani::ensures(|&result| Some(result) == proofs::padded_layout(self)))]
pub const fn pad_to_align(self) -> Self {
+ #[cfg(all(kani, kani_slow))]
+ #[kani::proof_for_contract(DstLayout::pad_to_align)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ let _ = kani::any::<DstLayout>().pad_to_align();
+ }
+
use util::padding_needed_for;
let (static_padding, size_info) = match self.size_info {
@@ -504,10 +1369,69 @@ pub const fn pad_to_align(self) -> Self {
(padding, SizeInfo::Sized { size })
}
// For DST layouts, trailing padding depends on the length of the
- // trailing DST and is computed at runtime. This does not alter the
- // offset or element size of the layout, so we leave `size_info`
- // unchanged.
- size_info @ SizeInfo::SliceDst(_) => (0, size_info),
+ // trailing DST and is computed at runtime. Normalize the composed
+ // rounding operation back into the representation documented on
+ // `TrailingSliceLayout`.
+ SizeInfo::SliceDst(mut trailing) => {
+ let (size_align, size_phase) = trailing.size_rounding_align_and_phase.components();
+ if size_align.get() < self.align.get() {
+ // For a trailing-slice byte count `trailing_size`, the
+ // input layout describes:
+ //
+ // trailing.size_base
+ // + round_up(size_phase + trailing_size, size_align)
+ //
+ // Here, `size_align < self.align`. Both alignments are
+ // powers of two, so rounding the complete input size to
+ // `self.align` is equivalent to:
+ //
+ // rounded_base = round_up(trailing.size_base, size_align)
+ // fixed_contribution = rounded_base + size_phase
+ // new_phase = fixed_contribution % self.align
+ // new_base = fixed_contribution - new_phase
+ // new_base + round_up(new_phase + trailing_size, self.align)
+ //
+ // `prove_dst_layout_pad_to_align_dst_outer_alignment_transition`
+ // checks the exact field transition and preserved
+ // invariants. Its `_formula` companion proves this
+ // identity for every representable trailing byte count.
+ let base_padding = padding_needed_for(trailing.size_base, size_align);
+ let rounded_base = match trailing.size_base.checked_add(base_padding) {
+ Some(base) => base,
+ None => const_panic!("Adding padding caused size to overflow `usize`."),
+ };
+ let fixed_contribution = match rounded_base.checked_add(size_phase) {
+ Some(bytes) => bytes,
+ None => const_panic!("Adding padding caused size to overflow `usize`."),
+ };
+ #[allow(clippy::arithmetic_side_effects)]
+ let phase = fixed_contribution & (self.align.get() - 1);
+ // Rounding down removes `phase`, the fixed contribution
+ // modulo `self.align`, leaving the new base.
+ let size_base = util::round_down_to_next_multiple_of_alignment(
+ fixed_contribution,
+ self.align,
+ );
+ trailing.size_base = size_base;
+ trailing.size_rounding_align_and_phase =
+ RoundingAlignAndPhase::new(self.align, phase);
+ } else {
+ // `size_align` is a multiple of `self.align`, so
+ // the inner rounded size is already aligned to
+ // `self.align`. Only `size_base` needs to be rounded.
+ // `prove_dst_layout_pad_to_align_dst_inner_rounding_transition`
+ // checks the exact field transition and preserved
+ // invariants. Its `_formula` companion proves size
+ // equivalence for every representable trailing byte count.
+ let padding = padding_needed_for(trailing.size_base, self.align);
+ trailing.size_base = match trailing.size_base.checked_add(padding) {
+ Some(base) => base,
+ None => const_panic!("Adding padding caused size to overflow `usize`."),
+ };
+ }
+
+ (0, SizeInfo::SliceDst(trailing))
+ }
};
let statically_shallow_unpadded = self.statically_shallow_unpadded && static_padding == 0;
@@ -515,48 +1439,127 @@ pub const fn pad_to_align(self) -> Self {
DstLayout { align: self.align, size_info, statically_shallow_unpadded }
}
- /// Produces `true` if `self` requires static padding; otherwise `false`.
+ /// Produces `false` when this layout is marked as having no static shallow
+ /// padding, and `true` otherwise.
+ ///
+ /// A `true` result is conservative: it does not establish that the type
+ /// contains padding. For example, this method produces `true` for
+ /// `DstLayout::for_type::<u8>()`, since `for_type` does not record whether
+ /// static shallow padding is absent.
#[must_use]
#[inline(always)]
+ #[cfg_attr(kani, kani::ensures(|&padding| padding == !self.statically_shallow_unpadded))]
pub const fn requires_static_padding(self) -> bool {
+ #[cfg(kani)]
+ #[kani::proof_for_contract(DstLayout::requires_static_padding)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ let _ = kani::any::<DstLayout>().requires_static_padding();
+ }
+
!self.statically_shallow_unpadded
}
- /// Produces `true` if there exists any metadata for which a type of layout
- /// `self` would require dynamic trailing padding; otherwise `false`.
+ /// Produces `false` only if every valid metadata describes an instance
+ /// which requires no dynamic trailing padding.
+ ///
+ /// A `true` result conservatively selects the dynamic-padding path; it does
+ /// not guarantee that any valid metadata actually requires padding.
#[must_use]
#[inline(always)]
+ #[cfg_attr(kani, kani::ensures(|&padding| match self.size_info {
+ SizeInfo::Sized { .. } => !padding,
+ SizeInfo::SliceDst(trailing) => padding == (
+ proofs::slice_dst_size_for_trailing_bytes(trailing, 0) != Some(trailing.offset)
+ || trailing.elem_size % trailing.size_rounding_align_and_phase.align().get() != 0
+ ),
+ }))]
pub const fn requires_dynamic_padding(self) -> bool {
- // A `% self.align.get()` cannot panic, since `align` is non-zero.
- #[allow(clippy::arithmetic_side_effects)]
+ #[cfg(all(kani, kani_slow))]
+ #[kani::proof_for_contract(DstLayout::requires_dynamic_padding)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ let _ = kani::any::<DstLayout>().requires_dynamic_padding();
+ }
+
match self.size_info {
SizeInfo::Sized { .. } => false,
SizeInfo::SliceDst(trailing_slice_layout) => {
- // SAFETY: This predicate is formally proved sound by
- // `proofs::prove_requires_dynamic_padding`.
- trailing_slice_layout.offset % self.align.get() != 0
- || trailing_slice_layout.elem_size % self.align.get() != 0
+ // SAFETY: `proofs::prove_requires_dynamic_padding` formally
+ // proves the safety-critical direction: when this predicate
+ // returns false, no valid metadata requires dynamic padding.
+ match trailing_slice_layout.size_for_elems(0) {
+ Some(initial_size) => {
+ #[allow(clippy::arithmetic_side_effects)]
+ let has_static_stride = trailing_slice_layout.elem_size
+ % trailing_slice_layout.size_rounding_align_and_phase.align().get()
+ == 0;
+ initial_size != trailing_slice_layout.offset || !has_static_stride
+ }
+ None => true,
+ }
+ }
+ }
+ }
+
+ /// Returns the largest trailing slice length whose object size is exactly
+ /// `size`.
+ ///
+ /// Returns `None` if there is no such length, if this describes a sized
+ /// type, or if the trailing slice element is zero-sized.
+ #[inline(always)]
+ #[cfg_attr(kani, kani::ensures(|&result| {
+ result == match self.size_info {
+ SizeInfo::SliceDst(trailing) if trailing.elem_size != 0 => {
+ proofs::max_trailing_bytes(trailing, size)
+ .map(|bytes| bytes / trailing.elem_size)
+ .filter(|&elems| trailing.size_for_elems(elems) == Some(size))
+ }
+ _ => None,
+ }
+ }))]
+ const fn metadata_for_exact_size(&self, size: usize) -> Option<usize> {
+ #[cfg(all(kani, kani_slow))]
+ #[kani::proof_for_contract(DstLayout::metadata_for_exact_size)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ kani::any::<DstLayout>().metadata_for_exact_size(kani::any());
+ }
+
+ match self.size_info {
+ SizeInfo::Sized { .. }
+ | SizeInfo::SliceDst(TrailingSliceLayout { elem_size: 0, .. }) => None,
+ SizeInfo::SliceDst(_) => {
+ // Address zero is aligned to every supported alignment, and
+ // `0 + size` cannot overflow. The validator returns the
+ // largest metadata whose object fits in `size`; accepting it
+ // only when the resulting object consumes all `size` bytes
+ // turns that bounded query into an exact-size query.
+ match self.validate_cast_and_convert_metadata(0, size, CastType::Prefix) {
+ Ok((elems, object_size)) if object_size == size => Some(elems),
+ _ => None,
+ }
}
}
}
- /// Validates that a cast is sound from a layout perspective.
+ /// Given a completed type layout, produces metadata and a split index for
+ /// a prefix or suffix cast that satisfies the type's size and alignment
+ /// requirements.
///
- /// Validates that the size and alignment requirements of a type with the
- /// layout described in `self` would not be violated by performing a
- /// `cast_type` cast from a pointer with address `addr` which refers to a
- /// memory region of size `bytes_len`.
+ /// `addr` and `bytes_len` describe the address and size of the source
+ /// memory region, and `cast_type` selects its prefix or suffix.
///
/// If the cast is valid, `validate_cast_and_convert_metadata` returns
- /// `(elems, split_at)`. If `self` describes a dynamically-sized type, then
- /// `elems` is the maximum number of trailing slice elements for which a
- /// cast would be valid (for sized types, `elem` is meaningless and should
- /// be ignored). `split_at` is the index at which to split the memory region
- /// in order for the prefix (suffix) to contain the result of the cast, and
- /// in order for the remaining suffix (prefix) to contain the leftover
- /// bytes.
+ /// `Ok((elems, split_at))`. For a dynamically-sized type, `elems` is the
+ /// maximum number of trailing slice elements for which the cast is valid.
+ /// For sized types, `elems` is meaningless and should be ignored.
+ /// `split_at` is the index at which to split the memory region so that the
+ /// prefix (suffix) contains the result of the cast and the remaining
+ /// suffix (prefix) contains the leftover bytes.
///
/// There are three conditions under which a cast can fail:
+ ///
/// - The smallest possible value for the type is larger than the provided
/// memory region
/// - A prefix cast is requested, and `addr` does not satisfy `self`'s
@@ -568,10 +1571,10 @@ pub const fn requires_dynamic_padding(self) -> bool {
///
/// # Safety
///
- /// The caller may assume that this implementation is correct, and may rely
- /// on that assumption for the soundness of their code. In particular, the
- /// caller may assume that, if `validate_cast_and_convert_metadata` returns
- /// `Some((elems, split_at))`, then:
+ /// When `self` accurately describes a completed type layout, including
+ /// trailing padding, and `addr + bytes_len` fits in `usize`, callers may
+ /// rely on the following guarantees for `Ok((elems, split_at))`:
+ ///
/// - A pointer to the type (for dynamically sized types, this includes
/// `elems` as its pointer metadata) describes an object of size `size <=
/// bytes_len`
@@ -582,6 +1585,11 @@ pub const fn requires_dynamic_padding(self) -> bool {
/// - `split_at == bytes_len - size`
/// - `addr + split_at` satisfies `self`'s alignment
///
+ /// These guarantees require the completed-layout premise; `DstLayout`
+ /// also permits unfinished layout fragments and arbitrary field values.
+ /// In particular, suffix alignment relies on every object size being a
+ /// multiple of the type's alignment.
+ ///
/// Note that this method does *not* ensure that a pointer constructed from
/// its return values will be a valid pointer. In particular, this method
/// does not reason about `isize` overflow, which is a requirement of many
@@ -606,12 +1614,57 @@ pub const fn requires_dynamic_padding(self) -> bool {
/// condition, even if `debug_assertions` are enabled.
#[allow(unused)]
#[inline(always)]
+ #[cfg_attr(kani, kani::requires(addr.checked_add(bytes_len).is_some()))]
+ #[cfg_attr(kani, kani::requires(!matches!(self.size_info,
+ SizeInfo::SliceDst(TrailingSliceLayout { elem_size: 0, .. }))))]
+ #[cfg_attr(kani, kani::ensures(|result| {
+ let endpoint = match cast_type {
+ CastType::Prefix => addr,
+ CastType::Suffix => addr + bytes_len,
+ };
+ if endpoint % self.align.get() != 0 {
+ return matches!(result, Err(MetadataCastError::Alignment));
+ }
+ let candidate = match self.size_info {
+ SizeInfo::Sized { size } => {
+ if size <= bytes_len { Some((0, size)) } else { None }
+ }
+ SizeInfo::SliceDst(trailing) => {
+ proofs::max_trailing_bytes(trailing, bytes_len).and_then(|bytes| {
+ let elems = bytes / trailing.elem_size;
+ trailing.size_for_elems(elems).map(|size| (elems, size))
+ })
+ }
+ };
+ match (result, candidate) {
+ (Ok((elems, split_at)), Some((expected_elems, size))) => {
+ *elems == expected_elems && *split_at == match cast_type {
+ CastType::Prefix => size,
+ CastType::Suffix => bytes_len - size,
+ }
+ }
+ (Err(MetadataCastError::Size), None) => true,
+ _ => false,
+ }
+ }))]
pub(crate) const fn validate_cast_and_convert_metadata(
&self,
addr: usize,
bytes_len: usize,
cast_type: CastType,
) -> Result<(usize, usize), MetadataCastError> {
+ #[cfg(all(kani, kani_slow))]
+ #[kani::proof_for_contract(DstLayout::validate_cast_and_convert_metadata)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ let cast_type = if kani::any() { CastType::Prefix } else { CastType::Suffix };
+ let _ = kani::any::<DstLayout>().validate_cast_and_convert_metadata(
+ kani::any(),
+ kani::any(),
+ cast_type,
+ );
+ }
+
// `debug_assert!`, but with `#[allow(clippy::arithmetic_side_effects)]`.
macro_rules! __const_debug_assert {
($e:expr $(, $msg:expr)?) => {
@@ -651,9 +1704,8 @@ macro_rules! __const_debug_assert {
// of this check is trivial - `addr` is the address the object will
// live at.
//
- // For a suffix cast, we know that all valid sizes for the type are
- // a multiple of the alignment (and by safety precondition, we know
- // `DstLayout` may only describe valid Rust types). Thus, a
+ // Under the documented completed-layout premise, all valid sizes
+ // for the type are multiples of its alignment. Thus, a
// validly-sized instance which lives at a validly-aligned address
// must also end at a validly-aligned address. Thus, if the end
// address for a suffix cast (`addr + bytes_len`) is not aligned,
@@ -666,7 +1718,7 @@ macro_rules! __const_debug_assert {
// Addition is guaranteed not to overflow because `offset <=
// bytes_len`, and `addr + bytes_len <= usize::MAX` is a
// precondition of this method. Modulus is guaranteed not to divide
- // by 0 because `align` is non-zero.
+ // by zero because `self.align` is non-zero.
#[allow(clippy::arithmetic_side_effects)]
if (addr + offset) % self.align.get() != 0 {
return Err(MetadataCastError::Alignment);
@@ -680,59 +1732,50 @@ macro_rules! __const_debug_assert {
}
(0, size)
}
- SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => {
- // Calculate the maximum number of bytes that could be consumed
- // - any number of bytes larger than this will either not be a
- // multiple of the alignment, or will be larger than
- // `bytes_len`.
- let max_total_bytes =
- util::round_down_to_next_multiple_of_alignment(bytes_len, self.align);
- // Calculate the maximum number of bytes that could be consumed
- // by the trailing slice.
- //
- // FIXME(#67): Once our MSRV is 1.65, use let-else:
- // https://blog.rust-lang.org/2022/11/03/Rust-1.65.0.html#let-else-statements
- let max_slice_and_padding_bytes = match max_total_bytes.checked_sub(offset) {
- Some(max) => max,
- // `bytes_len` too small even for 0 trailing slice elements.
+ SizeInfo::SliceDst(trailing) => {
+ let max_slice_bytes = match trailing.max_trailing_bytes(bytes_len) {
+ Some(bytes) => bytes,
None => return Err(MetadataCastError::Size),
};
-
// Calculate the number of elements that fit in
- // `max_slice_and_padding_bytes`; any remaining bytes will be
- // considered padding.
+ // `max_slice_bytes`. Any remaining input bytes stay outside
+ // the selected object unless the object's own rounding
+ // operation consumes them as padding.
//
// Guaranteed not to divide by zero: `elem_size` is non-zero.
- #[allow(clippy::arithmetic_side_effects)]
- let elems = max_slice_and_padding_bytes / elem_size.get();
- // Guaranteed not to overflow on multiplication: `usize::MAX >=
- // max_slice_and_padding_bytes >= (max_slice_and_padding_bytes /
- // elem_size) * elem_size`.
+ let (elems, _) = max_elems_for_bytes(max_slice_bytes, trailing.elem_size);
+ // The remainder is `max_slice_bytes - elems * elem_size`.
+ // Writing it as a remainder exposes its bound to the
+ // optimizer: when `elem_size <= size_align`, rounding the
+ // remainder down to `size_align` always gives zero.
+ let size_align = trailing.size_rounding_align_and_phase.align();
+ // `size_phase + max_slice_bytes == max_rounded_bytes`, an
+ // alignment multiple. The selected tail removes `unused_bytes`
+ // from this boundary; rounding back up removes only whole
+ // alignment units. Thus its normalized rounded size is:
//
- // Guaranteed not to overflow on addition:
- // - max_slice_and_padding_bytes == max_total_bytes - offset
- // - elems * elem_size <= max_slice_and_padding_bytes == max_total_bytes - offset
- // - elems * elem_size + offset <= max_total_bytes <= usize::MAX
- #[allow(clippy::arithmetic_side_effects)]
- let without_padding = offset + elems * elem_size.get();
- // `self_bytes` is equal to the offset bytes plus the bytes
- // consumed by the trailing slice plus any padding bytes
- // required to satisfy the alignment. Note that we have computed
- // the maximum number of trailing slice elements that could fit
- // in `self_bytes`, so any padding is guaranteed to be less than
- // the size of an extra element.
+ // round_up(size_phase + elems * elem_size, size_align)
+ // = max_rounded_bytes - round_down(unused_bytes, size_align)
//
- // Guaranteed not to overflow:
- // - By previous comment: without_padding == elems * elem_size +
- // offset <= max_total_bytes
- // - By construction, `max_total_bytes` is a multiple of
- // `self.align`.
- // - At most, adding padding needed to round `without_padding`
- // up to the next multiple of the alignment will bring
- // `self_bytes` up to `max_total_bytes`.
+ // `prove_slice_dst_validator_selected_size_fits` checks this
+ // identity and its intermediate bounds, using the relations
+ // in `prove_max_trailing_bytes_components`. The capacity is
+ // exact by `prove_max_trailing_bytes`; the element helper's
+ // contract establishes the largest whole-element count.
#[allow(clippy::arithmetic_side_effects)]
- let self_bytes =
- without_padding + util::padding_needed_for(without_padding, self.align);
+ let unused_bytes = max_slice_bytes % trailing.elem_size.get();
+ let unused_aligned_bytes =
+ util::round_down_to_next_multiple_of_alignment(unused_bytes, size_align);
+ #[allow(clippy::arithmetic_side_effects)]
+ let max_rounded_bytes = util::round_down_to_next_multiple_of_alignment(
+ bytes_len - trailing.size_base,
+ size_align,
+ );
+ // `unused_aligned_bytes <= unused_bytes <= max_slice_bytes
+ // <= max_rounded_bytes`, and `size_base + max_rounded_bytes
+ // <= bytes_len`, so neither operation can overflow.
+ #[allow(clippy::arithmetic_side_effects)]
+ let self_bytes = trailing.size_base + (max_rounded_bytes - unused_aligned_bytes);
(elems, self_bytes)
}
};
@@ -745,7 +1788,7 @@ macro_rules! __const_debug_assert {
// - In the `Sized` branch, only returns `size` if `size <=
// bytes_len`.
// - In the `SliceDst` branch, calculates `self_bytes <=
- // max_toatl_bytes`, which is upper-bounded by `bytes_len`.
+ // bytes_len`.
#[allow(clippy::arithmetic_side_effects)]
CastType::Suffix => bytes_len - self_bytes,
};
@@ -754,6 +1797,95 @@ macro_rules! __const_debug_assert {
}
}
+/// Computes the affine destination-metadata map.
+///
+/// # Safety
+///
+/// `base + metadata * multiple`, evaluated over the mathematical nonnegative
+/// integers, must not exceed `usize::MAX`.
+#[inline(always)]
+#[allow(unstable_name_collisions)]
+#[cfg_attr(kani, kani::requires(metadata.checked_mul(multiple)
+ .and_then(|scaled| base.checked_add(scaled)).is_some()))]
+#[cfg_attr(kani, kani::ensures(|&result| {
+ metadata.checked_mul(multiple).and_then(|scaled| base.checked_add(scaled)) == Some(result)
+}))]
+unsafe fn add_scaled_metadata(base: usize, metadata: usize, multiple: usize) -> usize {
+ // The call-site safety argument establishes that both operations are
+ // representable for valid source metadata. This harness verifies the
+ // production implementation selected by Kani's modern rustc against
+ // checked arithmetic under precisely that precondition. The distributive
+ // natural-number argument which establishes the precondition is kept
+ // beside the unsafe pointer projection where it can be reviewed with the
+ // layout premises that it consumes.
+ #[cfg(kani)]
+ #[kani::proof_for_contract(add_scaled_metadata)]
+ #[kani::solver(kissat)]
+ fn proof() {
+ let base: usize = kani::any();
+ let metadata: usize = kani::any();
+ let multiple: usize = kani::any();
+ let Some(scaled) = metadata.checked_mul(multiple) else {
+ kani::assume(false);
+ loop {}
+ };
+ let Some(expected) = base.checked_add(scaled) else {
+ kani::assume(false);
+ loop {}
+ };
+
+ // SAFETY: The checked operations above returned `Some`, so the
+ // operation is representable in `usize`. The standard-library
+ // contracts say that this rules out undefined behavior for both
+ // unchecked operations [1][2].
+ //
+ // [1] Per https://doc.rust-lang.org/1.93.1/std/primitive.usize.html#method.unchecked_mul:
+ //
+ // This results in undefined behavior [..] when `checked_mul` would
+ // return `None`.
+ //
+ // [2] Per https://doc.rust-lang.org/1.93.1/std/primitive.usize.html#method.unchecked_add:
+ //
+ // This results in undefined behavior [..] when `checked_add` would
+ // return `None`.
+ assert_eq!(unsafe { add_scaled_metadata(base, metadata, multiple) }, expected);
+ }
+
+ #[allow(unused_imports)]
+ use crate::util::polyfills::*;
+
+ // SAFETY: The caller promises that `base + metadata * multiple` is
+ // representable over the nonnegative integers. Thus `metadata * multiple`
+ // is representable too. On toolchains with inherent unchecked arithmetic,
+ // this is exactly the condition under which `usize::unchecked_mul` does
+ // not have undefined behavior [1]. On older supported toolchains, method
+ // resolution selects `polyfills::NumExt::unchecked_mul`, whose safety
+ // contract requires the same no-overflow condition. Kani checks the two
+ // implementations in separate harnesses.
+ //
+ // [1] Per https://doc.rust-lang.org/1.93.1/std/primitive.usize.html#method.unchecked_mul:
+ //
+ // This results in undefined behavior when `self * rhs > usize::MAX` or
+ // `self * rhs < usize::MIN`, i.e. when `checked_mul` would return `None`.
+ //
+ let scaled = unsafe { metadata.unchecked_mul(multiple) };
+
+ // SAFETY: The caller promises that `base + metadata * multiple` is
+ // representable in `usize`, and `scaled == metadata * multiple`. On
+ // toolchains with inherent unchecked arithmetic, that is exactly the
+ // condition under which `usize::unchecked_add` does not have undefined
+ // behavior [1]. On older supported toolchains, method resolution selects
+ // `polyfills::NumExt::unchecked_add`, whose safety contract requires the
+ // same no-overflow condition. Kani checks the two implementations in
+ // separate harnesses.
+ //
+ // [1] Per https://doc.rust-lang.org/1.93.1/std/primitive.usize.html#method.unchecked_add:
+ //
+ // This results in undefined behavior when `self + rhs > usize::MAX` or
+ // `self + rhs < usize::MIN`, i.e. when `checked_add` would return `None`.
+ unsafe { base.unchecked_add(scaled) }
+}
+
pub(crate) use cast_from::CastFrom;
mod cast_from {
use crate::*;
@@ -823,52 +1955,19 @@ enum CastParamsInner {
// - If this is possible, any information necessary to perform
// the `Src`->`Dst` metadata conversion at runtime.
//
- // Assume that `Src` and `Dst` are slice DSTs, and define:
- // - `S_OFF = Src::LAYOUT.size_info.offset`
- // - `S_ELEM = Src::LAYOUT.size_info.elem_size`
- // - `D_OFF = Dst::LAYOUT.size_info.offset`
- // - `D_ELEM = Dst::LAYOUT.size_info.elem_size`
- //
- // We are trying to solve the following equation:
- //
- // D_OFF + d_meta * D_ELEM = S_OFF + s_meta * S_ELEM
- //
- // At runtime, we will be attempting to compute `d_meta`, given
- // `s_meta` (a runtime value) and all other parameters (which
- // are compile-time values). We can solve like so:
- //
- // D_OFF + d_meta * D_ELEM = S_OFF + s_meta * S_ELEM
- //
- // d_meta * D_ELEM = S_OFF - D_OFF + s_meta * S_ELEM
- //
- // d_meta = (S_OFF - D_OFF + s_meta * S_ELEM)/D_ELEM
- //
- // Since `d_meta` will be a `usize`, we need the right-hand side
- // to be an integer, and this needs to hold for *any* value of
- // `s_meta` (in order for our conversion to be infallible - ie,
- // to not have to reject certain values of `s_meta` at runtime).
- // This means that:
- //
- // - `s_meta * S_ELEM` must be a multiple of `D_ELEM`
- // - Since this must hold for any value of `s_meta`, `S_ELEM`
- // must be a multiple of `D_ELEM`
- // - `S_OFF - D_OFF` must be a multiple of `D_ELEM`
- //
- // Thus, let `OFFSET_DELTA_ELEMS = (S_OFF - D_OFF)/D_ELEM` and
- // `ELEM_MULTIPLE = S_ELEM/D_ELEM`. We can rewrite the above
- // expression as:
- //
- // d_meta = (S_OFF - D_OFF + s_meta * S_ELEM)/D_ELEM
+ // For slice DSTs, destination metadata is an affine function
+ // of source metadata:
//
- // d_meta = OFFSET_DELTA_ELEMS + s_meta * ELEM_MULTIPLE
+ // dst_meta = offset_delta_elems + src_meta * elem_multiple
//
- // Thus, we just need to compute the following and confirm that
- // they have integer solutions in order to both a) determine
- // whether infallible `Src` -> `Dst` casts are possible and, b)
- // pre-compute the parameters necessary to perform those casts
- // at runtime. These parameters are encapsulated in
- // `CastParams`, which acts as a witness that such infallible
- // casts are possible.
+ // `elem_multiple` scales the destination's trailing element
+ // size to the source's. `offset_delta_elems` selects a
+ // destination size equal to the source's zero-element size.
+ // We then prove that the two complete rounded size formulas
+ // produce the same sequence for all metadata. This is
+ // necessary because a packed outer DST can retain a nested
+ // field's rounding operation; comparing only physical
+ // trailing-slice offsets is not sufficient.
/// The parameters required in order to perform an
/// unsized-to-unsized pointer cast from `Src` to `Dst` as
/// described above.
@@ -910,15 +2009,60 @@ fn clone(&self) -> Self {
}
impl<Src: ?Sized, Dst: ?Sized> CastParams<Src, Dst> {
+ /// Given a nonzero `dst.elem_size` that exactly divides
+ /// `src.elem_size`, produces `true` only when the following
+ /// metadata map preserves the complete object size for every
+ /// source element count `src_meta`:
+ ///
+ /// ```text
+ /// dst_meta = dst_base + src_meta * (src.elem_size / dst.elem_size)
+ /// ```
+ ///
+ /// This recognizes sufficient conditions for equality. A
+ /// `false` result also includes equivalent size sequences
+ /// that this test cannot recognize, or an adjustment whose
+ /// normalized components cannot be represented in `usize`.
+ const fn size_sequences_match(
+ src: TrailingSliceLayout,
+ dst: TrailingSliceLayout,
+ dst_base: usize,
+ ) -> bool {
+ let base_bytes = match dst_base.checked_mul(dst.elem_size) {
+ Some(bytes) => bytes,
+ None => return false,
+ };
+ let shifted_dst = match dst.advance(base_bytes, src.elem_size) {
+ Some(layout) => layout,
+ None => return false,
+ };
+ src.has_same_size_sequence(shifted_dst)
+ }
+
+ /// Given the complete layouts of `Src` and `Dst`, produces
+ /// parameters for a cast that preserves object size.
+ ///
+ /// A `Some` result establishes that `Src`'s alignment is at
+ /// least `Dst`'s and that the metadata map preserves size for
+ /// every valid source metadata value.
+ ///
+ /// Supports casts between sized types of equal size, from a
+ /// sized type to a slice DST, and between slice DSTs. A slice
+ /// DST destination must have nonzero-sized elements; between
+ /// slice DSTs, its element size must exactly divide the
+ /// source's nonzero element size.
+ ///
+ /// Produces `None` for unsupported casts or when this method
+ /// cannot establish a size-preserving metadata map. Rejection
+ /// does not establish that no such map exists.
const fn try_compute(
- src: &DstLayout,
- dst: &DstLayout,
+ src_layout: &DstLayout,
+ dst_layout: &DstLayout,
) -> Option<CastParams<Src, Dst>> {
- if src.align.get() < dst.align.get() {
+ if src_layout.align.get() < dst_layout.align.get() {
return None;
}
- let inner = match (src.size_info, dst.size_info) {
+ let inner = match (src_layout.size_info, dst_layout.size_info) {
(
SizeInfo::Sized { size: src_size },
SizeInfo::Sized { size: dst_size },
@@ -931,82 +2075,79 @@ const fn try_compute(
// dst_size`.
CastParamsInner::SizedToSized
}
- (SizeInfo::Sized { size: src_size }, SizeInfo::SliceDst(dst)) => {
- let offset_delta = if let Some(od) = src_size.checked_sub(dst.offset) {
- od
- } else {
- return None;
- };
-
- let dst_elem_size = if let Some(e) = NonZeroUsize::new(dst.elem_size) {
- e
- } else {
- return None;
+ (SizeInfo::Sized { size: src_size }, SizeInfo::SliceDst(_)) => {
+ let dst_meta = match dst_layout.metadata_for_exact_size(src_size) {
+ Some(meta) => meta,
+ None => return None,
};
- // PANICS: `dst_elem_size: NonZeroUsize`, so this won't
- // divide by zero.
- #[allow(clippy::arithmetic_side_effects)]
- let delta_mod_other_elem = offset_delta % dst_elem_size.get();
-
- if delta_mod_other_elem != 0 {
- return None;
- }
-
- // PANICS: `dst_elem_size: NonZeroUsize`, so this won't
- // divide by zero.
- #[allow(clippy::arithmetic_side_effects)]
- let dst_meta = offset_delta / dst_elem_size.get();
-
// SAFETY: The preceding math ensures that a `Dst`
// with `dst_meta` addresses `src_size` bytes.
CastParamsInner::SizedToUnsized { dst_meta }
}
(SizeInfo::SliceDst(src), SizeInfo::SliceDst(dst)) => {
- let offset_delta = if let Some(od) = src.offset.checked_sub(dst.offset)
- {
- od
- } else {
- return None;
- };
-
let dst_elem_size = if let Some(e) = NonZeroUsize::new(dst.elem_size) {
e
} else {
return None;
};
- // PANICS: `dst_elem_size: NonZeroUsize`, so this won't
- // divide by zero.
- #[allow(clippy::arithmetic_side_effects)]
- let delta_mod_other_elem = offset_delta % dst_elem_size.get();
-
- // PANICS: `dst_elem_size: NonZeroUsize`, so this won't
- // divide by zero.
- #[allow(clippy::arithmetic_side_effects)]
- let elem_remainder = src.elem_size % dst_elem_size.get();
-
- if delta_mod_other_elem != 0
- || src.elem_size < dst.elem_size
- || elem_remainder != 0
- {
+ if src.elem_size < dst.elem_size {
return None;
}
- // PANICS: `dst_elem_size: NonZeroUsize`, so this won't
- // divide by zero.
- #[allow(clippy::arithmetic_side_effects)]
- let offset_delta_elems = offset_delta / dst_elem_size.get();
+ let (elem_multiple, described_src_elem_size) =
+ super::max_elems_for_bytes(src.elem_size, dst_elem_size);
+ if described_src_elem_size != src.elem_size {
+ return None;
+ }
- // PANICS: `dst_elem_size: NonZeroUsize`, so this won't
- // divide by zero.
- #[allow(clippy::arithmetic_side_effects)]
- let elem_multiple = src.elem_size / dst_elem_size.get();
+ // Prefer an exact destination-element adjustment
+ // between `src.size_offset()` and
+ // `dst.size_offset()`. This preserves offset-based
+ // cast compatibility and metadata selection when
+ // several lengths have the same rounded size. It
+ // also avoids choosing a length whose subsequent
+ // padding sequence differs from the source's.
+ let exact_offset_delta_elems =
+ match src.size_offset().checked_sub(dst.size_offset()) {
+ Some(delta) => {
+ let (elems, described_delta) =
+ super::max_elems_for_bytes(delta, dst_elem_size);
+ if described_delta == delta {
+ Some(elems)
+ } else {
+ None
+ }
+ }
+ None => None,
+ };
+
+ let offset_delta_elems = match exact_offset_delta_elems {
+ Some(elems) if Self::size_sequences_match(src, dst, elems) => elems,
+ _ => {
+ let src_zero_size = match src.size_for_elems(0) {
+ Some(size) => size,
+ None => return None,
+ };
+ let elems =
+ match dst_layout.metadata_for_exact_size(src_zero_size) {
+ Some(elems) => elems,
+ None => return None,
+ };
+ if !Self::size_sequences_match(src, dst, elems) {
+ return None;
+ }
+ elems
+ }
+ };
CastParamsInner::UnsizedToUnsized {
- // SAFETY: We checked above that this is an exact ratio.
+ // SAFETY: `size_sequences_match` proves that
+ // this affine metadata map preserves size.
offset_delta_elems,
- // SAFETY: We checked above that this is an exact ratio.
+ // SAFETY: We checked above that this is an exact
+ // ratio of source to destination element size.
elem_multiple,
}
}
@@ -1019,40 +2160,77 @@ const fn try_compute(
}
impl<Src: KnownLayout + ?Sized, Dst: KnownLayout + ?Sized> CastParams<Src, Dst> {
+ /// Produces `Dst` metadata describing an object of the same
+ /// size as the `Src` described by `src_meta`.
+ ///
/// # Safety
///
/// `src_meta` describes a `Src` whose size is no larger than
/// `isize::MAX`.
- ///
- /// The returned metadata describes a `Dst` of the same size as
- /// the original `Src`.
#[inline(always)]
unsafe fn cast_metadata(
self,
src_meta: Src::PointerMetadata,
) -> Dst::PointerMetadata {
- #[allow(unused)]
- use crate::util::polyfills::*;
-
let dst_meta = match self.inner {
CastParamsInner::UnsizedToUnsized { offset_delta_elems, elem_multiple } => {
let src_meta = src_meta.to_elem_count();
- #[allow(
- unstable_name_collisions,
- clippy::multiple_unsafe_ops_per_block
- )]
- // SAFETY: `self` is a witness that the following
- // equation holds:
+ // SAFETY: `self` witnesses that this affine map
+ // makes `Src` and `Dst`'s complete rounded size
+ // formulas equal for every source metadata value.
+ // Interpret the arithmetic in this proof over the
+ // mathematical nonnegative integers. Since the
+ // caller promises that `src_meta` is valid `Src`
+ // metadata, the source object size `src_size` it
+ // describes is at most `isize::MAX`. Let
+ // `src_elem_size` and `dst_elem_size` be the
+ // respective trailing-slice element sizes. Then:
+ //
+ // src_meta * src_elem_size
+ // <= src_size <= isize::MAX.
+ //
+ // Since `elem_multiple` is the exact ratio
+ // `src_elem_size / dst_elem_size` and
+ // `dst_elem_size >= 1`,
//
- // D_OFF + d_meta * D_ELEM = S_OFF + s_meta * S_ELEM
+ // src_meta * elem_multiple
+ // <= src_meta * src_elem_size
+ // <= isize::MAX <= usize::MAX.
//
- // Since the caller promises that `src_meta` is
- // valid `Src` metadata, this math will not
- // overflow, and the returned value will describe a
- // `Dst` of the same size.
+ // Thus `scaled_elems = src_meta * elem_multiple`
+ // is representable, and its byte contribution is
+ // `scaled_elems * dst_elem_size = src_meta *
+ // src_elem_size`. Let `base_bytes =
+ // offset_delta_elems * dst_elem_size`.
+ // `size_sequences_match` compares the source
+ // formula with the destination formula advanced by
+ // `base_bytes`. The latter's unrounded term
+ // contains `base_bytes + src_meta * src_elem_size`,
+ // and its complete size is at least that large
+ // (formally checked by Kani in
+ // `prove_size_formula_bounds_size_offset`).
+ // Sequence equality therefore gives, over the
+ // mathematical nonnegative integers,
+ //
+ // base_bytes + src_meta * src_elem_size
+ // = (offset_delta_elems + scaled_elems)
+ // * dst_elem_size
+ // <= src_size.
+ //
+ // Since `dst_elem_size >= 1`, the metadata sum
+ // `offset_delta_elems + scaled_elems` is also at
+ // most `src_size`, and thus at most `isize::MAX <=
+ // usize::MAX`. The sum is therefore representable,
+ // and the returned metadata describes a `Dst` of
+ // the same size.
+ // SAFETY: The bounds above establish the helper's
+ // complete no-overflow precondition.
unsafe {
- offset_delta_elems
- .unchecked_add(src_meta.unchecked_mul(elem_multiple))
+ super::add_scaled_metadata(
+ offset_delta_elems,
+ src_meta,
+ elem_multiple,
+ )
}
}
CastParamsInner::SizedToUnsized { dst_meta } => dst_meta,
@@ -1095,6 +2273,55 @@ impl<Src, Dst> Params<Src> for Dst
}
}
+#[cfg(any(test, all(kani, feature = "derive")))]
+mod padding_testutil {
+ use crate::KnownLayout;
+
+ #[derive(KnownLayout)]
+ #[repr(C, align(4))]
+ pub(super) struct Aligned<Prefix, Tail: ?Sized> {
+ pub(super) prefix: Prefix,
+ pub(super) tail: Tail,
+ }
+
+ /// Compare direct padding with the complete size minus the slice end.
+ /// Counts whose complete object size exceeds `usize::MAX` are omitted.
+ pub(super) fn check_padding_for_elems<Target>(elems: usize) -> Option<usize>
+ where
+ Target: ?Sized + KnownLayout<PointerMetadata = usize>,
+ {
+ let layout = crate::trailing_slice_layout::<Target>();
+ let trailing_size = elems.checked_mul(layout.elem_size)?;
+
+ // Evaluate the complete rounded size independently of both helpers:
+ // `size_for_elems` delegates to the padding method being tested.
+ let (alignment, phase) = layout.size_rounding_align_and_phase.components();
+ #[allow(clippy::arithmetic_side_effects)]
+ let mask = alignment.get() - 1;
+ let unrounded = phase.checked_add(trailing_size)?;
+ let rounded = unrounded.checked_add(mask)? & !mask;
+ let object_size = layout.size_base.checked_add(rounded)?;
+ let trailing_end = layout.offset.checked_add(trailing_size).unwrap();
+ let expected = object_size.checked_sub(trailing_end).unwrap();
+
+ let padding = layout.padding_for_elems(elems);
+ assert_eq!(padding, expected);
+ Some(padding)
+ }
+
+ pub(super) fn check_layouts(elems: usize) {
+ // Slices have no trailing padding. The aligned fixtures cover
+ // zero and nonzero phases, a three-byte element whose padding
+ // cycles with the element count, and zero-sized elements.
+ let _ = check_padding_for_elems::<[u8]>(elems);
+ let _ = check_padding_for_elems::<[()]>(elems);
+ let _ = check_padding_for_elems::<Aligned<[u8; 8], [u8]>>(elems);
+ let _ = check_padding_for_elems::<Aligned<[u8; 9], [u8]>>(elems);
+ let _ = check_padding_for_elems::<Aligned<[u8; 9], [[u8; 3]]>>(elems);
+ let _ = check_padding_for_elems::<Aligned<[u8; 9], [()]>>(elems);
+ }
+}
+
// FIXME(#67): For some reason, on our MSRV toolchain, this `allow` isn't
// enforced despite having `#![allow(unknown_lints)]` at the crate root, but
// putting it here works. Once our MSRV is high enough that this bug has been
@@ -1103,12 +2330,560 @@ impl<Src, Dst> Params<Src> for Dst
#[cfg(test)]
mod tests {
use super::*;
+ use crate::PointerMetadata as _;
+
+ /// Models the size of nested `repr(C)` structs ending in a slice.
+ ///
+ /// `leading` lists structs from outermost to innermost. Each tuple gives
+ /// the packing factor, minimum alignment, and prefix size before padding
+ /// for the final field. `trailing_elem_size` and `trailing_alignment` give
+ /// the slice element size and alignment. An empty `leading` describes the
+ /// slice itself.
+ ///
+ /// The returned closure maps a slice length to the complete size, including
+ /// padding at every nesting level. It returns `None` only on `usize`
+ /// overflow; sizes above `isize::MAX` are permitted by this arithmetic model.
+ /// Packing affects field placement, preserving each field's internal
+ /// padding, even for descriptors that Rust would reject as types.
+ ///
+ /// # Panics
+ ///
+ /// Panics if any alignment or packing factor is not a power of two, a
+ /// minimum alignment exceeds its packing factor, or the element size is
+ /// not a multiple of the slice alignment.
+ fn size_for_metadata_model(
+ leading: &[(NonZeroUsize, NonZeroUsize, usize)],
+ trailing_elem_size: usize,
+ trailing_alignment: NonZeroUsize,
+ ) -> impl Fn(usize) -> Option<usize> + '_ {
+ // Rust requires power-of-two alignments and sizes divisible by their
+ // alignment [1]; alignment modifiers also take powers of two [2].
+ //
+ // [1] https://doc.rust-lang.org/1.93.0/reference/type-layout.html#size-and-alignment
+ // [2] https://doc.rust-lang.org/1.93.0/reference/type-layout.html#the-alignment-modifiers
+ assert!(trailing_alignment.get().is_power_of_two());
+ assert_eq!(trailing_elem_size % trailing_alignment, 0);
+ for &(packed, align, _) in leading {
+ assert!(packed.get().is_power_of_two());
+ assert!(align.get().is_power_of_two());
+ // The descriptor requires its minimum to fit under the packing
+ // cap. It also models combinations that Rust forbids, including
+ // simultaneous `align` and `packed` modifiers [2].
+ assert!(align <= packed);
+ }
+
+ move |elems| {
+ // A slice shares its array section's layout [3]. Array size is
+ // element size times length, with the element's alignment [4].
+ //
+ // [3] https://doc.rust-lang.org/1.93.0/reference/type-layout.html#slice-layout
+ // [4] https://doc.rust-lang.org/1.93.0/reference/type-layout.html#array-layout
+ let mut size = trailing_elem_size.checked_mul(elems)?;
+ let mut alignment = trailing_alignment;
+
+ // Work outward with each field's complete size, including its
+ // trailing padding: an enclosing representation "does not change
+ // the layout of the fields themselves" [5]. Thus packing a nested
+ // field preserves the padding already included in `size`.
+ //
+ // [5] https://doc.rust-lang.org/1.93.0/reference/type-layout.html#representations
+ for &(packed, align, prefix) in leading.iter().rev() {
+ // Packing caps the field's alignment for placement [2].
+ let field_align = alignment.min(packed);
+
+ // `repr(C)` places the next field at an aligned offset [6];
+ // packing requires the least sufficient inter-field padding
+ // [2]. Here `prefix` is the end of the preceding fields.
+ //
+ // [6] https://doc.rust-lang.org/1.93.0/reference/type-layout.html#reprc-structs
+ let offset = prefix.checked_add(util::padding_needed_for(prefix, field_align))?;
+
+ // `repr(C)` uses the greatest field alignment [6], and an
+ // explicit alignment can only raise it [2]. Here `align`
+ // summarizes the minimum from the prefix and any explicit
+ // alignment. Both operands are <= `packed`, so their maximum
+ // also respects the packing cap.
+ alignment = align.max(field_align);
+
+ // Advance past the complete field, then round to the struct's
+ // alignment, following `repr(C)`'s final sizing steps [6].
+ size = offset.checked_add(size)?;
+ size = size.checked_add(util::padding_needed_for(size, alignment))?;
+ }
+ // Each enclosing size is at least its field's size. Thus any
+ // checked multiplication or addition that overflowed above would
+ // also make the final size unrepresentable in `usize`.
+ Some(size)
+ }
+ }
+
+ /// Computes [`size_for_metadata_model`]'s size function using `DstLayout`.
+ ///
+ /// Layout construction uses [`DstLayout::for_repr_c_struct`], and the
+ /// returned closure uses [`crate::PointerMetadata::size_for_metadata`].
+ ///
+ /// # Panics
+ ///
+ /// Panics for invalid descriptors as in [`size_for_metadata_model`], or if
+ /// `DstLayout` cannot construct the layout. This includes static size
+ /// overflow and, in debug builds, alignments or packing factors above
+ /// [`DstLayout::CURRENT_MAX_ALIGN`].
+ fn size_for_metadata_via_dst_layout(
+ leading: &[(NonZeroUsize, NonZeroUsize, usize)],
+ trailing_elem_size: usize,
+ trailing_alignment: NonZeroUsize,
+ ) -> impl Fn(usize) -> Option<usize> + '_ {
+ assert_eq!(trailing_elem_size % trailing_alignment, 0);
+ let mut layout = DstLayout {
+ align: trailing_alignment,
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: 0,
+ elem_size: trailing_elem_size,
+ size_base: 0,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(trailing_alignment, 0),
+ }),
+ statically_shallow_unpadded: true,
+ };
+ for &(packed, align, prefix) in leading.iter().rev() {
+ assert!(align <= packed);
+ let prefix = DstLayout {
+ align: DstLayout::MIN_ALIGN,
+ size_info: SizeInfo::Sized { size: prefix },
+ statically_shallow_unpadded: true,
+ };
+ layout = DstLayout::for_repr_c_struct(Some(align), Some(packed), &[prefix, layout]);
+ }
+ move |elems: usize| elems.size_for_metadata(layout)
+ }
+
+ #[test]
+ fn test_model_via_dst_layout() {
+ let nz = |n| NonZeroUsize::new(n).unwrap();
+ let cases: &[&[_]] = &[
+ &[],
+ &[(nz(8), nz(1), 3)],
+ &[(nz(8), nz(8), 3)],
+ &[(nz(2), nz(1), 1), (nz(8), nz(8), 3)],
+ &[(nz(8), nz(8), 1), (nz(2), nz(1), 3), (nz(8), nz(8), 5)],
+ &[(nz(1), nz(1), 1), (nz(2), nz(2), 3), (nz(8), nz(8), 5)],
+ ];
+ for &leading in cases {
+ for &(elem_size, alignment) in &[(0, 1), (0, 8), (1, 1), (2, 2), (3, 1), (8, 8)] {
+ let alignment = nz(alignment);
+ let expected = size_for_metadata_model(leading, elem_size, alignment);
+ let actual = size_for_metadata_via_dst_layout(leading, elem_size, alignment);
+ let max_elems = usize::MAX / elem_size.max(1);
+ for elems in (0..32).chain([
+ max_elems - 1,
+ max_elems,
+ max_elems.saturating_add(1),
+ usize::MAX,
+ ]) {
+ assert_eq!(
+ actual(elems),
+ expected(elems),
+ "leading: {:?}, trailing: {:?}, elems: {}",
+ leading,
+ (elem_size, alignment),
+ elems,
+ );
+ }
+ }
+ }
+
+ for prefix in [usize::MAX - 1, usize::MAX] {
+ let leading = [(nz(1), nz(1), prefix)];
+ let expected = size_for_metadata_model(&leading, 1, nz(1));
+ let actual = size_for_metadata_via_dst_layout(&leading, 1, nz(1));
+ for elems in [0, 1, 2] {
+ assert_eq!(actual(elems), expected(elems));
+ }
+ }
+ }
+
+ // This exercises arithmetic rather than unsafe operations; keep the
+ // smaller comparison test enabled under Miri instead of this large search.
+ #[test]
+ #[cfg_attr(miri, ignore)]
+ fn test_model_via_dst_layout_combinations() {
+ use rand::{rngs::SmallRng, Rng as _, SeedableRng as _};
+
+ fn check(
+ leading: &[(NonZeroUsize, NonZeroUsize, usize)],
+ trailing_elem_size: usize,
+ trailing_alignment: NonZeroUsize,
+ ) {
+ let expected = size_for_metadata_model(leading, trailing_elem_size, trailing_alignment);
+ assert!(
+ expected(0).is_some(),
+ "unrepresentable test layout: {:?}, {:?}",
+ leading,
+ (trailing_elem_size, trailing_alignment)
+ );
+ let actual = std::panic::catch_unwind(|| {
+ size_for_metadata_via_dst_layout(leading, trailing_elem_size, trailing_alignment)
+ })
+ .unwrap_or_else(|_| {
+ panic!(
+ "DstLayout rejected leading: {:?}, trailing: {:?}",
+ leading,
+ (trailing_elem_size, trailing_alignment)
+ )
+ });
+ let compare = |elems| {
+ let expected = expected(elems);
+ assert_eq!(
+ actual(elems),
+ expected,
+ "leading: {:?}, trailing: {:?}, elems: {}",
+ leading,
+ (trailing_elem_size, trailing_alignment),
+ elems,
+ );
+ expected
+ };
+
+ let first_sixteen = 0..=16;
+ let pointer_width = (0..POINTER_WIDTH_BITS).step_by(4).flat_map(|shift| {
+ let elems = 1usize << shift;
+ [elems - 1, elems, elems + 1]
+ });
+ let last_two = (usize::MAX - 1)..=usize::MAX;
+ let elemses = first_sixteen.chain(pointer_width).chain(last_two);
+ for elems in elemses {
+ let _ = compare(elems);
+ }
+
+ // The modeled size is nondecreasing. Find the last fitting length,
+ // comparing every probe, then check both sides of that boundary.
+ // Zero-sized elements can remain representable at usize::MAX.
+ let (mut low, mut high) = (0, usize::MAX);
+ while low < high {
+ let distance = high - low;
+ let mid = low + distance / 2 + distance % 2;
+ if compare(mid).is_some() {
+ low = mid;
+ } else {
+ high = mid - 1;
+ }
+ }
+ assert!(compare(low).is_some());
+ if let Some(previous) = low.checked_sub(1) {
+ assert!(compare(previous).is_some());
+ }
+ if let Some(next) = low.checked_add(1) {
+ assert_eq!(compare(next), None);
+ }
+ }
+
+ let nz = |n| NonZeroUsize::new(n).unwrap();
+ let check_small_tails = |leading: &[_]| {
+ for alignment in [1, 2, 4, 8] {
+ for multiple in [0, 1, 2, 3, 7] {
+ check(leading, multiple * alignment, nz(alignment));
+ }
+ }
+ };
+ let mut fragments = Vec::new();
+ for packed in [1, 2, 4] {
+ for align in [1, 2, 4].iter().copied().filter(|&align| align <= packed) {
+ for prefix in 0..=4 {
+ fragments.push((nz(packed), nz(align), prefix));
+ }
+ }
+ }
+
+ // Exhaust every combination in this small domain at depths 0, 1, 2.
+ check_small_tails(&[]);
+ for &outer in &fragments {
+ check_small_tails(&[outer]);
+ for &inner in &fragments {
+ check_small_tails(&[outer, inner]);
+ }
+ }
+
+ let mut rng = SmallRng::seed_from_u64(0);
+ for depth in (0usize..=128).chain([256, 512, 1024]) {
+ // A small prefix adds at most 4 * max_align bytes per layer.
+ // Reserve half of usize::MAX for a large outermost prefix so
+ // construction remains representable even at the largest depth.
+ let align_limit =
+ DstLayout::CURRENT_MAX_ALIGN.get().min(usize::MAX / (8 * depth.max(1)));
+ let alignments: Vec<_> = (0..POINTER_WIDTH_BITS)
+ .map(|shift| 1usize << shift)
+ .take_while(|&align| align <= align_limit)
+ .collect();
+ let last = alignments.len() - 1;
+ let max_align = alignments[last];
+ for case in 0..40 {
+ let mut leading = Vec::with_capacity(depth);
+ for level in 0..depth {
+ // Mix arbitrary modifiers, increasing and decreasing
+ // alignments, alternating packing, and fully packed trees.
+ let (packed, align) = match case / 8 {
+ 0 => {
+ let packed = rng.gen_range(0..=last);
+ (packed, rng.gen_range(0..=packed))
+ }
+ 1 => (last, level % alignments.len()),
+ 2 => (last, last - level % alignments.len()),
+ 3 => {
+ let index = if level % 2 == 0 { 0 } else { last };
+ (index, index)
+ }
+ _ => (0, 0),
+ };
+ let (packed, align) = (alignments[packed], alignments[align]);
+ let prefix = if level == 0 && case % 4 == 3 {
+ usize::MAX / 2 + rng.gen_range(0..=2) - 1
+ } else {
+ match rng.gen_range(0..8) {
+ 0 => 0,
+ 1 => 1,
+ 2 => align - 1,
+ 3 => align,
+ 4 => align + 1,
+ 5 => packed - 1,
+ 6 => packed + 1,
+ _ => rng.gen_range(0..=2 * max_align),
+ }
+ };
+ leading.push((nz(packed), nz(align), prefix));
+ }
+ let alignment = alignments[rng.gen_range(0..=last)];
+ let elem_size = match case % 8 {
+ 0 => 0,
+ 1 => alignment,
+ 2 => 2 * alignment,
+ 3 => 3 * alignment,
+ 4 => 7 * alignment,
+ 5 => 15 * alignment,
+ 6 => (usize::MAX / 2 / alignment) * alignment,
+ _ => (usize::MAX / alignment) * alignment,
+ };
+ check(&leading, elem_size, nz(alignment));
+ }
+ }
+ }
+
+ const TEST_SIZE_ROUNDING_ALIGN_AND_PHASE_ALIGN: NonZeroUsize = match NonZeroUsize::new(8) {
+ Some(align) => align,
+ None => const_unreachable!(),
+ };
+ const TEST_SIZE_ROUNDING_ALIGN_AND_PHASE: RoundingAlignAndPhase =
+ RoundingAlignAndPhase::new(TEST_SIZE_ROUNDING_ALIGN_AND_PHASE_ALIGN, 3);
+ const TEST_SIZE_ROUNDING_ALIGN_AND_PHASE_DECODED_ALIGN: usize =
+ TEST_SIZE_ROUNDING_ALIGN_AND_PHASE.align().get();
+ const TEST_SIZE_ROUNDING_ALIGN_AND_PHASE_DECODED_PHASE: usize =
+ TEST_SIZE_ROUNDING_ALIGN_AND_PHASE.components().1;
+
+ #[test]
+ #[allow(clippy::arithmetic_side_effects)]
+ fn test_size_rounding_align_and_phase_encoding() {
+ // These constants ensure that encoding and decoding remain evaluable
+ // on every supported compiler, including the MSRV.
+ assert_eq!(TEST_SIZE_ROUNDING_ALIGN_AND_PHASE_DECODED_ALIGN, 8);
+ assert_eq!(TEST_SIZE_ROUNDING_ALIGN_AND_PHASE_DECODED_PHASE, 3);
+
+ for shift in 0..POINTER_WIDTH_BITS {
+ #[allow(clippy::arithmetic_side_effects)]
+ let align = NonZeroUsize::new(1usize << shift).unwrap();
+ for phase in [0, align.get() / 2, align.get() - 1] {
+ let rounding = RoundingAlignAndPhase::new(align, phase);
+ assert_eq!(rounding.align(), align);
+ assert_eq!(rounding.components().1, phase);
+ assert_eq!(rounding.0.get(), align.get() | phase);
+ }
+ }
+
+ let max_align = DstLayout::THEORETICAL_MAX_ALIGN;
+ let max_phase = max_align.get() - 1;
+ let max = RoundingAlignAndPhase::new(max_align, max_phase);
+ assert_eq!(max.0.get(), usize::MAX);
+ assert_eq!(max.align(), max_align);
+ assert_eq!(max.components().1, max_phase);
+
+ assert_eq!(mem::size_of::<TrailingSliceLayout>(), 4 * mem::size_of::<usize>());
+ }
+
+ #[test]
+ #[allow(clippy::arithmetic_side_effects)]
+ fn test_trailing_slice_layout_normal_form() {
+ // Compare the stored base-and-phase formula with
+ // `prefix + round_up(offset + elems * elem_size, align)`, where
+ // `prefix < align`. Move whole alignment-sized chunks of `offset`
+ // into `size_base` and retain the remainder as the phase. Also verify
+ // that `size_offset()` recovers the exact `offset` used by casts to
+ // choose between metadata values with the same rounded object size.
+ for align in [1, 2, 4, 8, 16] {
+ let align = NonZeroUsize::new(align).unwrap();
+ for prefix in 0..align.get() {
+ for offset in 0..32 {
+ let phase = offset % align;
+ let size_base = (offset - phase) | prefix;
+ let layout = TrailingSliceLayout {
+ offset: 0,
+ elem_size: 3,
+ size_base,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(align, phase),
+ };
+ assert_eq!(layout.size_offset(), offset);
+
+ for elems in 0..32 {
+ let unrounded = offset + elems * layout.elem_size;
+ let old_size =
+ prefix + unrounded + util::padding_needed_for(unrounded, align);
+ assert_eq!(layout.size_for_elems(elems), Some(old_size));
+ }
+ }
+ }
+ }
+ }
+
+ #[test]
+ #[allow(clippy::arithmetic_side_effects)]
+ fn test_normalized_size_bounds() {
+ // Use checked evaluation as an independent oracle, including layouts
+ // whose empty object is already too large to represent.
+ let checked_size = |layout: TrailingSliceLayout, trailing_bytes: usize| {
+ let (alignment, phase) = layout.size_rounding_align_and_phase.components();
+ let unrounded = phase.checked_add(trailing_bytes)?;
+ let rounded = unrounded.checked_add(util::padding_needed_for(unrounded, alignment))?;
+ layout.size_base.checked_add(rounded)
+ };
+ for exponent in 0..POINTER_WIDTH_BITS {
+ let alignment = NonZeroUsize::new(1usize << exponent).unwrap();
+ for phase in [0, alignment.get() / 2, alignment.get() - 1] {
+ for base in [0, 1, alignment.get() - 1, alignment.get(), usize::MAX - 1, usize::MAX]
+ {
+ let mut layout = TrailingSliceLayout {
+ offset: 0,
+ elem_size: 1,
+ size_base: base,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(alignment, phase),
+ };
+ for available in [0, 1, alignment.get() - 1, alignment.get(), usize::MAX] {
+ match layout.max_trailing_bytes(available) {
+ Some(max_bytes) => {
+ assert!(checked_size(layout, max_bytes).unwrap() <= available);
+ if let Some(next_bytes) = max_bytes.checked_add(1) {
+ assert!(checked_size(layout, next_bytes)
+ .map(|size| size > available)
+ .unwrap_or(true));
+ }
+ }
+ None => {
+ assert!(checked_size(layout, 0)
+ .map(|size| size > available)
+ .unwrap_or(true));
+ }
+ }
+ }
+
+ for elem_size in [0, 1, 3, alignment.get(), usize::MAX] {
+ layout.elem_size = elem_size;
+ let max_elems = layout
+ .max_trailing_bytes(usize::MAX)
+ .and_then(|bytes| bytes.checked_div(elem_size))
+ .unwrap_or(usize::MAX);
+ for elems in [
+ 0,
+ 1,
+ max_elems.saturating_sub(1),
+ max_elems,
+ max_elems.saturating_add(1),
+ usize::MAX,
+ ] {
+ let expected = elems
+ .checked_mul(elem_size)
+ .and_then(|bytes| checked_size(layout, bytes));
+ // The physical slice offset does not affect
+ // the size formula. Raw layouts may place the
+ // slice beyond the object, and adding its byte
+ // count to the offset may overflow.
+ for offset in [0, usize::MAX] {
+ layout.offset = offset;
+ assert_eq!(layout.size_for_elems(elems), expected);
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+
+ #[test]
+ fn test_padding_for_elems_for_rust_layouts() {
+ use super::padding_testutil::{check_layouts, check_padding_for_elems, Aligned};
+
+ for elems in 0..256 {
+ check_layouts(elems);
+ }
+ let size_limit = usize::MAX;
+ for remaining in 0..16 {
+ check_layouts(size_limit - remaining);
+ check_layouts(size_limit / 3 - remaining);
+ }
+ check_layouts(usize::MAX);
+
+ for (elems, expected) in [(0, 3), (1, 0), (2, 1), (3, 2)] {
+ assert_eq!(
+ check_padding_for_elems::<Aligned<[u8; 9], [[u8; 3]]>>(elems),
+ Some(expected)
+ );
+ }
+ // Every count is valid for zero-sized trailing elements.
+ assert_eq!(check_padding_for_elems::<Aligned<[u8; 9], [()]>>(usize::MAX), Some(3));
+ }
+
+ #[test]
+ fn test_padding_for_elems() {
+ for exponent in 0..POINTER_WIDTH_BITS {
+ let alignment = NonZeroUsize::new(1usize << exponent).unwrap();
+ let mask = alignment.get() - 1;
+ for phase in [0, alignment.get() / 2, mask] {
+ // Include zero-sized elements, a negative fixed term, fixed
+ // padding larger than the alignment, and overflowing sums
+ // and products. Raw layouts need not describe Rust types.
+ for (offset, elem_size, size_base) in [
+ (0, 0, 0),
+ (0, 1, 1),
+ (9, 3, 8),
+ (3, 3, 0),
+ (9, 3, 12),
+ (usize::MAX, usize::MAX, usize::MAX),
+ (usize::MAX, 0, usize::MAX),
+ ] {
+ let layout = TrailingSliceLayout {
+ offset,
+ elem_size,
+ size_base,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(alignment, phase),
+ };
+ for elems in [0, 1, 2, 3, alignment.get(), usize::MAX] {
+ // Independently reconstruct the complete rounded
+ // size and slice end, then subtract. Round up by
+ // adding the mask and clearing the low bits.
+ let trailing_bytes = elems.wrapping_mul(elem_size);
+ let unrounded = phase.wrapping_add(trailing_bytes);
+ let rounded = unrounded.wrapping_add(mask) & !mask;
+ let object_size = size_base.wrapping_add(rounded);
+ let trailing_end = offset.wrapping_add(trailing_bytes);
+ assert_eq!(
+ layout.padding_for_elems(elems),
+ object_size.wrapping_sub(trailing_end),
+ );
+ }
+ }
+ }
+ }
+ }
#[test]
fn test_dst_layout_for_slice() {
let layout = DstLayout::for_slice::<u32>();
match layout.size_info {
- SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => {
+ SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size, .. }) => {
assert_eq!(offset, 0);
assert_eq!(elem_size, 4);
}
@@ -1121,17 +2896,18 @@ fn test_dst_layout_for_slice() {
#[allow(clippy::decimal_literal_representation)]
#[test]
fn test_dst_layout_extend_sized_with_sized() {
- // This macro constructs a layout corresponding to a `u8` and extends it
- // with a zero-sized trailing field of given alignment `n`. The macro
- // tests that the resulting layout has both size and alignment `min(n,
- // P)` for all valid values of `repr(packed(P))`.
+ // This macro constructs a layout corresponding to a `u8` and extends
+ // it with a zero-sized trailing field of the specified alignment. The
+ // resulting size and alignment must both be the field's alignment
+ // capped by the packing limit. Exercise every supported packing limit
+ // as well as the absence of a packing limit.
macro_rules! test_align_is_size {
($n:expr) => {
let base = DstLayout::for_type::<u8>();
let trailing_field = DstLayout::for_type::<elain::Align<$n>>();
- let packs =
- core::iter::once(None).chain((0..29).map(|p| NonZeroUsize::new(2usize.pow(p))));
+ let packs = core::iter::once(None)
+ .chain((0..=29).map(|p| NonZeroUsize::new(2usize.pow(p))));
for pack in packs {
let composite = base.extend(trailing_field, pack);
@@ -1179,6 +2955,7 @@ macro_rules! test_align_is_size {
test_align_is_size!(33554432);
test_align_is_size!(134217728);
test_align_is_size!(268435456);
+ test_align_is_size!(536870912);
}
/// Tests of when a sized `DstLayout` is extended with a DST field.
@@ -1192,15 +2969,29 @@ fn test_dst_layout_extend_sized_with_dst() {
let aligns = (0..29).map(|p| NonZeroUsize::new(2usize.pow(p)).unwrap());
let packs = core::iter::once(None).chain(aligns.clone().map(Some));
- for align in aligns {
+ for field_type_align in aligns {
for pack in packs.clone() {
let base = DstLayout::for_type::<u8>();
let elem_size = 42;
let trailing_field_offset = 11;
+ #[allow(clippy::arithmetic_side_effects)]
+ let trailing_phase = trailing_field_offset % field_type_align;
+ let trailing_base = util::round_down_to_next_multiple_of_alignment(
+ trailing_field_offset,
+ field_type_align,
+ );
let trailing_field = DstLayout {
- align,
- size_info: SizeInfo::SliceDst(TrailingSliceLayout { elem_size, offset: 11 }),
+ align: field_type_align,
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout {
+ elem_size,
+ offset: trailing_field_offset,
+ size_base: trailing_base,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ field_type_align,
+ trailing_phase,
+ ),
+ }),
statically_shallow_unpadded: false,
};
@@ -1208,7 +2999,9 @@ fn test_dst_layout_extend_sized_with_dst() {
let max_align = pack.unwrap_or(DstLayout::CURRENT_MAX_ALIGN).get();
- let align = align.get().min(max_align);
+ let align = field_type_align.get().min(max_align);
+ let field_offset = align;
+ let size_base = field_offset + trailing_base;
assert_eq!(
composite,
@@ -1216,7 +3009,12 @@ fn test_dst_layout_extend_sized_with_dst() {
align: NonZeroUsize::new(align).unwrap(),
size_info: SizeInfo::SliceDst(TrailingSliceLayout {
elem_size,
- offset: align + trailing_field_offset,
+ offset: field_offset + trailing_field_offset,
+ size_base,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ field_type_align,
+ trailing_phase,
+ ),
}),
statically_shallow_unpadded: false,
}
@@ -1292,7 +3090,8 @@ macro_rules! test {
=> padded { size: current_max_align * 2, align: current_max_align });
}
- /// Tests that calling `pad_to_align` on a DST `DstLayout` is a no-op.
+ /// Tests that calling `pad_to_align` on a DST `DstLayout` incorporates the
+ /// type's trailing-padding operation into its runtime size formula.
#[test]
fn test_dst_layout_pad_to_align_with_dst() {
for align in (0..29).map(|p| NonZeroUsize::new(2usize.pow(p)).unwrap()) {
@@ -1300,42 +3099,183 @@ fn test_dst_layout_pad_to_align_with_dst() {
for elem_size in 0..10 {
let layout = DstLayout {
align,
- size_info: SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }),
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout {
+ offset,
+ elem_size,
+ size_base: offset,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ DstLayout::MIN_ALIGN,
+ 0,
+ ),
+ }),
statically_shallow_unpadded: false,
};
- assert_eq!(layout.pad_to_align(), layout);
+ assert_eq!(
+ layout.pad_to_align(),
+ DstLayout {
+ align,
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout {
+ offset,
+ elem_size,
+ size_base: offset - offset % align,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ align,
+ offset % align
+ ),
+ }),
+ statically_shallow_unpadded: false,
+ }
+ );
}
}
}
- }
-
- // This test takes a long time when running under Miri, so we skip it in
- // that case. This is acceptable because this is a logic test that doesn't
- // attempt to expose UB.
- #[test]
- #[cfg_attr(miri, ignore)]
- fn test_validate_cast_and_convert_metadata() {
- #[allow(non_local_definitions)]
- impl From<usize> for SizeInfo {
- fn from(size: usize) -> SizeInfo {
- SizeInfo::Sized { size }
- }
- }
-
- #[allow(non_local_definitions)]
- impl From<(usize, usize)> for SizeInfo {
- fn from((offset, elem_size): (usize, usize)) -> SizeInfo {
- SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size })
- }
- }
- fn layout<S: Into<SizeInfo>>(s: S, align: usize) -> DstLayout {
+ // Exercise the branch where the existing inner rounding alignment is
+ // at least the enclosing alignment. The inner rounded term is already
+ // aligned to two, so padding only advances `size_base` from three to
+ // four.
+ let inner_rounding_dominates = DstLayout {
+ align: NonZeroUsize::new(2).unwrap(),
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: 3,
+ elem_size: 1,
+ size_base: 3,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ NonZeroUsize::new(4).unwrap(),
+ 1,
+ ),
+ }),
+ statically_shallow_unpadded: false,
+ };
+ assert_eq!(
+ inner_rounding_dominates.pad_to_align(),
DstLayout {
- size_info: s.into(),
- align: NonZeroUsize::new(align).unwrap(),
+ align: NonZeroUsize::new(2).unwrap(),
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: 3,
+ elem_size: 1,
+ size_base: 4,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ NonZeroUsize::new(4).unwrap(),
+ 1
+ ),
+ }),
statically_shallow_unpadded: false,
}
- }
+ );
+
+ // Exercise the branch where the enclosing alignment is larger. For
+ // a trailing-slice byte count `trailing_size`, rounding
+ // `6 + round_up(1 + trailing_size, 4)` to eight gives the equivalent
+ // formula `8 + round_up(1 + trailing_size, 8)`.
+ let outer_rounding_dominates = DstLayout {
+ align: NonZeroUsize::new(8).unwrap(),
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: 7,
+ elem_size: 1,
+ size_base: 6,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ NonZeroUsize::new(4).unwrap(),
+ 1,
+ ),
+ }),
+ statically_shallow_unpadded: false,
+ };
+ assert_eq!(
+ outer_rounding_dominates.pad_to_align(),
+ DstLayout {
+ align: NonZeroUsize::new(8).unwrap(),
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: 7,
+ elem_size: 1,
+ size_base: 8,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ NonZeroUsize::new(8).unwrap(),
+ 1
+ ),
+ }),
+ statically_shallow_unpadded: false,
+ }
+ );
+ }
+
+ #[test]
+ fn test_trailing_slice_layout_size_sequence_equivalence() {
+ // These are the two normalized descriptions produced by a
+ // `repr(C, packed(2))` wrapper around slice DSTs whose element types
+ // both have size four but alignments four and two respectively. Their
+ // formulas differ structurally, but both simplify to `2 + 4 * elems`.
+ let align4 = TrailingSliceLayout {
+ offset: 2,
+ elem_size: 4,
+ size_base: 2,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ NonZeroUsize::new(4).unwrap(),
+ 0,
+ ),
+ };
+ let align2 = TrailingSliceLayout {
+ offset: 2,
+ elem_size: 4,
+ size_base: 2,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ NonZeroUsize::new(2).unwrap(),
+ 0,
+ ),
+ };
+ assert!(align4.has_same_size_sequence(align2));
+ for elems in 0..8 {
+ assert_eq!(align4.size_for_elems(elems), Some(2 + 4 * elems));
+ assert_eq!(align4.size_for_elems(elems), align2.size_for_elems(elems));
+ }
+
+ let dynamically_padded = TrailingSliceLayout { elem_size: 1, ..align4 };
+ assert!(!dynamically_padded.has_same_size_sequence(align2));
+ }
+
+ // This test takes a long time when running under Miri, so we skip it in
+ // that case. This is acceptable because this is a logic test that doesn't
+ // attempt to expose UB.
+ #[test]
+ #[cfg_attr(miri, ignore)]
+ fn test_validate_cast_and_convert_metadata() {
+ #[allow(non_local_definitions)]
+ impl From<usize> for SizeInfo {
+ fn from(size: usize) -> SizeInfo {
+ SizeInfo::Sized { size }
+ }
+ }
+
+ #[allow(non_local_definitions)]
+ impl From<(usize, usize)> for SizeInfo {
+ fn from((offset, elem_size): (usize, usize)) -> SizeInfo {
+ SizeInfo::SliceDst(TrailingSliceLayout {
+ offset,
+ elem_size,
+ size_base: offset,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ DstLayout::MIN_ALIGN,
+ 0,
+ ),
+ })
+ }
+ }
+
+ fn layout<S: Into<SizeInfo>>(s: S, align: usize) -> DstLayout {
+ let align = NonZeroUsize::new(align).unwrap();
+ let size_info = match s.into() {
+ SizeInfo::SliceDst(mut trailing) => {
+ let phase = trailing.size_base % align;
+ trailing.size_base =
+ util::round_down_to_next_multiple_of_alignment(trailing.size_base, align);
+ trailing.size_rounding_align_and_phase =
+ RoundingAlignAndPhase::new(align, phase);
+ SizeInfo::SliceDst(trailing)
+ }
+ size_info => size_info,
+ };
+ DstLayout { size_info, align, statically_shallow_unpadded: false }
+ }
/// This macro accepts arguments in the form of:
///
@@ -1531,10 +3471,18 @@ fn validate_behavior(
let resulting_size = match layout.size_info {
SizeInfo::Sized { size } => size,
- SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => {
+ SizeInfo::SliceDst(TrailingSliceLayout {
+ elem_size,
+ size_base,
+ size_rounding_align_and_phase,
+ ..
+ }) => {
+ let (size_align, size_phase) = size_rounding_align_and_phase.components();
let padded_size = |elems| {
- let without_padding = offset + elems * elem_size;
- without_padding + util::padding_needed_for(without_padding, align)
+ let without_padding = size_phase + elems * elem_size;
+ size_base
+ + without_padding
+ + util::padding_needed_for(without_padding, size_align)
};
let resulting_size = padded_size(elems);
@@ -1562,8 +3510,13 @@ fn validate_behavior(
} else {
let min_size = match layout.size_info {
SizeInfo::Sized { size } => size,
- SizeInfo::SliceDst(TrailingSliceLayout { offset, .. }) => {
- offset + util::padding_needed_for(offset, layout.align)
+ SizeInfo::SliceDst(TrailingSliceLayout {
+ size_base,
+ size_rounding_align_and_phase,
+ ..
+ }) => {
+ let (size_align, size_phase) = size_rounding_align_and_phase.components();
+ size_base + size_phase + util::padding_needed_for(size_phase, size_align)
}
};
@@ -1592,6 +3545,35 @@ fn validate_behavior(
.map(|(size_info, align)| layout(size_info, align));
itertools::iproduct!(layouts, 0..8, 0..8, [CastType::Prefix, CastType::Suffix])
.for_each(validate_behavior);
+
+ // Exercise a normalized formula which cannot be represented as
+ // `round_up(offset + elems * elem_size, align)`: the physical trailing
+ // slice begins at offset 7, while an inner aligned DST contributes the
+ // size formula `2 + round_up(5 + elems, 4)` to a packed outer type.
+ let nested_packed = DstLayout {
+ align: NonZeroUsize::new(2).unwrap(),
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: 7,
+ elem_size: 1,
+ size_base: 6,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ NonZeroUsize::new(4).unwrap(),
+ 1,
+ ),
+ }),
+ statically_shallow_unpadded: false,
+ };
+ itertools::iproduct!([nested_packed], 0..8, 0..24, [CastType::Prefix, CastType::Suffix])
+ .for_each(validate_behavior);
+
+ assert!(matches!(
+ nested_packed.validate_cast_and_convert_metadata(0, 8, CastType::Prefix),
+ Err(MetadataCastError::Size)
+ ));
+ assert!(matches!(
+ nested_packed.validate_cast_and_convert_metadata(0, 10, CastType::Prefix),
+ Ok((3, 10))
+ ));
}
#[test]
@@ -1631,9 +3613,18 @@ fn test<T: ?Sized, W: Fn(usize) -> NonNull<T>>(
let dst = args.elem_size.is_some();
let layout = {
let size_info = match args.elem_size {
- Some(elem_size) => {
- SizeInfo::SliceDst(TrailingSliceLayout { offset: args.offset, elem_size })
- }
+ Some(elem_size) => SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: args.offset,
+ elem_size,
+ size_base: util::round_down_to_next_multiple_of_alignment(
+ args.offset,
+ args.align,
+ ),
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ args.align,
+ args.offset % args.align,
+ ),
+ }),
None => SizeInfo::Sized {
// Rust only supports types whose sizes are a multiple
// of their alignment. If the macro created a type like
@@ -1951,70 +3942,748 @@ macro_rules! validate_against_rust {
#[cfg(kani)]
mod proofs {
+ //! Arithmetic models and input generators for the layout proofs.
+ //!
+ //! The contract models specify expected results without calling the
+ //! operation being checked. The composition models (`extended_layout`,
+ //! `padded_layout`, and `repr_c_layout`) also define the contracts' input
+ //! domains: `Some` supplies an expected layout, while `None` excludes an
+ //! input from the corresponding non-panicking contract. These models check
+ //! representability in `usize`, without imposing Rust's `isize::MAX` bound.
+
use core::alloc::Layout;
use super::*;
- impl kani::Arbitrary for DstLayout {
- fn any() -> Self {
- let align: NonZeroUsize = kani::any();
- let size_info: SizeInfo = kani::any();
-
- kani::assume(align.is_power_of_two());
- kani::assume(align < DstLayout::THEORETICAL_MAX_ALIGN);
-
- // For testing purposes, we most care about instantiations of
- // `DstLayout` that can correspond to actual Rust types. We use
- // `Layout` to verify that our `DstLayout` satisfies the validity
- // conditions of Rust layouts.
- kani::assume(
- match size_info {
- SizeInfo::Sized { size } => Layout::from_size_align(size, align.get()),
- SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size: _ }) => {
- // `SliceDst` cannot encode an exact size, but we know
- // it is at least `offset` bytes.
- Layout::from_size_align(offset, align.get())
+ /// Evaluates the normalized object size for a trailing slice of
+ /// `trailing_size` bytes:
+ ///
+ /// ```text
+ /// size_base + round_up(size_phase + trailing_size, size_align)
+ /// ```
+ ///
+ /// Here, `size_align` and `size_phase` come from `layout.size_rounding_align_and_phase`.
+ /// `round_up` rounds its input up to a multiple of its alignment. Returns
+ /// `None` if the size is not representable in `usize`.
+ ///
+ /// Using byte counts separates rounding from element-count multiplication
+ /// and permits proofs over byte counts that are not whole elements.
+ pub(super) fn slice_dst_size_for_trailing_bytes<ElementSize>(
+ layout: TrailingSliceLayout<ElementSize>,
+ trailing_size: usize,
+ ) -> Option<usize> {
+ let without_padding =
+ layout.size_rounding_align_and_phase.components().1.checked_add(trailing_size)?;
+ let padding =
+ util::padding_needed_for(without_padding, layout.size_rounding_align_and_phase.align());
+ let rounded = without_padding.checked_add(padding)?;
+ layout.size_base.checked_add(rounded)
+ }
+
+ /// Returns the least multiple of `align` greater than or equal to `bytes`,
+ /// or `None` if that multiple exceeds `usize::MAX`.
+ ///
+ /// Requires a power-of-two `align`. Composition models use this checked
+ /// operation to identify inputs for which adding padding would overflow.
+ fn round_up(bytes: usize, align: NonZeroUsize) -> Option<usize> {
+ bytes.checked_add(util::padding_needed_for(bytes, align))
+ }
+
+ /// Models the largest trailing-slice byte count whose object size fits in
+ /// `available` bytes. Returns `None` if even an empty trailing slice cannot
+ /// fit. The result need not be a multiple of the element size.
+ ///
+ /// Subtracting `size_base` leaves the budget for the rounded contribution.
+ /// Rounding that budget down to `size_align` and subtracting `size_phase`
+ /// gives the largest byte count whose `slice_dst_size_for_trailing_bytes`
+ /// result is at most `available`. Metadata and casting contracts use
+ /// this bound without calling the production capacity calculation.
+ pub(super) fn max_trailing_bytes(
+ trailing: TrailingSliceLayout,
+ available: usize,
+ ) -> Option<usize> {
+ let (align, phase) = trailing.size_rounding_align_and_phase.components();
+ let capacity = available.checked_sub(trailing.size_base)?;
+ util::round_down_to_next_multiple_of_alignment(capacity, align).checked_sub(phase)
+ }
+
+ /// Models appending `field` to a sized `prefix` for `DstLayout::extend`.
+ ///
+ /// Places the field at the first offset after the prefix that satisfies the
+ /// field's alignment, capped by `packed` when present. The resulting
+ /// alignment is the maximum of the prefix's alignment and this effective
+ /// field alignment. A DST field retains its inner rounding operation, with
+ /// its slice offset and size base shifted by the field's placement.
+ /// Trailing padding for the enclosing layout is left to `padded_layout`.
+ ///
+ /// Returns `None` for a non-power-of-two alignment, an unsized prefix, or
+ /// arithmetic overflow. The `statically_shallow_unpadded` flag is true only
+ /// if both inputs have it set and field placement introduces no gap.
+ pub(super) fn extended_layout(
+ prefix: DstLayout,
+ field: DstLayout,
+ packed: Option<NonZeroUsize>,
+ ) -> Option<DstLayout> {
+ if !prefix.align.is_power_of_two()
+ || !field.align.is_power_of_two()
+ || packed.map_or(false, |align| !align.is_power_of_two())
+ {
+ return None;
+ }
+ let SizeInfo::Sized { size: prefix_size } = prefix.size_info else { return None };
+ let field_align = field.align.min(packed.unwrap_or(DstLayout::THEORETICAL_MAX_ALIGN));
+ let field_offset = round_up(prefix_size, field_align)?;
+ let size_info = match field.size_info {
+ SizeInfo::Sized { size } => SizeInfo::Sized { size: field_offset.checked_add(size)? },
+ SizeInfo::SliceDst(trailing) => SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: field_offset.checked_add(trailing.offset)?,
+ size_base: field_offset.checked_add(trailing.size_base)?,
+ ..trailing
+ }),
+ };
+ Some(DstLayout {
+ align: prefix.align.max(field_align),
+ size_info,
+ statically_shallow_unpadded: prefix.statically_shallow_unpadded
+ && field.statically_shallow_unpadded
+ && field_offset == prefix_size,
+ })
+ }
+
+ /// Models adding trailing padding for `DstLayout::pad_to_align`.
+ ///
+ /// A sized layout rounds its size up to a multiple of `layout.align`. A DST
+ /// instead composes that rounding with its existing size expression:
+ ///
+ /// ```text
+ /// round_up(size_base + round_up(size_phase + trailing_size, size_align),
+ /// layout.align)
+ /// ```
+ ///
+ /// Here, `size_align` and `size_phase` come from the DST's
+ /// `size_rounding_align_and_phase`. The result encodes this expression with
+ /// one base, phase, and rounding alignment, preserving the physical slice
+ /// offset and element size. The `statically_shallow_unpadded` flag is
+ /// cleared only when a sized layout gains padding; DST padding is accounted
+ /// for by the size expression.
+ ///
+ /// Returns `None` if `layout.align` is not a power of two or a checked
+ /// intermediate overflows. The contract compares this expected encoding
+ /// with the production result; the separate padding formula proofs check
+ /// that the encoding preserves the rounded size.
+ pub(super) fn padded_layout(layout: DstLayout) -> Option<DstLayout> {
+ if !layout.align.is_power_of_two() {
+ return None;
+ }
+ let (size_info, no_static_padding) = match layout.size_info {
+ SizeInfo::Sized { size } => {
+ let rounded = round_up(size, layout.align)?;
+ (SizeInfo::Sized { size: rounded }, rounded == size)
+ }
+ SizeInfo::SliceDst(trailing) => {
+ let (inner_align, phase) = trailing.size_rounding_align_and_phase.components();
+ let (size_base, size_rounding_align_and_phase) = if inner_align >= layout.align {
+ (
+ round_up(trailing.size_base, layout.align)?,
+ trailing.size_rounding_align_and_phase,
+ )
+ } else {
+ let fixed = round_up(trailing.size_base, inner_align)?.checked_add(phase)?;
+ let new_phase = fixed & (layout.align.get() - 1);
+ (fixed - new_phase, RoundingAlignAndPhase::new(layout.align, new_phase))
+ };
+ (
+ SizeInfo::SliceDst(TrailingSliceLayout {
+ size_base,
+ size_rounding_align_and_phase,
+ ..trailing
+ }),
+ true,
+ )
+ }
+ };
+ Some(DstLayout {
+ size_info,
+ statically_shallow_unpadded: layout.statically_shallow_unpadded && no_static_padding,
+ ..layout
+ })
+ }
+
+ /// Models a complete layout for `DstLayout::for_repr_c_struct` by composing
+ /// the field-placement and trailing-padding specifications.
+ ///
+ /// Starts with an empty prefix of alignment `align`, or one if absent.
+ /// Appends fields in order, with `packed` as an optional cap on field
+ /// alignment; the final layout is padded to its resulting alignment. Only
+ /// the last field may be unsized, since field extension requires a sized
+ /// prefix.
+ ///
+ /// Returns `None` if the initial alignment is not a power of two or any
+ /// field extension or final padding step returns `None`.
+ pub(super) fn repr_c_layout(
+ align: Option<NonZeroUsize>,
+ packed: Option<NonZeroUsize>,
+ fields: &[DstLayout],
+ ) -> Option<DstLayout> {
+ let align = align.unwrap_or(DstLayout::MIN_ALIGN);
+ if !align.is_power_of_two() {
+ return None;
+ }
+ let mut layout = DstLayout {
+ align,
+ size_info: SizeInfo::Sized { size: 0 },
+ statically_shallow_unpadded: true,
+ };
+ for &field in fields {
+ layout = extended_layout(layout, field, packed)?;
+ }
+ padded_layout(layout)
+ }
+
+ // The inline contract harnesses use unrestricted `Arbitrary` values.
+ // These generators retain the narrower domains of the Rust-layout proofs.
+
+ /// Generates any power-of-two alignment representable in `usize`, including
+ /// `DstLayout::THEORETICAL_MAX_ALIGN`. This restricts the alignment alone;
+ /// callers must impose any constraints relating it to a layout's size.
+ fn any_layout_align() -> NonZeroUsize {
+ let exponent: u8 = kani::any();
+ kani::assume(usize::from(exponent) < POINTER_WIDTH_BITS);
+
+ // `exponent < POINTER_WIDTH_BITS`, so the shift is in range and its
+ // result is non-zero. This construction is surjective over every
+ // power-of-two alignment up to and including
+ // `THEORETICAL_MAX_ALIGN`. `Layout` admits that maximum alignment for
+ // zero-sized layouts, so the validity assumptions below—not this
+ // generator—filter combinations that cannot describe Rust types.
+ NonZeroUsize::new(1usize << exponent).unwrap()
+ }
+
+ /// Generates layouts satisfying the arithmetic restrictions used by the
+ /// standalone Rust-layout proofs.
+ ///
+ /// The fixed size, or the zero-element DST size, must be accepted by
+ /// `Layout::from_size_align`. For a DST, `size_base + size_phase` must also
+ /// be representable and at least the physical slice offset. Component
+ /// bounds come from `any_bounded_size_info`; `statically_shallow_unpadded`
+ /// remains arbitrary. These conditions do not establish that a Rust type
+ /// with this layout exists.
+ fn any_valid_dst_layout() -> DstLayout {
+ let align = any_layout_align();
+ let size_info = any_bounded_size_info();
+
+ kani::assume(
+ match size_info {
+ SizeInfo::Sized { size } => Layout::from_size_align(size, align.get()),
+ SizeInfo::SliceDst(trailing) => {
+ // `SliceDst` cannot encode one exact size. Validate its
+ // minimum size and the invariants required of the size
+ // formula for layouts of real Rust types.
+ let min_size = trailing.size_for_elems(0);
+ let unrounded_min_size = trailing
+ .size_base
+ .checked_add(trailing.size_rounding_align_and_phase.components().1);
+
+ kani::assume(matches!(
+ unrounded_min_size,
+ Some(size) if size >= trailing.offset
+ ));
+
+ match min_size {
+ Some(min_size) => Layout::from_size_align(min_size, align.get()),
+ None => Layout::from_size_align(usize::MAX, align.get()),
}
}
- .is_ok(),
- );
+ }
+ .is_ok(),
+ );
+
+ DstLayout { align: align, size_info: size_info, statically_shallow_unpadded: kani::any() }
+ }
+
+ /// Generates either a fixed size at most `DstLayout::MAX_SIZE`, or a DST
+ /// description from `any_bounded_trailing_layout`.
+ ///
+ /// Supplies bounded size components to `any_valid_dst_layout`, which adds
+ /// constraints involving the enclosing layout's alignment.
+ fn any_bounded_size_info() -> SizeInfo {
+ let is_sized: bool = kani::any();
+
+ match is_sized {
+ true => {
+ let size: usize = kani::any();
+
+ kani::assume(size <= DstLayout::MAX_SIZE);
+
+ SizeInfo::Sized { size }
+ }
+ false => SizeInfo::SliceDst(any_bounded_trailing_layout()),
+ }
+ }
+
+ /// Generates a trailing-slice description whose element size, slice offset,
+ /// and size base are each at most `DstLayout::MAX_SIZE` (`isize::MAX`).
+ ///
+ /// The rounding alignment spans all representable powers of two, and the
+ /// phase spans all values below that alignment. No further relationships
+ /// between these components are imposed: their combined size may overflow
+ /// or fail to contain the slice. Standalone proofs add the relationships
+ /// they require; contract harnesses use unrestricted `Arbitrary` values
+ /// instead.
+ fn any_bounded_trailing_layout() -> TrailingSliceLayout {
+ let elem_size: usize = kani::any();
+ let offset: usize = kani::any();
+ let size_base: usize = kani::any();
+ let size_align = any_layout_align();
+ let raw_size_phase: usize = kani::any();
+ // Since `size_align` is a power of two, masking is surjective over
+ // precisely the values in `0..size_align`.
+ #[allow(clippy::arithmetic_side_effects)]
+ let size_phase = raw_size_phase & (size_align.get() - 1);
+
+ kani::assume(elem_size <= DstLayout::MAX_SIZE);
+ kani::assume(offset <= DstLayout::MAX_SIZE);
+ kani::assume(size_base <= DstLayout::MAX_SIZE);
+
+ TrailingSliceLayout {
+ elem_size,
+ offset,
+ size_base,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(size_align, size_phase),
+ }
+ }
+
+ #[cfg(feature = "derive")]
+ #[kani::proof]
+ fn prove_padding_for_elems_for_rust_layouts() {
+ // For each fixture, check every element count whose complete size
+ // fits in `usize`, including all counts for zero-sized elements.
+ padding_testutil::check_layouts(kani::any());
+ }
+
+ #[cfg(kani_slow)]
+ #[kani::proof]
+ #[kani::solver(kissat)]
+ fn prove_padding_for_elems() {
+ // Check the modular result for every field value and element count,
+ // including layouts that cannot describe a Rust type.
+ let layout = TrailingSliceLayout {
+ offset: kani::any(),
+ elem_size: kani::any(),
+ size_base: kani::any(),
+ size_rounding_align_and_phase: RoundingAlignAndPhase(kani::any()),
+ };
+ let elems: usize = kani::any();
+ let (size_align, size_phase) = layout.size_rounding_align_and_phase.components();
+ let mask = size_align.get() - 1;
+
+ // Evaluate the complete size and trailing-slice end separately in
+ // modular arithmetic, without reducing the trailing bytes first or
+ // canceling them from the difference.
+ let trailing_bytes = elems.wrapping_mul(layout.elem_size);
+ let unrounded = size_phase.wrapping_add(trailing_bytes);
+ let rounded = unrounded.wrapping_add(mask) & !mask;
+ let object_size = layout.size_base.wrapping_add(rounded);
+ let trailing_end = layout.offset.wrapping_add(trailing_bytes);
+ assert_eq!(layout.padding_for_elems(elems), object_size.wrapping_sub(trailing_end));
+ }
+
+ #[kani::proof]
+ fn prove_size_rounding_align_and_phase_encoding() {
+ let encoded: usize = kani::any();
+ kani::assume(encoded != 0);
+ let encoded = NonZeroUsize::new(encoded).unwrap();
+ let rounding = RoundingAlignAndPhase(encoded);
+
+ let align = rounding.align();
+ let phase = rounding.components().1;
+ assert!(align.is_power_of_two());
+ assert!(phase < align.get());
+ assert_eq!(RoundingAlignAndPhase::new(align, phase), rounding);
+ }
+
+ #[kani::proof]
+ fn prove_size_rounding_align_and_phase_decoding() {
+ let align: NonZeroUsize = kani::any();
+ let phase: usize = kani::any();
+ kani::assume(align.is_power_of_two());
+ kani::assume(phase < align.get());
+
+ let rounding = RoundingAlignAndPhase::new(align, phase);
+ assert_eq!(rounding.components(), (align, phase));
+ }
+
+ #[cfg(kani_slow)]
+ #[kani::proof]
+ fn prove_trailing_slice_layout_normal_form() {
+ let rounding_word: usize = kani::any();
+ kani::assume(rounding_word != 0);
+ let align = RoundingAlignAndPhase(NonZeroUsize::new(rounding_word).unwrap()).align();
+
+ let raw_prefix: usize = kani::any();
+ let offset: usize = kani::any();
+ let trailing_size: usize = kani::any();
+ #[allow(clippy::arithmetic_side_effects)]
+ let prefix = raw_prefix & (align.get() - 1);
+ #[allow(clippy::arithmetic_side_effects)]
+ let phase = offset & (align.get() - 1);
+ #[allow(clippy::arithmetic_side_effects)]
+ let aligned_offset = offset - phase;
+ let layout = TrailingSliceLayout {
+ offset: 0,
+ elem_size: 1,
+ size_base: aligned_offset | prefix,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(align, phase),
+ };
+
+ assert_eq!(layout.size_offset(), offset);
+
+ let old_size = offset
+ .checked_add(trailing_size)
+ .and_then(|without_padding| {
+ without_padding.checked_add(util::padding_needed_for(without_padding, align))
+ })
+ .and_then(|rounded| prefix.checked_add(rounded));
+ assert_eq!(layout.size_for_elems(trailing_size), old_size);
+ assert_eq!(slice_dst_size_for_trailing_bytes(layout, trailing_size), old_size);
+ }
+
+ #[cfg(kani_slow)]
+ #[kani::proof]
+ #[kani::solver(kissat)]
+ fn prove_element_product_preserves_alignment() {
+ let elem_size: usize = kani::any();
+ let elems: usize = kani::any();
+ let align = any_layout_align();
+ #[allow(clippy::arithmetic_side_effects)]
+ let align_mask = align.get() - 1;
+
+ kani::assume(elem_size & align_mask == 0);
+ let Some(bytes) = elem_size.checked_mul(elems) else {
+ kani::assume(false);
+ loop {}
+ };
+ assert_eq!(bytes & align_mask, 0);
+ }
+
+ #[kani::proof]
+ fn prove_trailing_slice_layout_advance() {
+ let layout: TrailingSliceLayout = any_bounded_trailing_layout();
+ let bytes: usize = kani::any();
+ let elem_size: usize = kani::any();
+ let trailing_size: usize = kani::any();
+
+ let size_offset = layout.size_offset();
+ let advanced_offset = size_offset.checked_add(bytes);
+ let advanced = layout.advance(bytes, elem_size);
+ assert_eq!(advanced.is_some(), advanced_offset.is_some());
+ let Some(advanced_offset) = advanced_offset else {
+ kani::assume(false);
+ loop {}
+ };
+ let Some(advanced) = advanced else { unreachable!() };
+
+ assert_eq!(advanced.offset, layout.offset);
+ assert_eq!(advanced.size_offset(), advanced_offset);
+ assert_eq!(advanced.elem_size, elem_size);
+ assert_eq!(
+ advanced.size_rounding_align_and_phase.align(),
+ layout.size_rounding_align_and_phase.align()
+ );
+ #[allow(clippy::arithmetic_side_effects)]
+ let size_align_mask = advanced.size_rounding_align_and_phase.align().get() - 1;
+ assert_eq!(advanced.size_base & size_align_mask, layout.size_base & size_align_mask);
+
+ let size_align = layout.size_rounding_align_and_phase.align();
+ #[allow(clippy::arithmetic_side_effects)]
+ let size_prefix = layout.size_base & (size_align.get() - 1);
+ let expected_size = advanced_offset
+ .checked_add(trailing_size)
+ .and_then(|without_padding| {
+ without_padding.checked_add(util::padding_needed_for(without_padding, size_align))
+ })
+ .and_then(|rounded| size_prefix.checked_add(rounded));
+ assert_eq!(slice_dst_size_for_trailing_bytes(advanced, trailing_size), expected_size);
+ }
+
+ #[cfg(kani_slow)]
+ #[kani::proof]
+ fn prove_has_same_size_sequence() {
+ let left: TrailingSliceLayout = any_bounded_trailing_layout();
+ let right: TrailingSliceLayout = any_bounded_trailing_layout();
+ let trailing_size: usize = kani::any();
+
+ kani::assume(left.has_same_size_sequence(right));
+ let left_align = left.size_rounding_align_and_phase.align().get();
+ let right_align = right.size_rounding_align_and_phase.align().get();
+ let max_align = if left_align > right_align { left_align } else { right_align };
+ #[allow(clippy::arithmetic_side_effects)]
+ let max_align_mask = max_align - 1;
+ if left.elem_size & max_align_mask == 0 {
+ // In this branch `has_same_size_sequence` relies on every actual
+ // trailing byte count being a multiple of both alignments. Prove
+ // the formula for that entire byte class, which includes every
+ // representable `elems * elem_size`, without introducing a
+ // nonlinear symbolic multiplication into the harness.
+ kani::assume(trailing_size & max_align_mask == 0);
+ }
+ assert_eq!(
+ slice_dst_size_for_trailing_bytes(left, trailing_size),
+ slice_dst_size_for_trailing_bytes(right, trailing_size)
+ );
+ }
+
+ #[kani::proof]
+ #[kani::solver(kissat)]
+ fn prove_max_trailing_bytes() {
+ let trailing: TrailingSliceLayout = any_bounded_trailing_layout();
+ let available_bytes: usize = kani::any();
+ let trailing_bytes: usize = kani::any();
+ let fits = slice_dst_size_for_trailing_bytes(trailing, trailing_bytes)
+ .map(|size| size <= available_bytes)
+ .unwrap_or(false);
+ match trailing.max_trailing_bytes(available_bytes) {
+ Some(max_bytes) => assert_eq!(trailing_bytes <= max_bytes, fits),
+ None => assert!(!fits),
+ }
+ }
+
+ #[kani::proof]
+ #[kani::solver(kissat)]
+ fn prove_max_trailing_bytes_components() {
+ let trailing: TrailingSliceLayout = any_bounded_trailing_layout();
+ let bytes_len: usize = kani::any();
+ let Some(max_slice_bytes) = trailing.max_trailing_bytes(bytes_len) else {
+ kani::assume(false);
+ loop {}
+ };
+ let (size_align, size_phase) = trailing.size_rounding_align_and_phase.components();
+ let Some(bytes_after_base) = bytes_len.checked_sub(trailing.size_base) else {
+ unreachable!()
+ };
+ let max_rounded_bytes =
+ util::round_down_to_next_multiple_of_alignment(bytes_after_base, size_align);
+ assert_eq!(size_phase.checked_add(max_slice_bytes), Some(max_rounded_bytes));
+ assert!(trailing.size_base.checked_add(max_rounded_bytes).unwrap() <= bytes_len);
+ }
+
+ #[kani::proof]
+ #[kani::solver(kissat)]
+ fn prove_slice_dst_validator_selected_size_fits() {
+ let size_align = any_layout_align();
+ #[allow(clippy::arithmetic_side_effects)]
+ let align_mask = size_align.get() - 1;
+ let raw_phase: usize = kani::any();
+ let size_phase = raw_phase & align_mask;
+ let raw_capacity: usize = kani::any();
+ let max_rounded_bytes = raw_capacity & !align_mask;
+ let size_base: usize = kani::any();
+ let bytes_len: usize = kani::any();
+ let trailing_size: usize = kani::any();
+
+ // These are exactly the relations established for every successful
+ // capacity query by `prove_max_trailing_bytes_components`. Isolate
+ // them from the helper's implementation to keep this arithmetic
+ // identity independent of how that helper computes its bound.
+ let Some(max_slice_bytes) = max_rounded_bytes.checked_sub(size_phase) else {
+ kani::assume(false);
+ loop {}
+ };
+ let Some(max_object_size) = size_base.checked_add(max_rounded_bytes) else {
+ kani::assume(false);
+ loop {}
+ };
+ kani::assume(max_object_size <= bytes_len);
+ kani::assume(trailing_size <= max_slice_bytes);
+
+ // For a whole-element count selected by the validator, this unused
+ // byte count is also `max_slice_bytes % elem_size`. The identity here
+ // holds for every byte count satisfying the capacity bound.
+ #[allow(clippy::arithmetic_side_effects)]
+ let unused_bytes = max_slice_bytes - trailing_size;
+ let unused_aligned_bytes =
+ util::round_down_to_next_multiple_of_alignment(unused_bytes, size_align);
+ #[allow(clippy::arithmetic_side_effects)]
+ let object_size = size_base + (max_rounded_bytes - unused_aligned_bytes);
+
+ let unrounded = size_phase.checked_add(trailing_size).unwrap();
+ let rounded =
+ unrounded.checked_add(util::padding_needed_for(unrounded, size_align)).unwrap();
+ assert_eq!(Some(object_size), size_base.checked_add(rounded));
+ assert!(object_size <= bytes_len);
+ }
- Self { align: align, size_info: size_info, statically_shallow_unpadded: kani::any() }
+ #[kani::proof]
+ #[kani::solver(kissat)]
+ fn prove_slice_dst_validator_rejects_larger_trailing_size() {
+ let trailing: TrailingSliceLayout = any_bounded_trailing_layout();
+ let bytes_len: usize = kani::any();
+ let larger_trailing_size: usize = kani::any();
+ let Some(max_slice_bytes) = trailing.max_trailing_bytes(bytes_len) else {
+ kani::assume(false);
+ loop {}
+ };
+ kani::assume(larger_trailing_size > max_slice_bytes);
+ if let Some(object_size) = slice_dst_size_for_trailing_bytes(trailing, larger_trailing_size)
+ {
+ assert!(object_size > bytes_len);
}
}
- impl kani::Arbitrary for SizeInfo {
- fn any() -> Self {
- let is_sized: bool = kani::any();
+ #[kani::proof]
+ fn prove_slice_formula_contains_trailing_slice() {
+ let trailing: TrailingSliceLayout = any_bounded_trailing_layout();
+ let trailing_size: usize = kani::any();
+ let (_, size_phase) = trailing.size_rounding_align_and_phase.components();
+
+ let Some(unrounded_min) = trailing.size_base.checked_add(size_phase) else {
+ kani::assume(false);
+ loop {}
+ };
+ kani::assume(unrounded_min >= trailing.offset);
+ let Some(object_size) = slice_dst_size_for_trailing_bytes(trailing, trailing_size) else {
+ kani::assume(false);
+ loop {}
+ };
+
+ let Some(trailing_end) = trailing.offset.checked_add(trailing_size) else { unreachable!() };
+ assert!(trailing_end <= object_size);
+ }
+
+ #[kani::proof]
+ fn prove_finalized_slice_formula_is_aligned() {
+ let trailing: TrailingSliceLayout = any_bounded_trailing_layout();
+ let outer_align = any_layout_align();
+ let trailing_size: usize = kani::any();
+ kani::assume(trailing.size_rounding_align_and_phase.align() >= outer_align);
+ #[allow(clippy::arithmetic_side_effects)]
+ let outer_align_mask = outer_align.get() - 1;
+ kani::assume(trailing.size_base & outer_align_mask == 0);
+ let Some(object_size) = slice_dst_size_for_trailing_bytes(trailing, trailing_size) else {
+ kani::assume(false);
+ loop {}
+ };
- match is_sized {
- true => {
- let size: usize = kani::any();
+ assert_eq!(object_size & outer_align_mask, 0);
+ }
- kani::assume(size <= DstLayout::MAX_SIZE);
+ #[kani::proof]
+ fn prove_validator_checks_selected_endpoint_alignment() {
+ let align = any_layout_align();
+ let addr: usize = kani::any();
+ let bytes_len: usize = kani::any();
+ let Some(end) = addr.checked_add(bytes_len) else {
+ kani::assume(false);
+ loop {}
+ };
+ let suffix: bool = kani::any();
+ let cast_type = if suffix { CastType::Suffix } else { CastType::Prefix };
+ let selected_endpoint = if suffix { end } else { addr };
+ #[allow(clippy::arithmetic_side_effects)]
+ let align_mask = align.get() - 1;
- SizeInfo::Sized { size }
+ // The endpoint check precedes and is independent of the size-info
+ // branch. A zero-sized layout makes every aligned case succeed and
+ // isolates that control flow, including prefix/suffix split selection.
+ let layout = DstLayout {
+ align,
+ size_info: SizeInfo::Sized { size: 0 },
+ statically_shallow_unpadded: false,
+ };
+ match layout.validate_cast_and_convert_metadata(addr, bytes_len, cast_type) {
+ Ok((elems, split_at)) => {
+ assert_eq!(selected_endpoint & align_mask, 0);
+ assert_eq!(elems, 0);
+ if suffix {
+ assert_eq!(split_at, bytes_len);
+ } else {
+ assert_eq!(split_at, 0);
}
- false => SizeInfo::SliceDst(kani::any()),
}
+ Err(MetadataCastError::Alignment) => {
+ assert_ne!(selected_endpoint & align_mask, 0);
+ }
+ Err(MetadataCastError::Size) => unreachable!(),
}
}
- impl kani::Arbitrary for TrailingSliceLayout {
- fn any() -> Self {
- let elem_size: usize = kani::any();
- let offset: usize = kani::any();
+ #[kani::proof]
+ fn prove_aligned_suffix_end_and_size_imply_aligned_start() {
+ let align = any_layout_align();
+ #[allow(clippy::arithmetic_side_effects)]
+ let align_mask = align.get() - 1;
+ let addr: usize = kani::any();
+ let bytes_len: usize = kani::any();
+ let object_size: usize = kani::any();
+ let Some(end) = addr.checked_add(bytes_len) else {
+ kani::assume(false);
+ loop {}
+ };
+ kani::assume(end & align_mask == 0);
+ kani::assume(object_size & align_mask == 0);
+ kani::assume(object_size <= bytes_len);
- kani::assume(elem_size < DstLayout::MAX_SIZE);
- kani::assume(offset < DstLayout::MAX_SIZE);
+ #[allow(clippy::arithmetic_side_effects)]
+ let split_at = bytes_len - object_size;
+ let Some(object_start) = addr.checked_add(split_at) else { unreachable!() };
+ assert_eq!(object_start & align_mask, 0);
+ }
- TrailingSliceLayout { elem_size, offset }
- }
+ #[cfg(kani_slow)]
+ #[kani::proof]
+ #[kani::solver(kissat)]
+ fn prove_size_formula_bounds_size_offset() {
+ // Prove the bound for arbitrary byte counts, including counts that
+ // are not whole elements. The `size_for_elems` contract relates this
+ // formula to checked element products. This harness checks the
+ // formula's bound independently of that implementation.
+ let trailing: TrailingSliceLayout = kani::any();
+ let trailing_size: usize = kani::any();
+ let Some(object_size) = slice_dst_size_for_trailing_bytes(trailing, trailing_size) else {
+ kani::assume(false);
+ loop {}
+ };
+ let size_offset = trailing.size_offset();
+ let Some(without_padding) = size_offset.checked_add(trailing_size) else { unreachable!() };
+
+ assert!(without_padding <= object_size);
+ }
+
+ // Older supported compilers resolve the unchecked operation syntax in
+ // `add_scaled_metadata` to `polyfills::NumExt`; newer compilers prefer the
+ // inherent methods. Use UFCS to verify the fallback implementation which
+ // Kani's modern rustc cannot select in the production helper.
+ #[kani::proof]
+ #[kani::solver(kissat)]
+ fn prove_num_ext_affine_fallback_matches_checked_arithmetic() {
+ let base: usize = kani::any();
+ let metadata: usize = kani::any();
+ let multiple: usize = kani::any();
+ let Some(scaled) = metadata.checked_mul(multiple) else {
+ kani::assume(false);
+ loop {}
+ };
+ let Some(expected) = base.checked_add(scaled) else {
+ kani::assume(false);
+ loop {}
+ };
+
+ // SAFETY: The checked multiplication returned `Some`, satisfying
+ // `NumExt::unchecked_mul`'s no-overflow contract.
+ let scaled =
+ unsafe { <usize as crate::util::polyfills::NumExt>::unchecked_mul(metadata, multiple) };
+ // SAFETY: The checked addition returned `Some`, and `scaled` is the
+ // checked product, satisfying `NumExt::unchecked_add`'s no-overflow
+ // contract.
+ let actual =
+ unsafe { <usize as crate::util::polyfills::NumExt>::unchecked_add(base, scaled) };
+ assert_eq!(actual, expected);
}
+ #[cfg(kani_slow)]
#[kani::proof]
fn prove_requires_dynamic_padding() {
- let layout: DstLayout = kani::any();
+ let layout: DstLayout = any_valid_dst_layout();
let SizeInfo::SliceDst(size_info) = layout.size_info else {
kani::assume(false);
@@ -2029,31 +4698,35 @@ fn prove_requires_dynamic_padding() {
loop {}
};
- let Some(unpadded_size) = size_info.offset.checked_add(trailing_slice_size) else {
- // The `unpadded_size` exceeds `usize::MAX`; `meta`` is invalid.
+ let Some(trailing_slice_end) = size_info.offset.checked_add(trailing_slice_size) else {
+ // The trailing slice end exceeds `usize::MAX`; `meta` is invalid.
kani::assume(false);
loop {}
};
- if unpadded_size >= DstLayout::MAX_SIZE {
- // The `unpadded_size` exceeds `isize::MAX`; `meta` is invalid.
+ let Some(size) = slice_dst_size_for_trailing_bytes(size_info, trailing_slice_size) else {
kani::assume(false);
loop {}
- }
+ };
- let trailing_padding = util::padding_needed_for(unpadded_size, layout.align);
+ if size > DstLayout::MAX_SIZE || trailing_slice_end > size {
+ // This metadata cannot describe a valid instance.
+ kani::assume(false);
+ loop {}
+ }
if !layout.requires_dynamic_padding() {
- assert!(trailing_padding == 0);
+ assert_eq!(trailing_slice_end, size);
}
}
+ #[cfg(kani_slow)]
#[kani::proof]
fn prove_dst_layout_extend() {
use crate::util::{max, min, padding_needed_for};
- let base: DstLayout = kani::any();
- let field: DstLayout = kani::any();
+ let base: DstLayout = any_valid_dst_layout();
+ let field: DstLayout = any_valid_dst_layout();
let packed: Option<NonZeroUsize> = kani::any();
if let Some(max_align) = packed {
@@ -2069,9 +4742,6 @@ fn prove_dst_layout_extend() {
unreachable!();
};
- // Under the above conditions, `DstLayout::extend` will not panic.
- let composite = base.extend(field, packed);
-
// The field's alignment is clamped by `max_align` (i.e., the
// `packed` attribute, if any) [1].
//
@@ -2082,10 +4752,6 @@ fn prove_dst_layout_extend() {
// alignment of the field's type.
let field_align = min(field.align, packed.unwrap_or(DstLayout::THEORETICAL_MAX_ALIGN));
- // The struct's alignment is the maximum of its previous alignment and
- // `field_align`.
- assert_eq!(composite.align, max(base.align, field_align));
-
// Compute the minimum amount of inter-field padding needed to
// satisfy the field's alignment, and offset of the trailing field.
// [1]
@@ -2095,7 +4761,38 @@ fn prove_dst_layout_extend() {
// Inter-field padding is guaranteed to be the minimum required in
// order to satisfy each field's (possibly altered) alignment.
let padding = padding_needed_for(base_size, field_align);
- let offset = base_size + padding;
+ let offset = match base_size.checked_add(padding) {
+ Some(offset) => offset,
+ None => {
+ kani::assume(false);
+ loop {}
+ }
+ };
+
+ // Restrict the proof to extensions whose zero-metadata composite size
+ // is representable. This semantic condition implies the
+ // representability of each implementation intermediate below; prove
+ // those implications rather than assuming the intermediates.
+ match field.size_info {
+ SizeInfo::Sized { size } => kani::assume(offset.checked_add(size).is_some()),
+ SizeInfo::SliceDst(trailing) => {
+ let Some(field_min) = trailing.size_for_elems(0) else {
+ kani::assume(false);
+ loop {}
+ };
+ kani::assume(offset.checked_add(field_min).is_some());
+
+ assert!(offset.checked_add(trailing.offset).is_some());
+ assert!(offset.checked_add(trailing.size_base).is_some());
+ }
+ }
+
+ // Under the above conditions, `DstLayout::extend` will not panic.
+ let composite = base.extend(field, packed);
+
+ // The struct's alignment is the maximum of its previous alignment and
+ // `field_align`.
+ assert_eq!(composite.align, max(base.align, field_align));
// For testing purposes, we'll also construct `alloc::Layout`
// stand-ins for `DstLayout`, and show that `extend` behaves
@@ -2132,29 +4829,64 @@ fn prove_dst_layout_extend() {
panic!("The composite of two sized layouts must be sized.")
}
}
- SizeInfo::SliceDst(TrailingSliceLayout {
- offset: field_offset,
- elem_size: field_elem_size,
- }) => {
- if let SizeInfo::SliceDst(TrailingSliceLayout {
- offset: composite_offset,
- elem_size: composite_elem_size,
- }) = composite.size_info
- {
+ SizeInfo::SliceDst(field_trailing) => {
+ if let SizeInfo::SliceDst(composite_trailing) = composite.size_info {
// The offset of the trailing slice component is the sum
// of the offset of the trailing field and the trailing
// slice offset within that field.
- assert_eq!(composite_offset, offset + field_offset);
+ assert_eq!(composite_trailing.offset, offset + field_trailing.offset);
// The elem size is unchanged.
- assert_eq!(composite_elem_size, field_elem_size);
+ assert_eq!(composite_trailing.elem_size, field_trailing.elem_size);
+ // Extension preserves the invariant that the physical
+ // trailing-slice field fits within the unrounded minimum
+ // size. This is one of the premises consumed by the
+ // validator soundness proof.
+ let composite_phase =
+ composite_trailing.size_rounding_align_and_phase.components().1;
+ let Some(composite_unrounded_min) =
+ composite_trailing.size_base.checked_add(composite_phase)
+ else {
+ unreachable!()
+ };
+ assert!(composite_unrounded_min >= composite_trailing.offset);
+
+ // The normalized formula must continue to describe the
+ // size of the field at every trailing byte contribution
+ // for which the checked size computation succeeds. This
+ // is stronger than considering only products of a slice
+ // length and `elem_size`, and avoids a nonlinear
+ // multiplication in the verifier.
+ let trailing_size: usize = kani::any();
+ let field_size =
+ match slice_dst_size_for_trailing_bytes(field_trailing, trailing_size) {
+ Some(size) => size,
+ None => {
+ kani::assume(false);
+ loop {}
+ }
+ };
+ let expected_size = match offset.checked_add(field_size) {
+ Some(size) => size,
+ None => {
+ kani::assume(false);
+ loop {}
+ }
+ };
+ assert_eq!(
+ slice_dst_size_for_trailing_bytes(composite_trailing, trailing_size),
+ Some(expected_size)
+ );
+ // `alloc::Layout` permits sizes which are not a multiple
+ // of alignment, so this represents the unpadded prefix of
+ // the DST through the start of its trailing slice.
let field_analog =
- Layout::from_size_align(field_offset, field_align.get()).unwrap();
+ Layout::from_size_align(field_trailing.offset, field_align.get()).unwrap();
if let Ok((actual_composite, actual_offset)) = base_analog.extend(field_analog)
{
assert_eq!(actual_offset, offset);
- assert_eq!(actual_composite.size(), composite_offset);
+ assert_eq!(actual_composite.size(), composite_trailing.offset);
assert_eq!(actual_composite.align(), composite.align.get());
} else {
// An error here reflects that composite of `base`
@@ -2175,8 +4907,8 @@ fn prove_dst_layout_extend() {
#[kani::proof]
#[kani::should_panic]
fn prove_dst_layout_extend_dst_panics() {
- let base: DstLayout = kani::any();
- let field: DstLayout = kani::any();
+ let base: DstLayout = any_valid_dst_layout();
+ let field: DstLayout = any_valid_dst_layout();
let packed: Option<NonZeroUsize> = kani::any();
if let Some(max_align) = packed {
@@ -2189,39 +4921,266 @@ fn prove_dst_layout_extend_dst_panics() {
let _ = base.extend(field, packed);
}
+ #[cfg(kani_slow)]
+ fn prove_dst_layout_pad_to_align_dst_invariants(
+ layout: DstLayout,
+ unpadded_trailing: TrailingSliceLayout,
+ padded: DstLayout,
+ ) {
+ // Calling `pad_to_align` does not alter the `DstLayout`'s alignment.
+ assert_eq!(padded.align, layout.align);
+ assert_eq!(padded.statically_shallow_unpadded, layout.statically_shallow_unpadded);
+
+ let SizeInfo::SliceDst(padded_trailing) = padded.size_info else {
+ panic!("The padding of a DST layout must result in a DST layout.")
+ };
+ assert_eq!(padded_trailing.offset, unpadded_trailing.offset);
+ assert_eq!(padded_trailing.elem_size, unpadded_trailing.elem_size);
+
+ // A finalized DST formula is visibly aligned: the rounding alignment
+ // is at least the outer alignment, and the fixed base is a multiple of
+ // the outer alignment. This is the invariant which lets suffix
+ // validation infer alignment of the object's start from alignment of
+ // its end.
+ assert!(padded_trailing.size_rounding_align_and_phase.align() >= padded.align);
+ #[allow(clippy::arithmetic_side_effects)]
+ let padded_align_mask = padded.align.get() - 1;
+ assert_eq!(padded_trailing.size_base & padded_align_mask, 0);
+ assert_eq!(padded.pad_to_align(), padded);
+ let padded_phase = padded_trailing.size_rounding_align_and_phase.components().1;
+ let Some(padded_unrounded_min) = padded_trailing.size_base.checked_add(padded_phase) else {
+ unreachable!()
+ };
+ assert!(padded_unrounded_min >= padded_trailing.offset);
+ }
+
#[kani::proof]
- fn prove_dst_layout_pad_to_align() {
+ fn prove_dst_layout_pad_to_align_sized() {
use crate::util::padding_needed_for;
- let layout: DstLayout = kani::any();
+ let layout: DstLayout = any_valid_dst_layout();
+ let SizeInfo::Sized { size: unpadded_size } = layout.size_info else {
+ kani::assume(false);
+ loop {}
+ };
let padded = layout.pad_to_align();
-
- // Calling `pad_to_align` does not alter the `DstLayout`'s alignment.
assert_eq!(padded.align, layout.align);
+ let SizeInfo::Sized { size: padded_size } = padded.size_info else {
+ panic!("The padding of a sized layout must result in a sized layout.")
+ };
- if let SizeInfo::Sized { size: unpadded_size } = layout.size_info {
- if let SizeInfo::Sized { size: padded_size } = padded.size_info {
- // If the layout is sized, it will remain sized after padding is
- // added. Its sum will be its unpadded size and the size of the
- // trailing padding needed to satisfy its alignment
- // requirements.
- let padding = padding_needed_for(unpadded_size, layout.align);
- assert_eq!(padded_size, unpadded_size + padding);
-
- // Prove that calling `DstLayout::pad_to_align` behaves
- // identically to `Layout::pad_to_align`.
- let layout_analog =
- Layout::from_size_align(unpadded_size, layout.align.get()).unwrap();
- let padded_analog = layout_analog.pad_to_align();
- assert_eq!(padded_analog.align(), layout.align.get());
- assert_eq!(padded_analog.size(), padded_size);
- } else {
- panic!("The padding of a sized layout must result in a sized layout.")
- }
- } else {
- // If the layout is a DST, padding cannot be statically added.
- assert_eq!(padded.size_info, layout.size_info);
- }
+ // If the layout is sized, it will remain sized after padding is added.
+ // Its sum will be its unpadded size and the size of the trailing
+ // padding needed to satisfy its alignment requirements.
+ let padding = padding_needed_for(unpadded_size, layout.align);
+ assert_eq!(padded_size, unpadded_size + padding);
+ assert_eq!(
+ padded.statically_shallow_unpadded,
+ layout.statically_shallow_unpadded && padding == 0
+ );
+
+ // Prove that calling `DstLayout::pad_to_align` behaves identically to
+ // `Layout::pad_to_align`.
+ let layout_analog = Layout::from_size_align(unpadded_size, layout.align.get()).unwrap();
+ let padded_analog = layout_analog.pad_to_align();
+ assert_eq!(padded_analog.align(), layout.align.get());
+ assert_eq!(padded_analog.size(), padded_size);
+ }
+
+ #[cfg(kani_slow)]
+ #[kani::proof]
+ fn prove_dst_layout_pad_to_align_dst_inner_rounding_transition() {
+ use crate::util::padding_needed_for;
+
+ let layout: DstLayout = any_valid_dst_layout();
+ let SizeInfo::SliceDst(trailing) = layout.size_info else {
+ kani::assume(false);
+ loop {}
+ };
+ kani::assume(trailing.size_rounding_align_and_phase.align() >= layout.align);
+
+ // `any_valid_dst_layout` requires the zero-metadata layout to be valid.
+ // Prove that this semantic condition bounds both the padded minimum
+ // and the implementation intermediate; do not assume the intermediate
+ // merely because the implementation uses it.
+ let Some(min_size) = trailing.size_for_elems(0) else { unreachable!() };
+ let min_padding = padding_needed_for(min_size, layout.align);
+ let Some(padded_min_size) = min_size.checked_add(min_padding) else { unreachable!() };
+ assert!(padded_min_size <= DstLayout::MAX_SIZE);
+
+ let padding = padding_needed_for(trailing.size_base, layout.align);
+ let Some(expected_base) = trailing.size_base.checked_add(padding) else { unreachable!() };
+
+ let padded = layout.pad_to_align();
+ let SizeInfo::SliceDst(padded_trailing) = padded.size_info else { unreachable!() };
+ assert_eq!(padded_trailing.size_base, expected_base);
+ assert_eq!(
+ padded_trailing.size_rounding_align_and_phase,
+ trailing.size_rounding_align_and_phase
+ );
+ prove_dst_layout_pad_to_align_dst_invariants(layout, trailing, padded);
+ }
+
+ #[kani::proof]
+ #[kani::solver(kissat)]
+ fn prove_dst_layout_pad_to_align_dst_inner_rounding_formula() {
+ use crate::util::padding_needed_for;
+
+ let size_align = any_layout_align();
+ let outer_align = any_layout_align();
+ kani::assume(size_align >= outer_align);
+ let size_base: usize = kani::any();
+ let raw_size_phase: usize = kani::any();
+ #[allow(clippy::arithmetic_side_effects)]
+ let size_phase = raw_size_phase & (size_align.get() - 1);
+ let trailing_size: usize = kani::any();
+
+ // Evaluate `size_base + round_up(size_phase + trailing_size,
+ // size_align)`, then round that complete size to `outer_align`.
+ // Compare with the equivalent expression obtained by rounding only
+ // `size_base` to `outer_align`, since `size_align` is a multiple of
+ // `outer_align`. Valid Rust objects have representable sizes; discard
+ // only trailing byte counts whose input or padded object size is not
+ // representable.
+ let Some(unrounded_inner) = size_phase.checked_add(trailing_size) else {
+ kani::assume(false);
+ loop {}
+ };
+ let Some(rounded_inner) =
+ unrounded_inner.checked_add(padding_needed_for(unrounded_inner, size_align))
+ else {
+ kani::assume(false);
+ loop {}
+ };
+ let Some(unpadded_size) = size_base.checked_add(rounded_inner) else {
+ kani::assume(false);
+ loop {}
+ };
+ let Some(expected_size) =
+ unpadded_size.checked_add(padding_needed_for(unpadded_size, outer_align))
+ else {
+ kani::assume(false);
+ loop {}
+ };
+
+ let Some(padded_base) = size_base.checked_add(padding_needed_for(size_base, outer_align))
+ else {
+ unreachable!()
+ };
+ let Some(actual_size) = padded_base.checked_add(rounded_inner) else { unreachable!() };
+ assert_eq!(actual_size, expected_size);
+ }
+
+ #[cfg(kani_slow)]
+ #[kani::proof]
+ fn prove_dst_layout_pad_to_align_dst_outer_alignment_transition() {
+ use crate::util::padding_needed_for;
+
+ let layout: DstLayout = any_valid_dst_layout();
+ let SizeInfo::SliceDst(trailing) = layout.size_info else {
+ kani::assume(false);
+ loop {}
+ };
+ let size_align = trailing.size_rounding_align_and_phase.align();
+ kani::assume(size_align < layout.align);
+
+ // `any_valid_dst_layout` requires the zero-metadata layout to be valid.
+ // Prove that this semantic condition bounds both the padded minimum
+ // and every implementation intermediate; do not assume the
+ // intermediates merely because the implementation uses them.
+ let Some(min_size) = trailing.size_for_elems(0) else { unreachable!() };
+ let min_padding = padding_needed_for(min_size, layout.align);
+ let Some(padded_min_size) = min_size.checked_add(min_padding) else { unreachable!() };
+ assert!(padded_min_size <= DstLayout::MAX_SIZE);
+
+ let base_padding = padding_needed_for(trailing.size_base, size_align);
+ let Some(rounded_base) = trailing.size_base.checked_add(base_padding) else {
+ unreachable!()
+ };
+ let Some(t) =
+ rounded_base.checked_add(trailing.size_rounding_align_and_phase.components().1)
+ else {
+ unreachable!()
+ };
+ #[allow(clippy::arithmetic_side_effects)]
+ let expected_phase = t & (layout.align.get() - 1);
+ #[allow(clippy::arithmetic_side_effects)]
+ let expected_base = t - expected_phase;
+
+ let padded = layout.pad_to_align();
+ let SizeInfo::SliceDst(padded_trailing) = padded.size_info else { unreachable!() };
+ assert_eq!(padded_trailing.size_base, expected_base);
+ assert_eq!(
+ padded_trailing.size_rounding_align_and_phase,
+ RoundingAlignAndPhase::new(layout.align, expected_phase)
+ );
+ prove_dst_layout_pad_to_align_dst_invariants(layout, trailing, padded);
+ }
+
+ #[kani::proof]
+ #[kani::solver(kissat)]
+ fn prove_dst_layout_pad_to_align_dst_outer_alignment_formula() {
+ use crate::util::padding_needed_for;
+
+ let size_align = any_layout_align();
+ let outer_align = any_layout_align();
+ kani::assume(size_align < outer_align);
+ let size_base: usize = kani::any();
+ let raw_size_phase: usize = kani::any();
+ #[allow(clippy::arithmetic_side_effects)]
+ let size_phase = raw_size_phase & (size_align.get() - 1);
+ let trailing_size: usize = kani::any();
+
+ // Evaluate `size_base + round_up(size_phase + trailing_size,
+ // size_align)`, then round that complete size to `outer_align`.
+ let Some(unrounded_inner) = size_phase.checked_add(trailing_size) else {
+ kani::assume(false);
+ loop {}
+ };
+ let Some(rounded_inner) =
+ unrounded_inner.checked_add(padding_needed_for(unrounded_inner, size_align))
+ else {
+ kani::assume(false);
+ loop {}
+ };
+ let Some(unpadded_size) = size_base.checked_add(rounded_inner) else {
+ kani::assume(false);
+ loop {}
+ };
+ let Some(expected_size) =
+ unpadded_size.checked_add(padding_needed_for(unpadded_size, outer_align))
+ else {
+ kani::assume(false);
+ loop {}
+ };
+
+ // Since `outer_align` is a larger multiple of `size_align`, absorb
+ // the inner rounding into the outer rounding and split the fixed
+ // contribution into a base and phase:
+ //
+ // fixed_contribution = round_up(size_base, size_align) + size_phase
+ // new_phase = fixed_contribution % outer_align
+ // new_base = fixed_contribution - new_phase
+ // new_base + round_up(new_phase + trailing_size, outer_align)
+ let Some(rounded_base) = size_base.checked_add(padding_needed_for(size_base, size_align))
+ else {
+ unreachable!()
+ };
+ let Some(t) = rounded_base.checked_add(size_phase) else { unreachable!() };
+ #[allow(clippy::arithmetic_side_effects)]
+ let new_phase = t & (outer_align.get() - 1);
+ #[allow(clippy::arithmetic_side_effects)]
+ let new_base = t - new_phase;
+ let Some(new_unrounded_inner) = new_phase.checked_add(trailing_size) else {
+ unreachable!()
+ };
+ let Some(new_rounded_inner) =
+ new_unrounded_inner.checked_add(padding_needed_for(new_unrounded_inner, outer_align))
+ else {
+ unreachable!()
+ };
+ let Some(actual_size) = new_base.checked_add(new_rounded_inner) else { unreachable!() };
+ assert_eq!(actual_size, expected_size);
}
}
diff --git a/rust/zerocopy/src/lib.rs b/rust/zerocopy/src/lib.rs
index 572f0563fe2f..6dbeeb7488be 100644
--- a/rust/zerocopy/src/lib.rs
+++ b/rust/zerocopy/src/lib.rs
@@ -253,7 +253,7 @@
)]
#![cfg_attr(
__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS,
- deny(fuzzy_provenance_casts, lossy_provenance_casts)
+ deny(implicit_provenance_casts)
)]
#![deny(
clippy::all,
@@ -315,10 +315,6 @@
clippy::indexing_slicing,
))]
#![cfg_attr(not(any(test, kani, feature = "std")), no_std)]
-#![cfg_attr(
- all(feature = "simd-nightly", target_arch = "arm"),
- feature(stdarch_arm_neon_intrinsics)
-)]
#![cfg_attr(
all(feature = "simd-nightly", any(target_arch = "powerpc", target_arch = "powerpc64")),
feature(stdarch_powerpc)
@@ -326,10 +322,6 @@
#![cfg_attr(feature = "float-nightly", feature(f16, f128))]
#![cfg_attr(doc_cfg, feature(doc_cfg))]
#![cfg_attr(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS, feature(coverage_attribute))]
-#![cfg_attr(
- any(__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS, miri),
- feature(layout_for_ptr)
-)]
#![cfg_attr(all(test, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), feature(test))]
// This is a hack to allow zerocopy-derive derives to work in this crate. They
@@ -401,7 +393,7 @@
wrappers::*,
};
-#[cfg(any(feature = "alloc", test, kani))]
+#[cfg(any(feature = "alloc", test))]
extern crate alloc;
#[cfg(any(feature = "alloc", test))]
use alloc::{boxed::Box, vec::Vec};
@@ -411,7 +403,7 @@
// Used by `KnownLayout`.
#[doc(hidden)]
pub use crate::layout::*;
-// Used by `TryFromBytes::is_bit_valid`.
+// Used by `TryFromBytes::is_safe`.
#[doc(hidden)]
pub use crate::pointer::{invariant::BecauseImmutable, Maybe, Ptr};
// For each trait polyfill, as soon as the corresponding feature is stable, the
@@ -786,10 +778,16 @@ fn only_derive_is_allowed_to_implement_this_trait()
/// where `size == size_of::<Self>()`
/// - If `Self` is a slice DST, then `LAYOUT.size_info ==
/// SizeInfo::SliceDst(slice_layout)` where:
+ /// - `slice_layout.size_rounding_align_and_phase` decodes to an alignment
+ /// that is a power of two and a phase which is less than that alignment
/// - The size, `size`, of an instance of `Self` with `elems` trailing
- /// slice elements is equal to `slice_layout.offset +
- /// slice_layout.elem_size * elems` rounded up to the nearest multiple
- /// of `LAYOUT.align`
+ /// slice elements is equal to `slice_layout.size_base +
+ /// round_up(size_phase + slice_layout.elem_size * elems, size_align)`,
+ /// where `(size_align, size_phase)` is decoded from
+ /// `slice_layout.size_rounding_align_and_phase`
+ /// - The trailing slice begins at byte offset `slice_layout.offset`
+ /// - `slice_layout.offset + slice_layout.elem_size * elems` does not
+ /// overflow and is no greater than `size`
/// - For such an instance, any bytes in the range `[slice_layout.offset +
/// slice_layout.elem_size * elems, size)` are padding and must not be
/// assumed to be initialized
@@ -1004,11 +1002,7 @@ fn to_elem_count(self) -> usize {
#[inline]
fn size_for_metadata(self, layout: DstLayout) -> Option<usize> {
match layout.size_info {
- SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }) => {
- let slice_len = elem_size.checked_mul(self)?;
- let without_padding = offset.checked_add(slice_len)?;
- without_padding.checked_add(util::padding_needed_for(without_padding, layout.align))
- }
+ SizeInfo::SliceDst(trailing) => trailing.size_for_elems(self),
// NOTE: This branch is unreachable, but we return `None` rather
// than `unreachable!()` to avoid generating panic paths.
SizeInfo::Sized { .. } => None,
@@ -1073,29 +1067,85 @@ fn raw_from_ptr_len(data: NonNull<u8>, elems: usize) -> NonNull<Self> {
#[inline(always)]
fn pointer_to_metadata(ptr: *mut [T]) -> usize {
- #[allow(clippy::as_conversions)]
- let slc = ptr as *const [()];
+ #[cfg(not(no_zerocopy_slice_ptr_len_1_79_0))]
+ {
+ ptr.len()
+ }
- // SAFETY:
- // - `()` has alignment 1, so `slc` is trivially aligned.
- // - `slc` was derived from a non-null pointer.
- // - The size is 0 regardless of the length, so it is sound to
- // materialize a reference regardless of location.
- // - By invariant, `self.ptr` has valid provenance.
- let slc = unsafe { &*slc };
-
- // This is correct because the preceding `as` cast preserves the number
- // of slice elements. [1]
- //
- // [1] Per https://doc.rust-lang.org/reference/expressions/operator-expr.html#pointer-to-pointer-cast:
- //
- // For slice types like `[T]` and `[U]`, the raw pointer types `*const
- // [T]`, `*mut [T]`, `*const [U]`, and `*mut [U]` encode the number of
- // elements in this slice. Casts between these raw pointer types
- // preserve the number of elements. ... The same holds for `str` and
- // any compound type whose unsized tail is a slice type, such as
- // struct `Foo(i32, [u8])` or `(u64, Foo)`.
- slc.len()
+ #[cfg(no_zerocopy_slice_ptr_len_1_79_0)]
+ {
+ // `*mut [T]::len` was not stable before Rust 1.79. In every Rust
+ // version from our 1.56 MSRV through 1.78, `Hash for *mut T`
+ // decomposes a raw pointer and passes its address and metadata to
+ // the hasher in that order [1]. `Hash for usize` passes its value
+ // to `Hasher::write_usize` [2]. Capture those two values to obtain
+ // a candidate for the slice length without dereferencing `ptr` or
+ // constructing a reference.
+ //
+ // This historical `Hash` implementation is only used to produce a
+ // candidate. Before returning it, we reconstruct a raw slice and
+ // authenticate the candidate with `ptr::eq`, which compares slice
+ // lengths as well as addresses [3]. Thus, an unexpected `Hash`
+ // implementation cannot cause us to return incorrect metadata; it
+ // can only fail to produce an authenticated candidate.
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/src/core/hash/mod.rs.html#776-782:
+ //
+ // let (address, metadata) = self.to_raw_parts();
+ // state.write_usize(address as usize);
+ // metadata.hash(state);
+ //
+ // [2] Per https://doc.rust-lang.org/1.56.0/src/core/hash/mod.rs.html#628-656:
+ //
+ // fn hash<H: Hasher>(&self, state: &mut H) {
+ // state.$meth(*self)
+ // }
+ // ...
+ // (usize, write_usize),
+ //
+ // [3] Per https://doc.rust-lang.org/1.56.0/std/ptr/fn.eq.html:
+ //
+ // Slices are also compared by their length (fat pointers).
+ struct MetadataHasher {
+ values: [usize; 2],
+ writes: usize,
+ valid: bool,
+ }
+
+ impl Hasher for MetadataHasher {
+ #[inline(always)]
+ fn finish(&self) -> u64 {
+ 0
+ }
+
+ #[inline(always)]
+ fn write(&mut self, _bytes: &[u8]) {
+ self.valid = false;
+ }
+
+ #[inline(always)]
+ fn write_usize(&mut self, value: usize) {
+ match self.values.get_mut(self.writes) {
+ Some(slot) => *slot = value,
+ None => self.valid = false,
+ }
+ self.writes = self.writes.saturating_add(1);
+ }
+ }
+
+ let mut hasher = MetadataHasher { values: [0; 2], writes: 0, valid: true };
+ core::hash::Hash::hash(&ptr, &mut hasher);
+ assert!(
+ hasher.valid && hasher.writes == 2,
+ "unexpected raw-pointer Hash implementation"
+ );
+
+ let elems = hasher.values[1];
+ #[allow(clippy::as_conversions)]
+ let reconstructed = ptr::slice_from_raw_parts_mut(ptr as *mut T, elems);
+ assert!(ptr::eq(ptr, reconstructed), "captured value is not raw-slice metadata");
+ elems
+ }
}
}
@@ -1182,11 +1232,29 @@ fn pointer_to_metadata(ptr: *mut [T]) -> usize {
#[doc(hidden)]
pub const REPR_C_UNION_VARIANT_ID: i128 = -3;
+/// Marker types used to disambiguate implementations of projection traits that
+/// would otherwise conflict.
+#[doc(hidden)]
+#[allow(missing_copy_implementations, missing_debug_implementations)]
+pub mod project_clients {
+ pub enum TryFromBytesDerive {}
+
+ pub enum ProjectDerive {}
+}
+
+#[cfg(any(feature = "derive", test))]
+#[cfg_attr(doc_cfg, doc(cfg(feature = "derive")))]
+#[doc(hidden)]
+pub use zerocopy_derive::Project;
+
/// # Safety
///
-/// `Self::ProjectToTag` must satisfy its safety invariant.
+/// `<Self as HasTag<Client>>::ProjectToTag` must satisfy its safety invariant.
+///
+/// The `Client` parameter exists solely to disambiguate between implementations
+/// of `HasTag` that would otherwise conflict.
#[doc(hidden)]
-pub unsafe trait HasTag {
+pub unsafe trait HasTag<Client = project_clients::TryFromBytesDerive> {
fn only_derive_is_allowed_to_implement_this_trait()
where
Self: Sized;
@@ -1198,28 +1266,35 @@ fn only_derive_is_allowed_to_implement_this_trait()
///
/// # Safety
///
- /// It must be the case that, for all `slf: Ptr<'_, Self, I>`, it is sound
- /// to project from `slf` to `Ptr<'_, Self::Tag, I>` using this projection.
+ /// It must be the case that, for all `slf: Ptr<'_, Self, I>` where
+ /// `I::Aliasing` is `Shared`, it is sound to project it using this
+ /// projection to a `Ptr<'_, Self::Tag, (Shared, I::Alignment,
+ /// I::Validity)>`.
type ProjectToTag: pointer::cast::Project<Self, Self::Tag>;
}
/// Projects a given field from `Self`.
///
-/// All implementations of `HasField` for a particular field `f` in `Self`
-/// should use the same `Field` type; this ensures that `Field` is inferable
-/// given an explicit `VARIANT_ID` and `FIELD_ID`.
+/// All implementations of `HasField` for a particular `Client` and field `f`
+/// in `Self` should use the same `Field` type; this ensures that `Field` is
+/// inferable given an explicit `Client`, `VARIANT_ID`, and `FIELD_ID`.
+///
+/// The `Client` parameter exists solely to disambiguate between implementations
+/// of `HasField` that would otherwise conflict.
///
/// # Safety
///
/// A field `f` is `HasField` for `Self` if and only if:
///
-/// - If `Self` has the layout of a struct or union type, then `VARIANT_ID` is
-/// `STRUCT_VARIANT_ID` or `UNION_VARIANT_ID` respectively; otherwise, if
-/// `Self` has the layout of an enum type, `VARIANT_ID` is the numerical index
-/// of the enum variant in which `f` appears. Note that `Self` does not need
-/// to actually *be* such a type – it just needs to have the same layout as
-/// such a type. For example, a `#[repr(transparent)]` wrapper around an enum
-/// has the same layout as that enum.
+/// - If `Self` has the layout of a struct type, `VARIANT_ID` is
+/// `STRUCT_VARIANT_ID`. If `Self` has the layout of a `repr(C)` union type,
+/// `VARIANT_ID` is `REPR_C_UNION_VARIANT_ID`; for other union layouts, it is
+/// `UNION_VARIANT_ID`. Otherwise, if `Self` has the layout of an enum type and
+/// `f` appears in a variant named `v`, `VARIANT_ID` is
+/// `zerocopy::ident_id!(v)`. Note that `Self` does not need to actually *be*
+/// such a type – it just needs to have the same layout as such a type. For
+/// example, a `#[repr(transparent)]` wrapper around an enum has the same
+/// layout as that enum.
/// - If `f` has name `n`, `FIELD_ID` is `zerocopy::ident_id!(n)`; otherwise,
/// if `f` is at index `i`, `FIELD_ID` is `zerocopy::ident_id!(i)`.
/// - `Field` is a type with the same visibility as `f`.
@@ -1232,8 +1307,8 @@ fn only_derive_is_allowed_to_implement_this_trait()
///
/// The implementation of `project` must satisfy its safety post-condition.
#[doc(hidden)]
-pub unsafe trait HasField<Field, const VARIANT_ID: i128, const FIELD_ID: i128>:
- HasTag
+pub unsafe trait HasField<Client, Field, const VARIANT_ID: i128, const FIELD_ID: i128>:
+ HasTag<Client>
{
fn only_derive_is_allowed_to_implement_this_trait()
where
@@ -1263,15 +1338,20 @@ fn only_derive_is_allowed_to_implement_this_trait()
/// other words, it is a type-level function over invariants; `I` goes in,
/// `Self::Invariants` comes out.
///
+/// The `Client` parameter exists solely to disambiguate between implementations
+/// of `ProjectField` (and their corresponding `HasField` implementations) that
+/// would otherwise conflict.
+///
/// # Safety
///
-/// `T: ProjectField<Field, I, VARIANT_ID, FIELD_ID>` if, for a
-/// `ptr: Ptr<'_, T, I>` such that `T::is_projectable(ptr).is_ok()`,
-/// `<T as HasField<Field, VARIANT_ID, FIELD_ID>>::project(ptr.as_inner())`
+/// `T: ProjectField<Client, Field, I, VARIANT_ID, FIELD_ID>` if, for a
+/// `ptr: Ptr<'_, T, I>` such that `T::is_projectable` returns `Ok(())`
+/// when passed the tag pointer projected from `ptr`,
+/// `<T as HasField<Client, Field, VARIANT_ID, FIELD_ID>>::project(ptr.as_inner())`
/// conforms to `T::Invariants`.
#[doc(hidden)]
-pub unsafe trait ProjectField<Field, I, const VARIANT_ID: i128, const FIELD_ID: i128>:
- HasField<Field, VARIANT_ID, FIELD_ID>
+pub unsafe trait ProjectField<Client, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128>:
+ HasField<Client, Field, VARIANT_ID, FIELD_ID>
where
I: invariant::Invariants,
{
@@ -1290,29 +1370,35 @@ fn only_derive_is_allowed_to_implement_this_trait()
/// Is the given field projectable from `ptr`?
///
- /// If a field with [`Self::Invariants`] is projectable from the referent,
- /// this function produces an `Ok(ptr)` from which the projection can be
- /// made; otherwise `Err`.
+ /// If a field with [`Self::Invariants`] is projectable from the containing
+ /// value whose projected tag is `ptr`, this function produces `Ok(())`;
+ /// otherwise it produces `Err`.
///
/// This method must be overriden if the field's projectability depends on
/// the value of the bytes in `ptr`.
#[inline(always)]
- fn is_projectable<'a>(_ptr: Ptr<'a, Self::Tag, I>) -> Result<(), Self::Error> {
+ fn is_projectable<'a>(
+ _ptr: Ptr<
+ 'a,
+ <Self as HasTag<Client>>::Tag,
+ (invariant::Shared, I::Alignment, I::Validity),
+ >,
+ ) -> Result<(), Self::Error> {
trait IsInfallible {
const IS_INFALLIBLE: bool;
}
- struct Projection<T, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128>(
- PhantomData<(Field, I, T)>,
+ struct Projection<T, Client, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128>(
+ PhantomData<(Client, Field, I, T)>,
)
where
- T: ?Sized + HasField<Field, VARIANT_ID, FIELD_ID>,
+ T: ?Sized + HasField<Client, Field, VARIANT_ID, FIELD_ID>,
I: invariant::Invariants;
- impl<T, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128> IsInfallible
- for Projection<T, Field, I, VARIANT_ID, FIELD_ID>
+ impl<T, Client, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128> IsInfallible
+ for Projection<T, Client, Field, I, VARIANT_ID, FIELD_ID>
where
- T: ?Sized + HasField<Field, VARIANT_ID, FIELD_ID>,
+ T: ?Sized + HasField<Client, Field, VARIANT_ID, FIELD_ID>,
I: invariant::Invariants,
{
const IS_INFALLIBLE: bool = {
@@ -1323,7 +1409,9 @@ impl<T, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128> IsInfallible
// referent. This default implementation of `is_projectable`
// is non-destructive, as it does not overwrite any part of
// the referent.
- crate::STRUCT_VARIANT_ID | crate::UNION_VARIANT_ID => true,
+ crate::STRUCT_VARIANT_ID
+ | crate::UNION_VARIANT_ID
+ | crate::REPR_C_UNION_VARIANT_ID => true,
_enum_variant => {
use crate::invariant::{Validity, ValidityKind};
match I::Validity::KIND {
@@ -1336,9 +1424,9 @@ impl<T, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128> IsInfallible
// type – the type itself is irrelevant.
ValidityKind::Uninit | ValidityKind::Initialized => true,
// The projectability of an enum field from an
- // `AsInitialized` or `Valid` state is a dynamic
+ // `AsInitialized` or `Safe` state is a dynamic
// property of its tag.
- ValidityKind::AsInitialized | ValidityKind::Valid => false,
+ ValidityKind::AsInitialized | ValidityKind::Safe => false,
}
}
};
@@ -1348,7 +1436,7 @@ impl<T, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128> IsInfallible
}
const_assert!(
- <Projection<Self, Field, I, VARIANT_ID, FIELD_ID> as IsInfallible>::IS_INFALLIBLE
+ <Projection<Self, Client, Field, I, VARIANT_ID, FIELD_ID> as IsInfallible>::IS_INFALLIBLE
);
Ok(())
@@ -1614,6 +1702,64 @@ fn only_derive_is_allowed_to_implement_this_trait()
/// }
/// ```
///
+#[cfg_attr(
+ zerocopy_unstable_ptr,
+ doc = r#"
+# Field invariants
+
+This experimental feature requires `--cfg zerocopy_unstable_ptr`.
+
+Named fields of structs, enum variants, and unions can specify additional
+runtime checks using `#[zerocopy(invariant(expression))]`:
+
+```
+# use zerocopy_derive::TryFromBytes;
+#[derive(TryFromBytes)]
+struct Foo {
+ a: u8,
+ #[zerocopy(invariant((*a.read() % 2) == (*b.read() as u8)))]
+ b: bool,
+ #[zerocopy(invariant(*c.read() > 0))]
+ c: i16,
+}
+```
+
+Each expression must return a `bool`. It has access to validated, read-only
+[`Ptr`]s to the current field and all preceding fields of the struct or
+variant, using their field names. A union's invariants have access only to
+the current field. The expression can use the existing [`Ptr`] APIs to
+inspect those fields. In this example, `read()` copies each field without
+requiring alignment, and dereferencing the resulting [`ReadOnly`] accesses
+the copied value.
+
+Rust's usual restrictions on local bindings apply; for example, a field name
+cannot shadow an in-scope constant.
+
+Fields are checked in declaration order. Each field's bit validity is
+checked before its invariants run. Multiple invariants on a field run in
+attribute order. For structs and enums, validation stops at the first invalid
+field or invariant that returns `false`. For unions, a failed bit-validity
+check or invariant causes validation to try the next field; validation
+succeeds as soon as one field and all its invariants pass. Each expression
+runs in its own closure; `return` returns from that expression, and panics
+propagate to the caller. Only the selected enum variant's fields and
+invariants are checked. Expressions may have arbitrary side effects, even
+when the conversion fails or its result is discarded.
+
+These predicates are additional acceptance checks beyond Rust's bit validity;
+bit-valid bytes may still be rejected. See [What is a "valid instance"?] for
+the distinction.
+
+[What is a "valid instance"?]: trait@TryFromBytes#what-is-a-valid-instance
+
+Invariants are not supported on tuple fields. Types with invariants cannot
+derive [`FromZeros`] or [`FromBytes`], whose conversions do not perform runtime
+validation. These checks apply to conversions through [`TryFromBytes`]; they
+do not restrict ordinary construction or mutation of Rust values.
+
+"#
+)]
+///
/// # Portability
///
/// To ensure consistent endianness for enums with multi-byte representations,
@@ -1708,6 +1854,14 @@ fn only_derive_is_allowed_to_implement_this_trait()
/// If you are negatively affected by lack of support for a particular type,
/// we encourage you to let us know by [filing an issue][github-repo].
///
+/// In this trait's conversion methods, a "valid instance" must also pass any
+/// configured field invariants, including those on nested fields. Rust bit
+/// validity alone does not guarantee that a conversion succeeds: an invariant
+/// may reject otherwise bit-valid bytes. Such predicates check acceptance at
+/// conversion time; they do not constrain subsequent mutation or ordinary Rust
+/// construction. See the [derive's field invariants][derive] documentation for
+/// the experimental attribute's syntax, evaluation order, and side effects.
+///
/// # `TryFromBytes` is not symmetrical with [`IntoBytes`]
///
/// There are some types which implement both `TryFromBytes` and [`IntoBytes`],
@@ -1754,25 +1908,25 @@ fn only_derive_is_allowed_to_implement_this_trait()
///
/// # Safety
///
- /// Unsafe code may assume that, if `is_bit_valid(candidate)` returns true,
+ /// Unsafe code may assume that, if `is_safe(candidate)` returns true,
/// `*candidate` contains a valid `Self`.
///
/// # Panics
///
- /// `is_bit_valid` may panic. Callers are responsible for ensuring that any
- /// `unsafe` code remains sound even in the face of `is_bit_valid`
- /// panicking. (We support user-defined validation routines; so long as
- /// these routines are not required to be `unsafe`, there is no way to
- /// ensure that these do not generate panics.)
+ /// `is_safe` may panic. Callers are responsible for ensuring that any
+ /// `unsafe` code remains sound even in the face of `is_safe` panicking. (We
+ /// support user-defined validation routines; so long as these routines are
+ /// not required to be `unsafe`, there is no way to ensure that these do not
+ /// generate panics.)
///
- /// Besides user-defined validation routines panicking, `is_bit_valid` will
- /// either panic or fail to compile if called on a pointer with [`Shared`]
- /// aliasing when `Self: !Immutable`.
+ /// Besides user-defined validation routines panicking, `is_safe` will either
+ /// panic or fail to compile if called on a pointer with [`Shared`] aliasing
+ /// when `Self: !Immutable`.
///
/// [`UnsafeCell`]: core::cell::UnsafeCell
/// [`Shared`]: invariant::Shared
#[doc(hidden)]
- fn is_bit_valid<A>(candidate: Maybe<'_, Self, A>) -> bool
+ fn is_safe<A>(candidate: Maybe<'_, Self, A>) -> bool
where
A: invariant::Alignment;
@@ -1872,7 +2026,7 @@ fn is_bit_valid<A>(candidate: Maybe<'_, Self, A>) -> bool
@variant "dynamic_padding"
]
)]
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_ref_from_bytes(source: &[u8]) -> Result<&Self, TryCastError<&[u8], Self>>
@@ -1885,7 +2039,7 @@ fn try_ref_from_bytes(source: &[u8]) -> Result<&Self, TryCastError<&[u8], Self>>
// This call may panic. If that happens, it doesn't cause any soundness
// issues, as we have not generated any invalid state which we need to
// fix before returning.
- match source.try_into_valid() {
+ match source.try_into_safe() {
Ok(valid) => Ok(valid.as_ref()),
Err(e) => {
Err(e.map_src(|src| src.as_bytes::<BecauseImmutable>().as_ref()).into())
@@ -1995,7 +2149,7 @@ fn try_ref_from_bytes(source: &[u8]) -> Result<&Self, TryCastError<&[u8], Self>>
@variant "dynamic_padding"
]
)]
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_ref_from_prefix(source: &[u8]) -> Result<(&Self, &[u8]), TryCastError<&[u8], Self>>
@@ -2105,7 +2259,7 @@ fn try_ref_from_prefix(source: &[u8]) -> Result<(&Self, &[u8]), TryCastError<&[u
@variant "dynamic_padding"
]
)]
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_ref_from_suffix(source: &[u8]) -> Result<(&[u8], &Self), TryCastError<&[u8], Self>>
@@ -2199,7 +2353,7 @@ fn try_ref_from_suffix(source: &[u8]) -> Result<(&[u8], &Self), TryCastError<&[u
#[doc = codegen_header!("h5", "try_mut_from_bytes")]
///
/// See [`TryFromBytes::try_ref_from_bytes`](#method.try_ref_from_bytes.codegen).
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_mut_from_bytes(bytes: &mut [u8]) -> Result<&mut Self, TryCastError<&mut [u8], Self>>
@@ -2212,7 +2366,7 @@ fn try_mut_from_bytes(bytes: &mut [u8]) -> Result<&mut Self, TryCastError<&mut [
// This call may panic. If that happens, it doesn't cause any soundness
// issues, as we have not generated any invalid state which we need to
// fix before returning.
- match source.try_into_valid() {
+ match source.try_into_safe() {
Ok(source) => Ok(source.as_mut()),
Err(e) => Err(e.map_src(|src| src.as_bytes().as_mut()).into()),
}
@@ -2308,7 +2462,7 @@ fn try_mut_from_bytes(bytes: &mut [u8]) -> Result<&mut Self, TryCastError<&mut [
#[doc = codegen_header!("h5", "try_mut_from_prefix")]
///
/// See [`TryFromBytes::try_ref_from_prefix`](#method.try_ref_from_prefix.codegen).
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_mut_from_prefix(
@@ -2408,7 +2562,7 @@ fn try_mut_from_prefix(
#[doc = codegen_header!("h5", "try_mut_from_suffix")]
///
/// See [`TryFromBytes::try_ref_from_suffix`](#method.try_ref_from_suffix.codegen).
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_mut_from_suffix(
@@ -2513,7 +2667,7 @@ fn try_mut_from_suffix(
@variant "dynamic_padding"
]
)]
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_ref_from_bytes_with_elems(
@@ -2529,7 +2683,7 @@ fn try_ref_from_bytes_with_elems(
// This call may panic. If that happens, it doesn't cause any soundness
// issues, as we have not generated any invalid state which we need to
// fix before returning.
- match source.try_into_valid() {
+ match source.try_into_safe() {
Ok(source) => Ok(source.as_ref()),
Err(e) => {
Err(e.map_src(|src| src.as_bytes::<BecauseImmutable>().as_ref()).into())
@@ -2634,7 +2788,7 @@ fn try_ref_from_bytes_with_elems(
@variant "dynamic_padding"
]
)]
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_ref_from_prefix_with_elems(
@@ -2742,7 +2896,7 @@ fn try_ref_from_prefix_with_elems(
@variant "dynamic_padding"
]
)]
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_ref_from_suffix_with_elems(
@@ -2838,7 +2992,7 @@ fn try_ref_from_suffix_with_elems(
#[doc = codegen_header!("h5", "try_mut_from_bytes_with_elems")]
///
/// See [`TryFromBytes::try_ref_from_bytes_with_elems`](#method.try_ref_from_bytes_with_elems.codegen).
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_mut_from_bytes_with_elems(
@@ -2854,7 +3008,7 @@ fn try_mut_from_bytes_with_elems(
// This call may panic. If that happens, it doesn't cause any soundness
// issues, as we have not generated any invalid state which we need to
// fix before returning.
- match source.try_into_valid() {
+ match source.try_into_safe() {
Ok(source) => Ok(source.as_mut()),
Err(e) => Err(e.map_src(|src| src.as_bytes().as_mut()).into()),
}
@@ -2949,7 +3103,7 @@ fn try_mut_from_bytes_with_elems(
#[doc = codegen_header!("h5", "try_mut_from_prefix_with_elems")]
///
/// See [`TryFromBytes::try_ref_from_prefix_with_elems`](#method.try_ref_from_prefix_with_elems.codegen).
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_mut_from_prefix_with_elems(
@@ -3049,7 +3203,7 @@ fn try_mut_from_prefix_with_elems(
#[doc = codegen_header!("h5", "try_mut_from_suffix_with_elems")]
///
/// See [`TryFromBytes::try_ref_from_suffix_with_elems`](#method.try_ref_from_suffix_with_elems.codegen).
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_mut_from_suffix_with_elems(
@@ -3116,24 +3270,14 @@ fn try_mut_from_suffix_with_elems(
bench = "try_read_from_bytes",
format = "coco_static_size",
)]
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_read_from_bytes(source: &[u8]) -> Result<Self, TryReadError<&[u8], Self>>
where
Self: Sized,
{
- // FIXME(#2981): If `align_of::<Self>() == 1`, validate `source` in-place.
-
- let candidate = match CoreMaybeUninit::<Self>::read_from_bytes(source) {
- Ok(candidate) => candidate,
- Err(e) => {
- return Err(TryReadError::Size(e.with_dst()));
- }
- };
- // SAFETY: `candidate` was copied from from `source: &[u8]`, so all of
- // its bytes are initialized.
- unsafe { try_read_from(source, candidate) }
+ try_read_from(source)
}
/// Attempts to read a `Self` from the prefix of the given `source`.
@@ -3194,24 +3338,24 @@ fn try_read_from_bytes(source: &[u8]) -> Result<Self, TryReadError<&[u8], Self>>
bench = "try_read_from_prefix",
format = "coco_static_size",
)]
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_read_from_prefix(source: &[u8]) -> Result<(Self, &[u8]), TryReadError<&[u8], Self>>
where
Self: Sized,
{
- // FIXME(#2981): If `align_of::<Self>() == 1`, validate `source` in-place.
-
- let (candidate, suffix) = match CoreMaybeUninit::<Self>::read_from_prefix(source) {
- Ok(candidate) => candidate,
- Err(e) => {
- return Err(TryReadError::Size(e.with_dst()));
- }
+ let (prefix, suffix) = match SplitAt::split_at(source, mem::size_of::<Self>()) {
+ Some(split) => split.via_immutable(),
+ None => return Err(SizeError::new(source).into()),
};
- // SAFETY: `candidate` was copied from from `source: &[u8]`, so all of
- // its bytes are initialized.
- unsafe { try_read_from(source, candidate).map(|slf| (slf, suffix)) }
+ match try_read_from(prefix) {
+ Ok(slf) => Ok((slf, suffix)),
+ Err(e) => Err(e.map_src(
+ #[inline(always)]
+ |_| source,
+ )),
+ }
}
/// Attempts to read a `Self` from the suffix of the given `source`.
@@ -3273,44 +3417,72 @@ fn try_read_from_prefix(source: &[u8]) -> Result<(Self, &[u8]), TryReadError<&[u
bench = "try_read_from_suffix",
format = "coco_static_size",
)]
- #[must_use = "has no side effects"]
+ #[must_use = "the conversion result must be checked"]
#[cfg_attr(zerocopy_inline_always, inline(always))]
#[cfg_attr(not(zerocopy_inline_always), inline)]
fn try_read_from_suffix(source: &[u8]) -> Result<(&[u8], Self), TryReadError<&[u8], Self>>
where
Self: Sized,
{
- // FIXME(#2981): If `align_of::<Self>() == 1`, validate `source` in-place.
-
- let (prefix, candidate) = match CoreMaybeUninit::<Self>::read_from_suffix(source) {
- Ok(candidate) => candidate,
- Err(e) => {
- return Err(TryReadError::Size(e.with_dst()));
- }
+ let split_at = match source.len().checked_sub(mem::size_of::<Self>()) {
+ Some(split_at) => split_at,
+ None => return Err(SizeError::new(source).into()),
};
- // SAFETY: `candidate` was copied from from `source: &[u8]`, so all of
- // its bytes are initialized.
- unsafe { try_read_from(source, candidate).map(|slf| (prefix, slf)) }
+ // SAFETY: `checked_sub` returned the difference without overflow [1],
+ // so `split_at = source.len() - size_of::<Self>() <= source.len()`.
+ // This satisfies `SplitAt::split_at_unchecked`'s precondition.
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/std/primitive.usize.html#method.checked_sub:
+ //
+ // Checked integer subtraction. Computes `self - rhs`, returning
+ // `None` if overflow occurred.
+ let (prefix, suffix) =
+ unsafe { SplitAt::split_at_unchecked(source, split_at) }.via_immutable();
+ match try_read_from(suffix) {
+ Ok(slf) => Ok((prefix, slf)),
+ Err(e) => Err(e.map_src(
+ #[inline(always)]
+ |_| source,
+ )),
+ }
}
}
+/// Validates and interprets the given affix of `source`'s bytes as a `&T`.
+///
+/// Returns the destination reference and excess bytes on success. All errors,
+/// including validity errors, contain the original `&S`.
#[inline(always)]
-fn try_ref_from_prefix_suffix<T: TryFromBytes + KnownLayout + Immutable + ?Sized>(
- source: &[u8],
+fn try_ref_from_prefix_suffix<S, T>(
+ source: &S,
cast_type: CastType,
meta: Option<T::PointerMetadata>,
-) -> Result<(&T, &[u8]), TryCastError<&[u8], T>> {
- match Ptr::from_ref(source).try_cast_into::<T, BecauseImmutable>(cast_type, meta) {
- Ok((source, prefix_suffix)) => {
+) -> Result<(&T, &[u8]), TryCastError<&S, T>>
+where
+ S: IntoBytes + Immutable + ?Sized,
+ T: TryFromBytes + KnownLayout + Immutable + ?Sized,
+{
+ match Ptr::from_ref(source.as_bytes()).try_cast_into::<T, BecauseImmutable>(cast_type, meta) {
+ Ok((candidate, prefix_suffix)) => {
// This call may panic. If that happens, it doesn't cause any soundness
// issues, as we have not generated any invalid state which we need to
// fix before returning.
- match source.try_into_valid() {
+ match candidate.try_into_safe() {
Ok(valid) => Ok((valid.as_ref(), prefix_suffix.as_ref())),
- Err(e) => Err(e.map_src(|src| src.as_bytes::<BecauseImmutable>().as_ref()).into()),
+ Err(e) => Err(e
+ .map_src(
+ #[inline(always)]
+ |_| source,
+ )
+ .into()),
}
}
- Err(e) => Err(e.map_src(Ptr::as_ref).into()),
+ Err(e) => Err(e
+ .map_src(
+ #[inline(always)]
+ |_| source,
+ )
+ .into()),
}
}
@@ -3325,7 +3497,7 @@ fn try_mut_from_prefix_suffix<T: IntoBytes + TryFromBytes + KnownLayout + ?Sized
// This call may panic. If that happens, it doesn't cause any soundness
// issues, as we have not generated any invalid state which we need to
// fix before returning.
- match candidate.try_into_valid() {
+ match candidate.try_into_safe() {
Ok(valid) => Ok((valid.as_mut(), prefix_suffix.as_mut())),
Err(e) => Err(e.map_src(|src| src.as_bytes().as_mut()).into()),
}
@@ -3339,49 +3511,112 @@ fn swap<T, U>((t, u): (T, U)) -> (U, T) {
(u, t)
}
-/// # Safety
-///
-/// All bytes of `candidate` must be initialized.
#[inline(always)]
-unsafe fn try_read_from<S, T: TryFromBytes>(
- source: S,
- mut candidate: CoreMaybeUninit<T>,
-) -> Result<T, TryReadError<S, T>> {
+fn try_read_from<S, T>(source: &S) -> Result<T, TryReadError<&S, T>>
+where
+ S: IntoBytes + Immutable + ?Sized,
+ T: TryFromBytes,
+{
+ let bytes = source.as_bytes();
+
+ if bytes.len() != mem::size_of::<T>() {
+ return Err(SizeError::new(source).into());
+ }
+
+ // FIXME(#2981): Avoid the validation copy when validation can safely use
+ // a pointer derived from `source`'s shared borrow.
+
+ // Initialize the candidate in its final location. A typed move of a
+ // `MaybeUninit<T>` may discard initialized bytes at `T`'s padding offsets
+ // [1], but validation requires every byte to be initialized. Do not move
+ // `candidate` between this copy and validation.
+ //
+ // [1] Per https://doc.rust-lang.org/1.93.1/std/mem/union.MaybeUninit.html#validity:
+ //
+ // Moving or copying a value of type `MaybeUninit<T>` (i.e., performing a
+ // "typed copy") will exactly preserve the contents, including the
+ // provenance, of all non-padding bytes of type `T` in the value's
+ // representation.
+ let mut candidate = CoreMaybeUninit::<T>::uninit();
+
+ // SAFETY: The earlier `if bytes.len() != mem::size_of::<T>()` returns on a
+ // size mismatch, so reaching this copy implies that `bytes.len()` equals
+ // `mem::size_of::<T>()`. Copying that many `u8`s cannot overrun `bytes`.
+ // `candidate.as_mut_ptr()` is writable for the same number of bytes because
+ // `MaybeUninit<T>` has `T`'s size. Both pointers are non-null and aligned
+ // for `u8`, including when `T` is zero-sized. The fresh local allocation
+ // cannot overlap `source`, which is an argument. Thus the read, write,
+ // alignment, and non-overlap requirements of [2] hold. Copying `u8`s
+ // initializes every destination byte, including those at `T`'s padding
+ // offsets.
+ //
+ // These writes cannot violate `candidate`'s bit validity because every bit
+ // pattern is valid for `MaybeUninit<T>` [3], even if it is invalid for `T`.
+ //
+ // [2] Per https://doc.rust-lang.org/1.56.0/std/ptr/fn.copy_nonoverlapping.html:
+ //
+ // Copies `count * size_of::<T>()` bytes from `src` to `dst`. The source
+ // and destination must *not* overlap.
+ //
+ // [3] Per https://doc.rust-lang.org/1.56.0/std/mem/union.MaybeUninit.html#layout:
+ //
+ // ... any bit value is valid for a `MaybeUninit<T>` ...
+ unsafe {
+ ptr::copy_nonoverlapping(
+ bytes.as_ptr(),
+ candidate.as_mut_ptr().cast::<u8>(),
+ mem::size_of::<T>(),
+ );
+ }
+
// We use `from_mut` despite not mutating via `c_ptr` so that we don't need
// to add a `T: Immutable` bound.
let c_ptr = Ptr::from_mut(&mut candidate);
+
// SAFETY: `c_ptr` has no uninitialized sub-ranges because it derived from
- // `candidate`, which the caller promises is entirely initialized. Since
- // `candidate` is a `MaybeUninit`, it has no validity requirements, and so
- // no values written to an `Initialized` `c_ptr` can violate its validity.
- // Since `c_ptr` has `Exclusive` aliasing, no mutations may happen except
- // via `c_ptr` so long as it is live, so we don't need to worry about the
- // fact that `c_ptr` may have more restricted validity than `candidate`.
+ // `candidate`, whose bytes were all initialized by the copy above and which
+ // has not been moved since. Since `candidate` is a `MaybeUninit`, it has no
+ // validity requirements, and so no values written to an `Initialized`
+ // `c_ptr` can violate its validity. Since `c_ptr` has `Exclusive` aliasing,
+ // no mutations may happen except via `c_ptr` so long as it is live, so we
+ // don't need to worry about the fact that `c_ptr` may have more restricted
+ // validity than `candidate`.
let c_ptr = unsafe { c_ptr.assume_validity::<invariant::Initialized>() };
- let mut c_ptr = c_ptr.cast::<_, crate::pointer::cast::CastSized, _>();
- // Since we don't have `T: KnownLayout`, we hack around that by using
- // `Wrapping<T>`, which implements `KnownLayout` even if `T` doesn't.
+ let c_ptr = c_ptr.cast::<_, crate::pointer::cast::CastSized, _>();
+
+ // SAFETY: `c_ptr` originated from a reference to `candidate`, so its
+ // address is aligned for `MaybeUninit<T>`, which has `T`'s alignment [1].
+ // `CastSized` preserves that address. `ReadOnly<T>` is `repr(transparent)`
+ // with a single `T` field, so it also has `T`'s alignment [2], including
+ // when `T` is zero-sized.
//
+ // [1] Per https://doc.rust-lang.org/1.56.0/std/mem/union.MaybeUninit.html#layout:
+ //
+ // `MaybeUninit<T>` is guaranteed to have the same size, alignment, and
+ // ABI as `T`:
+ //
+ // [2] Per https://doc.rust-lang.org/1.93.1/reference/type-layout.html#the-transparent-representation:
+ //
+ // ... same layout and ABI as the only non-size 0 non-alignment 1 field,
+ // if present, or unit otherwise.
+ let mut c_ptr = unsafe { c_ptr.assume_alignment::<invariant::Aligned>() };
+
// This call may panic. If that happens, it doesn't cause any soundness
// issues, as we have not generated any invalid state which we need to fix
// before returning.
- if !Wrapping::<T>::is_bit_valid(c_ptr.reborrow_shared().forget_aligned()) {
+ if !T::is_safe(c_ptr.reborrow_shared()) {
return Err(ValidityError::new(source).into());
}
- fn _assert_same_size_and_validity<T>()
- where
- Wrapping<T>: pointer::TransmuteFrom<T, invariant::Valid, invariant::Valid>,
- T: pointer::TransmuteFrom<Wrapping<T>, invariant::Valid, invariant::Valid>,
- {
- }
-
- _assert_same_size_and_validity::<T>();
-
- // SAFETY: We just validated that `candidate` contains a valid
- // `Wrapping<T>`, which has the same size and bit validity as `T`, as
- // guaranteed by the preceding type assertion.
+ // SAFETY: `T::is_safe` returned true for `candidate`'s initialized bytes,
+ // so it contains a valid `T`, as required by [1]. Validation used a shared
+ // `ReadOnly<T>` pointer, and the bytes have not been modified since.
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/std/mem/union.MaybeUninit.html#method.assume_init:
+ //
+ // It is up to the caller to guarantee that the `MaybeUninit<T>` really is
+ // in an initialized state.
Ok(unsafe { candidate.assume_init() })
}
@@ -5464,22 +5699,32 @@ fn read_from(source: &[u8]) -> Option<Self>
}
}
-/// Interprets the given affix of the given bytes as a `&Self`.
+/// Interprets the given affix of `source`'s bytes as a `&T`.
///
-/// This method computes the largest possible size of `Self` that can fit in the
-/// prefix or suffix bytes of `source`, then attempts to return both a reference
-/// to those bytes interpreted as a `Self`, and a reference to the excess bytes.
-/// If there are insufficient bytes, or if that affix of `source` is not
-/// appropriately aligned, this returns `Err`.
+/// This method uses `meta` if provided; otherwise, it computes the largest
+/// possible size of `T` that can fit in the prefix or suffix bytes of `source`.
+/// It returns both a reference to those bytes interpreted as a `T`, and a
+/// reference to the excess bytes. If there are insufficient bytes, or if that
+/// affix of `source` is not appropriately aligned, this returns `Err` containing
+/// the original `&S`.
#[inline(always)]
-fn ref_from_prefix_suffix<T: FromBytes + KnownLayout + Immutable + ?Sized>(
- source: &[u8],
+fn ref_from_prefix_suffix<S, T>(
+ source: &S,
meta: Option<T::PointerMetadata>,
cast_type: CastType,
-) -> Result<(&T, &[u8]), CastError<&[u8], T>> {
- let (slf, prefix_suffix) = Ptr::from_ref(source)
+) -> Result<(&T, &[u8]), CastError<&S, T>>
+where
+ S: IntoBytes + Immutable + ?Sized,
+ T: FromBytes + KnownLayout + Immutable + ?Sized,
+{
+ let (slf, prefix_suffix) = Ptr::from_ref(source.as_bytes())
.try_cast_into::<_, BecauseImmutable>(cast_type, meta)
- .map_err(|err| err.map_src(|s| s.as_ref()))?;
+ .map_err(|err| {
+ err.map_src(
+ #[inline(always)]
+ |_| source,
+ )
+ })?;
Ok((slf.recall_validity().as_ref(), prefix_suffix.as_ref()))
}
@@ -5602,6 +5847,10 @@ fn mut_from_prefix_suffix<T: FromBytes + IntoBytes + KnownLayout + ?Sized>(
/// if its field is [`IntoBytes`]; else,
/// - if the type has no generic parameters, it is [`IntoBytes`] if the type
/// is sized and has no padding bytes; else,
+/// - if the type is `repr(C)` without an `align(N)` modifier for `N > 1`
+/// (it may have `align(1)` or `packed(N)`), and every field is `T`,
+/// `[T; N]`, or a final `[T]` for the same type parameter `T`, it is
+/// [`IntoBytes`]; else,
/// - if the type is `repr(C)`, its fields must be [`Unaligned`].
/// - If the type is an enum:
/// - It must have a defined representation (`repr`s `C`, `u8`, `u16`, `u32`,
@@ -6493,6 +6742,39 @@ fn from_mut_slice(slc: &mut [u8]) -> &mut Unsized {
}
}
+ // These types exercise a nested slice DST whose inner trailing-padding
+ // operation cannot be flattened into the alignment of the packed outer
+ // type. For `NestedPackedDstOuter<[u8]>`, the trailing slice begins at
+ // offset 7, but the size for `elems` elements is
+ // `2 + round_up(5 + elems, 4)`.
+ //
+ // The Reference rejects directly placing an alignment-4 field in a
+ // `repr(packed(2))` type. Rust currently accepts this construction through
+ // generic indirection. So long as Rust accepts it, this test deliberately
+ // treats its layout as following the otherwise-applicable `repr(C)` and
+ // `repr(packed)` rules.
+ #[derive(FromBytes, Immutable, KnownLayout)]
+ #[repr(C, align(4))]
+ struct NestedPackedDstWord([u8; 4]);
+
+ #[derive(FromBytes, Immutable, KnownLayout)]
+ #[repr(C)]
+ struct NestedPackedDstInner<T: ?Sized> {
+ a: NestedPackedDstWord,
+ b: u8,
+ tail: T,
+ }
+
+ #[derive(FromBytes, Immutable, KnownLayout)]
+ #[repr(C, packed(2))]
+ struct NestedPackedDstOuter<T: ?Sized> {
+ z: u8,
+ inner: ManuallyDrop<NestedPackedDstInner<T>>,
+ }
+
+ #[repr(C, align(2))]
+ struct Align2<T>(T);
+
#[test]
fn test_known_layout() {
// Test that `$ty` and `ManuallyDrop<$ty>` have the expected layout.
@@ -6507,17 +6789,25 @@ macro_rules! test {
};
}
- let layout =
- |offset, align, trailing_slice_elem_size, statically_shallow_unpadded| DstLayout {
- align: NonZeroUsize::new(align).unwrap(),
+ let layout = |offset, align, trailing_slice_elem_size, statically_shallow_unpadded| {
+ let align = NonZeroUsize::new(align).unwrap();
+ let size_phase = offset % align;
+ DstLayout {
+ align,
size_info: match trailing_slice_elem_size {
None => SizeInfo::Sized { size: offset },
- Some(elem_size) => {
- SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size })
- }
+ Some(elem_size) => SizeInfo::SliceDst(TrailingSliceLayout {
+ offset,
+ elem_size,
+ size_base: offset - size_phase,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ align, size_phase,
+ ),
+ }),
},
statically_shallow_unpadded,
- };
+ }
+ };
test!((), layout(0, 1, None, false));
test!(u8, layout(1, 1, None, false));
@@ -6533,6 +6823,153 @@ macro_rules! test {
test!(str, layout(0, 1, Some(1), true));
}
+ #[test]
+ fn test_known_layout_pointer_to_metadata() {
+ fn test<T>(data: *mut T, elems: usize) {
+ let ptr = ptr::slice_from_raw_parts_mut(data, elems);
+ assert_eq!(<[T] as KnownLayout>::pointer_to_metadata(ptr), elems);
+ }
+
+ for elems in [0, 1, usize::MAX] {
+ test(ptr::null_mut::<u8>(), elems);
+ }
+
+ test(NonNull::<u8>::dangling().as_ptr(), 0);
+
+ let mut bytes = [1, 2, 3];
+ test(bytes.as_mut_ptr(), bytes.len());
+ assert_eq!(bytes, [1, 2, 3]);
+
+ test(NonNull::<()>::dangling().as_ptr(), usize::MAX);
+
+ // Retaining and inspecting a raw pointer after freeing its allocation
+ // is safe so long as the pointer is not dereferenced.
+ let deallocated = {
+ let mut byte = Box::new(0u8);
+ let ptr: *mut u8 = &mut *byte;
+ drop(byte);
+ ptr
+ };
+ test(deallocated, 3);
+
+ // Metadata extraction requires neither element storage nor alignment
+ // for `T`; it does not access the pointer's referent.
+ test(NonNull::<u8>::dangling().as_ptr().cast::<u64>(), usize::MAX);
+ }
+
+ #[cfg(feature = "derive")]
+ #[test]
+ fn test_known_layout_pointer_to_metadata_derive() {
+ #[derive(KnownLayout)]
+ #[repr(C)]
+ struct Dst {
+ prefix: u8,
+ trailing: [u8],
+ }
+
+ for elems in [0, 1, usize::MAX] {
+ let ptr = ptr::slice_from_raw_parts_mut(ptr::null_mut::<u8>(), elems);
+ #[allow(clippy::as_conversions)]
+ let ptr = ptr as *mut Dst;
+ assert_eq!(Dst::pointer_to_metadata(ptr), elems);
+ }
+ }
+
+ #[test]
+ fn test_nested_packed_dst_layout() {
+ fn assert_native_size<const N: usize>(expected: usize) {
+ let value = NestedPackedDstOuter {
+ z: 0,
+ inner: ManuallyDrop::new(NestedPackedDstInner {
+ a: NestedPackedDstWord([0; 4]),
+ b: 0,
+ tail: [0; N],
+ }),
+ };
+ let value: &NestedPackedDstOuter<[u8]> = &value;
+ assert_eq!(mem::size_of_val(value), expected);
+ assert_eq!(NestedPackedDstOuter::<[u8]>::size_for_metadata(N), Some(expected));
+ }
+
+ assert_native_size::<0>(10);
+ assert_native_size::<1>(10);
+ assert_native_size::<2>(10);
+ assert_native_size::<3>(10);
+ assert_native_size::<4>(14);
+ assert_native_size::<7>(14);
+ assert_native_size::<8>(18);
+
+ let layout = NestedPackedDstOuter::<[u8]>::LAYOUT;
+ assert_eq!(layout.align.get(), 2);
+ assert!(layout.requires_dynamic_padding());
+ assert_eq!(
+ layout.size_info,
+ SizeInfo::SliceDst(TrailingSliceLayout {
+ offset: 7,
+ elem_size: 1,
+ size_base: 6,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(
+ NonZeroUsize::new(4).unwrap(),
+ 1
+ ),
+ })
+ );
+
+ // Eight bytes used to be incorrectly accepted for a zero-element
+ // value. The actual Rust layout requires ten bytes.
+ let short = Align2([0u8; 8]);
+ assert!(NestedPackedDstOuter::<[u8]>::ref_from_bytes_with_elems(&short.0, 0).is_err());
+ assert!(pointer::Ptr::from_ref(&short.0[..])
+ .try_cast_into::<NestedPackedDstOuter<[u8]>, pointer::BecauseImmutable>(
+ CastType::Prefix,
+ Some(0),
+ )
+ .is_err());
+
+ let full = Align2([0, 0, 0, 0, 0, 0, 0, 0, b'S', b'!']);
+ let value = NestedPackedDstOuter::<[u8]>::ref_from_bytes_with_elems(&full.0, 0).unwrap();
+ assert_eq!(mem::size_of_val(value), 10);
+
+ // When metadata is inferred, lengths 0 through 3 all produce a
+ // ten-byte object. Parsing chooses the largest trailing-slice length
+ // which fits exactly.
+ let inferred = NestedPackedDstOuter::<[u8]>::ref_from_bytes(&full.0).unwrap();
+ assert_eq!(pointer::Ptr::from_ref(inferred).len(), 3);
+
+ // Trailing padding includes the inner type's padding, even when the
+ // outer type's alignment is reduced by packing.
+ let trailing_layout = trailing_slice_layout::<NestedPackedDstOuter<[u8]>>();
+ for (elems, expected) in [(0, 3), (1, 2), (2, 1), (3, 0), (4, 3)] {
+ assert_eq!(trailing_layout.padding_for_elems(elems), expected);
+ }
+ }
+
+ // Before Rust 1.76, rust-lang/rust#118537 caused `addr_of_mut!` to
+ // miscompute the offset of an unsized field in a packed struct. The fix is
+ // rust-lang/rust#118540. Keep the layout and parsing regression above on
+ // all supported compilers, and gate only the projection regression which
+ // necessarily exercises that independent compiler bug.
+ #[rustversion::since(1.76.0)]
+ #[test]
+ fn test_nested_packed_dst_projection() {
+ let full = Align2([0, 0, 0, 0, 0, 0, 0, 0, b'S', b'!']);
+
+ // The projected inner value occupies the final eight bytes of the full
+ // ten-byte input, including the two sentinel bytes.
+ let (value, rest) = pointer::Ptr::from_ref(&full.0[..])
+ .try_cast_into::<NestedPackedDstOuter<[u8]>, pointer::BecauseImmutable>(
+ CastType::Prefix,
+ Some(0),
+ )
+ .unwrap();
+ assert!(rest.as_ref().is_empty());
+ let inner = value.project::<project_clients::TryFromBytesDerive, _, { STRUCT_VARIANT_ID }, { ident_id!(inner) }>().unwrap();
+ assert_eq!(
+ inner.as_bytes::<pointer::BecauseImmutable>().as_ref(),
+ &[0, 0, 0, 0, 0, 0, b'S', b'!']
+ );
+ }
+
#[cfg(feature = "derive")]
#[test]
fn test_known_layout_derive() {
@@ -6588,10 +7025,19 @@ fn _assert_kl<T: ?Sized + KnownLayout>(_: &T) {}
statically_shallow_unpadded: false,
};
- let unsized_layout = |align, elem_size, offset, statically_shallow_unpadded| DstLayout {
- align: NonZeroUsize::new(align).unwrap(),
- size_info: SizeInfo::SliceDst(TrailingSliceLayout { offset, elem_size }),
- statically_shallow_unpadded,
+ let unsized_layout = |align, elem_size, offset, statically_shallow_unpadded| {
+ let align = NonZeroUsize::new(align).unwrap();
+ let size_phase = offset % align;
+ DstLayout {
+ align,
+ size_info: SizeInfo::SliceDst(TrailingSliceLayout {
+ offset,
+ elem_size,
+ size_base: offset - size_phase,
+ size_rounding_align_and_phase: RoundingAlignAndPhase::new(align, size_phase),
+ }),
+ statically_shallow_unpadded,
+ }
};
// | `repr(C)`? | generic? | `KnownLayout`? | `Sized`? | Type Name |
@@ -7152,6 +7598,112 @@ fn test_try_from_bytes_try_read_from() {
);
}
+ #[test]
+ fn test_try_read_from_typed_source() {
+ #[derive(IntoBytes, Immutable)]
+ #[repr(transparent)]
+ struct Source<T: ?Sized>(T);
+
+ fn check<S: IntoBytes + Immutable + ?Sized>(source: &S, valid: bool) {
+ match try_read_from::<_, [bool; 4]>(source) {
+ Ok(value) => {
+ assert!(valid);
+ assert_eq!(value, [false, true, false, true]);
+ }
+ Err(error) => {
+ assert!(!valid);
+ assert!(matches!(error, TryReadError::Validity(_)));
+ assert!(ptr::eq(error.into_src(), source));
+ }
+ }
+
+ let error = try_read_from::<_, [bool; 5]>(source).unwrap_err();
+ assert!(matches!(error, TryReadError::Size(_)));
+ assert!(ptr::eq(error.into_src(), source));
+ }
+
+ for (value, valid) in [(u16::from_ne_bytes([0, 1]), true), (0x0202, false)] {
+ let source = Source([value; 2]);
+ check(&source, valid);
+ let source: &Source<[u16]> = &source;
+ check(source, valid);
+ }
+
+ let source = Source([(); 2]);
+ assert!(try_read_from::<_, ()>(&source).is_ok());
+ let source: &Source<[()]> = &source;
+ assert!(try_read_from::<_, ()>(source).is_ok());
+ }
+
+ #[test]
+ fn test_try_read_initialized_padding() {
+ #[derive(KnownLayout, Immutable, Debug, PartialEq, Eq)]
+ #[repr(C, align(8))]
+ struct Padded(u8);
+
+ // SAFETY: `Padded` has only a `u8` field and padding, so all
+ // initialized byte sequences are valid. Per
+ // https://doc.rust-lang.org/1.93.1/reference/behavior-considered-undefined.html#invalid-values:
+ //
+ // An integer (`i*`/`u*`), floating point value (`f*`), or raw pointer
+ // must be initialized, i.e., must not be obtained from uninitialized
+ // memory.
+ //
+ // A `struct`, tuple, and array requires all fields/elements to be
+ // valid at their respective type.
+ const _: () = unsafe {
+ unsafe_impl!(=> TryFromBytes for Padded; |candidate| {
+ // Observe every byte while validation is running, including
+ // padding that a typed move of `MaybeUninit<Padded>` could
+ // discard. The returned `Padded` need not retain its padding.
+ assert_eq!(candidate.as_bytes::<BecauseImmutable>().as_ref(), &[0xA5; 8]);
+ true
+ })
+ };
+
+ let source = [0xA5; 8];
+ assert_eq!(Padded::try_read_from_bytes(&source), Ok(Padded(0xA5)));
+
+ let prefix_source = [0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0];
+ let (value, suffix) = Padded::try_read_from_prefix(&prefix_source).unwrap();
+ assert_eq!(value, Padded(0xA5));
+ assert!(ptr::eq(suffix, &prefix_source[8..]));
+
+ let suffix_source = [0, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5, 0xA5];
+ let (prefix, value) = Padded::try_read_from_suffix(&suffix_source).unwrap();
+ assert_eq!(value, Padded(0xA5));
+ assert!(ptr::eq(prefix, &suffix_source[..1]));
+ }
+
+ #[test]
+ fn test_try_read_error_sources_and_zero_sized() {
+ let invalid = [2, 3];
+ for (error, source) in [
+ (bool::try_read_from_bytes(&invalid[..1]).unwrap_err(), &invalid[..1]),
+ (bool::try_read_from_prefix(&invalid).unwrap_err(), &invalid[..]),
+ (bool::try_read_from_suffix(&invalid).unwrap_err(), &invalid[..]),
+ ] {
+ assert!(matches!(error, TryReadError::Validity(_)));
+ assert!(ptr::eq(error.into_src(), source));
+ }
+ for source in [&[][..], &invalid[..]] {
+ let error = bool::try_read_from_bytes(source).unwrap_err();
+ assert!(matches!(error, TryReadError::Size(_)));
+ assert!(ptr::eq(error.into_src(), source));
+ }
+ for error in [
+ bool::try_read_from_prefix(&invalid[..0]).unwrap_err(),
+ bool::try_read_from_suffix(&invalid[..0]).unwrap_err(),
+ ] {
+ assert!(matches!(error, TryReadError::Size(_)));
+ assert!(ptr::eq(error.into_src(), &invalid[..0]));
+ }
+
+ assert_eq!(<()>::try_read_from_bytes(&[]), Ok(()));
+ assert_eq!(<()>::try_read_from_prefix(&invalid), Ok(((), &invalid[..])));
+ assert_eq!(<()>::try_read_from_suffix(&invalid), Ok((&invalid[..], ())));
+ }
+
#[test]
fn test_ref_from_mut_from_bytes() {
// Test `FromBytes::{ref_from_bytes, mut_from_bytes}{,_prefix,Suffix}`
@@ -7502,6 +8054,16 @@ fn test_new_box_zeroed_with_elems() {
assert_eq!(&*s, &[0, 3, 0]);
}
+ #[test]
+ fn test_new_box_zeroed_with_elems_nested_packed_dst() {
+ let mut value = NestedPackedDstOuter::<[u8]>::new_box_zeroed_with_elems(0).unwrap();
+ assert_eq!(mem::size_of_val(&*value), 10);
+
+ // Exercise the whole-object write that exposed the former
+ // under-allocation under Miri and AddressSanitizer.
+ value.zero();
+ }
+
#[test]
fn test_new_box_zeroed_with_elems_empty() {
let s: Box<[u64]> = <[u64]>::new_box_zeroed_with_elems(0).unwrap();
@@ -7571,6 +8133,133 @@ fn test_deprecated_from_bytes() {
assert_eq!(rest.len(), 4);
}
+ #[test]
+ fn test_ref_from_prefix_suffix_typed_source() {
+ #[derive(IntoBytes, Immutable)]
+ #[repr(transparent)]
+ struct Source<T: ?Sized>(T);
+
+ fn check<S: IntoBytes + Immutable + ?Sized>(source: &S) {
+ let bytes = source.as_bytes();
+ assert_eq!(bytes.len(), 4);
+
+ for cast_type in [CastType::Prefix, CastType::Suffix] {
+ let (expected, expected_rest) = match cast_type {
+ CastType::Prefix => (&bytes[..3], &bytes[3..]),
+ CastType::Suffix => (&bytes[1..], &bytes[..1]),
+ };
+ for meta in [None, Some(1)] {
+ let (value, rest) =
+ ref_from_prefix_suffix::<_, [[u8; 3]]>(source, meta, cast_type).unwrap();
+ assert_eq!(value, &[expected]);
+ assert!(ptr::eq(value.as_bytes(), expected));
+ assert!(ptr::eq(rest, expected_rest));
+ }
+
+ let err =
+ ref_from_prefix_suffix::<_, [u8; 5]>(source, None, cast_type).unwrap_err();
+ assert!(matches!(err, CastError::Size(_)));
+ assert!(ptr::eq(err.into_src(), source));
+
+ let err =
+ ref_from_prefix_suffix::<_, [u8]>(source, Some(5), cast_type).unwrap_err();
+ assert!(matches!(err, CastError::Size(_)));
+ assert!(ptr::eq(err.into_src(), source));
+ }
+ }
+
+ check(&Source([0x0102u16, 0x0304]));
+ let source = Source([false, true, false, true]);
+ let source: &Source<[bool]> = &source;
+ check(source);
+ }
+
+ #[test]
+ fn test_ref_from_prefix_suffix_typed_source_alignment_error() {
+ let storage = Align::<[bool; 9], AU64>::new([false; 9]);
+ for cast_type in [CastType::Prefix, CastType::Suffix] {
+ let source = match cast_type {
+ CastType::Prefix => &storage.t[1..],
+ CastType::Suffix => &storage.t[..],
+ };
+ let err = ref_from_prefix_suffix::<_, AU64>(source, None, cast_type).unwrap_err();
+ assert!(matches!(err, CastError::Alignment(_)));
+ let recovered: &[bool] = err.into_src();
+ assert!(ptr::eq(recovered, source));
+
+ let err = try_ref_from_prefix_suffix::<_, AU64>(source, cast_type, None).unwrap_err();
+ assert!(matches!(err, TryCastError::Alignment(_)));
+ let recovered: &[bool] = err.into_src();
+ assert!(ptr::eq(recovered, source));
+ }
+ }
+
+ #[test]
+ fn test_try_ref_from_prefix_suffix_typed_source() {
+ #[derive(IntoBytes, Immutable)]
+ #[repr(transparent)]
+ struct Source<T: ?Sized>(T);
+
+ fn check<S: IntoBytes + Immutable + ?Sized>(source: &S) {
+ let bytes = source.as_bytes();
+ assert_eq!(bytes, [0, 1, 0, 1]);
+
+ for cast_type in [CastType::Prefix, CastType::Suffix] {
+ let (expected, expected_rest) = match cast_type {
+ CastType::Prefix => (&bytes[..3], &bytes[3..]),
+ CastType::Suffix => (&bytes[1..], &bytes[..1]),
+ };
+ for meta in [None, Some(1)] {
+ let (value, rest) =
+ try_ref_from_prefix_suffix::<_, [[bool; 3]]>(source, cast_type, meta)
+ .unwrap();
+ assert_eq!(value.as_bytes(), expected);
+ assert!(ptr::eq(value.as_bytes(), expected));
+ assert!(ptr::eq(rest, expected_rest));
+ }
+
+ let err = try_ref_from_prefix_suffix::<_, [bool; 5]>(source, cast_type, None)
+ .unwrap_err();
+ assert!(matches!(err, TryCastError::Size(_)));
+ assert!(ptr::eq(err.into_src(), source));
+
+ let err = try_ref_from_prefix_suffix::<_, [bool]>(source, cast_type, Some(5))
+ .unwrap_err();
+ assert!(matches!(err, TryCastError::Size(_)));
+ assert!(ptr::eq(err.into_src(), source));
+ }
+ }
+
+ check(&Source([u16::from_ne_bytes([0, 1]); 2]));
+ let source = Source([false, true, false, true]);
+ let source: &Source<[bool]> = &source;
+ check(source);
+ }
+
+ #[test]
+ fn test_try_ref_from_prefix_suffix_typed_source_validity_error() {
+ fn check<S: IntoBytes + Immutable + ?Sized>(source: &S) {
+ for cast_type in [CastType::Prefix, CastType::Suffix] {
+ let err =
+ try_ref_from_prefix_suffix::<_, bool>(source, cast_type, None).unwrap_err();
+ assert!(matches!(err, TryCastError::Validity(_)));
+ assert!(ptr::eq(err.into_src(), source));
+
+ for meta in [None, Some(1)] {
+ let err = try_ref_from_prefix_suffix::<_, [bool]>(source, cast_type, meta)
+ .unwrap_err();
+ assert!(matches!(err, TryCastError::Validity(_)));
+ assert!(ptr::eq(err.into_src(), source));
+ }
+ }
+ }
+
+ let source = [u16::from_ne_bytes([2, 2]); 2];
+ check(&source);
+ check(&source[..]);
+ check(source.as_bytes());
+ }
+
#[test]
fn test_try_ref_from_prefix_suffix() {
use crate::util::testutil::Align;
diff --git a/rust/zerocopy/src/macros.rs b/rust/zerocopy/src/macros.rs
index ec67c03a44fc..73f6dd0ca0ae 100644
--- a/rust/zerocopy/src/macros.rs
+++ b/rust/zerocopy/src/macros.rs
@@ -1033,7 +1033,7 @@ unsafe impl $(<$($tyvar),*>)? $crate::TryFromBytes for $name$(<$($tyvar),*>)?
)?
{
#[inline(always)]
- fn is_bit_valid<A>(_: $crate::Maybe<'_, Self, A>) -> bool
+ fn is_safe<A>(_: $crate::Maybe<'_, Self, A>) -> bool
where
A: $crate::invariant::Alignment,
{
@@ -1179,7 +1179,7 @@ unsafe impl $crate::TryFromBytes for $name
)*
{
#[inline(always)]
- fn is_bit_valid<A>(_: $crate::Maybe<'_, Self, A>) -> bool
+ fn is_safe<A>(_: $crate::Maybe<'_, Self, A>) -> bool
where
A: $crate::invariant::Alignment,
{
@@ -1411,6 +1411,38 @@ const fn transmute_ref<T, U>(t: &T) -> &U
let x: &SliceDst<U16, u8> = transmute_ref!(slice_dst_big);
assert_eq!(x, slice_dst_small);
+ // A packed outer DST can preserve the trailing field's stronger
+ // rounding alignment. These two types consequently have different
+ // normalized layout formulas even though both have size `2 + 4 * n`
+ // for every trailing slice length. Ensure the semantic size-sequence
+ // comparison accepts this representation-preserving transmutation.
+ #[derive(FromBytes, Immutable, IntoBytes, KnownLayout)]
+ #[repr(C, align(2))]
+ struct TransmuteAlign2([u8; 4]);
+
+ #[derive(FromBytes, Immutable, IntoBytes, KnownLayout)]
+ #[repr(C, packed(2))]
+ struct TransmutePacked4 {
+ head: u16,
+ tail: [u32],
+ }
+
+ #[derive(FromBytes, Immutable, IntoBytes, KnownLayout)]
+ #[repr(C, packed(2))]
+ struct TransmutePacked2 {
+ head: u16,
+ tail: [TransmuteAlign2],
+ }
+
+ #[repr(C, align(2))]
+ struct TransmuteBytes([u8; 10]);
+
+ let bytes = TransmuteBytes([0; 10]);
+ let src = TransmutePacked4::ref_from_bytes(&bytes.0).unwrap();
+ let dst: &TransmutePacked2 = transmute_ref!(src);
+ assert_eq!(mem::size_of_val(src), 10);
+ assert_eq!(mem::size_of_val(dst), 10);
+
// Test that it's legal to transmute a reference while shrinking the
// lifetime (note that `X` has the lifetime `'static`).
let x: &[u8; 8] = transmute_ref!(X);
@@ -1471,6 +1503,19 @@ fn drop(&mut self) {
// its `PanicOnDrop` temporary, which would cause a panic.
assert_eq!(y.as_ref().map_err(|p| &p.src.0), Err::<&bool, _>(&2u8));
mem::forget(y);
+
+ #[derive(Debug, TryFromBytes)]
+ #[repr(C, align(8))]
+ struct PaddedBool(bool);
+
+ // A failed transmutation must return every source byte, including
+ // bytes which would be padding in the destination type.
+ let src = [2u8, 1, 2, 3, 4, 5, 6, 7];
+ let x: Result<PaddedBool, _> = try_transmute!(src);
+ assert_eq!(x.unwrap_err().into_src(), src);
+
+ let x: Result<PaddedBool, _> = try_transmute!([0u8; 8]);
+ assert!(!x.unwrap().0);
}
#[test]
@@ -1687,9 +1732,9 @@ fn test_macros_evaluate_args_once() {
#[test]
fn test_include_value() {
const AS_U32: u32 = include_value!("../testdata/include_value/data");
- assert_eq!(AS_U32, u32::from_ne_bytes([b'a', b'b', b'c', b'd']));
+ assert_eq!(AS_U32, u32::from_ne_bytes(*b"abcd"));
const AS_I32: i32 = include_value!("../testdata/include_value/data");
- assert_eq!(AS_I32, i32::from_ne_bytes([b'a', b'b', b'c', b'd']));
+ assert_eq!(AS_I32, i32::from_ne_bytes(*b"abcd"));
}
#[test]
diff --git a/rust/zerocopy/src/pointer/inner.rs b/rust/zerocopy/src/pointer/inner.rs
index 949b60a3f83e..8beb6267b217 100644
--- a/rust/zerocopy/src/pointer/inner.rs
+++ b/rust/zerocopy/src/pointer/inner.rs
@@ -289,10 +289,10 @@ impl<'a, T> PtrInner<'a, T>
///
/// Given `let (left, right) = ptr.split_at(l_len)`, it is guaranteed that
/// `left` and `right` are contiguous and non-overlapping if
- /// `l_len.padding_needed_for() == 0`. This is true for all `[T]`.
+ /// the left part has no trailing padding. This is true for all `[T]`.
///
- /// If `l_len.padding_needed_for() != 0`, then the left pointer will overlap
- /// the right pointer to satisfy `T`'s padding requirements.
+ /// Any trailing padding in the left part begins at the right part's
+ /// starting address and may overlap its referent.
#[inline]
#[must_use]
pub unsafe fn split_at_unchecked(
@@ -313,23 +313,19 @@ pub unsafe fn split_at_unchecked(
// Trivially, `slf.meta() <= slf.meta()`.
let right = unsafe { right.slice_unchecked(l_len..self.meta().get()) };
- // SAFETY: If `l_len.padding_needed_for() == 0`, then `left` and `right`
- // are non-overlapping. Proof: `left` is constructed `slf` with `l_len`
- // as its (exclusive) upper bound. If `l_len.padding_needed_for() == 0`,
- // then `left` requires no trailing padding following its final element.
- // Since `right` is constructed from `slf`'s trailing slice with `l_len`
- // as its (inclusive) lower bound, no byte is referred to by both
- // pointers.
+ // SAFETY: If the left part has no trailing padding, then `left` and
+ // `right` are non-overlapping. Proof: `left` has `l_len` trailing
+ // elements, and `right` starts at element `l_len` of the original
+ // trailing slice. Without padding after `left`'s final element, no
+ // byte is referred to by both pointers.
//
- // Conversely, `l_len.padding_needed_for() == N`, where `N
- // > 0`, `left` requires `N` bytes of trailing padding following its
- // final element. Since `right` is constructed from the trailing slice
- // of `slf` with `l_len` as its (inclusive) lower bound, the first `N`
- // bytes of `right` are aliased by `left`.
+ // Any trailing padding in `left` begins at `right`'s starting
+ // address. If `right` is nonzero sized, the referents overlap.
(left, right)
}
- /// Produces the trailing slice of `self`.
+ /// Produces the trailing slice of `self` using `T::Elem` as its element
+ /// representation.
#[inline]
#[must_use]
pub fn trailing_slice(self) -> PtrInner<'a, [T::Elem]>
@@ -368,9 +364,10 @@ pub fn trailing_slice(self) -> PtrInner<'a, [T::Elem]>
// 1. If `ptr`'s referent is not zero sized, then `ptr` has valid
// provenance for `A` because `raw` is derived from the same
// allocated object as `self` via provenance-preserving operations.
- // 2. If `ptr`'s referent is not zero sized, then `ptr` addresses a byte
- // range which is entirely contained in `A`, by previous safety proof
- // on `bytes`.
+ // 2. `T: SplitAt` guarantees that `T::Elem` has the size of the actual
+ // trailing element. The new slice has the same length and start as
+ // the original trailing slice, so it covers the same byte range.
+ // If nonzero sized, that range is entirely contained in `A`.
// 3. `ptr` addresses a byte range whose length fits in an `isize`, by
// consequence of #2.
// 4. `ptr` addresses a byte range which does not wrap around the
diff --git a/rust/zerocopy/src/pointer/invariant.rs b/rust/zerocopy/src/pointer/invariant.rs
index 7ff0d43dad5e..8dfaf82e134f 100644
--- a/rust/zerocopy/src/pointer/invariant.rs
+++ b/rust/zerocopy/src/pointer/invariant.rs
@@ -47,7 +47,7 @@ pub trait Alignment: Sealed {
fn read<T, I, R>(ptr: crate::Ptr<'_, T, I>) -> T
where
T: Copy + Read<I::Aliasing, R>,
- I: Invariants<Alignment = Self, Validity = Valid>,
+ I: Invariants<Alignment = Self, Validity = Safe>,
I::Aliasing: Reference;
}
@@ -64,35 +64,39 @@ fn read<T, I, R>(ptr: crate::Ptr<'_, T, I>) -> T
/// all `T: ?Sized`. Any `V: Validity` must guarantee that this set is only a
/// function of the *bit validity* of the referent type, `T`, and not of any
/// other property of `T`. As a consequence, given `V: Validity`, `T`, and `U`
-/// where `T` and `U` have the same bit validity, `S(V, T) = S(V, U)`.
+/// where `T` and `U` have the same bit validity, `S(T, V) = S(U, V)`.
///
/// It is guaranteed that the referent of any `ptr: Ptr<T, V>` is a member of
/// `S(T, V)`. Unsafe code must ensure that this guarantee will be upheld for
/// any existing `Ptr`s or any `Ptr`s that that code creates.
///
-/// An important implication of this guarantee is that it restricts what
-/// transmutes are sound, where "transmute" is used in this context to refer to
-/// changing the referent type or validity invariant of a `Ptr`, as either
-/// change may change the set of bit values permitted to appear in the referent.
-/// In particular, the following are necessary (but not sufficient) conditions
-/// in order for a transmute from `src: Ptr<T, V>` to `dst: Ptr<U, W>` to be
-/// sound:
-/// - If `S(T, V) = S(U, W)`, then no restrictions apply; otherwise,
+/// An important implication of this guarantee is that it restricts which exact
+/// reinterpretations are sound. Here, an exact reinterpretation changes the
+/// referent type or validity invariant of a `Ptr` while preserving the exact
+/// byte range. In particular, given `src: Ptr<T, V>` and `dst: Ptr<U, W>` which
+/// refer to the same byte range, the following are necessary (but not
+/// sufficient) conditions:
+/// - If `S(T, V) = S(U, W)`, then no additional validity-preservation
+/// restrictions apply; otherwise,
/// - If `dst` permits mutation of its referent (e.g. via `Exclusive` aliasing
/// or interior mutation under `Shared` aliasing), then it must hold that
-/// `S(T, V) ⊇ S(U, W)` - in other words, the transmute must not expand the
-/// set of allowed referent bit patterns. A violation of this requirement
+/// `S(T, V) ⊇ S(U, W)` - in other words, the reinterpretation must not expand
+/// the set of allowed referent bit patterns. A violation of this requirement
/// would permit using `dst` to write `x` where `x ∈ S(U, W)` but `x ∉ S(T,
/// V)`, which would violate the guarantee that `src`'s referent may only
/// contain values in `S(T, V)`.
/// - If the referent may be mutated without going through `dst` while `dst` is
/// live (e.g. via interior mutation on a `Shared`-aliased `Ptr` or `&`
/// reference), then it must hold that `S(T, V) ⊆ S(U, W)` - in other words,
-/// the transmute must not shrink the set of allowed referent bit patterns. A
-/// violation of this requirement would permit using `src` or another
-/// mechanism (e.g. a `&` reference used to derive `src`) to write `x` where
-/// `x ∈ S(T, V)` but `x ∉ S(U, W)`, which would violate the guarantee that
-/// `dst`'s referent may only contain values in `S(U, W)`.
+/// the reinterpretation must not shrink the set of allowed referent bit
+/// patterns. A violation of this requirement would permit using `src` or
+/// another mechanism (e.g. a `&` reference used to derive `src`) to write `x`
+/// where `x ∈ S(T, V)` but `x ∉ S(U, W)`, which would violate the guarantee
+/// that `dst`'s referent may only contain values in `S(U, W)`.
+///
+/// These conditions describe only exact reinterpretations. Shrinking
+/// projections may have validity that depends on the relationship between the
+/// projected region and bytes outside it, and require additional reasoning.
pub unsafe trait Validity: Sealed {
const KIND: ValidityKind;
}
@@ -101,7 +105,7 @@ pub enum ValidityKind {
Uninit,
AsInitialized,
Initialized,
- Valid,
+ Safe,
}
/// An [`Aliasing`] invariant which is either [`Shared`] or [`Exclusive`].
@@ -147,7 +151,7 @@ impl Alignment for Unaligned {
fn read<T, I, R>(ptr: crate::Ptr<'_, T, I>) -> T
where
T: Copy + Read<I::Aliasing, R>,
- I: Invariants<Alignment = Self, Validity = Valid>,
+ I: Invariants<Alignment = Self, Validity = Safe>,
I::Aliasing: Reference,
{
(*ptr.into_unalign().as_ref()).into_inner()
@@ -162,7 +166,7 @@ impl Alignment for Aligned {
fn read<T, I, R>(ptr: crate::Ptr<'_, T, I>) -> T
where
T: Copy + Read<I::Aliasing, R>,
- I: Invariants<Alignment = Self, Validity = Valid>,
+ I: Invariants<Alignment = Self, Validity = Safe>,
I::Aliasing: Reference,
{
*ptr.as_ref()
@@ -227,13 +231,20 @@ unsafe impl Validity for Initialized {
/// The referent of a `Ptr<T>` is valid for `T`, upholding bit validity and any
/// library safety invariants.
-pub enum Valid {}
-// SAFETY: `Valid`'s validity is well-defined for all `T: ?Sized`, and is not a
+pub enum Safe {}
+// SAFETY: `Safe`'s validity is well-defined for all `T: ?Sized`, and is not a
// function of any property of `T` other than its bit validity.
-unsafe impl Validity for Valid {
- const KIND: ValidityKind = ValidityKind::Valid;
+unsafe impl Validity for Safe {
+ const KIND: ValidityKind = ValidityKind::Safe;
}
+/// Proof helper for casts whose validity requirement does not depend on the
+/// referent type.
+///
+/// Currently this covers `Uninit`, which permits any byte state, and
+/// `Initialized`, which requires every referent byte to be initialized
+/// regardless of referent type.
+///
/// # Safety
///
/// `DT: CastableFrom<ST, SV, DV>` is sound if `SV = DV = Uninit` or `SV = DV =
@@ -289,7 +300,7 @@ impl Sealed for Aligned {}
impl Sealed for Uninit {}
impl Sealed for AsInitialized {}
impl Sealed for Initialized {}
- impl Sealed for Valid {}
+ impl Sealed for Safe {}
impl<A: Sealed, AA: Sealed, V: Sealed> Sealed for (A, AA, V) {}
diff --git a/rust/zerocopy/src/pointer/mod.rs b/rust/zerocopy/src/pointer/mod.rs
index d6eacc52febe..0ce543391530 100644
--- a/rust/zerocopy/src/pointer/mod.rs
+++ b/rust/zerocopy/src/pointer/mod.rs
@@ -25,9 +25,9 @@
use crate::wrappers::ReadOnly;
/// A shorthand for a maybe-valid, maybe-aligned reference. Used as the argument
-/// to [`TryFromBytes::is_bit_valid`].
+/// to [`TryFromBytes::is_safe`].
///
-/// [`TryFromBytes::is_bit_valid`]: crate::TryFromBytes::is_bit_valid
+/// [`TryFromBytes::is_safe`]: crate::TryFromBytes::is_safe
pub type Maybe<'a, T, Alignment = invariant::Unaligned> =
Ptr<'a, ReadOnly<T>, (invariant::Shared, Alignment, invariant::Initialized)>;
@@ -47,10 +47,7 @@ pub(crate) fn is_zeroed<T, I>(ptr: Ptr<'_, T, I>) -> bool
pub mod cast {
use core::{marker::PhantomData, mem};
- use crate::{
- layout::{SizeInfo, TrailingSliceLayout},
- HasField, KnownLayout, PtrInner,
- };
+ use crate::{layout::SizeInfo, HasField, KnownLayout, PtrInner};
/// A pointer cast or projection.
///
@@ -182,11 +179,9 @@ pub enum CastUnsized {}
// SAFETY: By the `static_assert!`, `Src` and `Dst` are either:
// - Both sized and equal in size
- // - Both slice DSTs with the same trailing slice offset and element size
- // and with align_of::<Src>() == align_of::<Dst>(). These ensure that any
- // given pointer metadata encodes the same size for both `Src` and `Dst`
- // (note that the alignment is required as it affects the amount of
- // trailing padding). Thus, `project` preserves the set of referent bytes.
+ // - Both slice DSTs with the same trailing-slice offset, the same object
+ // size for every metadata value, and with align_of::<Src>() ==
+ // align_of::<Dst>(). Thus, `project` preserves the set of referent bytes.
unsafe impl<Src, Dst> Project<Src, Dst> for CastUnsized
where
Src: ?Sized + KnownLayout,
@@ -201,10 +196,11 @@ fn project(src: PtrInner<'_, Src>) -> *mut Dst {
let dst = <Dst as KnownLayout>::LAYOUT;
match (src.size_info, dst.size_info) {
(SizeInfo::Sized { size: src_size }, SizeInfo::Sized { size: dst_size }) => src_size == dst_size,
- (
- SizeInfo::SliceDst(TrailingSliceLayout { offset: src_offset, elem_size: src_elem_size }),
- SizeInfo::SliceDst(TrailingSliceLayout { offset: dst_offset, elem_size: dst_elem_size })
- ) => src.align.get() == dst.align.get() && src_offset == dst_offset && src_elem_size == dst_elem_size,
+ (SizeInfo::SliceDst(src_trailing), SizeInfo::SliceDst(dst_trailing)) => {
+ src.align.get() == dst.align.get()
+ && src_trailing.offset == dst_trailing.offset
+ && src_trailing.has_same_size_sequence(dst_trailing)
+ },
_ => false,
}
});
@@ -225,10 +221,8 @@ unsafe impl<Src, Dst> Cast<Src, Dst> for CastUnsized
// SAFETY: By the `static_assert!` in `Project::project`, `Src` and `Dst`
// are either:
// - Both sized and equal in size
- // - Both slice DSTs with the same alignment, trailing slice offset, and
- // element size. These ensure that any given pointer metadata encodes the
- // same size for both `Src` and `Dst` (note that the alignment is required
- // as it affects the amount of trailing padding).
+ // - Both slice DSTs with the same alignment and trailing-slice offset, and
+ // with equal sizes for every metadata value.
unsafe impl<Src, Dst> CastExact<Src, Dst> for CastUnsized
where
Src: ?Sized + KnownLayout,
@@ -240,18 +234,22 @@ unsafe impl<Src, Dst> CastExact<Src, Dst> for CastUnsized
///
/// A `Projection` is a [`Project`] which implements projection by
/// delegating to an implementation of [`HasField::project`].
+ ///
+ /// The `Client` parameter exists to select between implementations of
+ /// [`HasField`] that would otherwise conflict.
#[allow(missing_debug_implementations, missing_copy_implementations)]
- pub struct Projection<F: ?Sized, const VARIANT_ID: i128, const FIELD_ID: i128> {
+ pub struct Projection<Client, F: ?Sized, const VARIANT_ID: i128, const FIELD_ID: i128> {
_never: core::convert::Infallible,
+ _client: PhantomData<Client>,
_phantom: PhantomData<F>,
}
// SAFETY: `HasField::project` has the same safety post-conditions as
// `Project::project`.
- unsafe impl<T: ?Sized, F, const VARIANT_ID: i128, const FIELD_ID: i128> Project<T, T::Type>
- for Projection<F, VARIANT_ID, FIELD_ID>
+ unsafe impl<T: ?Sized, Client, F, const VARIANT_ID: i128, const FIELD_ID: i128>
+ Project<T, T::Type> for Projection<Client, F, VARIANT_ID, FIELD_ID>
where
- T: HasField<F, VARIANT_ID, FIELD_ID>,
+ T: HasField<Client, F, VARIANT_ID, FIELD_ID>,
{
#[inline(always)]
fn project(src: PtrInner<'_, T>) -> *mut T::Type {
@@ -292,10 +290,10 @@ fn project(src: PtrInner<'_, T>) -> *mut T::Type {
//
// FIXME(https://github.com/rust-lang/unsafe-code-guidelines/issues/595):
// Cite the documentation once it's updated.
- unsafe impl<T: ?Sized, F, const FIELD_ID: i128> Cast<T, T::Type>
- for Projection<F, { crate::REPR_C_UNION_VARIANT_ID }, FIELD_ID>
+ unsafe impl<T: ?Sized, Client, F, const FIELD_ID: i128> Cast<T, T::Type>
+ for Projection<Client, F, { crate::REPR_C_UNION_VARIANT_ID }, FIELD_ID>
where
- T: HasField<F, { crate::REPR_C_UNION_VARIANT_ID }, FIELD_ID>,
+ T: HasField<Client, F, { crate::REPR_C_UNION_VARIANT_ID }, FIELD_ID>,
{
}
diff --git a/rust/zerocopy/src/pointer/ptr.rs b/rust/zerocopy/src/pointer/ptr.rs
index 7213f6f4a04e..0fcd2ed11d37 100644
--- a/rust/zerocopy/src/pointer/ptr.rs
+++ b/rust/zerocopy/src/pointer/ptr.rs
@@ -169,7 +169,7 @@ mod _conversions {
use crate::pointer::cast::{CastExact, CastSized, IdCast};
/// `&'a T` → `Ptr<'a, T>`
- impl<'a, T> Ptr<'a, T, (Shared, Aligned, Valid)>
+ impl<'a, T> Ptr<'a, T, (Shared, Aligned, Safe)>
where
T: 'a + ?Sized,
{
@@ -183,20 +183,19 @@ pub fn from_ref(ptr: &'a T) -> Self {
// 1. `ptr`, by invariant on `&'a T`, conforms to the alignment
// invariant of `Aligned`.
// 2. `ptr`'s referent, by invariant on `&'a T`, is a bit-valid `T`.
- // This satisfies the requirement that a `Ptr<T, (_, _, Valid)>`
+ // This satisfies the requirement that a `Ptr<T, (_, _, Safe)>`
// point to a bit-valid `T`. Even if `T` permits interior
- // mutation, this invariant guarantees that the returned `Ptr`
- // can only ever be used to modify the referent to store
- // bit-valid `T`s, which ensures that the returned `Ptr` cannot
- // be used to violate the soundness of the original `ptr: &'a T`
- // or of any other references that may exist to the same
- // referent.
+ // mutation, this invariant guarantees that the returned `Ptr` can
+ // only ever be used to modify the referent to store bit-valid
+ // `T`s, which ensures that the returned `Ptr` cannot be used to
+ // violate the soundness of the original `ptr: &'a T` or of any
+ // other references that may exist to the same referent.
unsafe { Self::from_inner(inner) }
}
}
/// `&'a mut T` → `Ptr<'a, T>`
- impl<'a, T> Ptr<'a, T, (Exclusive, Aligned, Valid)>
+ impl<'a, T> Ptr<'a, T, (Exclusive, Aligned, Safe)>
where
T: 'a + ?Sized,
{
@@ -211,7 +210,7 @@ pub fn from_mut(ptr: &'a mut T) -> Self {
// invariant of `Aligned`.
// 2. `ptr`'s referent, by invariant on `&'a mut T`, is a bit-valid
// `T`. This satisfies the requirement that a `Ptr<T, (_, _,
- // Valid)>` point to a bit-valid `T`. This invariant guarantees
+ // Safe)>` point to a bit-valid `T`. This invariant guarantees
// that the returned `Ptr` can only ever be used to modify the
// referent to store bit-valid `T`s, which ensures that the
// returned `Ptr` cannot be used to violate the soundness of the
@@ -224,7 +223,7 @@ pub fn from_mut(ptr: &'a mut T) -> Self {
impl<'a, T, I> Ptr<'a, T, I>
where
T: 'a + ?Sized,
- I: Invariants<Alignment = Aligned, Validity = Valid>,
+ I: Invariants<Alignment = Aligned, Validity = Safe>,
I::Aliasing: Reference,
{
/// Converts `self` to a shared reference.
@@ -239,7 +238,7 @@ pub fn as_ref(self) -> &'a T {
let raw = self.as_inner().as_non_null();
// SAFETY: `self` satisfies the `Aligned` invariant, so we know that
// `raw` is validly-aligned for `T`.
- #[cfg(miri)]
+ #[cfg(all(miri, __ZEROCOPY_INTERNAL_USE_ONLY_DEV_MODE))]
unsafe {
crate::util::miri_promise_symbolic_alignment(
raw.as_ptr().cast(),
@@ -264,7 +263,7 @@ pub fn as_ref(self) -> &'a T {
//
// 3. The pointer must point to a validly-initialized instance of
// `T`. This is ensured by-contract on `Ptr`, because the
- // `I::Validity` is `Valid`.
+ // `I::Validity` is `Safe`.
//
// 4. You must enforce Rust’s aliasing rules. This is ensured by
// contract on `Ptr`, because `I::Aliasing: Reference`. Either it
@@ -377,7 +376,7 @@ pub fn reborrow_shared<'b>(&'b mut self) -> Ptr<'b, T, (Shared, I::Alignment, I:
}
/// `Ptr<'a, T>` → `&'a mut T`
- impl<'a, T> Ptr<'a, T, (Exclusive, Aligned, Valid)>
+ impl<'a, T> Ptr<'a, T, (Exclusive, Aligned, Safe)>
where
T: 'a + ?Sized,
{
@@ -389,7 +388,7 @@ pub fn as_mut(self) -> &'a mut T {
let mut raw = self.as_inner().as_non_null();
// SAFETY: `self` satisfies the `Aligned` invariant, so we know that
// `raw` is validly-aligned for `T`.
- #[cfg(miri)]
+ #[cfg(all(miri, __ZEROCOPY_INTERNAL_USE_ONLY_DEV_MODE))]
unsafe {
crate::util::miri_promise_symbolic_alignment(
raw.as_ptr().cast(),
@@ -413,8 +412,8 @@ pub fn as_mut(self) -> &'a mut T {
// This is ensured by contract on all `PtrInner`s.
//
// 3. The pointer must point to a validly-initialized instance of
- // `T`. This is ensured by-contract on `Ptr`, because the
- // validity invariant is `Valid`.
+ // `T`. This is ensured by-contract on `Ptr`, because the validity
+ // invariant is `Safe`.
//
// 4. You must enforce Rust’s aliasing rules. This is ensured by
// contract on `Ptr`, because the `ALIASING_INVARIANT` is
@@ -431,6 +430,10 @@ impl<'a, T: ?Sized, I> Ptr<'a, T, I>
where
I: Invariants,
{
+ /// Reinterprets the same byte region as `U` with validity `V`.
+ ///
+ /// This preserves the aliasing invariant, uses [`SizeEq`] to select a
+ /// [`CastExact`] implementation, and conservatively forgets alignment.
#[must_use]
#[inline(always)]
pub fn transmute<U, V, R>(self) -> Ptr<'a, U, (I::Aliasing, Unaligned, V)>
@@ -443,6 +446,12 @@ pub fn transmute<U, V, R>(self) -> Ptr<'a, U, (I::Aliasing, Unaligned, V)>
self.transmute_with::<U, V, <U as SizeEq<T>>::CastFrom, R>()
}
+ /// Reinterprets the same byte region as `U` with validity `V` using
+ /// `C`.
+ ///
+ /// `C: CastExact` preserves the byte region. The aliasing invariant is
+ /// preserved, while alignment is conservatively forgotten because the
+ /// destination type may have a different alignment requirement.
#[inline]
#[must_use]
pub fn transmute_with<U, V, C, R>(self) -> Ptr<'a, U, (I::Aliasing, Unaligned, V)>
@@ -462,11 +471,15 @@ pub fn transmute_with<U, V, C, R>(self) -> Ptr<'a, U, (I::Aliasing, Unaligned, V
// at the same time, as neither can perform interior mutation
// - It is directly guaranteed that it is sound for shared code to
// operate on these references simultaneously
- // - By `U: TransmuteFromPtr<T, I::Aliasing, I::Validity, C, V>`, it
+ // - By `U: TransmuteFromPtr<T, I::Aliasing, I::Validity, V, C>`, it
// is sound to perform this transmute using `C`.
unsafe { self.project_transmute_unchecked::<_, _, C>() }
}
+ /// Changes only the validity invariant to `V`.
+ ///
+ /// The referent type and byte region are unchanged, so this preserves
+ /// the existing aliasing and alignment invariants.
#[inline]
#[must_use]
pub fn recall_validity<V, R>(self) -> Ptr<'a, T, (I::Aliasing, I::Alignment, V)>
@@ -562,7 +575,7 @@ pub fn into_unalign(
// FIXME(#1359): This should be a `transmute_with` call.
// Unfortunately, to avoid blanket impl conflicts, we only implement
// `TransmuteFrom<T>` for `Unalign<T>` (and vice versa) specifically
- // for `Valid` validity, not for all validity types.
+ // for `Safe` validity, not for all validity types.
// SAFETY:
// - By `CastSized: Cast`, `CastSized` preserves referent address,
@@ -589,7 +602,7 @@ pub fn into_unalign(
impl<'a, T, I> Ptr<'a, T, I>
where
T: ?Sized,
- I: Invariants<Validity = Valid>,
+ I: Invariants<Validity = Safe>,
I::Aliasing: Reference,
{
/// Reads the referent.
@@ -656,6 +669,22 @@ pub fn unify_invariants<
unsafe { self.assume_invariants::<H>() }
}
+ /// Downgrades `self` to shared aliasing.
+ #[inline]
+ #[must_use]
+ pub(super) fn into_shared(self) -> Ptr<'a, T, (Shared, I::Alignment, I::Validity)>
+ where
+ I::Aliasing: Reference,
+ {
+ // SAFETY: `I::Aliasing: Reference` guarantees that the source
+ // aliasing is either `Shared` or `Exclusive`. `Shared` requires no
+ // transition. If it is `Exclusive`, consuming `self` ends the only
+ // permitted access to the referent, so the returned pointer can
+ // soundly carry `Shared` aliasing. The referent type, alignment,
+ // and validity invariants are unchanged.
+ unsafe { self.assume_invariants() }
+ }
+
/// Assumes that `self`'s referent is validly-aligned for `T` if
/// required by `A`.
///
@@ -738,40 +767,40 @@ pub unsafe fn assume_initialized(
unsafe { self.assume_validity::<Initialized>() }
}
- /// A shorthand for `self.assume_validity<Valid>()`.
+ /// A shorthand for `self.assume_validity<Safe>()`.
///
/// # Safety
///
/// The caller promises to uphold the safety preconditions of
- /// `self.assume_validity<Valid>()`.
+ /// `self.assume_validity<Safe>()`.
#[must_use]
#[inline]
- pub unsafe fn assume_valid(self) -> Ptr<'a, T, (I::Aliasing, I::Alignment, Valid)> {
+ pub unsafe fn assume_safe(self) -> Ptr<'a, T, (I::Aliasing, I::Alignment, Safe)> {
// SAFETY: The caller has promised to uphold the safety
// preconditions.
- unsafe { self.assume_validity::<Valid>() }
+ unsafe { self.assume_validity::<Safe>() }
}
/// Checks that `self`'s referent is validly initialized for `T`,
- /// returning a `Ptr` with `Valid` on success.
+ /// returning a `Ptr` with `Safe` on success.
///
/// # Panics
///
/// This method will panic if
- /// [`T::is_bit_valid`][TryFromBytes::is_bit_valid] panics.
+ /// [`T::is_safe`][TryFromBytes::is_safe] panics.
///
/// # Safety
///
/// On error, unsafe code may rely on this method's returned
/// `ValidityError` containing `self`.
#[inline]
- pub fn try_into_valid<R, S>(
+ pub fn try_into_safe<R, S>(
mut self,
- ) -> Result<Ptr<'a, T, (I::Aliasing, I::Alignment, Valid)>, ValidityError<Self, T>>
+ ) -> Result<Ptr<'a, T, (I::Aliasing, I::Alignment, Safe)>, ValidityError<Self, T>>
where
T: TryFromBytes
+ Read<I::Aliasing, R>
- + TryTransmuteFromPtr<T, I::Aliasing, I::Validity, Valid, IdCast, S>,
+ + TryTransmuteFromPtr<T, I::Aliasing, I::Validity, Safe, IdCast, S>,
ReadOnly<T>: Read<I::Aliasing, R>,
I::Aliasing: Reference,
I: Invariants<Validity = Initialized>,
@@ -779,14 +808,13 @@ pub fn try_into_valid<R, S>(
// This call may panic. If that happens, it doesn't cause any
// soundness issues, as we have not generated any invalid state
// which we need to fix before returning.
- if T::is_bit_valid(self.reborrow().transmute::<_, _, _>().reborrow_shared()) {
- // SAFETY: If `T::is_bit_valid`, code may assume that `self`
- // contains a bit-valid instance of `T`. By `T:
- // TryTransmuteFromPtr<T, I::Aliasing, I::Validity, Valid>`, so
- // long as `self`'s referent conforms to the `Valid` validity
- // for `T` (which we just confirmed), then this transmute is
- // sound.
- Ok(unsafe { self.assume_valid() })
+ if T::is_safe(self.reborrow().transmute::<_, _, _>().reborrow_shared()) {
+ // SAFETY: If `T::is_safe` returns true, code may assume that
+ // `self` contains a valid `T`, so its referent conforms to
+ // `Safe` for `T`. By `T: TryTransmuteFromPtr<T, I::Aliasing,
+ // I::Validity, Safe>`, given that condition, changing `self`'s
+ // validity to `Safe` is sound.
+ Ok(unsafe { self.assume_safe() })
} else {
Err(ValidityError::new(self))
}
@@ -803,7 +831,7 @@ pub fn forget_aligned(self) -> Ptr<'a, T, (I::Aliasing, Unaligned, I::Validity)>
}
/// Casts of the referent type.
-#[cfg_attr(not(zerocopy_unstable_ptr), allow(unreachable_pub))]
+#[allow(unreachable_pub)] // False positive on MSRV
pub use _casts::TryWithError;
mod _casts {
use core::cell::UnsafeCell;
@@ -875,19 +903,22 @@ pub fn cast<U, C, R>(self) -> Ptr<'a, U, (I::Aliasing, Unaligned, I::Validity)>
}
#[inline(always)]
- pub fn project<F, const VARIANT_ID: i128, const FIELD_ID: i128>(
+ pub fn project<Client, F, const VARIANT_ID: i128, const FIELD_ID: i128>(
mut self,
) -> Result<Ptr<'a, T::Type, T::Invariants>, T::Error>
where
- T: ProjectField<F, I, VARIANT_ID, FIELD_ID>,
+ T: ProjectField<Client, F, I, VARIANT_ID, FIELD_ID>,
I::Aliasing: Reference,
{
use crate::pointer::cast::Projection;
- match T::is_projectable(self.reborrow().project_tag()) {
+ match <T as ProjectField<Client, F, I, VARIANT_ID, FIELD_ID>>::is_projectable(
+ self.reborrow().project_tag::<Client>(),
+ ) {
Ok(()) => {
let inner = self.as_inner();
- let projected = inner.project::<_, Projection<F, VARIANT_ID, FIELD_ID>>();
- // SAFETY: By `T: ProjectField<F, I, VARIANT_ID, FIELD_ID>`,
+ let projected =
+ inner.project::<_, Projection<Client, F, VARIANT_ID, FIELD_ID>>();
+ // SAFETY: By `T: ProjectField<Client, F, I, VARIANT_ID, FIELD_ID>`,
// for `self: Ptr<'_, T, I>` such that `T::is_projectable`
// (which we've verified in this match arm),
// `T::project(self.as_inner())` conforms to
@@ -908,17 +939,22 @@ pub fn project<F, const VARIANT_ID: i128, const FIELD_ID: i128>(
#[must_use]
#[inline(always)]
- pub(crate) fn project_tag(self) -> Ptr<'a, T::Tag, I>
+ pub fn project_tag<Client>(
+ self,
+ ) -> Ptr<'a, <T as HasTag<Client>>::Tag, (Shared, I::Alignment, I::Validity)>
where
- T: HasTag,
+ T: HasTag<Client>,
+ I::Aliasing: Reference,
{
- // SAFETY: By invariant on `Self::ProjectToTag`, this is a sound
- // projection.
- let tag = unsafe { self.project_transmute_unchecked::<_, _, T::ProjectToTag>() };
- // SAFETY: By invariant on `Self::ProjectToTag`, the projected
- // pointer has the same alignment as `ptr`.
- let tag = unsafe { tag.assume_alignment() };
- tag.unify_invariants()
+ let ptr = self.into_shared();
+ // SAFETY: By invariant on `ProjectToTag`, it is sound to project
+ // `ptr` to a shared tag pointer with the same validity invariant.
+ let ptr = unsafe {
+ ptr.project_transmute_unchecked::<_, _, <T as HasTag<Client>>::ProjectToTag>()
+ };
+ // SAFETY: By invariant on `ProjectToTag`, the projected pointer has
+ // the same alignment as `self`.
+ unsafe { ptr.assume_alignment() }
}
/// Attempts to transform the pointer, restoring the original on
@@ -1015,11 +1051,11 @@ impl<'a, T, I> Ptr<'a, T, I>
#[allow(clippy::wrong_self_convention)]
#[must_use]
#[inline]
- pub fn as_bytes<R>(self) -> Ptr<'a, [u8], (I::Aliasing, Aligned, Valid)>
+ pub fn as_bytes<R>(self) -> Ptr<'a, [u8], (I::Aliasing, Aligned, Safe)>
where
- [u8]: TransmuteFromPtr<T, I::Aliasing, I::Validity, Valid, AsBytesCast, R>,
+ [u8]: TransmuteFromPtr<T, I::Aliasing, I::Validity, Safe, AsBytesCast, R>,
{
- self.transmute_with::<[u8], Valid, AsBytesCast, _>().bikeshed_recall_aligned()
+ self.transmute_with::<[u8], Safe, AsBytesCast, _>().bikeshed_recall_aligned()
}
}
@@ -1071,7 +1107,7 @@ pub fn as_slice(self) -> Ptr<'a, [T], I> {
/// alignment of `[u8]` is 1.
impl<'a, I> Ptr<'a, [u8], I>
where
- I: Invariants<Validity = Valid>,
+ I: Invariants<Validity = Safe>,
{
/// Attempts to cast `self` to a `U` using the given cast type.
///
@@ -1135,7 +1171,7 @@ pub fn try_cast_into<U, R>(
// it is derived from `try_cast_into`, which promises that the
// object described by `target` is validly aligned for `U`.
// 2. By trait bound, `self` - and thus `target` - is a bit-valid
- // `[u8]`. `Ptr<[u8], (_, _, Valid)>` and `Ptr<_, (_, _,
+ // `[u8]`. `Ptr<[u8], (_, _, Safe)>` and `Ptr<_, (_, _,
// Initialized)>` have the same bit validity, and so neither
// `self` nor `res` can be used to write a value to the referent
// which violates the other's validity invariant.
@@ -1143,14 +1179,14 @@ pub fn try_cast_into<U, R>(
// SAFETY:
// 0. `self` and `remainder` both have the type `[u8]`. Thus, they
- // have `UnsafeCell`s at the same locations. Type casting does
- // not affect aliasing.
+ // have `UnsafeCell`s at the same locations. Type casting does not
+ // affect aliasing.
// 1. `[u8]` has no alignment requirement.
- // 2. `self` has validity `Valid` and has type `[u8]`. Since
+ // 2. `self` has validity `Safe` and has type `[u8]`. Since
// `remainder` references a subset of `self`'s referent, it is
// also a bit-valid `[u8]`. Thus, neither `self` nor `remainder`
- // can be used to write a value to the referent which violates
- // the other's validity invariant.
+ // can be used to write a value to the referent which violates the
+ // other's validity invariant.
let remainder = unsafe { Ptr::from_inner(remainder) };
Ok((res, remainder))
@@ -1333,6 +1369,23 @@ mod tests {
use crate::util::AsAddress;
use crate::{pointer::BecauseImmutable, util::testutil::AU64, FromBytes, Immutable};
+ #[test]
+ fn test_project_tag_downgrades_aliasing() {
+ #[allow(dead_code)]
+ #[derive(zerocopy_derive::Project)]
+ #[repr(u8)]
+ enum Enum {
+ Variant(u8),
+ }
+
+ let mut value = Enum::Variant(0);
+ let _: Ptr<
+ '_,
+ <Enum as crate::HasTag<crate::project_clients::ProjectDerive>>::Tag,
+ (Shared, Aligned, Safe),
+ > = Ptr::from_mut(&mut value).project_tag::<crate::project_clients::ProjectDerive>();
+ }
+
mod test_ptr_try_cast_into_soundness {
use super::*;
diff --git a/rust/zerocopy/src/pointer/transmute.rs b/rust/zerocopy/src/pointer/transmute.rs
index ef9836698203..75f9312f622e 100644
--- a/rust/zerocopy/src/pointer/transmute.rs
+++ b/rust/zerocopy/src/pointer/transmute.rs
@@ -24,14 +24,14 @@
FromBytes, Immutable, IntoBytes, Unalign,
};
-/// Transmutations which are sound to attempt, conditional on validating the bit
-/// validity of the destination type.
+/// Exact pointer reinterpretations which are sound to attempt, conditional on
+/// establishing the destination validity invariant.
///
/// If a `Ptr` transmutation is `TryTransmuteFromPtr`, then it is sound to
/// perform that transmutation so long as some additional mechanism is used to
-/// validate that the referent is bit-valid for the destination type. That
-/// validation mechanism could be a type bound (such as `TransmuteFrom`) or a
-/// runtime validity check.
+/// establish that the referent is `DV`-valid for the destination type. That
+/// mechanism could be a type bound (such as `TransmuteFrom`) or a runtime
+/// validity check.
///
/// # Safety
///
@@ -49,14 +49,14 @@
/// Given `src: Ptr<Src, (A, _, SV)>` and `dst: Ptr<Dst, (A, Unaligned, DV)>`,
/// `Dst: TryTransmuteFromPtr<Src, A, SV, DV, C, _>` is sound if all of the
/// following hold:
-/// - Forwards transmutation: Either of the following hold:
+/// - Preserve destination validity: Either of the following hold:
/// - So long as `dst` is active, no mutation of `dst`'s referent is allowed
/// except via `dst` itself
/// - The set of `DV`-valid referents of `dst` is a superset of the set of
/// `SV`-valid referents of `src` (NOTE: this condition effectively bans
/// shrinking or overwriting transmutes, which cannot satisfy this
/// condition)
-/// - Reverse transmutation: Either of the following hold:
+/// - Preserve source validity: Either of the following hold:
/// - `dst` does not permit mutation of its referent
/// - The set of `DV`-valid referents of `dst` is a subset of the set of
/// `SV`-valid referents of `src` (NOTE: this condition effectively bans
@@ -126,7 +126,7 @@ pub unsafe trait TryTransmuteFromPtr<
pub enum BecauseMutationCompatible {}
// SAFETY:
-// - Forwards transmutation: By `Dst: MutationCompatible<Src, A, SV, DV, _>`, we
+// - Preserve destination validity: By `Dst: MutationCompatible<Src, A, SV, DV, _>`, we
// know that at least one of the following holds:
// - So long as `dst: Ptr<Dst>` is active, no mutation of its referent is
// allowed except via `dst` itself if either of the following hold:
@@ -138,17 +138,18 @@ pub enum BecauseMutationCompatible {}
// referent as `src`. By `Dst: TransmuteFrom<Src, SV, DV>`, the set of
// `DV`-valid referents of `dst` is a superset of the set of `SV`-valid
// referents of `src`.
-// - Reverse transmutation: Since the underlying cast is size-preserving, `dst`
-// addresses the same referent as `src`. By `Src: TransmuteFrom<Dst, DV, SV>`,
-// the set of `DV`-valid referents of `src` is a subset of the set of
-// `SV`-valid referents of `dst`.
+// - Preserve source validity: Since the underlying cast is size-preserving,
+// `dst` addresses the same referent as `src`. By
+// `Src: TransmuteFrom<Dst, DV, SV>`, the set of `DV`-valid referents of `dst`
+// is a subset of the set of `SV`-valid referents of `src`.
// - No safe code, given access to `src` and `dst`, can cause undefined
// behavior: By `Dst: MutationCompatible<Src, A, SV, DV, _>`, at least one of
// the following holds:
// - `A` is `Exclusive`
// - `Src: Immutable` and `Dst: Immutable`
-// - `Dst: InvariantsEq<Src>`, which guarantees that `Src` and `Dst` have the
-// same invariants, and permit interior mutation on the same byte ranges
+// - `Dst: SharedCompatible<Src>`, which guarantees that it is sound for safe
+// code to operate on `&Src` and `&Dst` pointing to the same byte range at
+// the same time
unsafe impl<Src, Dst, SV, DV, A, C, R>
TryTransmuteFromPtr<Src, A, SV, DV, C, (BecauseMutationCompatible, R)> for Dst
where
@@ -162,9 +163,9 @@ unsafe impl<Src, Dst, SV, DV, A, C, R>
}
// SAFETY:
-// - Forwards transmutation: Since aliasing is `Shared` and `Src: Immutable`,
+// - Preserve destination validity: Since aliasing is `Shared` and `Src: Immutable`,
// `src` does not permit mutation of its referent.
-// - Reverse transmutation: Since aliasing is `Shared` and `Dst: Immutable`,
+// - Preserve source validity: Since aliasing is `Shared` and `Dst: Immutable`,
// `dst` does not permit mutation of its referent.
// - No safe code, given access to `src` and `dst`, can cause undefined
// behavior: `Src: Immutable` and `Dst: Immutable`
@@ -187,9 +188,9 @@ unsafe impl<Src, Dst, SV, DV, C> TryTransmuteFromPtr<Src, Shared, SV, DV, C, Bec
///
/// At least one of the following must hold:
/// - `Src: Read<A, _>` and `Self: Read<A, _>`
-/// - `Self: InvariantsEq<Src>`, and, for some `V`:
-/// - `Dst: TransmuteFrom<Src, V, V>`
-/// - `Src: TransmuteFrom<Dst, V, V>`
+/// - `Self: SharedCompatible<Src>`, and, for some `V`:
+/// - `Self: TransmuteFrom<Src, V, V>`
+/// - `Src: TransmuteFrom<Self, V, V>`
pub unsafe trait MutationCompatible<Src: ?Sized, A: Aliasing, SV, DV, R> {}
#[allow(missing_copy_implementations, missing_debug_implementations)]
@@ -204,38 +205,39 @@ unsafe impl<Src: ?Sized, Dst: ?Sized, A: Aliasing, SV: Validity, DV: Validity, R
{
}
-/// Denotes that two types have the same invariants.
+/// Denotes that shared references to two types may safely coexist on the same
+/// referent.
///
/// # Safety
///
/// It is sound for safe code to operate on a `&T` and a `&Self` pointing to the
/// same referent at the same time - no such safe code can cause undefined
/// behavior.
-pub unsafe trait InvariantsEq<T: ?Sized> {}
+pub unsafe trait SharedCompatible<T: ?Sized> {}
// SAFETY: Trivially sound to have multiple `&T` pointing to the same referent.
-unsafe impl<T: ?Sized> InvariantsEq<T> for T {}
+unsafe impl<T: ?Sized> SharedCompatible<T> for T {}
-// SAFETY: `Dst: InvariantsEq<Src> + TransmuteFrom<Src, SV, DV>`, and `Src:
+// SAFETY: `Dst: SharedCompatible<Src> + TransmuteFrom<Src, SV, DV>`, and `Src:
// TransmuteFrom<Dst, DV, SV>`.
unsafe impl<Src: ?Sized, Dst: ?Sized, A: Aliasing, SV: Validity, DV: Validity>
- MutationCompatible<Src, A, SV, DV, BecauseInvariantsEq> for Dst
+ MutationCompatible<Src, A, SV, DV, BecauseSharedCompatible> for Dst
where
Src: TransmuteFrom<Dst, DV, SV>,
- Dst: TransmuteFrom<Src, SV, DV> + InvariantsEq<Src>,
+ Dst: TransmuteFrom<Src, SV, DV> + SharedCompatible<Src>,
{
}
#[allow(missing_debug_implementations, missing_copy_implementations)]
-pub enum BecauseInvariantsEq {}
+pub enum BecauseSharedCompatible {}
-macro_rules! unsafe_impl_invariants_eq {
+macro_rules! unsafe_impl_shared_compatible {
($tyvar:ident => $t:ty, $u:ty) => {{
crate::util::macros::__unsafe();
// SAFETY: The caller promises that this is sound.
- unsafe impl<$tyvar> InvariantsEq<$t> for $u {}
+ unsafe impl<$tyvar> SharedCompatible<$t> for $u {}
// SAFETY: The caller promises that this is sound.
- unsafe impl<$tyvar> InvariantsEq<$u> for $t {}
+ unsafe impl<$tyvar> SharedCompatible<$u> for $t {}
}};
}
@@ -253,11 +255,11 @@ unsafe impl<$tyvar> InvariantsEq<$u> for $t {}
// validity as `T`
//
// [2] https://doc.rust-lang.org/1.81.0/std/mem/struct.ManuallyDrop.html#impl-Deref-for-ManuallyDrop%3CT%3E
-unsafe impl<T: ?Sized> InvariantsEq<T> for ManuallyDrop<T> {}
+unsafe impl<T: ?Sized> SharedCompatible<T> for ManuallyDrop<T> {}
// SAFETY: See previous safety comment.
-unsafe impl<T: ?Sized> InvariantsEq<ManuallyDrop<T>> for T {}
+unsafe impl<T: ?Sized> SharedCompatible<ManuallyDrop<T>> for T {}
-/// Transmutations which are always sound.
+/// Exact pointer reinterpretations which are always sound.
///
/// `TransmuteFromPtr` is a shorthand for [`TryTransmuteFromPtr`] and
/// [`TransmuteFrom`].
@@ -293,8 +295,12 @@ unsafe impl<
{
}
-/// Denotes that any `SV`-valid `Src` may soundly be transmuted into a
-/// `DV`-valid `Self`.
+/// Denotes that, for equally-sized referents, every bit pattern allowed for an
+/// `SV`-valid `Src` is also allowed for a `DV`-valid `Self`.
+///
+/// `TransmuteFrom` does not by itself authorize a pointer transmutation. It
+/// provides a directional validity implication used by the pointer-transmutation
+/// machinery.
///
/// # Safety
///
@@ -333,7 +339,7 @@ impl<T: ?Sized> SizeEq<T> for T {
// SAFETY: Since `Src: IntoBytes`, the set of valid `Src`'s is the set of
// initialized bit patterns, which is exactly the set allowed in the referent of
// any `Initialized` `Ptr`.
-unsafe impl<Src, Dst> TransmuteFrom<Src, Valid, Initialized> for Dst
+unsafe impl<Src, Dst> TransmuteFrom<Src, Safe, Initialized> for Dst
where
Src: IntoBytes + ?Sized,
Dst: ?Sized,
@@ -341,18 +347,19 @@ unsafe impl<Src, Dst> TransmuteFrom<Src, Valid, Initialized> for Dst
}
// SAFETY: Since `Dst: FromBytes`, any initialized bit pattern may appear in the
-// referent of a `Ptr<Dst, (_, _, Valid)>`. This is exactly equal to the set of
+// referent of a `Ptr<Dst, (_, _, Safe)>`. This is exactly equal to the set of
// bit patterns which may appear in the referent of any `Initialized` `Ptr`.
-unsafe impl<Src, Dst> TransmuteFrom<Src, Initialized, Valid> for Dst
+unsafe impl<Src, Dst> TransmuteFrom<Src, Initialized, Safe> for Dst
where
Src: ?Sized,
Dst: FromBytes + ?Sized,
{
}
-// FIXME(#2354): This seems like a smell - the soundness of this bound has
-// nothing to do with `Src` or `Dst` - we're basically just saying `[u8; N]` is
-// transmutable into `[u8; N]`.
+// FIXME(#2354): This relation does not depend on `Src` or `Dst`. That is
+// expected semantically: `Initialized` permits the same states regardless of
+// referent type. The awkward part is encoding that validity-state relation as a
+// trait relation between referent types.
// SAFETY: The set of allowed bit patterns in the referent of any `Initialized`
// `Ptr` is the same regardless of referent type.
@@ -363,9 +370,10 @@ unsafe impl<Src, Dst> TransmuteFrom<Src, Initialized, Initialized> for Dst
{
}
-// FIXME(#2354): This seems like a smell - the soundness of this bound has
-// nothing to do with `Dst` - we're basically just saying that any type is
-// transmutable into `MaybeUninit<[u8; N]>`.
+// FIXME(#2354): This relation does not depend on `Src` or `Dst`. That is
+// expected semantically: every validity state implies `Uninit`, which permits
+// any byte sequence. The awkward part is encoding that validity-state relation
+// as a trait relation between referent types.
// SAFETY: A `Dst` with validity `Uninit` permits any byte sequence, and
// therefore can be transmuted from any value.
@@ -400,7 +408,7 @@ unsafe impl<Src, Dst, V> TransmuteFrom<Src, V, Uninit> for Dst
// sound for these two references to exist at the same time since it's already
// possible for safe code to get into this state.
#[allow(clippy::multiple_unsafe_ops_per_block)]
-const _: () = unsafe { unsafe_impl_invariants_eq!(T => T, Unalign<T>) };
+const _: () = unsafe { unsafe_impl_shared_compatible!(T => T, Unalign<T>) };
// SAFETY:
// - `Wrapping<T>` has the same size as `T` [1].
@@ -433,7 +441,7 @@ unsafe impl<Src, Dst, V> TransmuteFrom<Src, V, Uninit> for Dst
// already possible for safe code to obtain a `&Wrapping<T>` and a `&T` pointing
// to the same referent at the same time. Thus, this must be sound.
#[allow(clippy::multiple_unsafe_ops_per_block)]
-const _: () = unsafe { unsafe_impl_invariants_eq!(T => T, Wrapping<T>) };
+const _: () = unsafe { unsafe_impl_shared_compatible!(T => T, Wrapping<T>) };
// SAFETY:
// - `UnsafeCell<T>` has the same size as `T` [1].
@@ -478,7 +486,7 @@ unsafe impl<Src, Dst, V> TransmuteFrom<Src, V, Uninit> for Dst
// explicitly guaranteed, but it's obvious from `MaybeUninit`'s documentation
// that this is the intention:
// https://doc.rust-lang.org/1.85.0/core/mem/union.MaybeUninit.html
-unsafe impl<T> TransmuteFrom<T, Uninit, Valid> for MaybeUninit<T> {}
+unsafe impl<T> TransmuteFrom<T, Uninit, Safe> for MaybeUninit<T> {}
impl<T> SizeEq<T> for MaybeUninit<T> {
type CastFrom = CastSizedExact;
diff --git a/rust/zerocopy/src/ref.rs b/rust/zerocopy/src/ref.rs
index e49f2a887ffa..7a8876a0587b 100644
--- a/rust/zerocopy/src/ref.rs
+++ b/rust/zerocopy/src/ref.rs
@@ -10,7 +10,7 @@
// those terms.
use super::*;
use crate::pointer::{
- BecauseInvariantsEq, BecauseMutationCompatible, MutationCompatible, TransmuteFromPtr,
+ BecauseMutationCompatible, BecauseSharedCompatible, MutationCompatible, TransmuteFromPtr,
};
mod def {
@@ -18,7 +18,7 @@ mod def {
use crate::{
ByteSlice, ByteSliceMut, CloneableByteSlice, CopyableByteSlice, IntoByteSlice,
- IntoByteSliceMut,
+ IntoByteSliceMut, KnownLayout, PointerMetadata,
};
/// A typed reference derived from a byte slice.
@@ -59,30 +59,83 @@ mod def {
/// }
/// }
/// ```
- pub struct Ref<B, T: ?Sized>(
- // INVARIANTS: The referent (via `.deref`, `.deref_mut`, `.into`) byte
- // slice is aligned to `T`'s alignment and its size corresponds to a
- // valid size for `T`.
- B,
- PhantomData<T>,
- );
-
- impl<B, T: ?Sized> Ref<B, T> {
+ pub struct Ref<B, T: ?Sized> {
+ // INVARIANTS:
+ // - The referent (via `.deref`, `.deref_mut`, `.into`) byte slice is
+ // aligned to `T`'s alignment, and its size is valid for `T`.
+ // - If `T: KnownLayout`, let `metadata` be
+ // `T::PointerMetadata::from_elem_count` applied to `elem_count`.
+ // Then `T::size_for_metadata(metadata)` is `Some(referent.len())`.
+ bytes: B,
+ // Ignored when `T` does not implement `KnownLayout`.
+ elem_count: usize,
+ _marker: PhantomData<T>,
+ }
+
+ impl<B, T> Ref<B, T> {
+ /// Constructs a new `Ref` to a sized `T`.
+ ///
+ /// # Safety
+ ///
+ /// `bytes` dereferences (via [`deref`], [`deref_mut`], and [`into`]) to
+ /// a byte slice which is aligned to `T`'s alignment and whose size is
+ /// `size_of::<T>()`.
+ ///
+ /// [`deref`]: core::ops::Deref::deref
+ /// [`deref_mut`]: core::ops::DerefMut::deref_mut
+ /// [`into`]: core::convert::Into::into
+ pub(super) unsafe fn new_sized_unchecked(bytes: B) -> Ref<B, T> {
+ // INVARIANTS: The caller has promised that `bytes`'s referent is
+ // validly aligned and has size `size_of::<T>()`. If `T:
+ // KnownLayout`, then, since `T` is sized, `T::PointerMetadata` is
+ // `()`. `from_elem_count` returns `()` for any element count, and
+ // `T::size_for_metadata(()) == Some(size_of::<T>())`. `size_of`
+ // reports this size in bytes [1].
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of.html:
+ //
+ // Returns the size of a type in bytes.
+ Ref { bytes, elem_count: 0, _marker: PhantomData }
+ }
+ }
+
+ impl<B, T: KnownLayout + ?Sized> Ref<B, T> {
/// Constructs a new `Ref`.
///
/// # Safety
///
/// `bytes` dereferences (via [`deref`], [`deref_mut`], and [`into`]) to
- /// a byte slice which is aligned to `T`'s alignment and whose size is a
- /// valid size for `T`.
+ /// a byte slice which is aligned to `T`'s alignment and satisfies
+ /// `T::size_for_metadata(metadata) == Some(referent.len())`.
///
/// [`deref`]: core::ops::Deref::deref
/// [`deref_mut`]: core::ops::DerefMut::deref_mut
/// [`into`]: core::convert::Into::into
- pub(crate) unsafe fn new_unchecked(bytes: B) -> Ref<B, T> {
+ pub(super) unsafe fn new_unchecked(bytes: B, metadata: T::PointerMetadata) -> Ref<B, T> {
// INVARIANTS: The caller has promised that `bytes`'s referent is
- // validly-aligned and has a valid size.
- Ref(bytes, PhantomData)
+ // validly aligned and has exactly the size described by `metadata`.
+ // `KnownLayout::PointerMetadata` is `()` for sized types and
+ // `usize` for slice DSTs. For both permitted metadata types,
+ // `from_elem_count(metadata.to_elem_count()) == metadata`: `()`
+ // round-trips through zero, while `usize` round-trips through
+ // itself. Thus, storing the element count preserves the metadata
+ // whose size the caller validated. This agrees with Rust's
+ // language-level definition of slice pointer metadata [1].
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/reference/dynamically-sized-types.html:
+ //
+ // Pointers to slices also store the number of elements of the
+ // slice.
+ Ref { bytes, elem_count: metadata.to_elem_count(), _marker: PhantomData }
+ }
+
+ /// Gets the pointer metadata represented by this `Ref`.
+ #[inline(always)]
+ pub(super) fn pointer_metadata(&self) -> T::PointerMetadata {
+ // INVARIANTS: `KnownLayout::PointerMetadata` is either `()` or
+ // `usize`. For `()`, `from_elem_count` ignores the element count;
+ // for `usize`, it returns the element count unchanged.
+ T::PointerMetadata::from_elem_count(self.elem_count)
}
}
@@ -96,16 +149,18 @@ impl<B: ByteSlice, T: ?Sized> Ref<B, T> {
/// `Any::downcast_ref`).
///
/// `as_byte_slice` promises to return a `ByteSlice` whose referent is
- /// validly-aligned for `T` and has a valid size for `T`.
+ /// validly aligned for `T` and has a valid size for `T`. If `T:
+ /// KnownLayout`, then `T::size_for_metadata(self.pointer_metadata())`
+ /// returns `Some(referent.len())`.
pub(crate) unsafe fn as_byte_slice(&self) -> &impl ByteSlice {
// INVARIANTS: The caller promises not to call methods other than
// those on `ByteSlice`. Since `B: ByteSlice`, dereference stability
// guarantees that calling `ByteSlice` methods will not change the
- // address or length of `self.0`'s referent.
+ // address or length of `self.bytes`'s referent.
//
- // SAFETY: By invariant on `self.0`, the alignment and size
- // post-conditions are upheld.
- &self.0
+ // SAFETY: By invariant on `self`, the alignment, size, and metadata
+ // postconditions are upheld.
+ &self.bytes
}
}
@@ -118,17 +173,20 @@ impl<B: ByteSliceMut, T: ?Sized> Ref<B, T> {
/// [`ByteSliceMut`] other than `ByteSliceMut` methods (for example, via
/// `Any::downcast_mut`).
///
- /// `as_byte_slice` promises to return a `ByteSlice` whose referent is
- /// validly-aligned for `T` and has a valid size for `T`.
+ /// `as_byte_slice_mut` promises to return a `ByteSliceMut` whose
+ /// referent is validly aligned for `T` and has a valid size for `T`. If
+ /// `T: KnownLayout`, then
+ /// `T::size_for_metadata(self.pointer_metadata())` returns
+ /// `Some(referent.len())`.
pub(crate) unsafe fn as_byte_slice_mut(&mut self) -> &mut impl ByteSliceMut {
// INVARIANTS: The caller promises not to call methods other than
// those on `ByteSliceMut`. Since `B: ByteSlice`, dereference
// stability guarantees that calling `ByteSlice` methods will not
- // change the address or length of `self.0`'s referent.
+ // change the address or length of `self.bytes`'s referent.
//
- // SAFETY: By invariant on `self.0`, the alignment and size
- // post-conditions are upheld.
- &mut self.0
+ // SAFETY: By invariant on `self`, the alignment, size, and metadata
+ // postconditions are upheld.
+ &mut self.bytes
}
}
@@ -141,17 +199,20 @@ impl<'a, B: IntoByteSlice<'a>, T: ?Sized> Ref<B, T> {
/// [`IntoByteSlice`] other than `IntoByteSlice` methods (for example,
/// via `Any::downcast_ref`).
///
- /// `as_byte_slice` promises to return a `ByteSlice` whose referent is
- /// validly-aligned for `T` and has a valid size for `T`.
+ /// `into_byte_slice` promises to return an `IntoByteSlice` whose
+ /// referent is validly aligned for `T` and has a valid size for `T`. If
+ /// `T: KnownLayout`, then
+ /// `T::size_for_metadata(self.pointer_metadata())` returns
+ /// `Some(referent.len())`.
pub(crate) unsafe fn into_byte_slice(self) -> impl IntoByteSlice<'a> {
// INVARIANTS: The caller promises not to call methods other than
// those on `IntoByteSlice`. Since `B: ByteSlice`, dereference
// stability guarantees that calling `ByteSlice` methods will not
- // change the address or length of `self.0`'s referent.
+ // change the address or length of `self.bytes`'s referent.
//
- // SAFETY: By invariant on `self.0`, the alignment and size
- // post-conditions are upheld.
- self.0
+ // SAFETY: By invariant on `self`, the alignment, size, and metadata
+ // postconditions are upheld.
+ self.bytes
}
}
@@ -164,33 +225,38 @@ impl<'a, B: IntoByteSliceMut<'a>, T: ?Sized> Ref<B, T> {
/// [`IntoByteSliceMut`] other than `IntoByteSliceMut` methods (for
/// example, via `Any::downcast_mut`).
///
- /// `as_byte_slice` promises to return a `ByteSlice` whose referent is
- /// validly-aligned for `T` and has a valid size for `T`.
+ /// `into_byte_slice_mut` promises to return an `IntoByteSliceMut` whose
+ /// referent is validly aligned for `T` and has a valid size for `T`. If
+ /// `T: KnownLayout`, then
+ /// `T::size_for_metadata(self.pointer_metadata())` returns
+ /// `Some(referent.len())`.
pub(crate) unsafe fn into_byte_slice_mut(self) -> impl IntoByteSliceMut<'a> {
// INVARIANTS: The caller promises not to call methods other than
// those on `IntoByteSliceMut`. Since `B: ByteSlice`, dereference
// stability guarantees that calling `ByteSlice` methods will not
- // change the address or length of `self.0`'s referent.
+ // change the address or length of `self.bytes`'s referent.
//
- // SAFETY: By invariant on `self.0`, the alignment and size
- // post-conditions are upheld.
- self.0
+ // SAFETY: By invariant on `self`, the alignment, size, and metadata
+ // postconditions are upheld.
+ self.bytes
}
}
impl<B: CloneableByteSlice + Clone, T: ?Sized> Clone for Ref<B, T> {
#[inline]
fn clone(&self) -> Ref<B, T> {
- // INVARIANTS: Since `B: CloneableByteSlice`, `self.0.clone()` has
- // the same address and length as `self.0`. Since `self.0` upholds
- // the field invariants, so does `self.0.clone()`.
- Ref(self.0.clone(), PhantomData)
+ // INVARIANTS: Since `B: CloneableByteSlice`, `self.bytes.clone()` has
+ // the same address and length as `self.bytes`. We copy
+ // `self.elem_count`, so the clone also has the same metadata. Since
+ // `self` upholds the field invariants, so does the clone.
+ Ref { bytes: self.bytes.clone(), elem_count: self.elem_count, _marker: PhantomData }
}
}
- // INVARIANTS: Since `B: CopyableByteSlice`, the copied `Ref`'s `.0` has the
- // same address and length as the original `Ref`'s `.0`. Since the original
- // upholds the field invariants, so does the copy.
+ // INVARIANTS: Since `B: CopyableByteSlice`, the copied `Ref`'s `bytes` has the
+ // same address and length as the original `Ref`'s `bytes`. The copied
+ // `elem_count` has the same metadata. Since the original upholds the field
+ // invariants, so does the copy.
impl<B: CopyableByteSlice + Copy, T: ?Sized> Copy for Ref<B, T> {}
}
@@ -198,7 +264,7 @@ impl<B: CopyableByteSlice + Copy, T: ?Sized> Copy for Ref<B, T> {}
pub use def::Ref;
use crate::pointer::{
- invariant::{Aligned, BecauseExclusive, Initialized, Unaligned, Valid},
+ invariant::{Aligned, BecauseExclusive, Initialized, Safe, Unaligned},
BecauseRead, PtrInner,
};
@@ -215,8 +281,14 @@ pub(crate) fn sized_from(bytes: B) -> Result<Ref<B, T>, CastError<B, T>> {
return Err(err.with_src(bytes).into());
}
- // SAFETY: We just validated size and alignment.
- Ok(unsafe { Ref::new_unchecked(bytes) })
+ // SAFETY: `size_of` returns `T`'s size in bytes [1]. The equality check
+ // above therefore establishes that `bytes` has exactly that byte
+ // length, and `validate_aligned_to` establishes alignment.
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of.html:
+ //
+ // Returns the size of a type in bytes.
+ Ok(unsafe { Ref::new_sized_unchecked(bytes) })
}
}
@@ -238,10 +310,15 @@ pub(crate) fn sized_from_prefix(bytes: B) -> Result<(Ref<B, T>, B), CastError<B,
)?;
// SAFETY: We just validated alignment and that `bytes` is at least as
// large as `T`. `bytes.split_at(mem::size_of::<T>())?` ensures that the
- // new `bytes` is exactly the size of `T`. By safety postcondition on
+ // new `bytes` is exactly the size of `T`, measured in bytes as specified
+ // for `size_of` [1]. By safety postcondition on
// `SplitByteSlice::split_at` we can rely on `split_at` to produce the
// correct `bytes` and `suffix`.
- let r = unsafe { Ref::new_unchecked(bytes) };
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of.html:
+ //
+ // Returns the size of a type in bytes.
+ let r = unsafe { Ref::new_sized_unchecked(bytes) };
Ok((r, suffix))
}
@@ -259,11 +336,16 @@ pub(crate) fn sized_from_suffix(bytes: B) -> Result<(B, Ref<B, T>), CastError<B,
}
// SAFETY: Since `split_at` is defined as `bytes_len - size_of::<T>()`,
// the `bytes` which results from `let (prefix, bytes) =
- // bytes.split_at(split_at)?` has length `size_of::<T>()`. After
- // constructing `bytes`, we validate that it has the proper alignment.
- // By safety postcondition on `SplitByteSlice::split_at` we can rely on
- // `split_at` to produce the correct `prefix` and `bytes`.
- let r = unsafe { Ref::new_unchecked(bytes) };
+ // bytes.split_at(split_at)?` has length `size_of::<T>()`, measured in
+ // bytes as specified for `size_of` [1]. After constructing `bytes`, we
+ // validate that it has the proper alignment. By safety postcondition on
+ // `SplitByteSlice::split_at` we can rely on `split_at` to produce the
+ // correct `prefix` and `bytes`.
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of.html:
+ //
+ // Returns the size of a type in bytes.
+ let r = unsafe { Ref::new_sized_unchecked(bytes) };
Ok((prefix, r))
}
}
@@ -306,17 +388,64 @@ impl<B, T> Ref<B, T>
///
/// let _ = Ref::<_, ZSTy>::from_bytes(&b"UU"[..]); // ⚠ Compile Error!
/// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_from_bytes",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
#[must_use = "has no side effects"]
#[inline]
pub fn from_bytes(source: B) -> Result<Ref<B, T>, CastError<B, T>> {
static_assert_dst_is_not_zst!(T);
- if let Err(e) =
- Ptr::from_ref(source.deref()).try_cast_into_no_leftover::<T, BecauseImmutable>(None)
+ let metadata = match Ptr::from_ref(source.deref())
+ .try_cast_into_no_leftover::<T, BecauseImmutable>(None)
{
- return Err(e.with_src(()).with_src(source));
- }
- // SAFETY: `try_cast_into_no_leftover` validates size and alignment.
- Ok(unsafe { Ref::new_unchecked(source) })
+ Ok(ptr) => T::pointer_to_metadata(ptr.as_inner().as_ptr()),
+ Err(e) => return Err(e.with_src(()).with_src(source)),
+ };
+ // SAFETY: By its safety postcondition, `try_cast_into_no_leftover`
+ // returned a `Ptr<T>` aligned as required for references [2], over the
+ // same byte range as `source.deref()`. `pointer_to_metadata` returns
+ // the metadata stored in that pointer, and `size_for_metadata` computes
+ // the corresponding dynamically-known byte size [1]. Thus,
+ // `T::size_for_metadata(metadata)` is `Some(source.len())`.
+ // `ByteSlice` and its applicable subtraits guarantee that the stored
+ // `source` continues to dereference or convert to that address and
+ // length. These facts establish both preconditions of `new_unchecked`.
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of_val.html:
+ //
+ // Returns the size of the pointed-to value in bytes.
+ //
+ // ... when `T` has no statically-known size ... `size_of_val` can
+ // be used to get the dynamically-known size.
+ //
+ // [2] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.align_of_val.html:
+ //
+ // Returns the ABI-required minimum alignment of the type of the
+ // value that `val` points to.
+ //
+ // Every reference to a value of the type `T` must be a multiple of
+ // this number.
+ Ok(unsafe { Ref::new_unchecked(source, metadata) })
}
}
@@ -360,14 +489,37 @@ impl<B, T> Ref<B, T>
///
/// let _ = Ref::<_, ZSTy>::from_prefix(&b"UU"[..]); // ⚠ Compile Error!
/// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_from_prefix",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
#[must_use = "has no side effects"]
#[inline]
pub fn from_prefix(source: B) -> Result<(Ref<B, T>, B), CastError<B, T>> {
static_assert_dst_is_not_zst!(T);
- let remainder = match Ptr::from_ref(source.deref())
+ let (metadata, remainder) = match Ptr::from_ref(source.deref())
.try_cast_into::<T, BecauseImmutable>(CastType::Prefix, None)
{
- Ok((_, remainder)) => remainder,
+ Ok((ptr, remainder)) => (T::pointer_to_metadata(ptr.as_inner().as_ptr()), remainder),
Err(e) => {
return Err(e.with_src(()).with_src(source));
}
@@ -381,11 +533,35 @@ pub fn from_prefix(source: B) -> Result<(Ref<B, T>, B), CastError<B, T>> {
#[allow(unstable_name_collisions)]
let split_at = unsafe { source.len().unchecked_sub(remainder.len()) };
let (bytes, suffix) = source.split_at(split_at).map_err(|b| SizeError::new(b).into())?;
- // SAFETY: `try_cast_into` validates size and alignment, and returns a
- // `split_at` that indicates how many bytes of `source` correspond to a
- // valid `T`. By safety postcondition on `SplitByteSlice::split_at` we
- // can rely on `split_at` to produce the correct `source` and `suffix`.
- let r = unsafe { Ref::new_unchecked(bytes) };
+ // SAFETY: By its safety postcondition, `try_cast_into` returned an
+ // `Ptr<T>` aligned as required for references [2], and a remainder
+ // which are non-overlapping, cover `source.deref()`, and place the
+ // `Ptr<T>` at `source.deref()`'s address.
+ // Therefore, `split_at` is the typed pointer's byte length. By the
+ // safety postcondition of `SplitByteSlice::split_at`, `bytes` has that
+ // length and address. `pointer_to_metadata` returns the metadata stored
+ // in the typed pointer, and `size_for_metadata` computes the
+ // corresponding dynamically-known byte size [1]. Thus,
+ // `T::size_for_metadata(metadata)` is `Some(bytes.len())`. The
+ // `SplitByteSlice` and `ByteSlice` contracts preserve the established
+ // address and length, satisfying both preconditions of
+ // `new_unchecked`.
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of_val.html:
+ //
+ // Returns the size of the pointed-to value in bytes.
+ //
+ // ... when `T` has no statically-known size ... `size_of_val` can
+ // be used to get the dynamically-known size.
+ //
+ // [2] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.align_of_val.html:
+ //
+ // Returns the ABI-required minimum alignment of the type of the
+ // value that `val` points to.
+ //
+ // Every reference to a value of the type `T` must be a multiple of
+ // this number.
+ let r = unsafe { Ref::new_unchecked(bytes, metadata) };
Ok((r, suffix))
}
@@ -425,14 +601,37 @@ pub fn from_prefix(source: B) -> Result<(Ref<B, T>, B), CastError<B, T>> {
///
/// let _ = Ref::<_, ZSTy>::from_suffix(&b"UU"[..]); // ⚠ Compile Error!
/// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_from_suffix",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
#[must_use = "has no side effects"]
#[inline]
pub fn from_suffix(source: B) -> Result<(B, Ref<B, T>), CastError<B, T>> {
static_assert_dst_is_not_zst!(T);
- let remainder = match Ptr::from_ref(source.deref())
+ let (metadata, remainder) = match Ptr::from_ref(source.deref())
.try_cast_into::<T, BecauseImmutable>(CastType::Suffix, None)
{
- Ok((_, remainder)) => remainder,
+ Ok((ptr, remainder)) => (T::pointer_to_metadata(ptr.as_inner().as_ptr()), remainder),
Err(e) => {
let e = e.with_src(());
return Err(e.with_src(source));
@@ -441,11 +640,39 @@ pub fn from_suffix(source: B) -> Result<(B, Ref<B, T>), CastError<B, T>> {
let split_at = remainder.len();
let (prefix, bytes) = source.split_at(split_at).map_err(|b| SizeError::new(b).into())?;
- // SAFETY: `try_cast_into` validates size and alignment, and returns a
- // `split_at` that indicates how many bytes of `source` correspond to a
- // valid `T`. By safety postcondition on `SplitByteSlice::split_at` we
- // can rely on `split_at` to produce the correct `prefix` and `bytes`.
- let r = unsafe { Ref::new_unchecked(bytes) };
+ // SAFETY: By its safety postcondition, `try_cast_into` returned an
+ // `Ptr<T>` aligned as required for references [2], and a remainder
+ // which are non-overlapping, cover `source.deref()`, and place the
+ // remainder at `source.deref()`'s address. More precisely,
+ // `PtrInner::try_cast_into` constructs the typed suffix from the
+ // `r_slice` produced by `split_at_unchecked(split_at)` and returns its
+ // `l_slice` as `remainder`. Thus, even when `T` is zero-sized, the typed
+ // pointer and the `bytes` produced by splitting `source` at
+ // `remainder.len()` have the same address and length. By the safety
+ // postcondition of `SplitByteSlice::split_at`, `bytes` retains that
+ // address and length. `pointer_to_metadata` returns the metadata stored
+ // in the typed pointer, and `size_for_metadata` computes the
+ // corresponding dynamically-known byte size [1]. Thus,
+ // `T::size_for_metadata(metadata)` is `Some(bytes.len())`. The
+ // `SplitByteSlice` and `ByteSlice` contracts preserve the established
+ // address and length, satisfying both preconditions of
+ // `new_unchecked`.
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of_val.html:
+ //
+ // Returns the size of the pointed-to value in bytes.
+ //
+ // ... when `T` has no statically-known size ... `size_of_val` can
+ // be used to get the dynamically-known size.
+ //
+ // [2] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.align_of_val.html:
+ //
+ // Returns the ABI-required minimum alignment of the type of the
+ // value that `val` points to.
+ //
+ // Every reference to a value of the type `T` must be a multiple of
+ // this number.
+ let r = unsafe { Ref::new_unchecked(bytes, metadata) };
Ok((prefix, r))
}
}
@@ -487,6 +714,24 @@ impl<B, T> Ref<B, T>
///
/// let _ = Ref::<_, ZSTy>::from_bytes_with_elems(&b"UU"[..], 42); // ⚠ Compile Error!
/// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_from_bytes_with_elems",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
#[inline]
pub fn from_bytes_with_elems(source: B, count: usize) -> Result<Ref<B, T>, CastError<B, T>> {
static_assert_dst_is_not_zst!(T);
@@ -497,7 +742,37 @@ pub fn from_bytes_with_elems(source: B, count: usize) -> Result<Ref<B, T>, CastE
if source.len() != expected_len {
return Err(SizeError::new(source).into());
}
- Self::from_bytes(source)
+ if let Err(e) = Ptr::from_ref(source.deref())
+ .try_cast_into_no_leftover::<T, BecauseImmutable>(Some(count))
+ {
+ return Err(e.with_src(()).with_src(source));
+ }
+ // SAFETY: `size_for_metadata(count)` returned `expected_len`, and the
+ // equality check established that this is `source.len()`. This is the
+ // dynamically-known byte size described by `size_of_val` [1]. By its
+ // safety postcondition, `try_cast_into_no_leftover(Some(count))` only
+ // succeeded after producing a `Ptr<T>` aligned as required for
+ // references [2], with that metadata over the same byte range as
+ // `source.deref()`. `ByteSlice` and its applicable subtraits guarantee
+ // that the stored `source` continues to dereference or convert to that
+ // address and length. These facts establish both preconditions of
+ // `new_unchecked`.
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of_val.html:
+ //
+ // Returns the size of the pointed-to value in bytes.
+ //
+ // ... when `T` has no statically-known size ... `size_of_val` can
+ // be used to get the dynamically-known size.
+ //
+ // [2] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.align_of_val.html:
+ //
+ // Returns the ABI-required minimum alignment of the type of the
+ // value that `val` points to.
+ //
+ // Every reference to a value of the type `T` must be a multiple of
+ // this number.
+ Ok(unsafe { Ref::new_unchecked(source, count) })
}
}
@@ -538,6 +813,24 @@ impl<B, T> Ref<B, T>
///
/// let _ = Ref::<_, ZSTy>::from_prefix_with_elems(&b"UU"[..], 42); // ⚠ Compile Error!
/// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_from_prefix_with_elems",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
#[inline]
pub fn from_prefix_with_elems(
source: B,
@@ -549,7 +842,7 @@ pub fn from_prefix_with_elems(
None => return Err(SizeError::new(source).into()),
};
let (prefix, bytes) = source.split_at(expected_len).map_err(SizeError::new)?;
- Self::from_bytes(prefix).map(move |l| (l, bytes))
+ Self::from_bytes_with_elems(prefix, count).map(move |l| (l, bytes))
}
/// Constructs a `Ref` from the suffix of the given bytes with DST length
@@ -584,6 +877,24 @@ pub fn from_prefix_with_elems(
///
/// let _ = Ref::<_, ZSTy>::from_suffix_with_elems(&b"UU"[..], 42); // ⚠ Compile Error!
/// ```
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_from_suffix_with_elems",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 2
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
#[inline]
pub fn from_suffix_with_elems(
source: B,
@@ -602,7 +913,7 @@ pub fn from_suffix_with_elems(
// SAFETY: The preceding `source.len().checked_sub(expected_len)`
// guarantees that `split_at` is in-bounds.
let (bytes, suffix) = unsafe { source.split_at_unchecked(split_at) };
- Self::from_bytes(suffix).map(move |l| (bytes, l))
+ Self::from_bytes_with_elems(suffix, count).map(move |l| (bytes, l))
}
}
@@ -618,12 +929,37 @@ impl<'a, B, T> Ref<B, T>
/// Note: this is an associated function, which means that you have to call
/// it as `Ref::into_ref(r)` instead of `r.into_ref()`. This is so that
/// there is no conflict with a method on the inner type.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_into_ref",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
#[must_use = "has no side effects"]
#[inline(always)]
pub fn into_ref(r: Self) -> &'a T {
// Presumably unreachable, since we've guarded each constructor of `Ref`.
static_assert_dst_is_not_zst!(T);
+ let metadata = r.pointer_metadata();
+
// SAFETY: We don't call any methods on `b` other than those provided by
// `IntoByteSlice`.
let b = unsafe { r.into_byte_slice() };
@@ -642,12 +978,12 @@ pub fn into_ref(r: Self) -> &'a T {
return ptr.as_ref();
}
- // PANICS: By post-condition on `into_byte_slice`, `b`'s size and
- // alignment are valid for `T`. By post-condition, `b.into_byte_slice()`
- // produces a byte slice with identical address and length to that
- // produced by `b.deref()`.
+ // PANICS: By post-condition on `into_byte_slice`, `b` is aligned for
+ // `T`, and its size is exactly the size encoded by `metadata`. By
+ // post-condition, `b.into_byte_slice()` produces a byte slice with
+ // identical address and length to that produced by `b.deref()`.
let ptr = Ptr::from_ref(b.into_byte_slice())
- .try_cast_into_no_leftover::<T, BecauseImmutable>(None)
+ .try_cast_into_no_leftover::<T, BecauseImmutable>(Some(metadata))
.expect("zerocopy internal error: into_ref should be infallible");
let ptr = ptr.recall_validity();
ptr.as_ref()
@@ -666,12 +1002,32 @@ impl<'a, B, T> Ref<B, T>
/// Note: this is an associated function, which means that you have to call
/// it as `Ref::into_mut(r)` instead of `r.into_mut()`. This is so that
/// there is no conflict with a method on the inner type.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_into_mut",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ]
+ )]
#[must_use = "has no side effects"]
#[inline(always)]
pub fn into_mut(r: Self) -> &'a mut T {
// Presumably unreachable, since we've guarded each constructor of `Ref`.
static_assert_dst_is_not_zst!(T);
+ let metadata = r.pointer_metadata();
+
// SAFETY: We don't call any methods on `b` other than those provided by
// `IntoByteSliceMut`.
let b = unsafe { r.into_byte_slice_mut() };
@@ -686,7 +1042,7 @@ pub fn into_mut(r: Self) -> &'a mut T {
T,
_,
(BecauseRead, BecauseExclusive),
- (BecauseMutationCompatible, BecauseInvariantsEq),
+ (BecauseMutationCompatible, BecauseSharedCompatible),
>(ptr)
};
@@ -697,13 +1053,13 @@ pub fn into_mut(r: Self) -> &'a mut T {
return ptr.as_mut();
}
- // PANICS: By post-condition on `into_byte_slice_mut`, `b`'s size and
- // alignment are valid for `T`. By post-condition,
- // `b.into_byte_slice_mut()` produces a byte slice with identical
- // address and length to that produced by `b.deref_mut()`.
+ // PANICS: By post-condition on `into_byte_slice_mut`, `b` is aligned
+ // for `T`, and its size is exactly the size encoded by `metadata`. By
+ // post-condition, `b.into_byte_slice_mut()` produces a byte slice with
+ // identical address and length to that produced by `b.deref_mut()`.
let ptr = Ptr::from_mut(b.into_byte_slice_mut())
- .try_cast_into_no_leftover::<T, BecauseExclusive>(None)
- .expect("zerocopy internal error: into_ref should be infallible");
+ .try_cast_into_no_leftover::<T, BecauseExclusive>(Some(metadata))
+ .expect("zerocopy internal error: into_mut should be infallible");
let ptr = ptr.recall_validity::<_, (_, (_, _))>();
ptr.as_mut()
}
@@ -719,6 +1075,29 @@ impl<B, T> Ref<B, T>
/// Note: this is an associated function, which means that you have to call
/// it as `Ref::bytes(r)` instead of `r.bytes()`. This is so that there is
/// no conflict with a method on the inner type.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_bytes",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
#[inline]
pub fn bytes(r: &Self) -> &[u8] {
// SAFETY: We don't call any methods on `b` other than those provided by
@@ -737,6 +1116,29 @@ impl<B, T> Ref<B, T>
/// Note: this is an associated function, which means that you have to call
/// it as `Ref::bytes_mut(r)` instead of `r.bytes_mut()`. This is so that
/// there is no conflict with a method on the inner type.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_bytes_mut",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
#[inline]
pub fn bytes_mut(r: &mut Self) -> &mut [u8] {
// SAFETY: We don't call any methods on `b` other than those provided by
@@ -755,6 +1157,12 @@ impl<B, T> Ref<B, T>
/// Note: this is an associated function, which means that you have to call
/// it as `Ref::read(r)` instead of `r.read()`. This is so that there is no
/// conflict with a method on the inner type.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_read_static_size",
+ format = "coco_static_size",
+ )]
#[must_use = "has no side effects"]
#[inline]
pub fn read(r: &Self) -> T {
@@ -780,14 +1188,20 @@ impl<B, T> Ref<B, T>
/// Note: this is an associated function, which means that you have to call
/// it as `Ref::write(r, t)` instead of `r.write(t)`. This is so that there
/// is no conflict with a method on the inner type.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_write_static_size",
+ format = "coco_static_size",
+ )]
#[inline]
pub fn write(r: &mut Self, t: T) {
// SAFETY: We don't call any methods on `b` other than those provided by
// `ByteSliceMut`.
let b = unsafe { r.as_byte_slice_mut() };
- // SAFETY: By postcondition on `as_byte_slice_mut`, we know that `b` is
- // a valid size and alignment for `T`. By safety invariant on
+ // SAFETY: By postcondition on `as_byte_slice_mut`, we know that `b` is a
+ // valid size and alignment for `T`. By safety invariant on
// `ByteSlice`, we know that this is preserved via `.deref()`. Writing
// `t` to the buffer will allow all of the bytes of `t` to be accessed
// as a `[u8]`, but because `T: IntoBytes`, we know that this is sound.
@@ -801,11 +1215,38 @@ impl<B, T> Deref for Ref<B, T>
T: FromBytes + KnownLayout + Immutable + ?Sized,
{
type Target = T;
+
+ /// Dereferences the value.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_deref",
+ format = "coco",
+ arity = 3,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ],
+ [
+ @index 3
+ @title "Dynamically Padded"
+ @variant "dynamic_padding"
+ ]
+ )]
#[inline]
fn deref(&self) -> &T {
// Presumably unreachable, since we've guarded each constructor of `Ref`.
static_assert_dst_is_not_zst!(T);
+ let metadata = self.pointer_metadata();
+
// SAFETY: We don't call any methods on `b` other than those provided by
// `ByteSlice`.
let b = unsafe { self.as_byte_slice() };
@@ -824,11 +1265,12 @@ fn deref(&self) -> &T {
return ptr.as_ref();
}
- // PANICS: By postcondition on `as_byte_slice`, `b`'s size and alignment
- // are valid for `T`, and by invariant on `ByteSlice`, these are
- // preserved through `.deref()`, so this `unwrap` will not panic.
+ // PANICS: By postcondition on `as_byte_slice`, `b` is aligned for `T`,
+ // and its size is exactly the size encoded by `metadata`. By invariant
+ // on `ByteSlice`, these are preserved through `.deref()`, so this
+ // `unwrap` will not panic.
let ptr = Ptr::from_ref(b)
- .try_cast_into_no_leftover::<T, BecauseImmutable>(None)
+ .try_cast_into_no_leftover::<T, BecauseImmutable>(Some(metadata))
.expect("zerocopy internal error: Deref::deref should be infallible");
let ptr = ptr.recall_validity();
ptr.as_ref()
@@ -843,11 +1285,32 @@ impl<B, T> DerefMut for Ref<B, T>
// add a separate inherent method for this?
T: FromBytes + IntoBytes + KnownLayout + Immutable + ?Sized,
{
+ /// Mutably dereferences the value.
+ ///
+ #[doc = codegen_section!(
+ header = "h5",
+ bench = "ref_type_deref_mut",
+ format = "coco",
+ arity = 2,
+ [
+ open
+ @index 1
+ @title "Sized"
+ @variant "static_size"
+ ],
+ [
+ @index 2
+ @title "Unsized"
+ @variant "dynamic_size"
+ ]
+ )]
#[inline]
fn deref_mut(&mut self) -> &mut T {
// Presumably unreachable, since we've guarded each constructor of `Ref`.
static_assert_dst_is_not_zst!(T);
+ let metadata = self.pointer_metadata();
+
// SAFETY: We don't call any methods on `b` other than those provided by
// `ByteSliceMut`.
let b = unsafe { self.as_byte_slice_mut() };
@@ -862,7 +1325,7 @@ fn deref_mut(&mut self) -> &mut T {
T,
_,
(BecauseRead, BecauseExclusive),
- (BecauseMutationCompatible, BecauseInvariantsEq),
+ (BecauseMutationCompatible, BecauseSharedCompatible),
>(ptr)
};
@@ -873,12 +1336,12 @@ fn deref_mut(&mut self) -> &mut T {
return ptr.as_mut();
}
- // PANICS: By postcondition on `as_byte_slice_mut`, `b`'s size and
- // alignment are valid for `T`, and by invariant on `ByteSlice`, these
- // are preserved through `.deref_mut()`, so this `unwrap` will not
- // panic.
+ // PANICS: By postcondition on `as_byte_slice_mut`, `b` is aligned for
+ // `T`, and its size is exactly the size encoded by `metadata`. By
+ // invariant on `ByteSlice`, these are preserved through `.deref_mut()`,
+ // so this `unwrap` will not panic.
let ptr = Ptr::from_mut(b)
- .try_cast_into_no_leftover::<T, BecauseExclusive>(None)
+ .try_cast_into_no_leftover::<T, BecauseExclusive>(Some(metadata))
.expect("zerocopy internal error: DerefMut::deref_mut should be infallible");
let ptr = ptr.recall_validity::<_, (_, (_, BecauseExclusive))>();
ptr.as_mut()
@@ -958,13 +1421,13 @@ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
/// `T: Sized` and `ptr`'s referent must have size `size_of::<T>()`.
#[inline(always)]
unsafe fn cast_for_sized<'a, T, A, R, S>(
- ptr: Ptr<'a, [u8], (A, Aligned, Valid)>,
-) -> Ptr<'a, T, (A, Unaligned, Valid)>
+ ptr: Ptr<'a, [u8], (A, Aligned, Safe)>,
+) -> Ptr<'a, T, (A, Unaligned, Safe)>
where
T: FromBytes + KnownLayout + ?Sized,
A: crate::invariant::Aliasing,
[u8]: MutationCompatible<T, A, Initialized, Initialized, R>,
- T: TransmuteFromPtr<T, A, Initialized, Valid, crate::pointer::cast::IdCast, S>,
+ T: TransmuteFromPtr<T, A, Initialized, Safe, crate::pointer::cast::IdCast, S>,
{
use crate::pointer::cast::{Cast, Project};
@@ -990,7 +1453,7 @@ unsafe impl<T: ?Sized + KnownLayout> Cast<[u8], T> for CastForSized {}
ptr.recall_validity::<Initialized, (_, (_, _))>()
.cast::<_, CastForSized, _>()
- .recall_validity::<Valid, _>()
+ .recall_validity::<Safe, _>()
}
#[cfg(test)]
@@ -1028,6 +1491,66 @@ fn test_address() {
assert_eq!(buf_ptr, deref_ptr);
}
+ #[test]
+ #[allow(clippy::clone_on_copy)] // Exercise `Ref`'s manual `Clone` impl.
+ fn test_with_elems_preserves_ambiguous_metadata() {
+ #[derive(FromBytes, Immutable, KnownLayout)]
+ #[repr(C, align(8))]
+ struct Frame {
+ header: [u8; 8],
+ body: [u8],
+ }
+
+ #[derive(FromBytes, Immutable, KnownLayout)]
+ #[repr(C, align(8))]
+ struct OddHeader {
+ header: [u8; 7],
+ body: [u8],
+ }
+
+ // Dynamic trailing padding makes these pairs of distinct element
+ // counts produce the same total byte size. Thus, the byte length alone
+ // cannot recover the count supplied to a `*_with_elems` constructor.
+ assert_eq!(Frame::size_for_metadata(1), Some(16));
+ assert_eq!(Frame::size_for_metadata(8), Some(16));
+ assert_eq!(OddHeader::size_for_metadata(0), Some(8));
+ assert_eq!(OddHeader::size_for_metadata(1), Some(8));
+
+ let bytes = Align::<[u8; 32], AU64>::default();
+
+ let whole = Ref::<_, Frame>::from_bytes_with_elems(&bytes.t[..16], 1).unwrap();
+ assert_eq!(whole.body.len(), 1);
+ assert_eq!(Ref::bytes(&whole).len(), 16);
+ assert_eq!(whole.clone().body.len(), 1);
+ let copied = whole;
+ assert_eq!(copied.body.len(), 1);
+ assert_eq!(Ref::into_ref(whole).body.len(), 1);
+
+ let (prefix, rest) = Ref::<_, Frame>::from_prefix_with_elems(&bytes.t[..], 1).unwrap();
+ assert_eq!(prefix.body.len(), 1);
+ assert_eq!(rest.len(), 16);
+
+ let (before, suffix) = Ref::<_, Frame>::from_suffix_with_elems(&bytes.t[..24], 1).unwrap();
+ assert_eq!(before.len(), 8);
+ assert_eq!(suffix.body.len(), 1);
+
+ // Zero is real metadata, not a sentinel for "infer from the bytes".
+ let empty = Ref::<_, OddHeader>::from_bytes_with_elems(&bytes.t[..8], 0).unwrap();
+ assert!(empty.body.is_empty());
+ assert!(Ref::into_ref(empty).body.is_empty());
+
+ // The implicit constructors still choose the largest fitting count,
+ // and retain the metadata selected during their validating cast.
+ let inferred = Ref::<_, Frame>::from_bytes(&bytes.t[..16]).unwrap();
+ assert_eq!(inferred.body.len(), 8);
+ let (inferred, rest) = Ref::<_, Frame>::from_prefix(&bytes.t[..16]).unwrap();
+ assert_eq!(inferred.body.len(), 8);
+ assert!(rest.is_empty());
+ let (before, inferred) = Ref::<_, Frame>::from_suffix(&bytes.t[..16]).unwrap();
+ assert!(before.is_empty());
+ assert_eq!(inferred.body.len(), 8);
+ }
+
// Verify that values written to a `Ref` are properly shared between the
// typed and untyped representations, that reads via `deref` and `read`
// behave the same, and that writes via `deref_mut` and `write` behave the
@@ -1251,6 +1774,13 @@ fn test_into_ref_mut() {
*rf = u64::MAX;
assert_eq!(buf.t, [0xFF; 8]);
+
+ // Exercise the unsized `into_mut` route, which must use the metadata
+ // retained by the constructor rather than infer fresh metadata.
+ let mut buf = Align::<[u8; 8], u16>::default();
+ let r = Ref::<_, [u16]>::from_bytes_with_elems(&mut buf.t[..], 4).unwrap();
+ let rf = Ref::into_mut(r);
+ assert_eq!(rf.len(), 4);
}
#[test]
@@ -1325,8 +1855,13 @@ fn bench_into_mut_sized(b: &mut Bencher) {
let _ = Ref::<_, AU64>::from_bytes(&mut *buf).unwrap();
b.iter(move || {
// SAFETY: The preceding `from_bytes` succeeded, and so we know that
- // `buf` is validly-aligned and has the correct length.
- let r = unsafe { Ref::<&mut [u8], AU64>::new_unchecked(&mut *buf) };
+ // `buf` is validly aligned and has the `size_of::<AU64>()` byte
+ // length required by `new_sized_unchecked` [1].
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of.html:
+ //
+ // Returns the size of a type in bytes.
+ let r = unsafe { Ref::<&mut [u8], AU64>::new_sized_unchecked(&mut *buf) };
test::black_box(Ref::into_mut(test::black_box(r)));
});
}
@@ -1349,8 +1884,13 @@ fn bench_deref_mut_sized(b: &mut Bencher) {
let _ = Ref::<_, AU64>::from_bytes(&mut *buf).unwrap();
b.iter(|| {
// SAFETY: The preceding `from_bytes` succeeded, and so we know that
- // `buf` is validly-aligned and has the correct length.
- let r = unsafe { Ref::<&mut [u8], AU64>::new_unchecked(&mut *buf) };
+ // `buf` is validly aligned and has the `size_of::<AU64>()` byte
+ // length required by `new_sized_unchecked` [1].
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/core/mem/fn.size_of.html:
+ //
+ // Returns the size of a type in bytes.
+ let r = unsafe { Ref::<&mut [u8], AU64>::new_sized_unchecked(&mut *buf) };
let mut temp = test::black_box(r);
test::black_box(temp.deref_mut());
});
diff --git a/rust/zerocopy/src/split_at.rs b/rust/zerocopy/src/split_at.rs
index d7778425a31d..4aeb602f69e0 100644
--- a/rust/zerocopy/src/split_at.rs
+++ b/rust/zerocopy/src/split_at.rs
@@ -10,13 +10,19 @@
// those terms.
use super::*;
-use crate::pointer::invariant::{Aligned, Exclusive, Invariants, Shared, Valid};
+use crate::pointer::invariant::{Aligned, Exclusive, Invariants, Safe, Shared};
/// Types that can be split in two.
///
/// This trait generalizes Rust's existing support for splitting slices to
/// support slices and slice-based dynamically-sized types ("slice DSTs").
///
+/// For a `repr(C, packed)` or `repr(C, packed(N))` struct, the derive wraps
+/// the trailing field's [`Elem`](Self::Elem) in [`Unalign`]. The returned
+/// slice can therefore be accessed even when packing misaligns the original
+/// elements. Each packed struct in the trailing-field chain adds one wrapper;
+/// unpacked structs and [`ManuallyDrop`] forward their trailing type's `Elem`.
+///
/// # Implementation
///
/// **Do not implement this trait yourself!** Instead, use
@@ -38,6 +44,28 @@
/// This derive performs a sophisticated, compile-time safety analysis to
/// determine whether a type is `SplitAt`.
///
+/// A packed struct exposes its trailing elements through `Unalign`:
+///
+/// ```
+/// use zerocopy::{FromBytes, SplitAt, Unalign};
+/// # use zerocopy_derive::*;
+///
+/// #[derive(FromBytes, IntoBytes, KnownLayout, SplitAt)]
+/// #[repr(C, packed)]
+/// struct Packet {
+/// tag: u8,
+/// words: [u16],
+/// }
+///
+/// let mut bytes = [0u8; 5];
+/// let packet = Packet::mut_from_bytes(&mut bytes[..]).unwrap();
+/// let (left, right): (&mut Packet, &mut [Unalign<u16>]) =
+/// packet.split_at_mut(1).unwrap().via_into_bytes();
+/// left.tag = 1;
+/// right[0] = Unalign::new(0x0202);
+/// assert_eq!(bytes, [1, 0, 0, 2, 2]);
+/// ```
+///
/// # Safety
///
/// This trait does not convey any safety guarantees to code outside this crate.
@@ -58,10 +86,20 @@
)]
// # Safety
//
-// The trailing slice is well-aligned for its element type. `Self` is `[T]`, or
-// a `repr(C)` or `repr(transparent)` slice DST.
+// `Self` is a slice, a `repr(C)` (possibly packed) or `repr(transparent)` slice
+// DST, or `ManuallyDrop<T>` for some `T: SplitAt`.
+//
+// `Self::Elem` has the same size, bit validity, and `UnsafeCell` coverage as
+// the actual trailing element type. Access through either representation
+// preserves the other's validity, including writes through mutable references
+// and interior mutation through shared references. For every aligned `Self`,
+// its trailing slice's address is aligned for `Self::Elem`.
pub unsafe trait SplitAt: KnownLayout<PointerMetadata = usize> {
- /// The element type of the trailing slice.
+ /// The element type exposed by the split's returned slice.
+ ///
+ /// For `[T]`, this is `T`. For a struct, this is the trailing field's `Elem`,
+ /// wrapped in [`Unalign`] if the struct is packed. Nested packed structs
+ /// produce nested `Unalign` wrappers.
type Elem;
#[doc(hidden)]
@@ -188,10 +226,7 @@ unsafe fn split_at_mut_unchecked(&mut self, l_len: usize) -> Split<&mut Self> {
/// Attempts to split `self` in two.
///
/// Returns `None` if `l_len` is greater than the length of `self`'s
- /// trailing slice, or if the given `l_len` would result in [the trailing
- /// padding](KnownLayout#slice-dst-layout) of the left portion overlapping
- /// the right portion.
- ///
+ /// trailing slice.
///
/// # Examples
///
@@ -249,7 +284,8 @@ fn split_at_mut(&mut self, l_len: usize) -> Option<Split<&mut Self>> {
}
}
-// SAFETY: `[T]`'s trailing slice is `[T]`, which is trivially aligned.
+// SAFETY: `[T]` exposes its own element type, preserving size, validity, and
+// `UnsafeCell` coverage. An aligned slice is aligned for its elements.
unsafe impl<T> SplitAt for [T] {
type Elem = T;
@@ -262,6 +298,29 @@ fn only_derive_is_allowed_to_implement_this_trait()
}
}
+// SAFETY: `ManuallyDrop<T>` has the same layout and bit validity as `T` [1].
+// Its `KnownLayout` implementation preserves `T`'s metadata and trailing-slice
+// layout. Forwarding `T::Elem` therefore preserves its size, validity, and
+// alignment guarantees. Its `UnsafeCell` coverage also matches `T`'s, as
+// established for `ManuallyDrop` in `impls.rs`; accesses through the forwarded
+// element representation preserve validity in both directions.
+//
+// [1] Per https://doc.rust-lang.org/1.93.0/std/mem/struct.ManuallyDrop.html:
+//
+// `ManuallyDrop<T>` is guaranteed to have the same layout and bit validity as
+// `T`
+unsafe impl<T: ?Sized + SplitAt> SplitAt for ManuallyDrop<T> {
+ type Elem = T::Elem;
+
+ #[inline]
+ #[allow(dead_code)]
+ fn only_derive_is_allowed_to_implement_this_trait()
+ where
+ Self: Sized,
+ {
+ }
+}
+
/// A `T` that has been split into two possibly-overlapping parts.
///
/// For some dynamically sized types, the padding that appears after the
@@ -270,10 +329,11 @@ fn only_derive_is_allowed_to_implement_this_trait()
/// requires trailing padding, the trailing padding of the left part of the
/// split `T` will overlap the right part. If `T` is a mutable reference or
/// permits interior mutation, you must ensure that the left and right parts do
-/// not overlap. You can do this at zero-cost using using
-/// [`Self::via_immutable`], [`Self::via_into_bytes`], or
-/// [`Self::via_unaligned`], or with a dynamic check by using
-/// [`Self::via_runtime_check`].
+/// not overlap. Use [`Self::via_into_bytes`] or
+/// [`Self::via_no_dynamic_padding`] to establish this without a runtime check,
+/// or [`Self::via_runtime_check`] to check a particular split. Shared references
+/// to an [`Immutable`] type may overlap; use [`Self::via_immutable`] in that
+/// case.
#[derive(Debug)]
pub struct Split<T> {
/// A pointer to the source slice DST.
@@ -304,7 +364,7 @@ impl<'a, T> Split<&'a T>
T: ?Sized + SplitAt,
{
#[inline(always)]
- fn into_ptr(self) -> Split<Ptr<'a, T, (Shared, Aligned, Valid)>> {
+ fn into_ptr(self) -> Split<Ptr<'a, T, (Shared, Aligned, Safe)>> {
let source = Ptr::from_ref(self.source);
// SAFETY: `Ptr::from_ref(self.source)` points to exactly `self.source`
// and thus maintains the invariants of `self` with respect to `l_len`.
@@ -423,59 +483,100 @@ pub fn via_into_bytes(self) -> (&'a T, &'a [T::Elem])
(l.as_ref(), r.as_ref())
}
- /// Produces the split parts of `self`, using [`Unaligned`] to ensure that
- /// it is sound to have concurrent references to both parts.
+ /// Produces the split parts of `self` using [`Self::via_no_dynamic_padding`].
+ #[deprecated(note = "use `Split::via_no_dynamic_padding` instead")]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn via_unaligned(self) -> (&'a T, &'a [T::Elem])
+ where
+ T: Unaligned,
+ {
+ self.via_no_dynamic_padding()
+ }
+
+ /// Produces the split parts of `self` using a compile-time assertion to
+ /// ensure that it is unconditionally sound to have concurrent references to
+ /// both parts.
+ ///
+ /// When possible, you should prefer using [`Self::via_into_bytes`], which
+ /// ensures this safety property through regular trait bounds.
///
/// # Examples
///
+ /// In the below example, a single byte of padding exists (on most
+ /// platforms) between `Packet<[u16]>`'s `length` and `body` fields,
+ /// precluding the use of [`Self::via_into_bytes`].
+ ///
/// ```
- /// use zerocopy::{SplitAt, FromBytes};
+ /// use zerocopy::SplitAt;
/// # use zerocopy_derive::*;
///
- /// #[derive(SplitAt, FromBytes, KnownLayout, Immutable, Unaligned)]
+ /// #[derive(SplitAt, KnownLayout)]
/// #[repr(C)]
- /// struct Packet {
+ /// struct Packet<B: ?Sized> {
/// length: u8,
- /// body: [u8],
+ /// body: B,
/// }
///
- /// // These bytes encode a `Packet`.
- /// let bytes = &[4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
- ///
- /// let packet = Packet::ref_from_bytes(bytes).unwrap();
+ /// let packet: &Packet<[u16]> = &Packet { length: 4, body: [1u16, 2, 3, 4, 5, 6, 7, 8, 9] };
///
/// assert_eq!(packet.length, 4);
/// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
///
- /// // Attempt to split `packet` at `length`.
- /// let split = packet.split_at(packet.length as usize).unwrap();
+ /// if let Some(split) = packet.split_at(packet.length as usize) {
+ /// // Every body length requires no trailing padding.
+ /// let (packet, rest) = split.via_no_dynamic_padding();
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4]);
+ /// assert_eq!(rest, [5, 6, 7, 8, 9]);
+ /// } else {
+ /// unreachable!("The packet's length field is within its body");
+ /// }
+ /// ```
///
- /// // Use the `Unaligned` bound on `Packet` to prove that it's okay to
- /// // return concurrent references to `packet` and `rest`.
- /// let (packet, rest) = split.via_unaligned();
+ /// # Compile-Time Assertions
///
- /// assert_eq!(packet.length, 4);
- /// assert_eq!(packet.body, [1, 2, 3, 4]);
- /// assert_eq!(rest, [5, 6, 7, 8, 9]);
+ /// This method rejects at compile-time any type that admits dynamic
+ /// trailing padding. In the below example, some `split_at` indices would
+ /// produce left parts whose trailing padding overlaps with the right part:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::SplitAt;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(SplitAt, KnownLayout)]
+ /// #[repr(C, align(2))]
+ /// struct Packet<B: ?Sized> {
+ /// length: u16,
+ /// body: B,
+ /// }
+ ///
+ /// let packet: &Packet<[u8]> = &Packet { length: 4, body: [1u8, 2, 3, 4, 5, 6, 7, 8, 9] };
+ ///
+ /// if let Some(split) = packet.split_at(packet.length as usize) {
+ /// let _ = split.via_no_dynamic_padding(); // ⚠ Compile Error!
+ /// } else {
+ /// unreachable!("The packet's length field is within its body");
+ /// }
/// ```
///
- #[doc = codegen_header!("h5", "split_via_unaligned")]
+ /// If you need to split such types, use [`Self::via_runtime_check`].
+ ///
+ #[doc = codegen_header!("h5", "split_via_no_dynamic_padding")]
///
/// See [`Split::via_immutable`](#method.split_via_immutable.codegen).
#[must_use = "has no side effects"]
#[inline(always)]
- pub fn via_unaligned(self) -> (&'a T, &'a [T::Elem])
- where
- T: Unaligned,
- {
- let (l, r) = self.into_ptr().via_unaligned();
+ pub fn via_no_dynamic_padding(self) -> (&'a T, &'a [T::Elem]) {
+ let (l, r) = self.into_ptr().via_no_dynamic_padding();
(l.as_ref(), r.as_ref())
}
/// Produces the split parts of `self`, using a dynamic check to ensure that
/// it is sound to have concurrent references to both parts. You should
/// prefer using [`Self::via_immutable`], [`Self::via_into_bytes`], or
- /// [`Self::via_unaligned`], which have no runtime cost.
+ /// [`Self::via_no_dynamic_padding`], which perform no runtime check.
///
/// Note that this check is overly conservative if `T` is [`Immutable`]; for
/// some types, this check will reject some splits which
@@ -484,52 +585,40 @@ pub fn via_unaligned(self) -> (&'a T, &'a [T::Elem])
/// # Examples
///
/// ```
- /// use zerocopy::{SplitAt, FromBytes, IntoBytes, network_endian::U16};
+ /// use zerocopy::SplitAt;
/// # use zerocopy_derive::*;
///
- /// #[derive(SplitAt, FromBytes, KnownLayout, Immutable, Debug)]
+ /// #[derive(SplitAt, KnownLayout)]
/// #[repr(C, align(2))]
- /// struct Packet {
- /// length: U16,
- /// body: [u8],
+ /// struct Packet<B: ?Sized> {
+ /// length: u16,
+ /// body: B,
/// }
///
- /// // These bytes encode a `Packet`.
- /// let bytes = [
- /// 4u16.to_be(),
- /// 1u16.to_be(),
- /// 2u16.to_be(),
- /// 3u16.to_be(),
- /// 4u16.to_be()
- /// ];
- ///
- /// let packet = Packet::ref_from_bytes(bytes.as_bytes()).unwrap();
+ /// let packet: &Packet<[u8]> = &Packet { length: 4, body: [1u8, 2, 3, 4, 5, 6, 7, 8, 9] };
///
/// assert_eq!(packet.length, 4);
- /// assert_eq!(packet.body, [0, 1, 0, 2, 0, 3, 0, 4]);
- ///
- /// // Attempt to split `packet` at `length`.
- /// let split = packet.split_at(packet.length.into()).unwrap();
+ /// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
///
- /// // Use a dynamic check to prove that it's okay to return concurrent
- /// // references to `packet` and `rest`.
- /// let (packet, rest) = split.via_runtime_check().unwrap();
+ /// // The two-byte prefix and four-byte body need no trailing padding.
+ /// if let Some(split) = packet.split_at(packet.length as usize) {
+ /// if let Ok((packet, rest)) = split.via_runtime_check() {
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4]);
+ /// assert_eq!(rest, [5, 6, 7, 8, 9]);
+ /// } else {
+ /// unreachable!("A four-byte body requires no trailing padding");
+ /// }
+ /// } else {
+ /// unreachable!("The packet's length field is within its body");
+ /// }
///
- /// assert_eq!(packet.length, 4);
- /// assert_eq!(packet.body, [0, 1, 0, 2]);
- /// assert_eq!(rest, [0, 3, 0, 4]);
- ///
- /// // Attempt to split `packet` at `length - 1`.
- /// let idx = packet.length.get() - 1;
- /// let split = packet.split_at(idx as usize).unwrap();
- ///
- /// // Attempt (and fail) to use a dynamic check to prove that it's okay
- /// // to return concurrent references to `packet` and `rest`. Note that
- /// // this is a case of `via_runtime_check` being overly conservative.
- /// // Although the left and right parts indeed overlap, the `Immutable`
- /// // bound ensures that concurrently referencing these overlapping
- /// // parts is sound.
- /// assert!(split.via_runtime_check().is_err());
+ /// // A three-byte body needs a padding byte that would overlap `rest`.
+ /// if let Some(split) = packet.split_at((packet.length - 1) as usize) {
+ /// assert!(split.via_runtime_check().is_err());
+ /// } else {
+ /// unreachable!("One less than the packet's length is within its body");
+ /// }
/// ```
///
#[doc = codegen_section!(
@@ -602,7 +691,7 @@ impl<'a, T> Split<&'a mut T>
T: ?Sized + SplitAt,
{
#[inline(always)]
- fn into_ptr(self) -> Split<Ptr<'a, T, (Exclusive, Aligned, Valid)>> {
+ fn into_ptr(self) -> Split<Ptr<'a, T, (Exclusive, Aligned, Safe)>> {
let source = Ptr::from_mut(self.source);
// SAFETY: `Ptr::from_mut(self.source)` points to exactly `self.source`,
// and thus maintains the invariants of `self` with respect to `l_len`.
@@ -665,101 +754,135 @@ pub fn via_into_bytes(self) -> (&'a mut T, &'a mut [T::Elem])
(l.as_mut(), r.as_mut())
}
- /// Produces the split parts of `self`, using [`Unaligned`] to ensure that
- /// it is sound to have concurrent references to both parts.
+ /// Produces the split parts of `self` using [`Self::via_no_dynamic_padding`].
+ #[deprecated(note = "use `Split::via_no_dynamic_padding` instead")]
+ #[doc(hidden)]
+ #[must_use = "has no side effects"]
+ #[inline(always)]
+ pub fn via_unaligned(self) -> (&'a mut T, &'a mut [T::Elem])
+ where
+ T: Unaligned,
+ {
+ self.via_no_dynamic_padding()
+ }
+
+ /// Produces the split parts of `self` using a compile-time assertion to
+ /// ensure that it is unconditionally sound to have concurrent references to
+ /// both parts.
+ ///
+ /// When possible, you should prefer using [`Self::via_into_bytes`], which
+ /// ensures this safety property through regular trait bounds.
///
/// # Examples
///
+ /// In the below example, a single byte of padding exists (on most
+ /// platforms) between `Packet<[u16]>`'s `length` and `body` fields,
+ /// precluding the use of [`Self::via_into_bytes`].
+ ///
/// ```
- /// use zerocopy::{SplitAt, FromBytes};
+ /// use zerocopy::SplitAt;
/// # use zerocopy_derive::*;
///
- /// #[derive(SplitAt, FromBytes, KnownLayout, IntoBytes, Unaligned)]
+ /// #[derive(SplitAt, KnownLayout)]
/// #[repr(C)]
/// struct Packet<B: ?Sized> {
/// length: u8,
/// body: B,
/// }
///
- /// // These bytes encode a `Packet`.
- /// let mut bytes = &mut [4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
- ///
- /// let packet = Packet::<[u8]>::mut_from_bytes(bytes).unwrap();
+ /// let packet: &mut Packet<[u16]> = &mut Packet { length: 4, body: [1u16, 2, 3, 4, 5, 6, 7, 8, 9] };
///
/// assert_eq!(packet.length, 4);
/// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
///
- /// {
- /// // Attempt to split `packet` at `length`.
- /// let split = packet.split_at_mut(packet.length as usize).unwrap();
- ///
- /// // Use the `Unaligned` bound on `Packet` to prove that it's okay to
- /// // return concurrent references to `packet` and `rest`.
- /// let (packet, rest) = split.via_unaligned();
- ///
+ /// // Every body length requires no trailing padding.
+ /// if let Some(split) = packet.split_at_mut(packet.length as usize) {
+ /// let (packet, rest) = split.via_no_dynamic_padding();
/// assert_eq!(packet.length, 4);
/// assert_eq!(packet.body, [1, 2, 3, 4]);
/// assert_eq!(rest, [5, 6, 7, 8, 9]);
- ///
/// rest.fill(0);
+ /// } else {
+ /// unreachable!("The packet's length field is within its body");
/// }
///
/// assert_eq!(packet.length, 4);
/// assert_eq!(packet.body, [1, 2, 3, 4, 0, 0, 0, 0, 0]);
/// ```
///
+ /// # Compile-Time Assertions
+ ///
+ /// This method rejects at compile-time any type that admits dynamic
+ /// trailing padding. In the below example, some `split_at` indices would
+ /// produce left parts whose trailing padding overlaps with the right part:
+ ///
+ /// ```compile_fail,E0080
+ /// use zerocopy::SplitAt;
+ /// # use zerocopy_derive::*;
+ ///
+ /// #[derive(SplitAt, KnownLayout)]
+ /// #[repr(C, align(2))]
+ /// struct Packet<B: ?Sized> {
+ /// length: u16,
+ /// body: B,
+ /// }
+ ///
+ /// let packet: &mut Packet<[u8]> = &mut Packet { length: 4, body: [1u8, 2, 3, 4, 5, 6, 7, 8, 9] };
+ ///
+ /// if let Some(split) = packet.split_at_mut(packet.length as usize) {
+ /// let _ = split.via_no_dynamic_padding(); // ⚠ Compile Error!
+ /// } else {
+ /// unreachable!("The packet's length field is within its body");
+ /// }
+ /// ```
+ ///
+ /// If you need to split such types, use [`Self::via_runtime_check`].
+ ///
/// # Code Generation
///
/// See [`Split::via_immutable`](#method.split_via_immutable.codegen).
#[must_use = "has no side effects"]
#[inline(always)]
- pub fn via_unaligned(self) -> (&'a mut T, &'a mut [T::Elem])
- where
- T: Unaligned,
- {
- let (l, r) = self.into_ptr().via_unaligned();
+ pub fn via_no_dynamic_padding(self) -> (&'a mut T, &'a mut [T::Elem]) {
+ let (l, r) = self.into_ptr().via_no_dynamic_padding();
(l.as_mut(), r.as_mut())
}
/// Produces the split parts of `self`, using a dynamic check to ensure that
/// it is sound to have concurrent references to both parts. You should
- /// prefer using [`Self::via_into_bytes`] or [`Self::via_unaligned`], which
- /// have no runtime cost.
+ /// prefer using [`Self::via_into_bytes`] or
+ /// [`Self::via_no_dynamic_padding`], which perform no runtime check.
///
/// # Examples
///
/// ```
- /// use zerocopy::{SplitAt, FromBytes};
+ /// use zerocopy::SplitAt;
/// # use zerocopy_derive::*;
///
- /// #[derive(SplitAt, FromBytes, KnownLayout, IntoBytes, Debug)]
- /// #[repr(C)]
+ /// #[derive(SplitAt, KnownLayout)]
+ /// #[repr(C, align(2))]
/// struct Packet<B: ?Sized> {
- /// length: u8,
+ /// length: u16,
/// body: B,
/// }
///
- /// // These bytes encode a `Packet`.
- /// let mut bytes = &mut [4, 1, 2, 3, 4, 5, 6, 7, 8, 9][..];
- ///
- /// let packet = Packet::<[u8]>::mut_from_bytes(bytes).unwrap();
+ /// let packet: &mut Packet<[u8]> = &mut Packet { length: 4, body: [1u8, 2, 3, 4, 5, 6, 7, 8, 9] };
///
/// assert_eq!(packet.length, 4);
/// assert_eq!(packet.body, [1, 2, 3, 4, 5, 6, 7, 8, 9]);
///
- /// {
- /// // Attempt to split `packet` at `length`.
- /// let split = packet.split_at_mut(packet.length as usize).unwrap();
- ///
- /// // Use a dynamic check to prove that it's okay to return concurrent
- /// // references to `packet` and `rest`.
- /// let (packet, rest) = split.via_runtime_check().unwrap();
- ///
- /// assert_eq!(packet.length, 4);
- /// assert_eq!(packet.body, [1, 2, 3, 4]);
- /// assert_eq!(rest, [5, 6, 7, 8, 9]);
- ///
- /// rest.fill(0);
+ /// // The two-byte prefix and four-byte body need no trailing padding.
+ /// if let Some(split) = packet.split_at_mut(packet.length as usize) {
+ /// if let Ok((packet, rest)) = split.via_runtime_check() {
+ /// assert_eq!(packet.length, 4);
+ /// assert_eq!(packet.body, [1, 2, 3, 4]);
+ /// assert_eq!(rest, [5, 6, 7, 8, 9]);
+ /// rest.fill(0);
+ /// } else {
+ /// unreachable!("A four-byte body requires no trailing padding");
+ /// }
+ /// } else {
+ /// unreachable!("The packet's length field is within its body");
/// }
///
/// assert_eq!(packet.length, 4);
@@ -805,7 +928,7 @@ pub unsafe fn via_unchecked(self) -> (&'a mut T, &'a mut [T::Elem]) {
impl<'a, T, I> Split<Ptr<'a, T, I>>
where
T: ?Sized + SplitAt,
- I: Invariants<Alignment = Aligned, Validity = Valid>,
+ I: Invariants<Alignment = Aligned, Validity = Safe>,
{
fn into_ref(self) -> Split<&'a T>
where
@@ -860,37 +983,57 @@ fn via_into_bytes(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>)
unsafe { self.via_unchecked() }
}
- /// Produces the split parts of `self`, using [`Unaligned`] to ensure that
- /// it is sound to have concurrent references to both parts.
+ /// Produces the split parts of `self` after the conservative layout
+ /// predicate establishes that `T` never requires dynamic trailing padding.
#[inline(always)]
- fn via_unaligned(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>)
- where
- T: Unaligned,
- {
- // SAFETY: By `T: SplitAt + Unaligned`, `T` is either a slice or a
- // `repr(C)` or `repr(transparent)` slice DST that is well-aligned at
- // any address and length. If `T` is a slice DST with alignment 1,
- // `repr(C)` or `repr(transparent)` ensures that no padding is placed
- // after the final element of the trailing slice. Consequently, `T` can
- // be split into strictly non-overlapping parts any any index.
+ fn via_no_dynamic_padding(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>) {
+ static_assert!(
+ T: ?Sized + KnownLayout => !T::LAYOUT.requires_dynamic_padding(),
+ "`Split::via_no_dynamic_padding` cannot be used with a type whose layout may require dynamic trailing padding; use `Split::via_runtime_check` instead"
+ );
+
+ // SAFETY: The assertion establishes that `requires_dynamic_padding()`
+ // is false. By that method's guarantee, every valid metadata—including
+ // `self.l_len()`, which is valid by `Split`'s invariant—describes an
+ // object with no trailing padding. By the postcondition of
+ // `PtrInner::split_at_unchecked`, the two returned referents are then
+ // contiguous and non-overlapping. Since the source already conforms to
+ // `I::Aliasing`, splitting it into disjoint byte ranges preserves that
+ // invariant: under `Exclusive`, neither result aliases or accesses the
+ // other's bytes; under `Shared`, interior mutation through one result
+ // cannot mutate bytes pointed to by the other. This satisfies Rust's
+ // reference aliasing rules [1].
+ //
+ // [1] Per https://doc.rust-lang.org/1.93.1/std/ptr/index.html#pointer-to-reference-conversion:
+ //
+ // When creating a mutable reference, then while this reference
+ // exists, the memory it points to must not get accessed (read or
+ // written) through any other pointer or reference not derived from
+ // this reference.
+ //
+ // When creating a shared reference, then while this reference
+ // exists, the memory it points to must not get mutated (except inside
+ // `UnsafeCell`).
unsafe { self.via_unchecked() }
}
/// Produces the split parts of `self`, using a dynamic check to ensure that
/// it is sound to have concurrent references to both parts. You should
/// prefer using [`Self::via_immutable`], [`Self::via_into_bytes`], or
- /// [`Self::via_unaligned`], which have no runtime cost.
+ /// [`Self::via_no_dynamic_padding`], which perform no runtime check.
#[inline(always)]
fn via_runtime_check(self) -> Result<(Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>), Self> {
let l_len = self.l_len();
+ // `l_len` witnesses an object size at most `isize::MAX`, and
+ // `KnownLayout::LAYOUT` guarantees that it contains the trailing
+ // slice. Thus `padding_for_elems` returns the exact trailing padding.
+ let trailing_padding = crate::trailing_slice_layout::<T>().padding_for_elems(l_len.get());
// FIXME(#1290): Once we require `KnownLayout` on all fields, add an
// `IS_IMMUTABLE` associated const, and add `T::IS_IMMUTABLE ||` to the
// below check.
- if l_len.padding_needed_for() == 0 {
- // SAFETY: By `T: SplitAt`, `T` is either `[T]`, or a `repr(C)` or
- // `repr(transparent)` slice DST, for which the trailing padding
- // needed to accommodate `l_len` trailing elements is
- // `l_len.padding_needed_for()`. If no trailing padding is required,
+ if trailing_padding == 0 {
+ // SAFETY: As established above, `trailing_padding` is the exact
+ // padding after the left part's trailing slice. If it is zero,
// the left and right parts are strictly non-overlapping.
Ok(unsafe { self.via_unchecked() })
} else {
@@ -903,7 +1046,7 @@ fn via_runtime_check(self) -> Result<(Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>), Sel
/// # Safety
///
/// The caller promises that if `I::Aliasing` is [`Exclusive`] or `T`
- /// permits interior mutation, then `l_len.padding_needed_for() == 0`.
+ /// permits interior mutation, then the left part has no trailing padding.
#[inline(always)]
unsafe fn via_unchecked(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>) {
let l_len = self.l_len();
@@ -915,11 +1058,11 @@ unsafe fn via_unchecked(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>) {
// Lemma 0: `left` and `right` conform to the aliasing invariant
// `I::Aliasing`. Proof: If `I::Aliasing` is `Exclusive` or `T` permits
- // interior mutation, the caller promises that `l_len.padding_needed_for()
- // == 0`. Consequently, by post-condition on `PtrInner::split_at_unchecked`,
- // there is no trailing padding after `left`'s final element that would
- // overlap into `right`. If `I::Aliasing` is shared and `T` forbids interior
- // mutation, then overlap between their referents is permissible.
+ // interior mutation, the caller promises that the left part has no
+ // trailing padding. Consequently, by the postcondition of
+ // `PtrInner::split_at_unchecked`, `left` and `right` do not overlap.
+ // If `I::Aliasing` is shared and `T` forbids interior mutation, then
+ // overlap between their referents is permissible.
// SAFETY:
// 0. `left` conforms to the aliasing invariant of `I::Aliasing`, by Lemma 0.
@@ -933,17 +1076,19 @@ unsafe fn via_unchecked(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>) {
// SAFETY:
// 0. `right` conforms to the aliasing invariant of `I::Aliasing`, by Lemma
// 0.
- // 1. `right` conforms to the alignment invariant of `I::Alignment, because
- // if `ptr` with `I::Alignment = Aligned`, then by invariant on `T:
- // SplitAt`, the trailing slice of `ptr` (from which `right` is derived)
- // will also be well-aligned.
+ // 1. `right` conforms to the alignment invariant of `I::Alignment`:
+ // if the source is aligned, `T: SplitAt` guarantees that its
+ // trailing slice's address is aligned for `T::Elem`. Advancing by
+ // whole elements preserves that alignment.
// 2. `right` conforms to the validity invariant of `I::Validity`,
- // because `right: [T::Elem]` is derived from the trailing slice of
- // `ptr`, which, by contract on `T: SplitAt::Elem`, has type
- // `[T::Elem]`. The `left` part cannot be used to invalidate `right`,
- // because the caller promises that if `I::Aliasing` is `Exclusive`
- // or `T` permits interior mutation, then `l_len.padding_needed_for()
- // == 0` and thus the parts will be non-overlapping.
+ // because `T: SplitAt` guarantees that `T::Elem` has the size and
+ // bit validity of the original trailing element. It also guarantees
+ // compatible shared and mutable access, so writes through `right`
+ // preserve the original elements' validity and interior mutation
+ // respects the original `UnsafeCell` coverage. The left part
+ // cannot invalidate `right`: when the references are exclusive or
+ // permit interior mutation, the caller guarantees no trailing
+ // padding and hence no overlap.
let right = unsafe { Ptr::from_inner(right) };
(left, right)
@@ -952,6 +1097,185 @@ unsafe fn via_unchecked(self) -> (Ptr<'a, T, I>, Ptr<'a, [T::Elem], I>) {
#[cfg(test)]
mod tests {
+ use core::{cell::Cell, mem::ManuallyDrop};
+
+ use crate::{Immutable, KnownLayout, SplitAt, Unalign, Unaligned};
+
+ #[derive(KnownLayout, SplitAt, Immutable, Unaligned)]
+ #[repr(C, packed)]
+ struct Packed<T: ?Sized> {
+ prefix: u8,
+ tail: ManuallyDrop<T>,
+ }
+
+ #[derive(KnownLayout, SplitAt, Immutable)]
+ #[repr(C, packed(2))]
+ struct Packed2<T: ?Sized> {
+ prefix: u8,
+ tail: ManuallyDrop<T>,
+ }
+
+ #[derive(KnownLayout, SplitAt, Immutable)]
+ #[repr(C)]
+ struct Inner<T: ?Sized> {
+ prefix: u32,
+ tail: T,
+ }
+
+ #[test]
+ fn test_split_at_packed() {
+ use crate::{FromBytes, IntoBytes};
+
+ #[derive(FromBytes, IntoBytes, KnownLayout, SplitAt, Immutable, Unaligned)]
+ #[repr(C, packed)]
+ struct Packet {
+ prefix: u8,
+ tail: [u16],
+ }
+
+ // The tail starts at an odd address, exercising unaligned reads and
+ // writes on targets where `u16` has alignment 2.
+ #[repr(align(8))]
+ struct Bytes([u8; 9]);
+
+ for i in 0..=4 {
+ let mut bytes = Bytes([1; 9]);
+ let packet = Packet::ref_from_bytes(&bytes.0).unwrap();
+ let (left, right): (&Packet, &[Unalign<u16>]) =
+ packet.split_at(i).unwrap().via_immutable();
+ assert_eq!(core::mem::size_of_val(left), 1 + 2 * i);
+ assert_eq!(right.len(), 4 - i);
+ assert!(right.iter().all(|elem| elem.get() == 0x0101));
+ assert!(left.split_at(i).is_some());
+ assert!(left.split_at(i + 1).is_none());
+ assert!(packet.split_at(5).is_none());
+
+ let (_, right) = packet.split_at(i).unwrap().via_into_bytes();
+ assert_eq!(right.len(), 4 - i);
+ let (_, right) = packet.split_at(i).unwrap().via_no_dynamic_padding();
+ assert_eq!(right.len(), 4 - i);
+
+ let packet = Packet::mut_from_bytes(&mut bytes.0).unwrap();
+ let (left, right): (&mut Packet, &mut [Unalign<u16>]) =
+ packet.split_at_mut(i).unwrap().via_runtime_check().ok().unwrap();
+ left.prefix = 2;
+ for elem in right {
+ *elem = Unalign::new(0x0202);
+ }
+ assert_eq!(bytes.0[0], 2);
+ assert!(bytes.0[1..1 + 2 * i].iter().all(|&byte| byte == 1));
+ assert!(bytes.0[1 + 2 * i..].iter().all(|&byte| byte == 2));
+ }
+ }
+
+ #[test]
+ fn test_split_at_nested_packed() {
+ // A packing factor greater than 1 still exposes `Unalign` elements,
+ // including when the factor is below the original element alignment.
+ let mut words = Packed2 { prefix: 0, tail: ManuallyDrop::new([1u32, 2, 3]) };
+ let dst: &mut Packed2<[u32]> = &mut words;
+ let (left, right): (&mut _, &mut [Unalign<u32>]) =
+ dst.split_at_mut(1).unwrap().via_no_dynamic_padding();
+ left.prefix = 4;
+ right[0] = Unalign::new(5);
+ let dst: &Packed2<[u32]> = &words;
+ let (left, right) = dst.split_at(0).unwrap().via_immutable();
+ assert_eq!(left.prefix, 4);
+ assert_eq!(right[0].get(), 1);
+ assert_eq!(right[1].get(), 5);
+ assert_eq!(right[2].get(), 3);
+
+ let mut packet = Inner {
+ prefix: 0,
+ tail: Packed2 {
+ prefix: 1,
+ tail: ManuallyDrop::new(Packed {
+ prefix: 2,
+ tail: ManuallyDrop::new([3u16, 4, 5]),
+ }),
+ },
+ };
+ let dst: &Inner<Packed2<Packed<[u16]>>> = &packet;
+ let (left, right): (&_, &[Unalign<Unalign<u16>>]) =
+ dst.split_at(1).unwrap().via_immutable();
+ assert_eq!(left.prefix, 0);
+ assert_eq!(right.len(), 2);
+ assert_eq!(right[0].get().get(), 4);
+ assert_eq!(right[1].get().get(), 5);
+
+ // Split the packed field separately: the unpacked outer struct's
+ // padding does not constrain this split.
+ let dst: &mut Packed2<Packed<[u16]>> = &mut packet.tail;
+ let (left, right): (&mut _, &mut [Unalign<Unalign<u16>>]) =
+ dst.split_at_mut(1).unwrap().via_no_dynamic_padding();
+ left.prefix = 6;
+ right[0] = Unalign::new(Unalign::new(7));
+ let dst: &Inner<Packed2<Packed<[u16]>>> = &packet;
+ let (_, right) = dst.split_at(0).unwrap().via_immutable();
+ assert_eq!(right[0].get().get(), 3);
+ assert_eq!(right[1].get().get(), 7);
+ assert_eq!(right[2].get().get(), 5);
+ }
+
+ #[test]
+ fn test_split_at_packed_preserves_inner_padding() {
+ for i in 0..=4 {
+ let mut packet = Packed {
+ prefix: 1,
+ tail: ManuallyDrop::new(Inner { prefix: 0, tail: [2u8, 3, 4, 5] }),
+ };
+ let dst: &Packed<Inner<[u8]>> = &packet;
+ let (left, right): (&_, &[Unalign<u8>]) = dst.split_at(i).unwrap().via_immutable();
+ assert_eq!(right.len(), 4 - i);
+ assert_eq!(right.first().map(Unalign::get), [2, 3, 4, 5].get(i).copied());
+
+ // The inner `u32` prefix makes the tail start at byte 5 of the
+ // packed wrapper. Its size may still include inner padding.
+ let has_padding = core::mem::size_of_val(left) != 5 + i;
+ assert_eq!(dst.split_at(i).unwrap().via_runtime_check().is_err(), has_padding);
+
+ let dst: &mut Packed<Inner<[u8]>> = &mut packet;
+ let split = dst.split_at_mut(i).unwrap().via_runtime_check();
+ assert_eq!(split.is_err(), has_padding);
+ if let Ok((left, right)) = split {
+ left.prefix = 6;
+ for elem in right {
+ *elem = Unalign::new(7);
+ }
+ let dst: &Packed<Inner<[u8]>> = &packet;
+ let (left, right) = dst.split_at(i).unwrap().via_immutable();
+ assert_eq!(left.prefix, 6);
+ assert!(right.iter().all(|elem| elem.get() == 7));
+ }
+ }
+ }
+
+ #[test]
+ fn test_split_at_packed_interior_mutation_and_zsts() {
+ let packet = Packed {
+ prefix: 0,
+ tail: ManuallyDrop::new([Cell::new(1u8), Cell::new(2), Cell::new(3)]),
+ };
+ let dst: &Packed<[Cell<u8>]> = &packet;
+ let (left, right): (&_, &[Unalign<Cell<u8>>]) =
+ dst.split_at(1).unwrap().via_no_dynamic_padding();
+ left.tail[0].set(4);
+ right[0].try_deref().unwrap().set(5);
+ assert_eq!(dst.tail[0].get(), 4);
+ assert_eq!(dst.tail[1].get(), 5);
+ assert_eq!(dst.tail[2].get(), 3);
+
+ let mut packet = Packed { prefix: 0, tail: ManuallyDrop::new([(); 3]) };
+ for i in 0..=3 {
+ let dst: &mut Packed<[()]> = &mut packet;
+ let (left, right): (&mut _, &mut [Unalign<()>]) =
+ dst.split_at_mut(i).unwrap().via_no_dynamic_padding();
+ assert_eq!(left.tail.len(), i);
+ assert_eq!(right.len(), 3 - i);
+ assert_eq!(core::mem::size_of_val(left), 1);
+ }
+ }
+
#[cfg(feature = "derive")]
#[test]
fn test_split_at() {
@@ -1011,6 +1335,8 @@ struct SliceDst {
trailing: [u8],
}
+ assert!(SliceDst::LAYOUT.requires_dynamic_padding());
+
const N: usize = 16;
let arr = [1u16; N];
@@ -1070,21 +1396,79 @@ struct Packet {
assert_eq!(r, &[4]);
}
#[test]
+ #[allow(deprecated)]
fn test_split_at_via_unaligned() {
- use crate::{FromBytes, Immutable, IntoBytes, KnownLayout, SplitAt, Unaligned};
- #[derive(FromBytes, KnownLayout, SplitAt, IntoBytes, Immutable, Unaligned)]
+ use crate::{Immutable, KnownLayout, Split, SplitAt, Unaligned};
+
+ fn via_unaligned<'a, T>(split: Split<&'a T>) -> (&'a T, &'a [T::Elem])
+ where
+ T: ?Sized + SplitAt + Unaligned,
+ {
+ split.via_unaligned()
+ }
+
+ fn via_unaligned_mut<'a, T>(split: Split<&'a mut T>) -> (&'a mut T, &'a mut [T::Elem])
+ where
+ T: ?Sized + SplitAt + Unaligned,
+ {
+ split.via_unaligned()
+ }
+
+ #[derive(KnownLayout, SplitAt, Immutable, Unaligned)]
#[repr(C)]
- struct Packet {
- length: u8,
- body: [u8],
+ struct Packet<B: ?Sized> {
+ prefix: [u8; 2],
+ body: B,
}
- let arr = [1, 2, 3, 4];
- let packet = Packet::ref_from_bytes(&arr[..]).unwrap();
+ // Exercise generic callers using the original `T: Unaligned` bound.
+ assert!(!Packet::<[[u8; 2]]>::LAYOUT.requires_dynamic_padding());
+ let packet = Packet { prefix: [0, 1], body: [[2, 3], [4, 5], [6, 7]] };
+ let packet: &Packet<[[u8; 2]]> = &packet;
let split = packet.split_at(2).unwrap();
- let (l, r) = split.via_unaligned();
- assert_eq!(l.length, 1);
- assert_eq!(r, &[4]);
+ let (l, r) = via_unaligned(split);
+ assert_eq!(l.body, [[2, 3], [4, 5]]);
+ assert_eq!(r, &[[6, 7]]);
+
+ let mut packet = Packet { prefix: [0, 1], body: [[2, 3], [4, 5], [6, 7]] };
+ {
+ let packet: &mut Packet<[[u8; 2]]> = &mut packet;
+ let split = packet.split_at_mut(2).unwrap();
+ let (l, r) = via_unaligned_mut(split);
+ l.body[0] = [8, 9];
+ r[0] = [10, 11];
+ }
+ assert_eq!(packet.body, [[8, 9], [4, 5], [10, 11]]);
+ }
+
+ #[test]
+ fn test_split_at_via_no_dynamic_padding() {
+ use core::cell::Cell;
+
+ use crate::SplitAt;
+
+ // `u16` does not implement `Unaligned`, but `[u16]` never requires
+ // dynamic trailing padding.
+ let words = [1u16, 2, 3, 4];
+ let split = SplitAt::split_at(&words[..], 2).unwrap();
+ let (left, right) = split.via_no_dynamic_padding();
+ assert_eq!(left, [1, 2]);
+ assert_eq!(right, [3, 4]);
+
+ let mut words = [1u16, 2, 3, 4];
+ let split = SplitAt::split_at_mut(&mut words[..], 2).unwrap();
+ let (left, right) = split.via_no_dynamic_padding();
+ left[0] = 5;
+ right[0] = 6;
+ assert_eq!(words, [5, 2, 6, 4]);
+
+ // Exercise the safety-sensitive shared route with interior mutation.
+ let cells = [Cell::new(1u16), Cell::new(2), Cell::new(3)];
+ let split = SplitAt::split_at(&cells[..], 2).unwrap();
+ let (left, right) = split.via_no_dynamic_padding();
+ left[0].set(4);
+ right[0].set(5);
+ assert_eq!([cells[0].get(), cells[1].get(), cells[2].get()], [4, 2, 5]);
}
}
diff --git a/rust/zerocopy/src/util/macro_util.rs b/rust/zerocopy/src/util/macro_util.rs
index ceeb80432b0b..fdcaa58d418c 100644
--- a/rust/zerocopy/src/util/macro_util.rs
+++ b/rust/zerocopy/src/util/macro_util.rs
@@ -28,8 +28,8 @@
use crate::{
pointer::{
- cast::CastSized,
- invariant::{Aligned, Initialized, Valid},
+ cast::CastSizedExact,
+ invariant::{Aligned, Initialized, Safe},
BecauseImmutable,
},
FromBytes, Immutable, IntoBytes, KnownLayout, Ptr, ReadOnly, TryFromBytes, ValidityError,
@@ -518,9 +518,9 @@ pub const fn hash_name(name: &str) -> i128 {
///
/// `try_transmute` may either produce a post-monomorphization error or a panic
/// if `Dst` is bigger than `Src`. Otherwise, `try_transmute` panics under the
-/// same circumstances as [`is_bit_valid`].
+/// same circumstances as [`is_safe`].
///
-/// [`is_bit_valid`]: TryFromBytes::is_bit_valid
+/// [`is_safe`]: TryFromBytes::is_safe
#[inline(always)]
pub fn try_transmute<Src, Dst>(src: Src) -> Result<Dst, ValidityError<Src, Dst>>
where
@@ -529,30 +529,64 @@ pub fn try_transmute<Src, Dst>(src: Src) -> Result<Dst, ValidityError<Src, Dst>>
{
static_assert!(Src, Dst => mem::size_of::<Dst>() == mem::size_of::<Src>());
- let mu_src = mem::MaybeUninit::new(src);
- // SAFETY: `MaybeUninit` has no validity requirements.
- let mu_dst: mem::MaybeUninit<ReadOnly<Dst>> =
- unsafe { crate::util::transmute_unchecked(mu_src) };
-
- let ptr = Ptr::from_ref(&mu_dst);
-
- // SAFETY: Since `Src: IntoBytes`, and since `size_of::<Src>() ==
- // size_of::<Dst>()` by the preceding assertion, all of `mu_dst`'s bytes are
- // initialized. `MaybeUninit` has no validity requirements, so even if
- // `ptr` is used to mutate its referent (which it actually can't be - it's
- // a shared `ReadOnly` pointer), that won't violate its referent's validity.
- let ptr = unsafe { ptr.assume_validity::<Initialized>() };
- if Dst::is_bit_valid(ptr.cast::<_, CastSized, _>()) {
- // SAFETY: Since `Dst::is_bit_valid`, we know that `ptr`'s referent is
- // bit-valid for `Dst`. `ptr` points to `mu_dst`, and no intervening
- // operations have mutated it, so it is a bit-valid `Dst`.
- Ok(ReadOnly::into_inner(unsafe { mu_dst.assume_init() }))
+ let src = mem::ManuallyDrop::new(ReadOnly::new(src));
+ let mut ptr =
+ Ptr::from_ref(&*src).transmute_with::<ReadOnly<Dst>, Initialized, CastSizedExact, _>();
+ if Dst::is_safe(ptr.reborrow_shared()) {
+ // SAFETY: `read_unaligned` requires that `ptr` be valid for reads and
+ // point to a properly initialized `ReadOnly<Dst>` [1]. `ptr` was
+ // derived from a live reference to `src`; `CastSizedExact` preserves
+ // that reference's address, provenance, and exact byte extent. Thus,
+ // the entire range remains live and valid for reads. The reference is
+ // non-null even when `ReadOnly<Dst>` is zero-sized, and the cast
+ // preserves that address, satisfying [1]'s explicit ZST requirement.
+ //
+ // `Src: IntoBytes` and the `ReadOnly<Src>` bridge let `transmute_with`
+ // establish that every byte in that exact range is initialized.
+ // `Dst::is_safe` then established that the same bytes are bit-valid for
+ // `Dst`, and thus for `ReadOnly<Dst>`. The candidate was a shared
+ // `ReadOnly` pointer, so it could not mutate the bytes; no operation
+ // occurs between validation and this read which could do so. Alignment
+ // is not an additional precondition because [1] expressly permits
+ // unaligned pointers.
+ //
+ // `read_unaligned` leaves the source memory unchanged and makes a
+ // bitwise copy [1]. The source is wrapped in `ManuallyDrop`, which
+ // inhibits its destructor [2]. After this read, neither `src` nor
+ // `ptr` is used to access the inner value, and the wrapper's eventual
+ // drop does not drop it. Thus, only the returned `Dst` is subsequently
+ // used or dropped; there is no duplicate ownership use.
+ //
+ // [1] Per https://doc.rust-lang.org/1.56.0/std/ptr/fn.read_unaligned.html:
+ //
+ // Reads the value from `src` without moving it. This leaves the
+ // memory in `src` unchanged.
+ //
+ // Unlike `read`, `read_unaligned` works with unaligned pointers.
+ //
+ // Behavior is undefined if any of the following conditions are
+ // violated:
+ //
+ // * `src` must be valid for reads.
+ //
+ // * `src` must point to a properly initialized value of type `T`.
+ //
+ // Like `read`, `read_unaligned` creates a bitwise copy of `T`,
+ // regardless of whether `T` is `Copy`. If `T` is not `Copy`, using
+ // both the returned value and the value at `*src` can violate
+ // memory safety.
+ //
+ // Note that even if `T` has size `0`, the pointer must be non-null.
+ //
+ // [2] Per https://doc.rust-lang.org/1.56.0/std/mem/struct.ManuallyDrop.html:
+ //
+ // A wrapper to inhibit compiler from automatically calling `T`'s
+ // destructor.
+ let dst = unsafe { core::ptr::read_unaligned(ptr.as_inner().as_ptr()) };
+ Ok(ReadOnly::into_inner(dst))
} else {
- // SAFETY: `MaybeUninit` has no validity requirements.
- let mu_src: mem::MaybeUninit<Src> = unsafe { crate::util::transmute_unchecked(mu_dst) };
- // SAFETY: `mu_dst`/`mu_src` was constructed from `src` and never
- // modified, so it is still bit-valid.
- Err(ValidityError::new(unsafe { mu_src.assume_init() }))
+ let src = ReadOnly::into_inner(mem::ManuallyDrop::into_inner(src));
+ Err(ValidityError::new(src))
}
}
@@ -599,7 +633,7 @@ fn try_transmute_ref(
let res = ptr.try_with(#[inline(always)] |ptr| {
let ptr = ptr.recall_validity::<Initialized, _>();
let ptr = ptr.cast::<_, crate::layout::CastFrom<Dst>, _>();
- ptr.try_into_valid()
+ ptr.try_into_safe()
});
match res {
Ok(ptr) => {
@@ -662,7 +696,7 @@ fn try_transmute_mut(
ptr.try_with_unchecked(#[inline(always)] |ptr| {
let ptr = ptr.recall_validity::<Initialized, (_, (_, _))>();
let ptr = ptr.cast::<_, crate::layout::CastFrom<Dst>, _>();
- ptr.try_into_valid()
+ ptr.try_into_safe()
})
};
match res {
@@ -878,7 +912,7 @@ fn transmute_ref(self) -> &'a Dst {
let ptr = Ptr::from_ref(self.0)
.recall_validity::<Initialized, _>()
.transmute_with::<Dst, Initialized, crate::layout::CastFrom<Dst>, (crate::pointer::BecauseMutationCompatible, _)>()
- .recall_validity::<Valid, _>();
+ .recall_validity::<Safe, _>();
static_assert!(Src: ?Sized + KnownLayout, Dst: ?Sized + KnownLayout => {
Src::LAYOUT.align.get() >= Dst::LAYOUT.align.get()
@@ -911,7 +945,7 @@ fn transmute_mut(self) -> &'a mut Dst {
let ptr = Ptr::from_mut(self.0)
.recall_validity::<Initialized, (_, (_, _))>()
.transmute_with::<Dst, Initialized, crate::layout::CastFrom<Dst>, _>()
- .recall_validity::<Valid, (_, (_, _))>();
+ .recall_validity::<Safe, (_, (_, _))>();
static_assert!(Src: ?Sized + KnownLayout, Dst: ?Sized + KnownLayout => {
Src::LAYOUT.align.get() >= Dst::LAYOUT.align.get()
@@ -946,6 +980,22 @@ pub mod mem {
}
}
+/// Projects `Self` to itself.
+///
+/// This is used by macros to have the compiler verify that two syntactically
+/// similar types actually resolve to the same type. The blanket implementation
+/// ensures that no other implementation can project a type to anything other
+/// than itself.
+#[doc(hidden)]
+pub trait Identity {
+ /// `Self`.
+ type Type: ?Sized;
+}
+
+impl<T: ?Sized> Identity for T {
+ type Type = T;
+}
+
#[cfg(test)]
mod tests {
use core::num::NonZeroUsize;
diff --git a/rust/zerocopy/src/util/macros.rs b/rust/zerocopy/src/util/macros.rs
index 7e63e3a54fc4..e514407ceeaa 100644
--- a/rust/zerocopy/src/util/macros.rs
+++ b/rust/zerocopy/src/util/macros.rs
@@ -15,21 +15,20 @@
/// The trait impl must be sound.
///
/// When implementing `TryFromBytes`:
-/// - If no `is_bit_valid` impl is provided, then it must be valid for
-/// `is_bit_valid` to unconditionally return `true`. In other words, it must
-/// be the case that any initialized sequence of bytes constitutes a valid
-/// instance of `$ty`.
-/// - If an `is_bit_valid` impl is provided, then the impl of `is_bit_valid`
-/// must only return `true` if its argument refers to a valid `$ty`.
+/// - If no `is_safe` impl is provided, then it must be valid for `is_safe` to
+/// unconditionally return `true`. In other words, it must be the case that any
+/// initialized sequence of bytes constitutes a valid instance of `$ty`.
+/// - If an `is_safe` impl is provided, then the impl of `is_safe` must only
+/// return `true` if its argument refers to a valid `$ty`.
macro_rules! unsafe_impl {
// Implement `$trait` for `$ty` with no bounds.
- ($(#[$attr:meta])* $ty:ty: $trait:ident $(; |$candidate:ident| $is_bit_valid:expr)?) => {{
+ ($(#[$attr:meta])* $ty:ty: $trait:ident $(; |$candidate:ident| $is_safe:expr)?) => {{
crate::util::macros::__unsafe();
$(#[$attr])*
// SAFETY: The caller promises that this is sound.
unsafe impl $trait for $ty {
- unsafe_impl!(@method $trait $(; |$candidate| $is_bit_valid)?);
+ unsafe_impl!(@method $trait $(; |$candidate| $is_safe)?);
}
}};
@@ -92,26 +91,26 @@ unsafe impl $trait for $ty {
$(#[$attr:meta])*
const $constname:ident : $constty:ident $(,)?
$($tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?),*
- => $trait:ident for $ty:ty $(; |$candidate:ident| $is_bit_valid:expr)?
+ => $trait:ident for $ty:ty $(; |$candidate:ident| $is_safe:expr)?
) => {
unsafe_impl!(
@inner
$(#[$attr])*
@const $constname: $constty,
$($tyvar $(: $(? $optbound +)* + $($bound +)*)?,)*
- => $trait for $ty $(; |$candidate| $is_bit_valid)?
+ => $trait for $ty $(; |$candidate| $is_safe)?
);
};
(
$(#[$attr:meta])*
$($tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?),*
- => $trait:ident for $ty:ty $(; |$candidate:ident| $is_bit_valid:expr)?
+ => $trait:ident for $ty:ty $(; |$candidate:ident| $is_safe:expr)?
) => {{
unsafe_impl!(
@inner
$(#[$attr])*
$($tyvar $(: $(? $optbound +)* + $($bound +)*)?,)*
- => $trait for $ty $(; |$candidate| $is_bit_valid)?
+ => $trait for $ty $(; |$candidate| $is_safe)?
);
}};
(
@@ -119,7 +118,7 @@ unsafe impl $trait for $ty {
$(#[$attr:meta])*
$(@const $constname:ident : $constty:ident,)*
$($tyvar:ident $(: $(? $optbound:ident +)* + $($bound:ident +)* )?,)*
- => $trait:ident for $ty:ty $(; |$candidate:ident| $is_bit_valid:expr)?
+ => $trait:ident for $ty:ty $(; |$candidate:ident| $is_safe:expr)?
) => {{
crate::util::macros::__unsafe();
@@ -127,21 +126,21 @@ unsafe impl $trait for $ty {
#[allow(non_local_definitions)]
// SAFETY: The caller promises that this is sound.
unsafe impl<$($tyvar $(: $(? $optbound +)* $($bound +)*)?),* $(, const $constname: $constty,)*> $trait for $ty {
- unsafe_impl!(@method $trait $(; |$candidate| $is_bit_valid)?);
+ unsafe_impl!(@method $trait $(; |$candidate| $is_safe)?);
}
}};
- (@method TryFromBytes ; |$candidate:ident| $is_bit_valid:expr) => {
+ (@method TryFromBytes ; |$candidate:ident| $is_safe:expr) => {
#[allow(clippy::missing_inline_in_public_items, dead_code)]
#[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))]
fn only_derive_is_allowed_to_implement_this_trait() {}
#[inline]
- fn is_bit_valid<Alignment>($candidate: Maybe<'_, Self, Alignment>) -> bool
+ fn is_safe<Alignment>($candidate: Maybe<'_, Self, Alignment>) -> bool
where
Alignment: crate::invariant::Alignment,
{
- $is_bit_valid
+ $is_safe
}
};
(@method TryFromBytes) => {
@@ -149,7 +148,7 @@ fn is_bit_valid<Alignment>($candidate: Maybe<'_, Self, Alignment>) -> bool
#[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))]
fn only_derive_is_allowed_to_implement_this_trait() {}
#[inline(always)]
- fn is_bit_valid<Alignment>(_candidate: Maybe<'_, Self, Alignment>) -> bool
+ fn is_safe<Alignment>(_candidate: Maybe<'_, Self, Alignment>) -> bool
where
Alignment: crate::invariant::Alignment,
{
@@ -161,16 +160,16 @@ fn is_bit_valid<Alignment>(_candidate: Maybe<'_, Self, Alignment>) -> bool
#[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))]
fn only_derive_is_allowed_to_implement_this_trait() {}
};
- (@method $trait:ident; |$_candidate:ident| $_is_bit_valid:expr) => {
- compile_error!("Can't provide `is_bit_valid` impl for trait other than `TryFromBytes`");
+ (@method $trait:ident; |$_candidate:ident| $_is_safe:expr) => {
+ compile_error!("Can't provide `is_safe` impl for trait other than `TryFromBytes`");
};
}
/// Implements `$trait` for `$ty` where `$ty: TransmuteFrom<$repr>` (and
/// vice-versa).
///
-/// Calling this macro is safe; the internals of the macro emit appropriate
-/// trait bounds which ensure that the given impl is sound.
+/// This macro is intended to be safe to call. The current proof is incomplete;
+/// see FIXME(#3691) below.
macro_rules! impl_for_transmute_from {
(
$(#[$attr:meta])*
@@ -181,26 +180,32 @@ macro_rules! impl_for_transmute_from {
$(#[$attr])*
#[allow(non_local_definitions)]
- // SAFETY: `is_trait<T, R>` (defined and used below) requires `T:
- // TransmuteFrom<R>`, `R: TransmuteFrom<T>`, and `R: $trait`. It is
- // called using `$ty` and `$repr`, ensuring that `$ty` and `$repr`
- // have equivalent bit validity, and ensuring that `$repr: $trait`.
- // The supported traits - `TryFromBytes`, `FromZeros`, `FromBytes`,
- // and `IntoBytes` - are defined only in terms of the bit validity
- // of a type. Therefore, `$repr: $trait` ensures that `$ty: $trait`
- // is sound.
+ // SAFETY: `is_trait<T, R>` (defined and used below) requires
+ // reciprocal `TransmuteFrom<_, Safe, Safe>` bounds and `R: $trait`.
+ // If `T` and `R` have the same size, the reciprocal bounds imply
+ // that they permit the same `Safe` bit patterns. The call below
+ // instantiates `T` with `$ty` and `R` with `$repr`, and establishes
+ // `$repr: $trait`. The supported traits - `TryFromBytes`,
+ // `FromZeros`, `FromBytes`, and `IntoBytes` - are defined only in
+ // terms of bit validity, so these premises are sufficient when
+ // `$ty` and `$repr` have the same size.
+ //
+ // FIXME(#3691): This macro does not establish that `$ty` and
+ // `$repr` have the same size. Without that premise, `TransmuteFrom`
+ // conveys no safety guarantee, so these bounds do not make
+ // arbitrary invocations of this macro sound.
unsafe impl<$($tyvar $(: $(? $optbound +)* $($bound +)*)?)?> $trait for $ty {
#[allow(dead_code, clippy::missing_inline_in_public_items)]
#[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))]
fn only_derive_is_allowed_to_implement_this_trait() {
- use crate::pointer::{*, invariant::Valid};
+ use crate::pointer::{*, invariant::Safe};
impl_for_transmute_from!(@assert_is_supported_trait $trait);
fn is_trait<T, R>()
where
- T: TransmuteFrom<R, Valid, Valid> + ?Sized,
- R: TransmuteFrom<T, Valid, Valid> + ?Sized,
+ T: TransmuteFrom<R, Safe, Safe> + ?Sized,
+ R: TransmuteFrom<T, Safe, Safe> + ?Sized,
R: $trait,
{
}
@@ -212,7 +217,7 @@ fn f<$($tyvar $(: $(? $optbound +)* $($bound +)*)?)?>() {
}
impl_for_transmute_from!(
- @is_bit_valid
+ @is_safe
$(<$tyvar $(: $(? $optbound +)* $($bound +)*)?>)?
$trait for $ty [$repr]
);
@@ -224,27 +229,27 @@ fn f<$($tyvar $(: $(? $optbound +)* $($bound +)*)?)?>() {
(@assert_is_supported_trait FromBytes) => {};
(@assert_is_supported_trait IntoBytes) => {};
(
- @is_bit_valid
+ @is_safe
$(<$tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?>)?
TryFromBytes for $ty:ty [$repr:ty]
) => {
#[inline(always)]
- fn is_bit_valid<Alignment>(candidate: $crate::Maybe<'_, Self, Alignment>) -> bool
+ fn is_safe<Alignment>(candidate: $crate::Maybe<'_, Self, Alignment>) -> bool
where
Alignment: $crate::invariant::Alignment,
{
// SAFETY: This macro ensures that `$repr` and `Self` have the same
// size and bit validity. Thus, a bit-valid instance of `$repr` is
// also a bit-valid instance of `Self`.
- <$repr as TryFromBytes>::is_bit_valid(candidate.transmute::<_, _, BecauseImmutable>())
+ <$repr as TryFromBytes>::is_safe(candidate.transmute::<_, _, BecauseImmutable>())
}
};
(
- @is_bit_valid
+ @is_safe
$(<$tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?>)?
$trait:ident for $ty:ty [$repr:ty]
) => {
- // Trait other than `TryFromBytes`; no `is_bit_valid` impl.
+ // Trait other than `TryFromBytes`; no `is_safe` impl.
};
}
@@ -269,33 +274,33 @@ fn is_bit_valid<Alignment>(candidate: $crate::Maybe<'_, Self, Alignment>) -> boo
macro_rules! unsafe_impl_for_power_set {
(
$first:ident $(, $rest:ident)* $(-> $ret:ident)? => $trait:ident for $macro:ident!(...)
- $(; |$candidate:ident| $is_bit_valid:expr)?
+ $(; |$candidate:ident| $is_safe:expr)?
) => {
unsafe_impl_for_power_set!(
$($rest),* $(-> $ret)? => $trait for $macro!(...)
- $(; |$candidate| $is_bit_valid)?
+ $(; |$candidate| $is_safe)?
);
unsafe_impl_for_power_set!(
@impl $first $(, $rest)* $(-> $ret)? => $trait for $macro!(...)
- $(; |$candidate| $is_bit_valid)?
+ $(; |$candidate| $is_safe)?
);
};
(
$(-> $ret:ident)? => $trait:ident for $macro:ident!(...)
- $(; |$candidate:ident| $is_bit_valid:expr)?
+ $(; |$candidate:ident| $is_safe:expr)?
) => {
unsafe_impl_for_power_set!(
@impl $(-> $ret)? => $trait for $macro!(...)
- $(; |$candidate| $is_bit_valid)?
+ $(; |$candidate| $is_safe)?
);
};
(
@impl $($vars:ident),* $(-> $ret:ident)? => $trait:ident for $macro:ident!(...)
- $(; |$candidate:ident| $is_bit_valid:expr)?
+ $(; |$candidate:ident| $is_safe:expr)?
) => {
unsafe_impl!(
$($vars,)* $($ret)? => $trait for $macro!($($vars),* $(-> $ret)?)
- $(; |$candidate| $is_bit_valid)?
+ $(; |$candidate| $is_safe)?
);
};
}
@@ -355,8 +360,8 @@ macro_rules! opt_unsafe_fn {
/// impl in this case, and also provides useful documentation for readers of the
/// code.
///
-/// Finally, if a `TryFromBytes::is_bit_valid` impl is provided, it must adhere
-/// to the safety preconditions of [`unsafe_impl!`].
+/// Finally, if a `TryFromBytes::is_safe` impl is provided, it must adhere to the
+/// safety preconditions of [`unsafe_impl!`].
///
/// ## Example
///
@@ -396,11 +401,11 @@ impl<const $constname: $constty, $($tyvar $(: $(? $optbound +)* $($bound +)*)?),
};
(
$($tyvar:ident $(: $(? $optbound:ident $(+)?)* $($bound:ident $(+)?)* )?),*
- => $trait:ident for $ty:ty $(; |$candidate:ident| $is_bit_valid:expr)?
+ => $trait:ident for $ty:ty $(; |$candidate:ident| $is_safe:expr)?
) => {
impl_or_verify!(@impl { unsafe_impl!(
$($tyvar $(: $(? $optbound +)* $($bound +)*)?),* => $trait for $ty
- $(; |$candidate| $is_bit_valid)?
+ $(; |$candidate| $is_safe)?
); });
impl_or_verify!(@verify $trait, {
impl<$($tyvar $(: $(? $optbound +)* $($bound +)*)?),*> Subtrait for $ty {}
@@ -495,7 +500,7 @@ macro_rules! unsafe_impl_known_layout {
#[allow(non_local_definitions)]
// SAFETY: The caller promises that this is sound.
unsafe impl<$($tyvar: ?Sized + KnownLayout)?> KnownLayout for $ty {
- #[allow(clippy::missing_inline_in_public_items, dead_code)]
+ #[allow(dead_code, clippy::missing_inline_in_public_items)]
#[cfg_attr(all(coverage_nightly, __ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS), coverage(off))]
fn only_derive_is_allowed_to_implement_this_trait() {}
@@ -742,14 +747,14 @@ macro_rules! unsafe_impl_for_transparent_wrapper {
($vis:vis T $(: ?$optbound:ident)? => $wrapper:ident<T>) => {{
crate::util::macros::__unsafe();
- use crate::pointer::{TransmuteFrom, cast::{CastExact, TransitiveProject}, SizeEq, invariant::Valid};
+ use crate::pointer::{TransmuteFrom, cast::{CastExact, TransitiveProject}, SizeEq, invariant::Safe};
use crate::wrappers::ReadOnly;
// SAFETY: The caller promises that `T` and `$wrapper<T>` have the same
// bit validity.
- unsafe impl<T $(: ?$optbound)?> TransmuteFrom<T, Valid, Valid> for $wrapper<T> {}
+ unsafe impl<T $(: ?$optbound)?> TransmuteFrom<T, Safe, Safe> for $wrapper<T> {}
// SAFETY: See previous safety comment.
- unsafe impl<T $(: ?$optbound)?> TransmuteFrom<$wrapper<T>, Valid, Valid> for T {}
+ unsafe impl<T $(: ?$optbound)?> TransmuteFrom<$wrapper<T>, Safe, Safe> for T {}
// SAFETY: The caller promises that a `T` to `$wrapper<T>` cast is
// size-preserving.
define_cast!(unsafe { $vis CastToWrapper<T $(: ?$optbound)? > = T => $wrapper<T> });
@@ -805,7 +810,7 @@ impl<T $(: ?$optbound)?> SizeEq<ReadOnly<$wrapper<T>>> for ReadOnly<T> {
macro_rules! impl_transitive_transmute_from {
($($tyvar:ident $(: ?$optbound:ident)?)? => $t:ty => $u:ty => $v:ty) => {
const _: () = {
- use crate::pointer::{TransmuteFrom, SizeEq, invariant::Valid};
+ use crate::pointer::{TransmuteFrom, SizeEq, invariant::Safe};
impl<$($tyvar $(: ?$optbound)?)?> SizeEq<$t> for $v
where
@@ -819,14 +824,14 @@ impl<$($tyvar $(: ?$optbound)?)?> SizeEq<$t> for $v
>;
}
- // SAFETY: Since `$u: TransmuteFrom<$t, Valid, Valid>`, it is sound
- // to transmute a bit-valid `$t` to a bit-valid `$u`. Since `$v:
- // TransmuteFrom<$u, Valid, Valid>`, it is sound to transmute that
+ // SAFETY: Since `$u: TransmuteFrom<$t, Safe, Safe>`, it is sound to
+ // transmute a bit-valid `$t` to a bit-valid `$u`. Since `$v:
+ // TransmuteFrom<$u, Safe, Safe>`, it is sound to transmute that
// bit-valid `$u` to a bit-valid `$v`.
- unsafe impl<$($tyvar $(: ?$optbound)?)?> TransmuteFrom<$t, Valid, Valid> for $v
+ unsafe impl<$($tyvar $(: ?$optbound)?)?> TransmuteFrom<$t, Safe, Safe> for $v
where
- $u: TransmuteFrom<$t, Valid, Valid>,
- $v: TransmuteFrom<$u, Valid, Valid>,
+ $u: TransmuteFrom<$t, Safe, Safe>,
+ $v: TransmuteFrom<$u, Safe, Safe>,
{}
};
};
diff --git a/rust/zerocopy/src/util/mod.rs b/rust/zerocopy/src/util/mod.rs
index 02fd4ed62741..646f45739802 100644
--- a/rust/zerocopy/src/util/mod.rs
+++ b/rust/zerocopy/src/util/mod.rs
@@ -23,7 +23,7 @@
use super::*;
use crate::pointer::{
- invariant::{Exclusive, Shared, Valid},
+ invariant::{Exclusive, Safe, Shared},
SizeEq, TransmuteFromPtr,
};
@@ -108,7 +108,7 @@ fn addr(self) -> usize {
#[allow(clippy::as_conversions)]
#[cfg_attr(
__ZEROCOPY_INTERNAL_USE_ONLY_NIGHTLY_FEATURES_IN_TESTS,
- allow(lossy_provenance_casts)
+ allow(implicit_provenance_casts)
)]
return self.cast::<()>() as usize;
}
@@ -144,9 +144,10 @@ pub(crate) fn validate_aligned_to<T: AsAddress, U>(t: T) -> Result<(), Alignment
/// on the answer it gives if this is not the case.
#[cfg_attr(
kani,
- kani::requires(len <= DstLayout::MAX_SIZE),
kani::requires(align.is_power_of_two()),
- kani::ensures(|&p| (len + p) % align.get() == 0),
+ // A power-of-two alignment divides the `usize` modulus, so wrapping
+ // preserves congruence even when the next aligned value exceeds `usize`.
+ kani::ensures(|&p| len.wrapping_add(p) % align.get() == 0),
// Ensures that we add the minimum required padding.
kani::ensures(|&p| p < align.get()),
)]
@@ -340,7 +341,7 @@ pub(crate) unsafe fn transmute_ref<Src, Dst, R>(src: &Src) -> &Dst
where
Src: ?Sized,
Dst: SizeEq<Src>
- + TransmuteFromPtr<Src, Shared, Valid, Valid, <Dst as SizeEq<Src>>::CastFrom, R>
+ + TransmuteFromPtr<Src, Shared, Safe, Safe, <Dst as SizeEq<Src>>::CastFrom, R>
+ ?Sized,
{
let dst = Ptr::from_ref(src).transmute();
@@ -357,7 +358,7 @@ pub(crate) unsafe fn transmute_mut<Src, Dst, R>(src: &mut Src) -> &mut Dst
where
Src: ?Sized,
Dst: SizeEq<Src>
- + TransmuteFromPtr<Src, Exclusive, Valid, Valid, <Dst as SizeEq<Src>>::CastFrom, R>
+ + TransmuteFromPtr<Src, Exclusive, Safe, Safe, <Dst as SizeEq<Src>>::CastFrom, R>
+ ?Sized,
{
let dst = Ptr::from_mut(src).transmute();
@@ -562,31 +563,6 @@ pub(crate) fn get(&self) -> T::PointerMetadata
self.meta
}
- #[inline]
- pub(crate) fn padding_needed_for(&self) -> usize
- where
- T: KnownLayout<PointerMetadata = usize>,
- {
- let trailing_slice_layout = crate::trailing_slice_layout::<T>();
-
- // FIXME(#67): Remove this allow. See NumExt for more details.
- #[allow(
- unstable_name_collisions,
- clippy::incompatible_msrv,
- clippy::multiple_unsafe_ops_per_block
- )]
- // SAFETY: By invariant on `self`, a `&T` with metadata `self.meta`
- // describes an object of size `<= isize::MAX`. This computes the
- // size of such a `&T` without any trailing padding, and so neither
- // the multiplication nor the addition will overflow.
- let unpadded_size = unsafe {
- let trailing_size = self.meta.unchecked_mul(trailing_slice_layout.elem_size);
- trailing_size.unchecked_add(trailing_slice_layout.offset)
- };
-
- util::padding_needed_for(unpadded_size, T::LAYOUT.align)
- }
-
#[inline(always)]
pub(crate) fn validate_cast_and_convert_metadata(
addr: usize,
@@ -895,6 +871,8 @@ mod tests {
#[test]
fn test_round_down_to_next_multiple_of_alignment() {
+ // Division by `NonZeroUsize` unsupported on our MSRV.
+ #[allow(clippy::needless_nonzero_get)]
fn alt_impl(n: usize, align: NonZeroUsize) -> usize {
let mul = n / align.get();
mul * align.get()
diff --git a/rust/zerocopy/src/wrappers.rs b/rust/zerocopy/src/wrappers.rs
index 1a8cf2b41d55..d0baa901f10c 100644
--- a/rust/zerocopy/src/wrappers.rs
+++ b/rust/zerocopy/src/wrappers.rs
@@ -8,10 +8,10 @@
// This file may not be copied, modified, or distributed except according to
// those terms.
-use core::{fmt, hash::Hash};
+use core::{borrow::Borrow, fmt, hash::Hash};
use super::*;
-use crate::pointer::{invariant::Valid, SizeEq, TransmuteFrom};
+use crate::pointer::{invariant::Safe, SizeEq, TransmuteFrom};
/// A type with no alignment requirement.
///
@@ -140,10 +140,10 @@
// Unaligned`.
// - `Unalign<T>` has the same bit validity as `T`, and so it is `FromZeros`,
// `FromBytes`, or `IntoBytes` exactly when `T` is as well.
-// - `Immutable`: `Unalign<T>` has the same fields as `T`, so it permits
-// interior mutation exactly when `T` does.
+// - `Immutable`: `Unalign<T>` has the same fields as `T`, so it permits interior
+// mutation exactly when `T` does.
// - `TryFromBytes`: `Unalign<T>` has the same the same bit validity as `T`, so
-// `T::is_bit_valid` is a sound implementation of `is_bit_valid`.
+// `T::is_safe` is a sound implementation of `is_safe`.
//
#[allow(clippy::multiple_unsafe_ops_per_block)]
const _: () = unsafe {
@@ -151,7 +151,7 @@
impl_or_verify!(T: Immutable => Immutable for Unalign<T>);
impl_or_verify!(
T: TryFromBytes => TryFromBytes for Unalign<T>;
- |c| T::is_bit_valid(c.transmute::<_, _, BecauseImmutable>())
+ |c| T::is_safe(c.transmute::<_, _, BecauseImmutable>())
);
impl_or_verify!(T: FromZeros => FromZeros for Unalign<T>);
impl_or_verify!(T: FromBytes => FromBytes for Unalign<T>);
@@ -613,7 +613,14 @@ mod read_only_def {
/// Note that `&mut ReadOnly<T>` still permits mutation – the read-only
/// property only applies to shared references.
///
+ /// `ReadOnly<T>` implements [`Copy`] and [`Clone`] when `T: Copy`. Cloning
+ /// copies the value without calling `T::clone`. Trait implementations that
+ /// expose or operate on a shared reference to `T` require `T: Immutable`.
+ /// Constructing a wrapper or accessing its contents through [`AsMut`]
+ /// does not require `T: Immutable`.
+ ///
/// [`Immutable`]: crate::Immutable
+ #[derive(Copy)]
#[repr(transparent)]
pub struct ReadOnly<T: ?Sized> {
// INVARIANT: `inner` is never mutated through a `&ReadOnly<T>`
@@ -665,7 +672,7 @@ pub(crate) fn as_mut(r: &mut ReadOnly<T>) -> &mut T {
// SAFETY:
// - `ReadOnly<T>` has the same alignment as `T`, and so it is `Unaligned`
// exactly when `T` is as well.
-// - `ReadOnly<T>` has the same bit validity as `T`, and so this `is_bit_valid`
+// - `ReadOnly<T>` has the same bit validity as `T`, and so this `is_safe`
// implementation is correct, and thus the `TryFromBytes` impl is sound.
// - `ReadOnly<T>` has the same bit validity as `T`, and so it is `FromZeros`,
// `FromBytes`, and `IntoBytes` exactly when `T` is as well.
@@ -673,7 +680,7 @@ pub(crate) fn as_mut(r: &mut ReadOnly<T>) -> &mut T {
unsafe_impl!(T: ?Sized + Unaligned => Unaligned for ReadOnly<T>);
unsafe_impl!(
T: ?Sized + TryFromBytes => TryFromBytes for ReadOnly<T>;
- |c| T::is_bit_valid(c.cast::<_, <ReadOnly<T> as SizeEq<ReadOnly<ReadOnly<T>>>>::CastFrom, _>())
+ |c| T::is_safe(c.cast::<_, <ReadOnly<T> as SizeEq<ReadOnly<ReadOnly<T>>>>::CastFrom, _>())
);
unsafe_impl!(T: ?Sized + FromZeros => FromZeros for ReadOnly<T>);
unsafe_impl!(T: ?Sized + FromBytes => FromBytes for ReadOnly<T>);
@@ -709,11 +716,34 @@ impl<T: ?Sized> SizeEq<T> for ReadOnly<T> {
// SAFETY: `ReadOnly<T>` is a `#[repr(transparent)]` wrapper around `T`, and so
// it has the same bit validity as `T`.
-unsafe impl<T: ?Sized> TransmuteFrom<T, Valid, Valid> for ReadOnly<T> {}
+unsafe impl<T: ?Sized> TransmuteFrom<T, Safe, Safe> for ReadOnly<T> {}
// SAFETY: `ReadOnly<T>` is a `#[repr(transparent)]` wrapper around `T`, and so
// it has the same bit validity as `T`.
-unsafe impl<T: ?Sized> TransmuteFrom<ReadOnly<T>, Valid, Valid> for T {}
+unsafe impl<T: ?Sized> TransmuteFrom<ReadOnly<T>, Safe, Safe> for T {}
+
+impl<T: Default> Default for ReadOnly<T> {
+ #[inline(always)]
+ fn default() -> Self {
+ Self::new(Default::default())
+ }
+}
+
+// Copying avoids calling `T::clone`, which could mutate through a shared
+// reference to `T` and violate `ReadOnly`'s invariant.
+impl<T: Copy> Clone for ReadOnly<T> {
+ #[inline(always)]
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+impl<T> From<T> for ReadOnly<T> {
+ #[inline(always)]
+ fn from(t: T) -> Self {
+ Self::new(t)
+ }
+}
impl<'a, T: ?Sized + Immutable> From<&'a T> for &'a ReadOnly<T> {
#[inline(always)]
@@ -743,15 +773,74 @@ fn deref_mut(&mut self) -> &mut Self::Target {
}
}
-impl<T: ?Sized + Immutable + Debug> Debug for ReadOnly<T> {
+impl<T: ?Sized + Immutable> AsRef<T> for ReadOnly<T> {
#[inline(always)]
- fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
- self.deref().fmt(f)
+ fn as_ref(&self) -> &T {
+ self.deref()
+ }
+}
+
+impl<T: ?Sized + Immutable> Borrow<T> for ReadOnly<T> {
+ #[inline(always)]
+ fn borrow(&self) -> &T {
+ self.deref()
+ }
+}
+
+impl<T: ?Sized> AsMut<T> for ReadOnly<T> {
+ #[inline(always)]
+ fn as_mut(&mut self) -> &mut T {
+ ReadOnly::as_mut(self)
+ }
+}
+
+// Delegate shared access through `Deref`, which requires `T: Immutable` and
+// thus prevents interior mutation of the wrapped value by these trait methods.
+impl<T: ?Sized + Immutable + PartialEq> PartialEq for ReadOnly<T> {
+ #[inline(always)]
+ fn eq(&self, other: &Self) -> bool {
+ self.deref().eq(other.deref())
+ }
+}
+
+impl<T: ?Sized + Immutable + Eq> Eq for ReadOnly<T> {}
+
+impl<T: ?Sized + Immutable + PartialOrd> PartialOrd for ReadOnly<T> {
+ #[inline(always)]
+ fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
+ self.deref().partial_cmp(other.deref())
}
}
+impl<T: ?Sized + Immutable + Ord> Ord for ReadOnly<T> {
+ #[inline(always)]
+ fn cmp(&self, other: &Self) -> Ordering {
+ self.deref().cmp(other.deref())
+ }
+}
+
+impl<T: ?Sized + Immutable + Hash> Hash for ReadOnly<T> {
+ #[inline(always)]
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ self.deref().hash(state);
+ }
+}
+
+macro_rules! impl_read_only_fmt {
+ ($($trait:ident),* $(,)?) => {$(
+ impl<T: ?Sized + Immutable + fmt::$trait> fmt::$trait for ReadOnly<T> {
+ #[inline(always)]
+ fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
+ fmt::$trait::fmt(self.deref(), f)
+ }
+ }
+ )*};
+}
+
+impl_read_only_fmt!(Debug, Display, Binary, Octal, LowerHex, UpperHex, LowerExp, UpperExp);
+
// SAFETY: See safety comment on `ProjectToTag`.
-unsafe impl<T: HasTag + ?Sized> HasTag for ReadOnly<T> {
+unsafe impl<T: HasTag<Client> + ?Sized, Client> HasTag<Client> for ReadOnly<T> {
#[allow(clippy::missing_inline_in_public_items)]
fn only_derive_is_allowed_to_implement_this_trait()
where
@@ -759,30 +848,33 @@ fn only_derive_is_allowed_to_implement_this_trait()
{
}
- type Tag = T::Tag;
+ type Tag = <T as HasTag<Client>>::Tag;
// SAFETY: `<T as SizeEq<ReadOnly<T>>>::CastFrom` is a no-op projection that
- // produces a pointer with the same referent. By invariant, for any `Ptr<'_,
- // T, I>` it is sound to use `T::ProjectToTag` to project to a `Ptr<'_,
- // T::Tag, I>`. Since `ReadOnly<T>` has the same layout and validity as `T`,
- // the same is true of projecting from a `Ptr<'_, ReadOnly<T>, I>`.
+ // produces a pointer with the same referent. By invariant, for any
+ // `Ptr<'_, T, I>` it is sound to use
+ // `<T as HasTag<Client>>::ProjectToTag` to project to a
+ // shared `Ptr<'_, <T as HasTag<Client>>::Tag,
+ // (_, I::Alignment, I::Validity)>`. Since `ReadOnly<T>` has the same
+ // layout and validity as `T`, the same is true of projecting from a
+ // `Ptr<'_, ReadOnly<T>, I>`.
type ProjectToTag = crate::pointer::cast::TransitiveProject<
T,
<T as SizeEq<ReadOnly<T>>>::CastFrom,
- T::ProjectToTag,
+ <T as HasTag<Client>>::ProjectToTag,
>;
}
// SAFETY: `ReadOnly<T>` is a `#[repr(transparent)]` wrapper around `T`, and so
// has the same fields at the same offsets. Thus, it satisfies the safety
-// invariants of `HasField<Field, VARIANT_ID, FIELD_ID>` for field `f` exactly
-// when `T` does, as guaranteed by the `T: HasField` bound:
-// - If `VARIANT_ID` is `STRUCT_VARIANT_ID` or `UNION_VARIANT_ID`, then `T` has
-// the layout of a struct or union type. Since `ReadOnly<T>` is a transparent
-// wrapper around `T`, it does too. Otherwise, if `VARIANT_ID` is an enum
-// variant index, then `T` has the layout of an enum type, and `ReadOnly<T>`
-// does too.
-// - By `T: HasField<_, _, FIELD_ID>`:
+// invariants of `HasField<Client, Field, VARIANT_ID, FIELD_ID>` for field `f`
+// exactly when `T` does, as guaranteed by the `T: HasField` bound:
+// - If `VARIANT_ID` is `STRUCT_VARIANT_ID`, `UNION_VARIANT_ID`, or
+// `REPR_C_UNION_VARIANT_ID`, then `T` has the layout of a struct, non-C union,
+// or C union type respectively. Since `ReadOnly<T>` is a transparent wrapper
+// around `T`, it does too. Otherwise, `VARIANT_ID` is the identifier ID of an
+// enum variant; `T` has the layout of an enum type, and `ReadOnly<T>` does too.
+// - By `T: HasField<_, _, _, FIELD_ID>`:
// - `T` has a field `f` with name `n` such that
// `FIELD_ID = zerocopy::ident_id!(n)` or at index `i` such that
// `FIELD_ID = zerocopy::ident_id!(i)`.
@@ -794,10 +886,10 @@ fn only_derive_is_allowed_to_implement_this_trait()
// refers to a non-strict subset of the bytes of `slf`'s referent, and has the
// same provenance as `slf` – because all intermediate operations satisfy those
// same conditions.
-unsafe impl<T, Field, const VARIANT_ID: i128, const FIELD_ID: i128>
- HasField<Field, VARIANT_ID, FIELD_ID> for ReadOnly<T>
+unsafe impl<T, Client, Field, const VARIANT_ID: i128, const FIELD_ID: i128>
+ HasField<Client, Field, VARIANT_ID, FIELD_ID> for ReadOnly<T>
where
- T: HasField<Field, VARIANT_ID, FIELD_ID> + ?Sized,
+ T: HasField<Client, Field, VARIANT_ID, FIELD_ID> + ?Sized,
{
#[allow(clippy::missing_inline_in_public_items)]
fn only_derive_is_allowed_to_implement_this_trait()
@@ -811,7 +903,7 @@ fn only_derive_is_allowed_to_implement_this_trait()
#[inline(always)]
fn project(slf: PtrInner<'_, Self>) -> *mut ReadOnly<T::Type> {
slf.project::<_, <T as SizeEq<ReadOnly<T>>>::CastFrom>()
- .project::<_, crate::pointer::cast::Projection<Field, VARIANT_ID, FIELD_ID>>()
+ .project::<_, crate::pointer::cast::Projection<Client, Field, VARIANT_ID, FIELD_ID>>()
.project::<_, <ReadOnly<T::Type> as SizeEq<T::Type>>::CastFrom>()
.as_non_null()
.as_ptr()
@@ -822,10 +914,10 @@ fn project(slf: PtrInner<'_, Self>) -> *mut ReadOnly<T::Type> {
// has the same fields at the same offsets. `is_projectable` simply delegates to
// `T::is_projectable`, which is sound because a `Ptr<'_, ReadOnly<T>, I>` will
// be projectable exactly when a `Ptr<'_, T, I>` referent is.
-unsafe impl<T, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128>
- ProjectField<Field, I, VARIANT_ID, FIELD_ID> for ReadOnly<T>
+unsafe impl<T, Client, Field, I, const VARIANT_ID: i128, const FIELD_ID: i128>
+ ProjectField<Client, Field, I, VARIANT_ID, FIELD_ID> for ReadOnly<T>
where
- T: ProjectField<Field, I, VARIANT_ID, FIELD_ID> + ?Sized,
+ T: ProjectField<Client, Field, I, VARIANT_ID, FIELD_ID> + ?Sized,
I: invariant::Invariants,
{
#[allow(clippy::missing_inline_in_public_items)]
@@ -835,13 +927,15 @@ fn only_derive_is_allowed_to_implement_this_trait()
{
}
- type Invariants = T::Invariants;
+ type Invariants = <T as ProjectField<Client, Field, I, VARIANT_ID, FIELD_ID>>::Invariants;
- type Error = T::Error;
+ type Error = <T as ProjectField<Client, Field, I, VARIANT_ID, FIELD_ID>>::Error;
#[inline(always)]
- fn is_projectable<'a>(ptr: Ptr<'a, Self::Tag, I>) -> Result<(), Self::Error> {
- T::is_projectable(ptr)
+ fn is_projectable<'a>(
+ ptr: Ptr<'a, <Self as HasTag<Client>>::Tag, (invariant::Shared, I::Alignment, I::Validity)>,
+ ) -> Result<(), Self::Error> {
+ <T as ProjectField<Client, Field, I, VARIANT_ID, FIELD_ID>>::is_projectable(ptr)
}
}
@@ -852,6 +946,96 @@ mod tests {
use super::*;
use crate::util::testutil::*;
+ #[test]
+ #[allow(clippy::clone_on_copy, clippy::non_canonical_clone_impl)]
+ fn test_read_only_copy_clone() {
+ // This type deliberately does not implement `Immutable`. Copying the
+ // wrapper must not require it or invoke the inner `Clone` impl.
+ #[derive(Copy, Debug, PartialEq, Eq, PartialOrd, Ord, Hash)]
+ struct PanickingClone(u16);
+
+ impl Clone for PanickingClone {
+ fn clone(&self) -> Self {
+ panic!("ReadOnly must not call the inner Clone implementation")
+ }
+ }
+
+ static_assertions::assert_not_impl_any!(
+ ReadOnly<PanickingClone>: Debug, PartialEq, Eq, PartialOrd, Ord,
+ Hash, AsRef<PanickingClone>, Borrow<PanickingClone>
+ );
+ static_assertions::assert_not_impl_any!(
+ ReadOnly<Cell<u8>>: Copy, Clone, Deref, AsRef<Cell<u8>>, Borrow<Cell<u8>>
+ );
+
+ let value = ReadOnly::new(PanickingClone(42));
+ let copy = value;
+ let clone = value.clone();
+ assert_eq!(ReadOnly::into_inner(copy).0, 42);
+ assert_eq!(ReadOnly::into_inner(clone).0, 42);
+ }
+
+ #[test]
+ fn test_read_only_mutable_access() {
+ // Construction and exclusive access also support interior mutability.
+ let mut value = ReadOnly::<Cell<u8>>::default();
+ let inner: &mut Cell<u8> = value.as_mut();
+ inner.set(42);
+ assert_eq!(ReadOnly::into_inner(value).get(), 42);
+
+ let value = ReadOnly::from(Cell::new(7));
+ assert_eq!(ReadOnly::into_inner(value).get(), 7);
+ }
+
+ #[test]
+ fn test_read_only_borrowed_lookup() {
+ // Borrowed lookups require equality, hashing, and ordering to agree
+ // between the wrapper and the inner value.
+ let mut hash_map = std::collections::HashMap::new();
+ hash_map.insert(ReadOnly::new(42u16), true);
+ assert_eq!(hash_map.get(&42u16), Some(&true));
+ assert_eq!(hash_map.get(&43u16), None);
+
+ let mut tree = std::collections::BTreeMap::new();
+ tree.insert(ReadOnly::new(42u16), true);
+ assert_eq!(tree.get(&42u16), Some(&true));
+ assert_eq!(tree.get(&43u16), None);
+
+ // Preserve partial orders, including incomparable values.
+ let nan = ReadOnly::new(f32::NAN);
+ assert_eq!(nan.partial_cmp(&nan), None);
+ assert_ne!(nan, nan);
+ }
+
+ #[test]
+ fn test_read_only_unsized() {
+ let bytes = [1u8, 2];
+ let value: &ReadOnly<[u8]> = (&bytes[..]).into();
+ let as_ref: &[u8] = value.as_ref();
+ let borrowed: &[u8] = value.borrow();
+ assert_eq!(as_ref, bytes);
+ assert_eq!(borrowed, bytes);
+ assert_eq!(value, value);
+ assert_eq!(value.cmp(value), Ordering::Equal);
+
+ let text: &ReadOnly<str> = "hello".into();
+ assert_eq!(format!("{:.3}", text), "hel");
+ }
+
+ #[test]
+ fn test_read_only_formatting() {
+ let value = ReadOnly::new(42u16);
+ // Check that formatting flags are forwarded as well as the value.
+ assert_eq!(format!("{:04?}", value), "0042");
+ assert_eq!(format!("{:04}", value), "0042");
+ assert_eq!(format!("{:#010b}", value), "0b00101010");
+ assert_eq!(format!("{:#06o}", value), "0o0052");
+ assert_eq!(format!("{:#06x}", value), "0x002a");
+ assert_eq!(format!("{:#06X}", value), "0x002A");
+ assert_eq!(format!("{:.2e}", value), "4.20e1");
+ assert_eq!(format!("{:.2E}", value), "4.20E1");
+ }
+
#[test]
fn test_unalign() {
// Test methods that don't depend on alignment.
--
2.56.0
^ permalink raw reply related [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-10-13 18:48 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-13 18:48 [PATCH 1/2] rust: zerocopy: fix link to upstream `README.md` Miguel Ojeda
2026-10-13 18:48 ` [PATCH 2/2] rust: zerocopy: update to v0.8.62 Miguel Ojeda
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.