Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Ariel Miculas (amiculas)]

Some of the patches were bounced by the rust-for-linux mailing list, you can see all the commits here:
https://github.com/Rust-for-Linux/linux/compare/rust-next...ariel-miculas:puzzlefs_rfc_v1

Regards,
Ariel


From: Ariel Miculas <amiculas@cisco.com>
Sent: Friday, June 9, 2023 9:29 AM
To: rust-for-linux@vger.kernel.org <rust-for-linux@vger.kernel.org>
Cc: Ariel Miculas (amiculas) <amiculas@cisco.com>
Subject: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver 
 
Hi all!

This is a proof of concept driver written for the PuzzleFS
next-generation container filesystem [1]. I've included a short abstract
about puzzlefs further below. This driver is based on the rust-next
branch, on top of which I've backported the filesystem abstractions from
Wedson Almeida Filho [2][3] and Miguel Ojeda's third-party crates
support: proc-macro2, quote, syn, serde and serde_derive [4]. I've added
the additional third-party crates serde_cbor[5] and hex [6]. Then I've
adapted the user space puzzlefs code [1] so that the puzzlefs kernel
module could present the directory hierarchy and implement the basic
read functionality.
For some additional context, puzzlefs was started by Tycho Andersen and
it's the successor of atomfs. This FOSDEM presentation from 2019 [12]
covers the rationale for a new oci image format and presents a higher
level overview of our goals with puzzlefs.
I've split the rest of the cover letter in following sections (using a
markdown style):
* Example: it describes a practical example of what was achieved
* Limitations: it presents the existing limitations of this POC
* Upstreaming steps: it describes the steps needed for upstreaming this
  driver
* Setup: it shows how to setup the necessary environment for testing the
  puzzlefs driver
* Puzzlefs abstract: it provides a short overview of puzzlefs

# Example
An example is provided below:

```
~ # cat /proc/filesystems | grep puzzlefs
nodev   puzzlefs
~ # mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o image_manifest="2d
6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" none /mnt
~ # ls -lR /mnt/
/mnt/:
total 0
drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-1
drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-2
drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-3
drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-4
-rw-r--r--    1 0        0                0 Jun  8 12:26 file1
-rw-r--r--    1 0        0                0 Jun  8 12:26 file2

/mnt/dir-1:
total 0

/mnt/dir-2:
total 0

/mnt/dir-3:
total 0

/mnt/dir-4:
total 0
~ # cat /mnt/file2
ana are mere bla bla bla
~ # wc /mnt/file1
      202       202      5454 /mnt/file1
```

In this example, /home/puzzlefs_oci is the puzzlefs oci directory:
```
~ # ls -lR /home/puzzlefs_oci/
/home/puzzlefs_oci/:
total 8
drwxr-xr-x    3 1000     1000             0 Jun  8 14:33 blobs
-rw-r--r--    1 1000     1000           266 Jun  8 14:33 index.json
-rw-r--r--    1 1000     1000            37 Jun  8 14:33 oci-layout

/home/puzzlefs_oci/blobs:
total 0
drwxr-xr-x    2 1000     1000             0 Jun  8 14:33 sha256

/home/puzzlefs_oci/blobs/sha256:
total 16
-rw-------    1 1000     1000            89 Jun  8 14:33 2d6602d678140540dc7e96de652a76a8b16eb
-rw-------    1 1000     1000           925 Jun  8 14:33 4df03518eea406343dbb55046720f6a478881
-rw-------    1 1000     1000          5479 Jun  8 14:33 d86a87b19bd9a2fec0d31687c1d669cdb59eb
```

`2d6602d678140540dc7e96de652a76a8b16eb` is the puzzlefs image manifest
hash for the first_try tag:
```
$ cat /tmp/oci-simple/index.json | jq .
{
  "schemaVersion": -1,
  "manifests": [
    {
      "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b",
      "size": 89,
      "media_type": "application/vnd.puzzlefs.image.rootfs.v1",
      "annotations": {
        "org.opencontainers.image.ref.name": "first_try"
      }
    }
  ],
  "annotations": {}
}
```

I will describe how to build a puzzlefs image in the `Setup` section, at
step 5.

# Limitations
One limitation is that the puzzlefs driver doesn't implement any lookup
functionality and instead it inserts every directory entry into the
dcache during init (see the `DCACHE_BASED` constant). This is similar to
how the sample `rust_fs` driver works, but the goal is to implement
proper lookup functions.  However, more filesystem abstractions need to
be implemented before this can be achieved.

# Upstreaming steps
Before the puzzlefs driver can be upstreamed, the following need to be
merged:
* Wedson's filesystem abstractions [3]
* the necessary third-party crates [4] (with the preliminary discussion
about whether this is desirable)

# Setup
My setup is based on Wedson's tutorial [8]. Next, I will describe the
necessary steps to build an initrd and run a custom kernel under qemu.

1. Get the linux rust-next branch [9] and apply this patchset

2. Follow the rust quickstart guide [10]

3. configure and build the kernel
```
$ make LLVM=1 allnoconfig qemu-busybox-min.config rust.config
$ make LLVM=1 -j$(nproc)
```

4. setup busybox
```
git clone git://git.busybox.net/busybox
cd busybox
make menuconfig # enable 'Build static binary' config
make
make install
```
This will create the `_install` directory with the rootfs inside it.

5. create a home directory in the rootfs and copy a puzzlefs oci
directory in home/puzzlefs_oci
To create a puzzlefs oci directory:
* download this custom puzzlefs repository [11] (it's custom because we
  want to build an image without verity data)
* run `make release`
* create a simple filesystem structure with a few directories and files
  (I've created one at ../test-puzzlefs/simple_rootfs)
* build a puzzlefs oci image at
  `~/work/busybox/_install/home/puzzlefs_oci` (replace this path with
  your busybox path) with the tag `first_try`:
```
$ target/release/puzzlefs build --omit-verity \
../test-puzzlefs/simple_rootfs ~/work/busybox/_install/home/puzzlefs_oci \
first_try
```
* get first_try's image manifest from index.json (inside `puzzlefs_oci`)
```
$ cat index.json | jq . | grep digest
      "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b",
```

6. add the following 'init' script in the busybox rootfs (rootfs path
defaults to `./_install'):
```
#!/bin/sh
mount -t devtmpfs none /dev
mkdir -p /proc
mount -t proc none /proc

ifconfig lo up
udhcpc -i eth0

mkdir /mnt
mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \
image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" \
none /mnt

setsid sh -c 'exec sh -l </dev/ttyS0 >/dev/ttyS0 2>&1'
```

Make sure to replace the `image_manifest` with your own digest. This
init script will be passed to rdinit in the kernel command line.

7. generate the initramfs

Assuming busybox is in `~/work/busybox`:
```
cd ~/work/busybox/_install && find . | cpio -H newc -o | gzip > ../ramdisk.img
```
This will generate a compressed ramdisk image in
`~/work/busybox/ramdisk.img`.

8. run with qemu (this assumes the linux tree is at '../linux' and busybox
is at '../busybox'):
```
qemu-system-x86_64 \
    -accel kvm \
    -cpu host \
    -m 4G \
    -initrd ../busybox/ramdisk.img \
    -kernel ../linux/arch/x86/boot/bzImage \
    -nographic \
    -append 'console=ttyS0 nokaslr debug rdinit=/init' \
    -nic user,model=rtl8139 \
    -no-reboot
```

9. Check whether puzzlefs has been successfully mounted
```
~ # mount | grep puzzlefs
none on /mnt type puzzlefs (rw,relatime)
~ # ls /mnt/
dir-1  dir-2  dir-3  dir-4  file1  file2
```


# Puzzlefs abstract
Puzzlefs [1] is a container filesystem designed to address the
limitations of the existing OCI format. The main goals of the project
are reduced duplication, reproducible image builds, direct mounting
support and memory safety guarantees, some inspired by the OCIv2 design
document [7].

Reduced duplication is achieved using the content defined chunking
algorithm FastCDC. This implementation allows chunks to be shared among
layers. Building a new layer starting from an existing one allows
reusing most of the chunks.

Another goal of the project is reproducible image builds, which is
achieved by defining a canonical representation of the image format.

Direct mounting support is a key feature of puzzlefs and, together with
fs-verity, it provides data integrity. Currently, puzzlefs is
implemented as a userspace filesystem (FUSE). A read-only kernel
filesystem driver is underway.

Lastly, memory safety is critical to puzzlefs, leading to the decision
to implement it in Rust. Another goal is to share the same code between
user space and kernel space in order to provide one secure
implementation.


[1] https://github.com/anuvu/puzzlefs
[2] https://github.com/wedsonaf/linux/tree/fs
[3] https://github.com/Rust-for-Linux/linux/issues/1004
[4] https://github.com/Rust-for-Linux/linux/pull/1007
[5] https://docs.rs/serde_cbor/latest/serde_cbor/
[6] https://docs.rs/hex/0.4.3/hex/
[7] https://hackmd.io/@cyphar/ociv2-brainstorm
[8] https://www.youtube.com/watch?v=tPs1uRqOnlk
[9] https://github.com/Rust-for-Linux/linux/tree/rust-next
[10] https://docs.kernel.org/rust/quick-start.html
[11] https://github.com/ariel-miculas/puzzlefs/tree/support-no-verity-data
[12] https://archive.fosdem.org/2019/schedule/event/containers_atomfs/

Ariel Miculas (58):
  rust: kernel: add libraries required by the filesystem abstractions
  rust: kernel: backport the delay module from the rust branch
  rust: kernel: add container_of macro
  rust: kernel: add offset_of macro
  drop: Add crate::pr_warn declaration
  rust: kernel: rename from_kernel_errno to from_errno
  rust: kernel: Rename from_pointer to from_foreing and into_pointer to
    into_foreign
  rust: kernel: add count_paren_items macro, needed by define_fs_params
    macro
  rust: helpers: add missing rust helper 'alloc_pages'
  kernel: configs: add qemu-busybox-min.config
  rust: kernel: format the rust code
  samples: puzzlefs: add initial puzzlefs sample, copied from rust_fs.rs
  kernel: configs: enable rust samples in rust.config
  Add SAMPLE_RUST_SERDE in rust.config
  rust: kernel: fix compile errors after rebase to rust-next
  rust: serde_cbor: import crate
  rust: serde_cbor: add SPDX License Identifiers
  rust: serde_cbor: add no_fp_fmt_parse support
  rust: Kbuild: enable serde_cbor
  samples: rust: add cbor serialize/deserialize example
  rust: serde_cbor: add support for serde_cbor's from_slice method by
    using a custom alloc_kernel feature
  rust: serde: add support for deserializing Vec with kernel_alloc
    feature
  rust: file: Replace UnsafeCell with Opaque for File
  rust: kernel: implement fmt::Debug for CString
  samples: puzzlefs: rename RustFs to PuzzleFs
  samples: puzzlefs: add basic deserializing support for the puzzlefs
    metadata
  rust: file: present the filesystem context to the open function
  rust: kernel: add an abstraction over vfsmount to allow cloning a new
    private mount
  rust: file: add from_path, from_path_in_root_mnt and read_with_offset
    methods to File
  samples: puzzlefs: pass the Vfsmount structure from open to read and
    return the contents of the data file inside /home/puzzlefs_oci
  rust: file: move from_path, from_path_in_root_mnt and read_with_offset
    methods to a RegularFile newtype
  rust: file: ensure RegularFile can only create regular files
  rust: file: add get_pos method to RegularFile
  rust: file: add methods read_to_end, get_file_size and update_pos to
    RegularFile
  rust: file: define a minimal Read trait and implement it for
    RegularFile
  samples: puzzlefs: add cbor_get_array_size method
  samples: puzzlefs: add KernelError to WireFormatError and implement
    From conversion
  samples: puzzlefs: implement new for MetadataBlob
  samples: puzzlefs: build puzzlefs into the kernel, thus avoiding the
    need to export rust symbols
  rust: alloc: add try_clone for Vec<T>
  rust: alloc: add from_iter_fallible for Vec<T>
  samples: puzzlefs: implement to_errno and from_errno for
    WireFormatError
  samples: puzzlefs: add TryReserveError (and from conversion) to
    WireFormatError
  samples: puzzlefs: add higher level inode related functionality
  samples: puzzlefs: populate the directory entries with the inodes from
    the puzzlefs metadata file
  rust: hex: import crate
  rust: hex: add SPDX license identifiers
  rust: Kbuild: enable `hex`
  rust: hex: implement FromHex trait and hex::decode using a custom
    kernel_alloc feature
  rust: hex: add encode_hex_iter and encode_hex_upper_iter methods
  rust: puzzlefs: add HexError to WireFormatError and implement the From
    conversion
  rust: puzzlefs: display the error value for
    WireFormatError::KernelError
  samples: puzzlefs: add Rootfs and Digest structs to types.rs
  samples: puzzlefs: implement the conversion from WireFormatError to
    kernel::error::Error
  rust: puzzlefs: read the puzzlefs image manifest instead of an
    individual metadata layer
  rust: puzzlefs: rename PuzzleFs to PuzzleFsModule to avoid confusion
    with the PuzzleFS struct
  rust: puzzlefs: add support for reading files
  rust: puzzlefs: add oci_root_dir and image_manifest filesystem
    parameters

Miguel Ojeda (15):
  rust: proc-macro2: import crate
  rust: proc-macro2: add SPDX License Identifiers
  rust: proc-macro2: remove `unicode_ident` dependency
  rust: quote: import crate
  rust: quote: add SPDX License Identifiers
  rust: syn: import crate
  rust: syn: add SPDX License Identifiers
  rust: syn: remove `unicode-ident` dependency
  rust: serde: import crate
  rust: serde: add `no_fp_fmt_parse` support
  rust: serde: add SPDX License Identifiers
  rust: serde_derive: import crate
  rust: serde_derive: add SPDX License Identifiers
  rust: Kbuild: enable `proc-macro2`, `quote`, `syn`, `serde` and
    `serde_derive`
  rust: test `serde` support

Wedson Almeida Filho (7):
  rust: add definitions for ref-counted inodes and dentries
  rust: add ability to register a file system
  rust: define fs context
  rust: add support for file system parameters
  rust: allow fs driver to initialise new superblocks
  rust: add `module_fs` macro
  WIP: rust: allow fs to be populated

 Makefile                                  |   14 +-
 arch/x86/configs/qemu-busybox-min.config  |   11 +
 kernel/configs/qemu-busybox-min.config    |   56 +
 kernel/configs/rust.config                |   11 +
 rust/.gitignore                           |    1 +
 rust/Makefile                             |  232 +-
 rust/alloc/vec/mod.rs                     |   48 +
 rust/bindings/bindings_helper.h           |   14 +
 rust/bindings/lib.rs                      |    5 +
 rust/helpers.c                            |   76 +
 rust/hex/error.rs                         |   78 +
 rust/hex/lib.rs                           |  506 +++
 rust/hex/serde.rs                         |  104 +
 rust/kernel/cred.rs                       |   46 +
 rust/kernel/delay.rs                      |  104 +
 rust/kernel/driver.rs                     |   28 +
 rust/kernel/error.rs                      |   52 +-
 rust/kernel/file.rs                       | 1117 ++++++
 rust/kernel/fs.rs                         | 1478 ++++++++
 rust/kernel/fs/param.rs                   |  558 +++
 rust/kernel/io_buffer.rs                  |  153 +
 rust/kernel/iov_iter.rs                   |   81 +
 rust/kernel/lib.rs                        |   83 +
 rust/kernel/mm.rs                         |  149 +
 rust/kernel/mount.rs                      |   66 +
 rust/kernel/pages.rs                      |  144 +
 rust/kernel/str.rs                        |    6 +
 rust/kernel/test_serde.rs                 |   26 +
 rust/kernel/test_serde/de.rs              |  439 +++
 rust/kernel/test_serde/error.rs           |   73 +
 rust/kernel/test_serde/ser.rs             |  466 +++
 rust/kernel/user_ptr.rs                   |  175 +
 rust/proc-macro2/detection.rs             |   77 +
 rust/proc-macro2/fallback.rs              | 1004 ++++++
 rust/proc-macro2/lib.rs                   | 1341 ++++++++
 rust/proc-macro2/marker.rs                |   20 +
 rust/proc-macro2/parse.rs                 |  874 +++++
 rust/proc-macro2/rcvec.rs                 |  144 +
 rust/proc-macro2/wrapper.rs               |  996 ++++++
 rust/quote/ext.rs                         |  112 +
 rust/quote/format.rs                      |  170 +
 rust/quote/ident_fragment.rs              |   88 +
 rust/quote/lib.rs                         | 1436 ++++++++
 rust/quote/runtime.rs                     |  440 +++
 rust/quote/spanned.rs                     |   45 +
 rust/quote/to_tokens.rs                   |  211 ++
 rust/serde/de/format.rs                   |   32 +
 rust/serde/de/ignored_any.rs              |  246 ++
 rust/serde/de/impls.rs                    | 2755 +++++++++++++++
 rust/serde/de/mod.rs                      | 2313 +++++++++++++
 rust/serde/de/seed.rs                     |   21 +
 rust/serde/de/utf8.rs                     |   48 +
 rust/serde/de/value.rs                    | 1718 ++++++++++
 rust/serde/integer128.rs                  |   84 +
 rust/serde/lib.rs                         |  351 ++
 rust/serde/macros.rs                      |  238 ++
 rust/serde/private/de.rs                  | 2997 ++++++++++++++++
 rust/serde/private/doc.rs                 |  161 +
 rust/serde/private/mod.rs                 |   52 +
 rust/serde/private/ser.rs                 | 1316 +++++++
 rust/serde/private/size_hint.rs           |   23 +
 rust/serde/ser/fmt.rs                     |  180 +
 rust/serde/ser/impls.rs                   |  987 ++++++
 rust/serde/ser/impossible.rs              |  218 ++
 rust/serde/ser/mod.rs                     | 1992 +++++++++++
 rust/serde/std_error.rs                   |   50 +
 rust/serde_cbor/de.rs                     | 1370 ++++++++
 rust/serde_cbor/error.rs                  |  320 ++
 rust/serde_cbor/lib.rs                    |  371 ++
 rust/serde_cbor/read.rs                   |  647 ++++
 rust/serde_cbor/ser.rs                    |  748 ++++
 rust/serde_cbor/tags.rs                   |  224 ++
 rust/serde_cbor/value/de.rs               |  168 +
 rust/serde_cbor/value/mod.rs              |  158 +
 rust/serde_cbor/value/ser.rs              |  447 +++
 rust/serde_cbor/write.rs                  |  177 +
 rust/serde_derive/bound.rs                |  408 +++
 rust/serde_derive/de.rs                   | 3148 +++++++++++++++++
 rust/serde_derive/dummy.rs                |   46 +
 rust/serde_derive/fragment.rs             |   76 +
 rust/serde_derive/internals/ast.rs        |  204 ++
 rust/serde_derive/internals/attr.rs       | 1908 +++++++++++
 rust/serde_derive/internals/case.rs       |  199 ++
 rust/serde_derive/internals/check.rs      |  445 +++
 rust/serde_derive/internals/ctxt.rs       |   64 +
 rust/serde_derive/internals/mod.rs        |   30 +
 rust/serde_derive/internals/receiver.rs   |  287 ++
 rust/serde_derive/internals/respan.rs     |   18 +
 rust/serde_derive/internals/symbol.rs     |   71 +
 rust/serde_derive/lib.rs                  |  112 +
 rust/serde_derive/pretend.rs              |  203 ++
 rust/serde_derive/ser.rs                  | 1342 ++++++++
 rust/serde_derive/this.rs                 |   34 +
 rust/serde_derive/try.rs                  |   26 +
 rust/syn/attr.rs                          |  664 ++++
 rust/syn/await.rs                         |    4 +
 rust/syn/bigint.rs                        |   68 +
 rust/syn/buffer.rs                        |  400 +++
 rust/syn/custom_keyword.rs                |  255 ++
 rust/syn/custom_punctuation.rs            |  302 ++
 rust/syn/data.rs                          |  495 +++
 rust/syn/derive.rs                        |  276 ++
 rust/syn/discouraged.rs                   |  196 ++
 rust/syn/error.rs                         |  430 +++
 rust/syn/export.rs                        |   41 +
 rust/syn/expr.rs                          | 3560 +++++++++++++++++++
 rust/syn/ext.rs                           |  141 +
 rust/syn/file.rs                          |  127 +
 rust/syn/gen/clone.rs                     | 2243 ++++++++++++
 rust/syn/gen/debug.rs                     | 3044 +++++++++++++++++
 rust/syn/gen/eq.rs                        | 2197 ++++++++++++
 rust/syn/gen/fold.rs                      | 3343 ++++++++++++++++++
 rust/syn/gen/hash.rs                      | 2871 ++++++++++++++++
 rust/syn/gen/visit.rs                     | 3788 +++++++++++++++++++++
 rust/syn/gen/visit_mut.rs                 | 3788 +++++++++++++++++++++
 rust/syn/gen_helper.rs                    |  156 +
 rust/syn/generics.rs                      | 1339 ++++++++
 rust/syn/group.rs                         |  284 ++
 rust/syn/ident.rs                         |  103 +
 rust/syn/item.rs                          | 3315 ++++++++++++++++++
 rust/syn/lib.rs                           |  985 ++++++
 rust/syn/lifetime.rs                      |  156 +
 rust/syn/lit.rs                           | 1602 +++++++++
 rust/syn/lookahead.rs                     |  171 +
 rust/syn/mac.rs                           |  221 ++
 rust/syn/macros.rs                        |  179 +
 rust/syn/op.rs                            |  236 ++
 rust/syn/parse.rs                         | 1316 +++++++
 rust/syn/parse_macro_input.rs             |  181 +
 rust/syn/parse_quote.rs                   |  169 +
 rust/syn/pat.rs                           |  929 +++++
 rust/syn/path.rs                          |  856 +++++
 rust/syn/print.rs                         |   18 +
 rust/syn/punctuated.rs                    | 1070 ++++++
 rust/syn/reserved.rs                      |   46 +
 rust/syn/sealed.rs                        |    6 +
 rust/syn/span.rs                          |   69 +
 rust/syn/spanned.rs                       |  116 +
 rust/syn/stmt.rs                          |  351 ++
 rust/syn/thread.rs                        |   43 +
 rust/syn/token.rs                         | 1015 ++++++
 rust/syn/tt.rs                            |  109 +
 rust/syn/ty.rs                            | 1288 +++++++
 rust/syn/verbatim.rs                      |   35 +
 rust/syn/whitespace.rs                    |   67 +
 samples/rust/Kconfig                      |   28 +
 samples/rust/Makefile                     |    3 +
 samples/rust/local_data_format/de.rs      |  422 +++
 samples/rust/local_data_format/error.rs   |   73 +
 samples/rust/local_data_format/ser.rs     |  443 +++
 samples/rust/puzzle.rs                    |    4 +
 samples/rust/puzzle/error.rs              |   91 +
 samples/rust/puzzle/inode.rs              |  150 +
 samples/rust/puzzle/oci.rs                |   71 +
 samples/rust/puzzle/types.rs              |  389 +++
 samples/rust/puzzle/types/cbor_helpers.rs |   50 +
 samples/rust/puzzlefs.rs                  |  220 ++
 samples/rust/rust_fs.rs                   |  105 +
 samples/rust/rust_serde.rs                |  125 +
 scripts/Makefile.build                    |    4 +-
 160 files changed, 89204 insertions(+), 29 deletions(-)
 create mode 100644 arch/x86/configs/qemu-busybox-min.config
 create mode 100644 kernel/configs/qemu-busybox-min.config
 create mode 100644 rust/hex/error.rs
 create mode 100644 rust/hex/lib.rs
 create mode 100644 rust/hex/serde.rs
 create mode 100644 rust/kernel/cred.rs
 create mode 100644 rust/kernel/delay.rs
 create mode 100644 rust/kernel/driver.rs
 create mode 100644 rust/kernel/file.rs
 create mode 100644 rust/kernel/fs.rs
 create mode 100644 rust/kernel/fs/param.rs
 create mode 100644 rust/kernel/io_buffer.rs
 create mode 100644 rust/kernel/iov_iter.rs
 create mode 100644 rust/kernel/mm.rs
 create mode 100644 rust/kernel/mount.rs
 create mode 100644 rust/kernel/pages.rs
 create mode 100644 rust/kernel/test_serde.rs
 create mode 100644 rust/kernel/test_serde/de.rs
 create mode 100644 rust/kernel/test_serde/error.rs
 create mode 100644 rust/kernel/test_serde/ser.rs
 create mode 100644 rust/kernel/user_ptr.rs
 create mode 100644 rust/proc-macro2/detection.rs
 create mode 100644 rust/proc-macro2/fallback.rs
 create mode 100644 rust/proc-macro2/lib.rs
 create mode 100644 rust/proc-macro2/marker.rs
 create mode 100644 rust/proc-macro2/parse.rs
 create mode 100644 rust/proc-macro2/rcvec.rs
 create mode 100644 rust/proc-macro2/wrapper.rs
 create mode 100644 rust/quote/ext.rs
 create mode 100644 rust/quote/format.rs
 create mode 100644 rust/quote/ident_fragment.rs
 create mode 100644 rust/quote/lib.rs
 create mode 100644 rust/quote/runtime.rs
 create mode 100644 rust/quote/spanned.rs
 create mode 100644 rust/quote/to_tokens.rs
 create mode 100644 rust/serde/de/format.rs
 create mode 100644 rust/serde/de/ignored_any.rs
 create mode 100644 rust/serde/de/impls.rs
 create mode 100644 rust/serde/de/mod.rs
 create mode 100644 rust/serde/de/seed.rs
 create mode 100644 rust/serde/de/utf8.rs
 create mode 100644 rust/serde/de/value.rs
 create mode 100644 rust/serde/integer128.rs
 create mode 100644 rust/serde/lib.rs
 create mode 100644 rust/serde/macros.rs
 create mode 100644 rust/serde/private/de.rs
 create mode 100644 rust/serde/private/doc.rs
 create mode 100644 rust/serde/private/mod.rs
 create mode 100644 rust/serde/private/ser.rs
 create mode 100644 rust/serde/private/size_hint.rs
 create mode 100644 rust/serde/ser/fmt.rs
 create mode 100644 rust/serde/ser/impls.rs
 create mode 100644 rust/serde/ser/impossible.rs
 create mode 100644 rust/serde/ser/mod.rs
 create mode 100644 rust/serde/std_error.rs
 create mode 100644 rust/serde_cbor/de.rs
 create mode 100644 rust/serde_cbor/error.rs
 create mode 100644 rust/serde_cbor/lib.rs
 create mode 100644 rust/serde_cbor/read.rs
 create mode 100644 rust/serde_cbor/ser.rs
 create mode 100644 rust/serde_cbor/tags.rs
 create mode 100644 rust/serde_cbor/value/de.rs
 create mode 100644 rust/serde_cbor/value/mod.rs
 create mode 100644 rust/serde_cbor/value/ser.rs
 create mode 100644 rust/serde_cbor/write.rs
 create mode 100644 rust/serde_derive/bound.rs
 create mode 100644 rust/serde_derive/de.rs
 create mode 100644 rust/serde_derive/dummy.rs
 create mode 100644 rust/serde_derive/fragment.rs
 create mode 100644 rust/serde_derive/internals/ast.rs
 create mode 100644 rust/serde_derive/internals/attr.rs
 create mode 100644 rust/serde_derive/internals/case.rs
 create mode 100644 rust/serde_derive/internals/check.rs
 create mode 100644 rust/serde_derive/internals/ctxt.rs
 create mode 100644 rust/serde_derive/internals/mod.rs
 create mode 100644 rust/serde_derive/internals/receiver.rs
 create mode 100644 rust/serde_derive/internals/respan.rs
 create mode 100644 rust/serde_derive/internals/symbol.rs
 create mode 100644 rust/serde_derive/lib.rs
 create mode 100644 rust/serde_derive/pretend.rs
 create mode 100644 rust/serde_derive/ser.rs
 create mode 100644 rust/serde_derive/this.rs
 create mode 100644 rust/serde_derive/try.rs
 create mode 100644 rust/syn/attr.rs
 create mode 100644 rust/syn/await.rs
 create mode 100644 rust/syn/bigint.rs
 create mode 100644 rust/syn/buffer.rs
 create mode 100644 rust/syn/custom_keyword.rs
 create mode 100644 rust/syn/custom_punctuation.rs
 create mode 100644 rust/syn/data.rs
 create mode 100644 rust/syn/derive.rs
 create mode 100644 rust/syn/discouraged.rs
 create mode 100644 rust/syn/error.rs
 create mode 100644 rust/syn/export.rs
 create mode 100644 rust/syn/expr.rs
 create mode 100644 rust/syn/ext.rs
 create mode 100644 rust/syn/file.rs
 create mode 100644 rust/syn/gen/clone.rs
 create mode 100644 rust/syn/gen/debug.rs
 create mode 100644 rust/syn/gen/eq.rs
 create mode 100644 rust/syn/gen/fold.rs
 create mode 100644 rust/syn/gen/hash.rs
 create mode 100644 rust/syn/gen/visit.rs
 create mode 100644 rust/syn/gen/visit_mut.rs
 create mode 100644 rust/syn/gen_helper.rs
 create mode 100644 rust/syn/generics.rs
 create mode 100644 rust/syn/group.rs
 create mode 100644 rust/syn/ident.rs
 create mode 100644 rust/syn/item.rs
 create mode 100644 rust/syn/lib.rs
 create mode 100644 rust/syn/lifetime.rs
 create mode 100644 rust/syn/lit.rs
 create mode 100644 rust/syn/lookahead.rs
 create mode 100644 rust/syn/mac.rs
 create mode 100644 rust/syn/macros.rs
 create mode 100644 rust/syn/op.rs
 create mode 100644 rust/syn/parse.rs
 create mode 100644 rust/syn/parse_macro_input.rs
 create mode 100644 rust/syn/parse_quote.rs
 create mode 100644 rust/syn/pat.rs
 create mode 100644 rust/syn/path.rs
 create mode 100644 rust/syn/print.rs
 create mode 100644 rust/syn/punctuated.rs
 create mode 100644 rust/syn/reserved.rs
 create mode 100644 rust/syn/sealed.rs
 create mode 100644 rust/syn/span.rs
 create mode 100644 rust/syn/spanned.rs
 create mode 100644 rust/syn/stmt.rs
 create mode 100644 rust/syn/thread.rs
 create mode 100644 rust/syn/token.rs
 create mode 100644 rust/syn/tt.rs
 create mode 100644 rust/syn/ty.rs
 create mode 100644 rust/syn/verbatim.rs
 create mode 100644 rust/syn/whitespace.rs
 create mode 100644 samples/rust/local_data_format/de.rs
 create mode 100644 samples/rust/local_data_format/error.rs
 create mode 100644 samples/rust/local_data_format/ser.rs
 create mode 100644 samples/rust/puzzle.rs
 create mode 100644 samples/rust/puzzle/error.rs
 create mode 100644 samples/rust/puzzle/inode.rs
 create mode 100644 samples/rust/puzzle/oci.rs
 create mode 100644 samples/rust/puzzle/types.rs
 create mode 100644 samples/rust/puzzle/types/cbor_helpers.rs
 create mode 100644 samples/rust/puzzlefs.rs
 create mode 100644 samples/rust/rust_fs.rs
 create mode 100644 samples/rust/rust_serde.rs

-- 
2.40.1

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Miguel Ojeda]

On Fri, Jun 9, 2023 at 9:37 AM Ariel Miculas (amiculas)
<amiculas@cisco.com> wrote:
>
> Some of the patches were bounced by the rust-for-linux mailing list, you can see all the commits here:
> https://github.com/Rust-for-Linux/linux/compare/rust-next...ariel-miculas:puzzlefs_rfc_v1

To ensure Lore sees them, you can add a Cc to patches@lists.linux.dev
for future submissions.

Hope that helps!

Cheers,
Miguel

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Ariel Miculas (amiculas)]

Thanks, good to know!

Regards,
Ariel


From: Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>
Sent: Friday, June 9, 2023 11:30 AM
To: Ariel Miculas (amiculas) <amiculas@cisco.com>
Cc: rust-for-linux@vger.kernel.org <rust-for-linux@vger.kernel.org>
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver 
 
On Fri, Jun 9, 2023 at 9:37 AM Ariel Miculas (amiculas)
<amiculas@cisco.com> wrote:
>
> Some of the patches were bounced by the rust-for-linux mailing list, you can see all the commits here:
> https://github.com/Rust-for-Linux/linux/compare/rust-next...ariel-miculas:puzzlefs_rfc_v1

To ensure Lore sees them, you can add a Cc to patches@lists.linux.dev
for future submissions.

Hope that helps!

Cheers,
Miguel

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Andreas Hindborg (Samsung)]

"Ariel Miculas (amiculas)" <amiculas@cisco.com> writes:

> Some of the patches were bounced by the rust-for-linux mailing list, you can see all the commits here:
> https://github.com/Rust-for-Linux/linux/compare/rust-next...ariel-miculas:puzzlefs_rfc_v1

This is super annoying when it happens. What is the reason, are they too
large? Miguel: can we do anything about this?

BR Andreas

>
> Regards,
> Ariel
>
>
> From: Ariel Miculas <amiculas@cisco.com>
> Sent: Friday, June 9, 2023 9:29 AM
> To: rust-for-linux@vger.kernel.org <rust-for-linux@vger.kernel.org>
> Cc: Ariel Miculas (amiculas) <amiculas@cisco.com>
> Subject: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver 
>  
> Hi all!
>
> This is a proof of concept driver written for the PuzzleFS
> next-generation container filesystem [1]. I've included a short abstract
> about puzzlefs further below. This driver is based on the rust-next
> branch, on top of which I've backported the filesystem abstractions from
> Wedson Almeida Filho [2][3] and Miguel Ojeda's third-party crates
> support: proc-macro2, quote, syn, serde and serde_derive [4]. I've added
> the additional third-party crates serde_cbor[5] and hex [6]. Then I've
> adapted the user space puzzlefs code [1] so that the puzzlefs kernel
> module could present the directory hierarchy and implement the basic
> read functionality.
> For some additional context, puzzlefs was started by Tycho Andersen and
> it's the successor of atomfs. This FOSDEM presentation from 2019 [12]
> covers the rationale for a new oci image format and presents a higher
> level overview of our goals with puzzlefs.
> I've split the rest of the cover letter in following sections (using a
> markdown style):
> * Example: it describes a practical example of what was achieved
> * Limitations: it presents the existing limitations of this POC
> * Upstreaming steps: it describes the steps needed for upstreaming this
>   driver
> * Setup: it shows how to setup the necessary environment for testing the
>   puzzlefs driver
> * Puzzlefs abstract: it provides a short overview of puzzlefs
>
> # Example
> An example is provided below:
>
> ```
> ~ # cat /proc/filesystems | grep puzzlefs
> nodev   puzzlefs
> ~ # mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o image_manifest="2d
> 6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" none /mnt
> ~ # ls -lR /mnt/
> /mnt/:
> total 0
> drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-1
> drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-2
> drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-3
> drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-4
> -rw-r--r--    1 0        0                0 Jun  8 12:26 file1
> -rw-r--r--    1 0        0                0 Jun  8 12:26 file2
>
> /mnt/dir-1:
> total 0
>
> /mnt/dir-2:
> total 0
>
> /mnt/dir-3:
> total 0
>
> /mnt/dir-4:
> total 0
> ~ # cat /mnt/file2
> ana are mere bla bla bla
> ~ # wc /mnt/file1
>       202       202      5454 /mnt/file1
> ```
>
> In this example, /home/puzzlefs_oci is the puzzlefs oci directory:
> ```
> ~ # ls -lR /home/puzzlefs_oci/
> /home/puzzlefs_oci/:
> total 8
> drwxr-xr-x    3 1000     1000             0 Jun  8 14:33 blobs
> -rw-r--r--    1 1000     1000           266 Jun  8 14:33 index.json
> -rw-r--r--    1 1000     1000            37 Jun  8 14:33 oci-layout
>
> /home/puzzlefs_oci/blobs:
> total 0
> drwxr-xr-x    2 1000     1000             0 Jun  8 14:33 sha256
>
> /home/puzzlefs_oci/blobs/sha256:
> total 16
> -rw-------    1 1000     1000            89 Jun  8 14:33 2d6602d678140540dc7e96de652a76a8b16eb
> -rw-------    1 1000     1000           925 Jun  8 14:33 4df03518eea406343dbb55046720f6a478881
> -rw-------    1 1000     1000          5479 Jun  8 14:33 d86a87b19bd9a2fec0d31687c1d669cdb59eb
> ```
>
> `2d6602d678140540dc7e96de652a76a8b16eb` is the puzzlefs image manifest
> hash for the first_try tag:
> ```
> $ cat /tmp/oci-simple/index.json | jq .
> {
>   "schemaVersion": -1,
>   "manifests": [
>     {
>       "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b",
>       "size": 89,
>       "media_type": "application/vnd.puzzlefs.image.rootfs.v1",
>       "annotations": {
>         "org.opencontainers.image.ref.name": "first_try"
>       }
>     }
>   ],
>   "annotations": {}
> }
> ```
>
> I will describe how to build a puzzlefs image in the `Setup` section, at
> step 5.
>
> # Limitations
> One limitation is that the puzzlefs driver doesn't implement any lookup
> functionality and instead it inserts every directory entry into the
> dcache during init (see the `DCACHE_BASED` constant). This is similar to
> how the sample `rust_fs` driver works, but the goal is to implement
> proper lookup functions.  However, more filesystem abstractions need to
> be implemented before this can be achieved.
>
> # Upstreaming steps
> Before the puzzlefs driver can be upstreamed, the following need to be
> merged:
> * Wedson's filesystem abstractions [3]
> * the necessary third-party crates [4] (with the preliminary discussion
> about whether this is desirable)
>
> # Setup
> My setup is based on Wedson's tutorial [8]. Next, I will describe the
> necessary steps to build an initrd and run a custom kernel under qemu.
>
> 1. Get the linux rust-next branch [9] and apply this patchset
>
> 2. Follow the rust quickstart guide [10]
>
> 3. configure and build the kernel
> ```
> $ make LLVM=1 allnoconfig qemu-busybox-min.config rust.config
> $ make LLVM=1 -j$(nproc)
> ```
>
> 4. setup busybox
> ```
> git clone git://git.busybox.net/busybox
> cd busybox
> make menuconfig # enable 'Build static binary' config
> make
> make install
> ```
> This will create the `_install` directory with the rootfs inside it.
>
> 5. create a home directory in the rootfs and copy a puzzlefs oci
> directory in home/puzzlefs_oci
> To create a puzzlefs oci directory:
> * download this custom puzzlefs repository [11] (it's custom because we
>   want to build an image without verity data)
> * run `make release`
> * create a simple filesystem structure with a few directories and files
>   (I've created one at ../test-puzzlefs/simple_rootfs)
> * build a puzzlefs oci image at
>   `~/work/busybox/_install/home/puzzlefs_oci` (replace this path with
>   your busybox path) with the tag `first_try`:
> ```
> $ target/release/puzzlefs build --omit-verity \
> ../test-puzzlefs/simple_rootfs ~/work/busybox/_install/home/puzzlefs_oci \
> first_try
> ```
> * get first_try's image manifest from index.json (inside `puzzlefs_oci`)
> ```
> $ cat index.json | jq . | grep digest
>       "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b",
> ```
>
> 6. add the following 'init' script in the busybox rootfs (rootfs path
> defaults to `./_install'):
> ```
> #!/bin/sh
> mount -t devtmpfs none /dev
> mkdir -p /proc
> mount -t proc none /proc
>
> ifconfig lo up
> udhcpc -i eth0
>
> mkdir /mnt
> mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \
> image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" \
> none /mnt
>
> setsid sh -c 'exec sh -l </dev/ttyS0 >/dev/ttyS0 2>&1'
> ```
>
> Make sure to replace the `image_manifest` with your own digest. This
> init script will be passed to rdinit in the kernel command line.
>
> 7. generate the initramfs
>
> Assuming busybox is in `~/work/busybox`:
> ```
> cd ~/work/busybox/_install && find . | cpio -H newc -o | gzip > ../ramdisk.img
> ```
> This will generate a compressed ramdisk image in
> `~/work/busybox/ramdisk.img`.
>
> 8. run with qemu (this assumes the linux tree is at '../linux' and busybox
> is at '../busybox'):
> ```
> qemu-system-x86_64 \
>     -accel kvm \
>     -cpu host \
>     -m 4G \
>     -initrd ../busybox/ramdisk.img \
>     -kernel ../linux/arch/x86/boot/bzImage \
>     -nographic \
>     -append 'console=ttyS0 nokaslr debug rdinit=/init' \
>     -nic user,model=rtl8139 \
>     -no-reboot
> ```
>
> 9. Check whether puzzlefs has been successfully mounted
> ```
> ~ # mount | grep puzzlefs
> none on /mnt type puzzlefs (rw,relatime)
> ~ # ls /mnt/
> dir-1  dir-2  dir-3  dir-4  file1  file2
> ```
>
>
> # Puzzlefs abstract
> Puzzlefs [1] is a container filesystem designed to address the
> limitations of the existing OCI format. The main goals of the project
> are reduced duplication, reproducible image builds, direct mounting
> support and memory safety guarantees, some inspired by the OCIv2 design
> document [7].
>
> Reduced duplication is achieved using the content defined chunking
> algorithm FastCDC. This implementation allows chunks to be shared among
> layers. Building a new layer starting from an existing one allows
> reusing most of the chunks.
>
> Another goal of the project is reproducible image builds, which is
> achieved by defining a canonical representation of the image format.
>
> Direct mounting support is a key feature of puzzlefs and, together with
> fs-verity, it provides data integrity. Currently, puzzlefs is
> implemented as a userspace filesystem (FUSE). A read-only kernel
> filesystem driver is underway.
>
> Lastly, memory safety is critical to puzzlefs, leading to the decision
> to implement it in Rust. Another goal is to share the same code between
> user space and kernel space in order to provide one secure
> implementation.
>
>
> [1] https://github.com/anuvu/puzzlefs
> [2] https://github.com/wedsonaf/linux/tree/fs
> [3] https://github.com/Rust-for-Linux/linux/issues/1004
> [4] https://github.com/Rust-for-Linux/linux/pull/1007
> [5] https://docs.rs/serde_cbor/latest/serde_cbor/
> [6] https://docs.rs/hex/0.4.3/hex/
> [7] https://hackmd.io/@cyphar/ociv2-brainstorm
> [8] https://www.youtube.com/watch?v=tPs1uRqOnlk
> [9] https://github.com/Rust-for-Linux/linux/tree/rust-next
> [10] https://docs.kernel.org/rust/quick-start.html
> [11] https://github.com/ariel-miculas/puzzlefs/tree/support-no-verity-data
> [12] https://archive.fosdem.org/2019/schedule/event/containers_atomfs/
>
> Ariel Miculas (58):
>   rust: kernel: add libraries required by the filesystem abstractions
>   rust: kernel: backport the delay module from the rust branch
>   rust: kernel: add container_of macro
>   rust: kernel: add offset_of macro
>   drop: Add crate::pr_warn declaration
>   rust: kernel: rename from_kernel_errno to from_errno
>   rust: kernel: Rename from_pointer to from_foreing and into_pointer to
>     into_foreign
>   rust: kernel: add count_paren_items macro, needed by define_fs_params
>     macro
>   rust: helpers: add missing rust helper 'alloc_pages'
>   kernel: configs: add qemu-busybox-min.config
>   rust: kernel: format the rust code
>   samples: puzzlefs: add initial puzzlefs sample, copied from rust_fs.rs
>   kernel: configs: enable rust samples in rust.config
>   Add SAMPLE_RUST_SERDE in rust.config
>   rust: kernel: fix compile errors after rebase to rust-next
>   rust: serde_cbor: import crate
>   rust: serde_cbor: add SPDX License Identifiers
>   rust: serde_cbor: add no_fp_fmt_parse support
>   rust: Kbuild: enable serde_cbor
>   samples: rust: add cbor serialize/deserialize example
>   rust: serde_cbor: add support for serde_cbor's from_slice method by
>     using a custom alloc_kernel feature
>   rust: serde: add support for deserializing Vec with kernel_alloc
>     feature
>   rust: file: Replace UnsafeCell with Opaque for File
>   rust: kernel: implement fmt::Debug for CString
>   samples: puzzlefs: rename RustFs to PuzzleFs
>   samples: puzzlefs: add basic deserializing support for the puzzlefs
>     metadata
>   rust: file: present the filesystem context to the open function
>   rust: kernel: add an abstraction over vfsmount to allow cloning a new
>     private mount
>   rust: file: add from_path, from_path_in_root_mnt and read_with_offset
>     methods to File
>   samples: puzzlefs: pass the Vfsmount structure from open to read and
>     return the contents of the data file inside /home/puzzlefs_oci
>   rust: file: move from_path, from_path_in_root_mnt and read_with_offset
>     methods to a RegularFile newtype
>   rust: file: ensure RegularFile can only create regular files
>   rust: file: add get_pos method to RegularFile
>   rust: file: add methods read_to_end, get_file_size and update_pos to
>     RegularFile
>   rust: file: define a minimal Read trait and implement it for
>     RegularFile
>   samples: puzzlefs: add cbor_get_array_size method
>   samples: puzzlefs: add KernelError to WireFormatError and implement
>     From conversion
>   samples: puzzlefs: implement new for MetadataBlob
>   samples: puzzlefs: build puzzlefs into the kernel, thus avoiding the
>     need to export rust symbols
>   rust: alloc: add try_clone for Vec<T>
>   rust: alloc: add from_iter_fallible for Vec<T>
>   samples: puzzlefs: implement to_errno and from_errno for
>     WireFormatError
>   samples: puzzlefs: add TryReserveError (and from conversion) to
>     WireFormatError
>   samples: puzzlefs: add higher level inode related functionality
>   samples: puzzlefs: populate the directory entries with the inodes from
>     the puzzlefs metadata file
>   rust: hex: import crate
>   rust: hex: add SPDX license identifiers
>   rust: Kbuild: enable `hex`
>   rust: hex: implement FromHex trait and hex::decode using a custom
>     kernel_alloc feature
>   rust: hex: add encode_hex_iter and encode_hex_upper_iter methods
>   rust: puzzlefs: add HexError to WireFormatError and implement the From
>     conversion
>   rust: puzzlefs: display the error value for
>     WireFormatError::KernelError
>   samples: puzzlefs: add Rootfs and Digest structs to types.rs
>   samples: puzzlefs: implement the conversion from WireFormatError to
>     kernel::error::Error
>   rust: puzzlefs: read the puzzlefs image manifest instead of an
>     individual metadata layer
>   rust: puzzlefs: rename PuzzleFs to PuzzleFsModule to avoid confusion
>     with the PuzzleFS struct
>   rust: puzzlefs: add support for reading files
>   rust: puzzlefs: add oci_root_dir and image_manifest filesystem
>     parameters
>
> Miguel Ojeda (15):
>   rust: proc-macro2: import crate
>   rust: proc-macro2: add SPDX License Identifiers
>   rust: proc-macro2: remove `unicode_ident` dependency
>   rust: quote: import crate
>   rust: quote: add SPDX License Identifiers
>   rust: syn: import crate
>   rust: syn: add SPDX License Identifiers
>   rust: syn: remove `unicode-ident` dependency
>   rust: serde: import crate
>   rust: serde: add `no_fp_fmt_parse` support
>   rust: serde: add SPDX License Identifiers
>   rust: serde_derive: import crate
>   rust: serde_derive: add SPDX License Identifiers
>   rust: Kbuild: enable `proc-macro2`, `quote`, `syn`, `serde` and
>     `serde_derive`
>   rust: test `serde` support
>
> Wedson Almeida Filho (7):
>   rust: add definitions for ref-counted inodes and dentries
>   rust: add ability to register a file system
>   rust: define fs context
>   rust: add support for file system parameters
>   rust: allow fs driver to initialise new superblocks
>   rust: add `module_fs` macro
>   WIP: rust: allow fs to be populated
>
>  Makefile                                  |   14 +-
>  arch/x86/configs/qemu-busybox-min.config  |   11 +
>  kernel/configs/qemu-busybox-min.config    |   56 +
>  kernel/configs/rust.config                |   11 +
>  rust/.gitignore                           |    1 +
>  rust/Makefile                             |  232 +-
>  rust/alloc/vec/mod.rs                     |   48 +
>  rust/bindings/bindings_helper.h           |   14 +
>  rust/bindings/lib.rs                      |    5 +
>  rust/helpers.c                            |   76 +
>  rust/hex/error.rs                         |   78 +
>  rust/hex/lib.rs                           |  506 +++
>  rust/hex/serde.rs                         |  104 +
>  rust/kernel/cred.rs                       |   46 +
>  rust/kernel/delay.rs                      |  104 +
>  rust/kernel/driver.rs                     |   28 +
>  rust/kernel/error.rs                      |   52 +-
>  rust/kernel/file.rs                       | 1117 ++++++
>  rust/kernel/fs.rs                         | 1478 ++++++++
>  rust/kernel/fs/param.rs                   |  558 +++
>  rust/kernel/io_buffer.rs                  |  153 +
>  rust/kernel/iov_iter.rs                   |   81 +
>  rust/kernel/lib.rs                        |   83 +
>  rust/kernel/mm.rs                         |  149 +
>  rust/kernel/mount.rs                      |   66 +
>  rust/kernel/pages.rs                      |  144 +
>  rust/kernel/str.rs                        |    6 +
>  rust/kernel/test_serde.rs                 |   26 +
>  rust/kernel/test_serde/de.rs              |  439 +++
>  rust/kernel/test_serde/error.rs           |   73 +
>  rust/kernel/test_serde/ser.rs             |  466 +++
>  rust/kernel/user_ptr.rs                   |  175 +
>  rust/proc-macro2/detection.rs             |   77 +
>  rust/proc-macro2/fallback.rs              | 1004 ++++++
>  rust/proc-macro2/lib.rs                   | 1341 ++++++++
>  rust/proc-macro2/marker.rs                |   20 +
>  rust/proc-macro2/parse.rs                 |  874 +++++
>  rust/proc-macro2/rcvec.rs                 |  144 +
>  rust/proc-macro2/wrapper.rs               |  996 ++++++
>  rust/quote/ext.rs                         |  112 +
>  rust/quote/format.rs                      |  170 +
>  rust/quote/ident_fragment.rs              |   88 +
>  rust/quote/lib.rs                         | 1436 ++++++++
>  rust/quote/runtime.rs                     |  440 +++
>  rust/quote/spanned.rs                     |   45 +
>  rust/quote/to_tokens.rs                   |  211 ++
>  rust/serde/de/format.rs                   |   32 +
>  rust/serde/de/ignored_any.rs              |  246 ++
>  rust/serde/de/impls.rs                    | 2755 +++++++++++++++
>  rust/serde/de/mod.rs                      | 2313 +++++++++++++
>  rust/serde/de/seed.rs                     |   21 +
>  rust/serde/de/utf8.rs                     |   48 +
>  rust/serde/de/value.rs                    | 1718 ++++++++++
>  rust/serde/integer128.rs                  |   84 +
>  rust/serde/lib.rs                         |  351 ++
>  rust/serde/macros.rs                      |  238 ++
>  rust/serde/private/de.rs                  | 2997 ++++++++++++++++
>  rust/serde/private/doc.rs                 |  161 +
>  rust/serde/private/mod.rs                 |   52 +
>  rust/serde/private/ser.rs                 | 1316 +++++++
>  rust/serde/private/size_hint.rs           |   23 +
>  rust/serde/ser/fmt.rs                     |  180 +
>  rust/serde/ser/impls.rs                   |  987 ++++++
>  rust/serde/ser/impossible.rs              |  218 ++
>  rust/serde/ser/mod.rs                     | 1992 +++++++++++
>  rust/serde/std_error.rs                   |   50 +
>  rust/serde_cbor/de.rs                     | 1370 ++++++++
>  rust/serde_cbor/error.rs                  |  320 ++
>  rust/serde_cbor/lib.rs                    |  371 ++
>  rust/serde_cbor/read.rs                   |  647 ++++
>  rust/serde_cbor/ser.rs                    |  748 ++++
>  rust/serde_cbor/tags.rs                   |  224 ++
>  rust/serde_cbor/value/de.rs               |  168 +
>  rust/serde_cbor/value/mod.rs              |  158 +
>  rust/serde_cbor/value/ser.rs              |  447 +++
>  rust/serde_cbor/write.rs                  |  177 +
>  rust/serde_derive/bound.rs                |  408 +++
>  rust/serde_derive/de.rs                   | 3148 +++++++++++++++++
>  rust/serde_derive/dummy.rs                |   46 +
>  rust/serde_derive/fragment.rs             |   76 +
>  rust/serde_derive/internals/ast.rs        |  204 ++
>  rust/serde_derive/internals/attr.rs       | 1908 +++++++++++
>  rust/serde_derive/internals/case.rs       |  199 ++
>  rust/serde_derive/internals/check.rs      |  445 +++
>  rust/serde_derive/internals/ctxt.rs       |   64 +
>  rust/serde_derive/internals/mod.rs        |   30 +
>  rust/serde_derive/internals/receiver.rs   |  287 ++
>  rust/serde_derive/internals/respan.rs     |   18 +
>  rust/serde_derive/internals/symbol.rs     |   71 +
>  rust/serde_derive/lib.rs                  |  112 +
>  rust/serde_derive/pretend.rs              |  203 ++
>  rust/serde_derive/ser.rs                  | 1342 ++++++++
>  rust/serde_derive/this.rs                 |   34 +
>  rust/serde_derive/try.rs                  |   26 +
>  rust/syn/attr.rs                          |  664 ++++
>  rust/syn/await.rs                         |    4 +
>  rust/syn/bigint.rs                        |   68 +
>  rust/syn/buffer.rs                        |  400 +++
>  rust/syn/custom_keyword.rs                |  255 ++
>  rust/syn/custom_punctuation.rs            |  302 ++
>  rust/syn/data.rs                          |  495 +++
>  rust/syn/derive.rs                        |  276 ++
>  rust/syn/discouraged.rs                   |  196 ++
>  rust/syn/error.rs                         |  430 +++
>  rust/syn/export.rs                        |   41 +
>  rust/syn/expr.rs                          | 3560 +++++++++++++++++++
>  rust/syn/ext.rs                           |  141 +
>  rust/syn/file.rs                          |  127 +
>  rust/syn/gen/clone.rs                     | 2243 ++++++++++++
>  rust/syn/gen/debug.rs                     | 3044 +++++++++++++++++
>  rust/syn/gen/eq.rs                        | 2197 ++++++++++++
>  rust/syn/gen/fold.rs                      | 3343 ++++++++++++++++++
>  rust/syn/gen/hash.rs                      | 2871 ++++++++++++++++
>  rust/syn/gen/visit.rs                     | 3788 +++++++++++++++++++++
>  rust/syn/gen/visit_mut.rs                 | 3788 +++++++++++++++++++++
>  rust/syn/gen_helper.rs                    |  156 +
>  rust/syn/generics.rs                      | 1339 ++++++++
>  rust/syn/group.rs                         |  284 ++
>  rust/syn/ident.rs                         |  103 +
>  rust/syn/item.rs                          | 3315 ++++++++++++++++++
>  rust/syn/lib.rs                           |  985 ++++++
>  rust/syn/lifetime.rs                      |  156 +
>  rust/syn/lit.rs                           | 1602 +++++++++
>  rust/syn/lookahead.rs                     |  171 +
>  rust/syn/mac.rs                           |  221 ++
>  rust/syn/macros.rs                        |  179 +
>  rust/syn/op.rs                            |  236 ++
>  rust/syn/parse.rs                         | 1316 +++++++
>  rust/syn/parse_macro_input.rs             |  181 +
>  rust/syn/parse_quote.rs                   |  169 +
>  rust/syn/pat.rs                           |  929 +++++
>  rust/syn/path.rs                          |  856 +++++
>  rust/syn/print.rs                         |   18 +
>  rust/syn/punctuated.rs                    | 1070 ++++++
>  rust/syn/reserved.rs                      |   46 +
>  rust/syn/sealed.rs                        |    6 +
>  rust/syn/span.rs                          |   69 +
>  rust/syn/spanned.rs                       |  116 +
>  rust/syn/stmt.rs                          |  351 ++
>  rust/syn/thread.rs                        |   43 +
>  rust/syn/token.rs                         | 1015 ++++++
>  rust/syn/tt.rs                            |  109 +
>  rust/syn/ty.rs                            | 1288 +++++++
>  rust/syn/verbatim.rs                      |   35 +
>  rust/syn/whitespace.rs                    |   67 +
>  samples/rust/Kconfig                      |   28 +
>  samples/rust/Makefile                     |    3 +
>  samples/rust/local_data_format/de.rs      |  422 +++
>  samples/rust/local_data_format/error.rs   |   73 +
>  samples/rust/local_data_format/ser.rs     |  443 +++
>  samples/rust/puzzle.rs                    |    4 +
>  samples/rust/puzzle/error.rs              |   91 +
>  samples/rust/puzzle/inode.rs              |  150 +
>  samples/rust/puzzle/oci.rs                |   71 +
>  samples/rust/puzzle/types.rs              |  389 +++
>  samples/rust/puzzle/types/cbor_helpers.rs |   50 +
>  samples/rust/puzzlefs.rs                  |  220 ++
>  samples/rust/rust_fs.rs                   |  105 +
>  samples/rust/rust_serde.rs                |  125 +
>  scripts/Makefile.build                    |    4 +-
>  160 files changed, 89204 insertions(+), 29 deletions(-)
>  create mode 100644 arch/x86/configs/qemu-busybox-min.config
>  create mode 100644 kernel/configs/qemu-busybox-min.config
>  create mode 100644 rust/hex/error.rs
>  create mode 100644 rust/hex/lib.rs
>  create mode 100644 rust/hex/serde.rs
>  create mode 100644 rust/kernel/cred.rs
>  create mode 100644 rust/kernel/delay.rs
>  create mode 100644 rust/kernel/driver.rs
>  create mode 100644 rust/kernel/file.rs
>  create mode 100644 rust/kernel/fs.rs
>  create mode 100644 rust/kernel/fs/param.rs
>  create mode 100644 rust/kernel/io_buffer.rs
>  create mode 100644 rust/kernel/iov_iter.rs
>  create mode 100644 rust/kernel/mm.rs
>  create mode 100644 rust/kernel/mount.rs
>  create mode 100644 rust/kernel/pages.rs
>  create mode 100644 rust/kernel/test_serde.rs
>  create mode 100644 rust/kernel/test_serde/de.rs
>  create mode 100644 rust/kernel/test_serde/error.rs
>  create mode 100644 rust/kernel/test_serde/ser.rs
>  create mode 100644 rust/kernel/user_ptr.rs
>  create mode 100644 rust/proc-macro2/detection.rs
>  create mode 100644 rust/proc-macro2/fallback.rs
>  create mode 100644 rust/proc-macro2/lib.rs
>  create mode 100644 rust/proc-macro2/marker.rs
>  create mode 100644 rust/proc-macro2/parse.rs
>  create mode 100644 rust/proc-macro2/rcvec.rs
>  create mode 100644 rust/proc-macro2/wrapper.rs
>  create mode 100644 rust/quote/ext.rs
>  create mode 100644 rust/quote/format.rs
>  create mode 100644 rust/quote/ident_fragment.rs
>  create mode 100644 rust/quote/lib.rs
>  create mode 100644 rust/quote/runtime.rs
>  create mode 100644 rust/quote/spanned.rs
>  create mode 100644 rust/quote/to_tokens.rs
>  create mode 100644 rust/serde/de/format.rs
>  create mode 100644 rust/serde/de/ignored_any.rs
>  create mode 100644 rust/serde/de/impls.rs
>  create mode 100644 rust/serde/de/mod.rs
>  create mode 100644 rust/serde/de/seed.rs
>  create mode 100644 rust/serde/de/utf8.rs
>  create mode 100644 rust/serde/de/value.rs
>  create mode 100644 rust/serde/integer128.rs
>  create mode 100644 rust/serde/lib.rs
>  create mode 100644 rust/serde/macros.rs
>  create mode 100644 rust/serde/private/de.rs
>  create mode 100644 rust/serde/private/doc.rs
>  create mode 100644 rust/serde/private/mod.rs
>  create mode 100644 rust/serde/private/ser.rs
>  create mode 100644 rust/serde/private/size_hint.rs
>  create mode 100644 rust/serde/ser/fmt.rs
>  create mode 100644 rust/serde/ser/impls.rs
>  create mode 100644 rust/serde/ser/impossible.rs
>  create mode 100644 rust/serde/ser/mod.rs
>  create mode 100644 rust/serde/std_error.rs
>  create mode 100644 rust/serde_cbor/de.rs
>  create mode 100644 rust/serde_cbor/error.rs
>  create mode 100644 rust/serde_cbor/lib.rs
>  create mode 100644 rust/serde_cbor/read.rs
>  create mode 100644 rust/serde_cbor/ser.rs
>  create mode 100644 rust/serde_cbor/tags.rs
>  create mode 100644 rust/serde_cbor/value/de.rs
>  create mode 100644 rust/serde_cbor/value/mod.rs
>  create mode 100644 rust/serde_cbor/value/ser.rs
>  create mode 100644 rust/serde_cbor/write.rs
>  create mode 100644 rust/serde_derive/bound.rs
>  create mode 100644 rust/serde_derive/de.rs
>  create mode 100644 rust/serde_derive/dummy.rs
>  create mode 100644 rust/serde_derive/fragment.rs
>  create mode 100644 rust/serde_derive/internals/ast.rs
>  create mode 100644 rust/serde_derive/internals/attr.rs
>  create mode 100644 rust/serde_derive/internals/case.rs
>  create mode 100644 rust/serde_derive/internals/check.rs
>  create mode 100644 rust/serde_derive/internals/ctxt.rs
>  create mode 100644 rust/serde_derive/internals/mod.rs
>  create mode 100644 rust/serde_derive/internals/receiver.rs
>  create mode 100644 rust/serde_derive/internals/respan.rs
>  create mode 100644 rust/serde_derive/internals/symbol.rs
>  create mode 100644 rust/serde_derive/lib.rs
>  create mode 100644 rust/serde_derive/pretend.rs
>  create mode 100644 rust/serde_derive/ser.rs
>  create mode 100644 rust/serde_derive/this.rs
>  create mode 100644 rust/serde_derive/try.rs
>  create mode 100644 rust/syn/attr.rs
>  create mode 100644 rust/syn/await.rs
>  create mode 100644 rust/syn/bigint.rs
>  create mode 100644 rust/syn/buffer.rs
>  create mode 100644 rust/syn/custom_keyword.rs
>  create mode 100644 rust/syn/custom_punctuation.rs
>  create mode 100644 rust/syn/data.rs
>  create mode 100644 rust/syn/derive.rs
>  create mode 100644 rust/syn/discouraged.rs
>  create mode 100644 rust/syn/error.rs
>  create mode 100644 rust/syn/export.rs
>  create mode 100644 rust/syn/expr.rs
>  create mode 100644 rust/syn/ext.rs
>  create mode 100644 rust/syn/file.rs
>  create mode 100644 rust/syn/gen/clone.rs
>  create mode 100644 rust/syn/gen/debug.rs
>  create mode 100644 rust/syn/gen/eq.rs
>  create mode 100644 rust/syn/gen/fold.rs
>  create mode 100644 rust/syn/gen/hash.rs
>  create mode 100644 rust/syn/gen/visit.rs
>  create mode 100644 rust/syn/gen/visit_mut.rs
>  create mode 100644 rust/syn/gen_helper.rs
>  create mode 100644 rust/syn/generics.rs
>  create mode 100644 rust/syn/group.rs
>  create mode 100644 rust/syn/ident.rs
>  create mode 100644 rust/syn/item.rs
>  create mode 100644 rust/syn/lib.rs
>  create mode 100644 rust/syn/lifetime.rs
>  create mode 100644 rust/syn/lit.rs
>  create mode 100644 rust/syn/lookahead.rs
>  create mode 100644 rust/syn/mac.rs
>  create mode 100644 rust/syn/macros.rs
>  create mode 100644 rust/syn/op.rs
>  create mode 100644 rust/syn/parse.rs
>  create mode 100644 rust/syn/parse_macro_input.rs
>  create mode 100644 rust/syn/parse_quote.rs
>  create mode 100644 rust/syn/pat.rs
>  create mode 100644 rust/syn/path.rs
>  create mode 100644 rust/syn/print.rs
>  create mode 100644 rust/syn/punctuated.rs
>  create mode 100644 rust/syn/reserved.rs
>  create mode 100644 rust/syn/sealed.rs
>  create mode 100644 rust/syn/span.rs
>  create mode 100644 rust/syn/spanned.rs
>  create mode 100644 rust/syn/stmt.rs
>  create mode 100644 rust/syn/thread.rs
>  create mode 100644 rust/syn/token.rs
>  create mode 100644 rust/syn/tt.rs
>  create mode 100644 rust/syn/ty.rs
>  create mode 100644 rust/syn/verbatim.rs
>  create mode 100644 rust/syn/whitespace.rs
>  create mode 100644 samples/rust/local_data_format/de.rs
>  create mode 100644 samples/rust/local_data_format/error.rs
>  create mode 100644 samples/rust/local_data_format/ser.rs
>  create mode 100644 samples/rust/puzzle.rs
>  create mode 100644 samples/rust/puzzle/error.rs
>  create mode 100644 samples/rust/puzzle/inode.rs
>  create mode 100644 samples/rust/puzzle/oci.rs
>  create mode 100644 samples/rust/puzzle/types.rs
>  create mode 100644 samples/rust/puzzle/types/cbor_helpers.rs
>  create mode 100644 samples/rust/puzzlefs.rs
>  create mode 100644 samples/rust/rust_fs.rs
>  create mode 100644 samples/rust/rust_serde.rs

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Miguel Ojeda]

On Sat, Jun 10, 2023 at 11:14 AM Andreas Hindborg (Samsung)
<nmi@metaspace.dk> wrote:
>
> This is super annoying when it happens. What is the reason, are they too
> large? Miguel: can we do anything about this?

Yes as far as I know, and yes, you should Cc patches@lists.linux.dev,
if they are going to be big (I replied above about this [1] -- did you
receive it?).

[1] https://lore.kernel.org/rust-for-linux/CANiq72njK=6TYt1z8WBpRFhPJY=yVtxytxDUi=_Jtp1dZ3hKLQ@mail.gmail.com/

Cheers,
Miguel

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Andreas Hindborg (Samsung)]

Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> writes:

> On Sat, Jun 10, 2023 at 11:14 AM Andreas Hindborg (Samsung)
> <nmi@metaspace.dk> wrote:
>>
>> This is super annoying when it happens. What is the reason, are they too
>> large? Miguel: can we do anything about this?
>
> Yes as far as I know, and yes, you should Cc patches@lists.linux.dev,
> if they are going to be big (I replied above about this [1] -- did you
> receive it?).

I got it. What I was trying to get at is: can we fix the rust-for-linux
list? Why is there a size limit? Can we increase it? Better to fix the
issue at the source than to add a workaround that people are bound to
forget before it is too late.

BR Andreas

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Miguel Ojeda]

On Sat, Jun 10, 2023 at 12:27 PM Andreas Hindborg (Samsung)
<nmi@metaspace.dk> wrote:
>
> I got it. What I was trying to get at is: can we fix the rust-for-linux
> list? Why is there a size limit? Can we increase it? Better to fix the
> issue at the source than to add a workaround that people are bound to
> forget before it is too late.

I have asked the admins about this. Back in August, the recommendation
I got was to Cc all patches to patches@lists.linux.dev. Also, the vger
services were planned to be transferred, which may have helped with
the size limits (not sure), but I was told that effort appeared to
have stalled.

Cheers,
Miguel

[RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Ariel Miculas]

Hi all!

This is a proof of concept driver written for the PuzzleFS
next-generation container filesystem [1]. I've included a short abstract
about puzzlefs further below. This driver is based on the rust-next
branch, on top of which I've backported the filesystem abstractions from
Wedson Almeida Filho [2][3] and Miguel Ojeda's third-party crates
support: proc-macro2, quote, syn, serde and serde_derive [4]. I've added
the additional third-party crates serde_cbor[5] and hex [6]. Then I've
adapted the user space puzzlefs code [1] so that the puzzlefs kernel
module could present the directory hierarchy and implement the basic
read functionality.
For some additional context, puzzlefs was started by Tycho Andersen and
it's the successor of atomfs. This FOSDEM presentation from 2019 [12]
covers the rationale for a new oci image format and presents a higher
level overview of our goals with puzzlefs.
I've split the rest of the cover letter in following sections (using a
markdown style):
* Example: it describes a practical example of what was achieved
* Limitations: it presents the existing limitations of this POC
* Upstreaming steps: it describes the steps needed for upstreaming this
  driver
* Setup: it shows how to setup the necessary environment for testing the
  puzzlefs driver
* Puzzlefs abstract: it provides a short overview of puzzlefs

# Example
An example is provided below:

```
~ # cat /proc/filesystems | grep puzzlefs
nodev   puzzlefs
~ # mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o image_manifest="2d
6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" none /mnt
~ # ls -lR /mnt/
/mnt/:
total 0
drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-1
drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-2
drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-3
drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-4
-rw-r--r--    1 0        0                0 Jun  8 12:26 file1
-rw-r--r--    1 0        0                0 Jun  8 12:26 file2

/mnt/dir-1:
total 0

/mnt/dir-2:
total 0

/mnt/dir-3:
total 0

/mnt/dir-4:
total 0
~ # cat /mnt/file2
ana are mere bla bla bla
~ # wc /mnt/file1
      202       202      5454 /mnt/file1
```

In this example, /home/puzzlefs_oci is the puzzlefs oci directory:
```
~ # ls -lR /home/puzzlefs_oci/
/home/puzzlefs_oci/:
total 8
drwxr-xr-x    3 1000     1000             0 Jun  8 14:33 blobs
-rw-r--r--    1 1000     1000           266 Jun  8 14:33 index.json
-rw-r--r--    1 1000     1000            37 Jun  8 14:33 oci-layout

/home/puzzlefs_oci/blobs:
total 0
drwxr-xr-x    2 1000     1000             0 Jun  8 14:33 sha256

/home/puzzlefs_oci/blobs/sha256:
total 16
-rw-------    1 1000     1000            89 Jun  8 14:33 2d6602d678140540dc7e96de652a76a8b16eb
-rw-------    1 1000     1000           925 Jun  8 14:33 4df03518eea406343dbb55046720f6a478881
-rw-------    1 1000     1000          5479 Jun  8 14:33 d86a87b19bd9a2fec0d31687c1d669cdb59eb
```

`2d6602d678140540dc7e96de652a76a8b16eb` is the puzzlefs image manifest
hash for the first_try tag:
```
$ cat /tmp/oci-simple/index.json | jq .
{
  "schemaVersion": -1,
  "manifests": [
    {
      "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b",
      "size": 89,
      "media_type": "application/vnd.puzzlefs.image.rootfs.v1",
      "annotations": {
        "org.opencontainers.image.ref.name": "first_try"
      }
    }
  ],
  "annotations": {}
}
```

I will describe how to build a puzzlefs image in the `Setup` section, at
step 5.

# Limitations
One limitation is that the puzzlefs driver doesn't implement any lookup
functionality and instead it inserts every directory entry into the
dcache during init (see the `DCACHE_BASED` constant). This is similar to
how the sample `rust_fs` driver works, but the goal is to implement
proper lookup functions.  However, more filesystem abstractions need to
be implemented before this can be achieved.

# Upstreaming steps
Before the puzzlefs driver can be upstreamed, the following need to be
merged:
* Wedson's filesystem abstractions [3]
* the necessary third-party crates [4] (with the preliminary discussion
about whether this is desirable)

# Setup
My setup is based on Wedson's tutorial [8]. Next, I will describe the
necessary steps to build an initrd and run a custom kernel under qemu.

1. Get the linux rust-next branch [9] and apply this patchset

2. Follow the rust quickstart guide [10]

3. configure and build the kernel
```
$ make LLVM=1 allnoconfig qemu-busybox-min.config rust.config
$ make LLVM=1 -j$(nproc)
```

4. setup busybox
```
git clone git://git.busybox.net/busybox
cd busybox
make menuconfig # enable 'Build static binary' config
make
make install
```
This will create the `_install` directory with the rootfs inside it.

5. create a home directory in the rootfs and copy a puzzlefs oci
directory in home/puzzlefs_oci
To create a puzzlefs oci directory:
* download this custom puzzlefs repository [11] (it's custom because we
  want to build an image without verity data)
* run `make release`
* create a simple filesystem structure with a few directories and files
  (I've created one at ../test-puzzlefs/simple_rootfs)
* build a puzzlefs oci image at
  `~/work/busybox/_install/home/puzzlefs_oci` (replace this path with
  your busybox path) with the tag `first_try`:
```
$ target/release/puzzlefs build --omit-verity \
../test-puzzlefs/simple_rootfs ~/work/busybox/_install/home/puzzlefs_oci \
first_try
```
* get first_try's image manifest from index.json (inside `puzzlefs_oci`)
```
$ cat index.json | jq . | grep digest
      "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b",
```

6. add the following 'init' script in the busybox rootfs (rootfs path
defaults to `./_install'):
```
#!/bin/sh
mount -t devtmpfs none /dev
mkdir -p /proc
mount -t proc none /proc

ifconfig lo up
udhcpc -i eth0

mkdir /mnt
mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \
image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" \
none /mnt

setsid sh -c 'exec sh -l </dev/ttyS0 >/dev/ttyS0 2>&1'
```

Make sure to replace the `image_manifest` with your own digest. This
init script will be passed to rdinit in the kernel command line.

7. generate the initramfs

Assuming busybox is in `~/work/busybox`:
```
cd ~/work/busybox/_install && find . | cpio -H newc -o | gzip > ../ramdisk.img
```
This will generate a compressed ramdisk image in
`~/work/busybox/ramdisk.img`.

8. run with qemu (this assumes the linux tree is at '../linux' and busybox
is at '../busybox'):
```
qemu-system-x86_64 \
    -accel kvm \
    -cpu host \
    -m 4G \
    -initrd ../busybox/ramdisk.img \
    -kernel ../linux/arch/x86/boot/bzImage \
    -nographic \
    -append 'console=ttyS0 nokaslr debug rdinit=/init' \
    -nic user,model=rtl8139 \
    -no-reboot
```

9. Check whether puzzlefs has been successfully mounted
```
~ # mount | grep puzzlefs
none on /mnt type puzzlefs (rw,relatime)
~ # ls /mnt/
dir-1  dir-2  dir-3  dir-4  file1  file2
```


# Puzzlefs abstract
Puzzlefs [1] is a container filesystem designed to address the
limitations of the existing OCI format. The main goals of the project
are reduced duplication, reproducible image builds, direct mounting
support and memory safety guarantees, some inspired by the OCIv2 design
document [7].

Reduced duplication is achieved using the content defined chunking
algorithm FastCDC. This implementation allows chunks to be shared among
layers. Building a new layer starting from an existing one allows
reusing most of the chunks.

Another goal of the project is reproducible image builds, which is
achieved by defining a canonical representation of the image format.

Direct mounting support is a key feature of puzzlefs and, together with
fs-verity, it provides data integrity. Currently, puzzlefs is
implemented as a userspace filesystem (FUSE). A read-only kernel
filesystem driver is underway.

Lastly, memory safety is critical to puzzlefs, leading to the decision
to implement it in Rust. Another goal is to share the same code between
user space and kernel space in order to provide one secure
implementation.


[1] https://github.com/anuvu/puzzlefs
[2] https://github.com/wedsonaf/linux/tree/fs
[3] https://github.com/Rust-for-Linux/linux/issues/1004
[4] https://github.com/Rust-for-Linux/linux/pull/1007
[5] https://docs.rs/serde_cbor/latest/serde_cbor/
[6] https://docs.rs/hex/0.4.3/hex/
[7] https://hackmd.io/@cyphar/ociv2-brainstorm
[8] https://www.youtube.com/watch?v=tPs1uRqOnlk
[9] https://github.com/Rust-for-Linux/linux/tree/rust-next
[10] https://docs.kernel.org/rust/quick-start.html
[11] https://github.com/ariel-miculas/puzzlefs/tree/support-no-verity-data
[12] https://archive.fosdem.org/2019/schedule/event/containers_atomfs/

Ariel Miculas (58):
  rust: kernel: add libraries required by the filesystem abstractions
  rust: kernel: backport the delay module from the rust branch
  rust: kernel: add container_of macro
  rust: kernel: add offset_of macro
  drop: Add crate::pr_warn declaration
  rust: kernel: rename from_kernel_errno to from_errno
  rust: kernel: Rename from_pointer to from_foreing and into_pointer to
    into_foreign
  rust: kernel: add count_paren_items macro, needed by define_fs_params
    macro
  rust: helpers: add missing rust helper 'alloc_pages'
  kernel: configs: add qemu-busybox-min.config
  rust: kernel: format the rust code
  samples: puzzlefs: add initial puzzlefs sample, copied from rust_fs.rs
  kernel: configs: enable rust samples in rust.config
  Add SAMPLE_RUST_SERDE in rust.config
  rust: kernel: fix compile errors after rebase to rust-next
  rust: serde_cbor: import crate
  rust: serde_cbor: add SPDX License Identifiers
  rust: serde_cbor: add no_fp_fmt_parse support
  rust: Kbuild: enable serde_cbor
  samples: rust: add cbor serialize/deserialize example
  rust: serde_cbor: add support for serde_cbor's from_slice method by
    using a custom alloc_kernel feature
  rust: serde: add support for deserializing Vec with kernel_alloc
    feature
  rust: file: Replace UnsafeCell with Opaque for File
  rust: kernel: implement fmt::Debug for CString
  samples: puzzlefs: rename RustFs to PuzzleFs
  samples: puzzlefs: add basic deserializing support for the puzzlefs
    metadata
  rust: file: present the filesystem context to the open function
  rust: kernel: add an abstraction over vfsmount to allow cloning a new
    private mount
  rust: file: add from_path, from_path_in_root_mnt and read_with_offset
    methods to File
  samples: puzzlefs: pass the Vfsmount structure from open to read and
    return the contents of the data file inside /home/puzzlefs_oci
  rust: file: move from_path, from_path_in_root_mnt and read_with_offset
    methods to a RegularFile newtype
  rust: file: ensure RegularFile can only create regular files
  rust: file: add get_pos method to RegularFile
  rust: file: add methods read_to_end, get_file_size and update_pos to
    RegularFile
  rust: file: define a minimal Read trait and implement it for
    RegularFile
  samples: puzzlefs: add cbor_get_array_size method
  samples: puzzlefs: add KernelError to WireFormatError and implement
    From conversion
  samples: puzzlefs: implement new for MetadataBlob
  samples: puzzlefs: build puzzlefs into the kernel, thus avoiding the
    need to export rust symbols
  rust: alloc: add try_clone for Vec<T>
  rust: alloc: add from_iter_fallible for Vec<T>
  samples: puzzlefs: implement to_errno and from_errno for
    WireFormatError
  samples: puzzlefs: add TryReserveError (and from conversion) to
    WireFormatError
  samples: puzzlefs: add higher level inode related functionality
  samples: puzzlefs: populate the directory entries with the inodes from
    the puzzlefs metadata file
  rust: hex: import crate
  rust: hex: add SPDX license identifiers
  rust: Kbuild: enable `hex`
  rust: hex: implement FromHex trait and hex::decode using a custom
    kernel_alloc feature
  rust: hex: add encode_hex_iter and encode_hex_upper_iter methods
  rust: puzzlefs: add HexError to WireFormatError and implement the From
    conversion
  rust: puzzlefs: display the error value for
    WireFormatError::KernelError
  samples: puzzlefs: add Rootfs and Digest structs to types.rs
  samples: puzzlefs: implement the conversion from WireFormatError to
    kernel::error::Error
  rust: puzzlefs: read the puzzlefs image manifest instead of an
    individual metadata layer
  rust: puzzlefs: rename PuzzleFs to PuzzleFsModule to avoid confusion
    with the PuzzleFS struct
  rust: puzzlefs: add support for reading files
  rust: puzzlefs: add oci_root_dir and image_manifest filesystem
    parameters

Miguel Ojeda (15):
  rust: proc-macro2: import crate
  rust: proc-macro2: add SPDX License Identifiers
  rust: proc-macro2: remove `unicode_ident` dependency
  rust: quote: import crate
  rust: quote: add SPDX License Identifiers
  rust: syn: import crate
  rust: syn: add SPDX License Identifiers
  rust: syn: remove `unicode-ident` dependency
  rust: serde: import crate
  rust: serde: add `no_fp_fmt_parse` support
  rust: serde: add SPDX License Identifiers
  rust: serde_derive: import crate
  rust: serde_derive: add SPDX License Identifiers
  rust: Kbuild: enable `proc-macro2`, `quote`, `syn`, `serde` and
    `serde_derive`
  rust: test `serde` support

Wedson Almeida Filho (7):
  rust: add definitions for ref-counted inodes and dentries
  rust: add ability to register a file system
  rust: define fs context
  rust: add support for file system parameters
  rust: allow fs driver to initialise new superblocks
  rust: add `module_fs` macro
  WIP: rust: allow fs to be populated

 Makefile                                  |   14 +-
 arch/x86/configs/qemu-busybox-min.config  |   11 +
 kernel/configs/qemu-busybox-min.config    |   56 +
 kernel/configs/rust.config                |   11 +
 rust/.gitignore                           |    1 +
 rust/Makefile                             |  232 +-
 rust/alloc/vec/mod.rs                     |   48 +
 rust/bindings/bindings_helper.h           |   14 +
 rust/bindings/lib.rs                      |    5 +
 rust/helpers.c                            |   76 +
 rust/hex/error.rs                         |   78 +
 rust/hex/lib.rs                           |  506 +++
 rust/hex/serde.rs                         |  104 +
 rust/kernel/cred.rs                       |   46 +
 rust/kernel/delay.rs                      |  104 +
 rust/kernel/driver.rs                     |   28 +
 rust/kernel/error.rs                      |   52 +-
 rust/kernel/file.rs                       | 1117 ++++++
 rust/kernel/fs.rs                         | 1478 ++++++++
 rust/kernel/fs/param.rs                   |  558 +++
 rust/kernel/io_buffer.rs                  |  153 +
 rust/kernel/iov_iter.rs                   |   81 +
 rust/kernel/lib.rs                        |   83 +
 rust/kernel/mm.rs                         |  149 +
 rust/kernel/mount.rs                      |   66 +
 rust/kernel/pages.rs                      |  144 +
 rust/kernel/str.rs                        |    6 +
 rust/kernel/test_serde.rs                 |   26 +
 rust/kernel/test_serde/de.rs              |  439 +++
 rust/kernel/test_serde/error.rs           |   73 +
 rust/kernel/test_serde/ser.rs             |  466 +++
 rust/kernel/user_ptr.rs                   |  175 +
 rust/proc-macro2/detection.rs             |   77 +
 rust/proc-macro2/fallback.rs              | 1004 ++++++
 rust/proc-macro2/lib.rs                   | 1341 ++++++++
 rust/proc-macro2/marker.rs                |   20 +
 rust/proc-macro2/parse.rs                 |  874 +++++
 rust/proc-macro2/rcvec.rs                 |  144 +
 rust/proc-macro2/wrapper.rs               |  996 ++++++
 rust/quote/ext.rs                         |  112 +
 rust/quote/format.rs                      |  170 +
 rust/quote/ident_fragment.rs              |   88 +
 rust/quote/lib.rs                         | 1436 ++++++++
 rust/quote/runtime.rs                     |  440 +++
 rust/quote/spanned.rs                     |   45 +
 rust/quote/to_tokens.rs                   |  211 ++
 rust/serde/de/format.rs                   |   32 +
 rust/serde/de/ignored_any.rs              |  246 ++
 rust/serde/de/impls.rs                    | 2755 +++++++++++++++
 rust/serde/de/mod.rs                      | 2313 +++++++++++++
 rust/serde/de/seed.rs                     |   21 +
 rust/serde/de/utf8.rs                     |   48 +
 rust/serde/de/value.rs                    | 1718 ++++++++++
 rust/serde/integer128.rs                  |   84 +
 rust/serde/lib.rs                         |  351 ++
 rust/serde/macros.rs                      |  238 ++
 rust/serde/private/de.rs                  | 2997 ++++++++++++++++
 rust/serde/private/doc.rs                 |  161 +
 rust/serde/private/mod.rs                 |   52 +
 rust/serde/private/ser.rs                 | 1316 +++++++
 rust/serde/private/size_hint.rs           |   23 +
 rust/serde/ser/fmt.rs                     |  180 +
 rust/serde/ser/impls.rs                   |  987 ++++++
 rust/serde/ser/impossible.rs              |  218 ++
 rust/serde/ser/mod.rs                     | 1992 +++++++++++
 rust/serde/std_error.rs                   |   50 +
 rust/serde_cbor/de.rs                     | 1370 ++++++++
 rust/serde_cbor/error.rs                  |  320 ++
 rust/serde_cbor/lib.rs                    |  371 ++
 rust/serde_cbor/read.rs                   |  647 ++++
 rust/serde_cbor/ser.rs                    |  748 ++++
 rust/serde_cbor/tags.rs                   |  224 ++
 rust/serde_cbor/value/de.rs               |  168 +
 rust/serde_cbor/value/mod.rs              |  158 +
 rust/serde_cbor/value/ser.rs              |  447 +++
 rust/serde_cbor/write.rs                  |  177 +
 rust/serde_derive/bound.rs                |  408 +++
 rust/serde_derive/de.rs                   | 3148 +++++++++++++++++
 rust/serde_derive/dummy.rs                |   46 +
 rust/serde_derive/fragment.rs             |   76 +
 rust/serde_derive/internals/ast.rs        |  204 ++
 rust/serde_derive/internals/attr.rs       | 1908 +++++++++++
 rust/serde_derive/internals/case.rs       |  199 ++
 rust/serde_derive/internals/check.rs      |  445 +++
 rust/serde_derive/internals/ctxt.rs       |   64 +
 rust/serde_derive/internals/mod.rs        |   30 +
 rust/serde_derive/internals/receiver.rs   |  287 ++
 rust/serde_derive/internals/respan.rs     |   18 +
 rust/serde_derive/internals/symbol.rs     |   71 +
 rust/serde_derive/lib.rs                  |  112 +
 rust/serde_derive/pretend.rs              |  203 ++
 rust/serde_derive/ser.rs                  | 1342 ++++++++
 rust/serde_derive/this.rs                 |   34 +
 rust/serde_derive/try.rs                  |   26 +
 rust/syn/attr.rs                          |  664 ++++
 rust/syn/await.rs                         |    4 +
 rust/syn/bigint.rs                        |   68 +
 rust/syn/buffer.rs                        |  400 +++
 rust/syn/custom_keyword.rs                |  255 ++
 rust/syn/custom_punctuation.rs            |  302 ++
 rust/syn/data.rs                          |  495 +++
 rust/syn/derive.rs                        |  276 ++
 rust/syn/discouraged.rs                   |  196 ++
 rust/syn/error.rs                         |  430 +++
 rust/syn/export.rs                        |   41 +
 rust/syn/expr.rs                          | 3560 +++++++++++++++++++
 rust/syn/ext.rs                           |  141 +
 rust/syn/file.rs                          |  127 +
 rust/syn/gen/clone.rs                     | 2243 ++++++++++++
 rust/syn/gen/debug.rs                     | 3044 +++++++++++++++++
 rust/syn/gen/eq.rs                        | 2197 ++++++++++++
 rust/syn/gen/fold.rs                      | 3343 ++++++++++++++++++
 rust/syn/gen/hash.rs                      | 2871 ++++++++++++++++
 rust/syn/gen/visit.rs                     | 3788 +++++++++++++++++++++
 rust/syn/gen/visit_mut.rs                 | 3788 +++++++++++++++++++++
 rust/syn/gen_helper.rs                    |  156 +
 rust/syn/generics.rs                      | 1339 ++++++++
 rust/syn/group.rs                         |  284 ++
 rust/syn/ident.rs                         |  103 +
 rust/syn/item.rs                          | 3315 ++++++++++++++++++
 rust/syn/lib.rs                           |  985 ++++++
 rust/syn/lifetime.rs                      |  156 +
 rust/syn/lit.rs                           | 1602 +++++++++
 rust/syn/lookahead.rs                     |  171 +
 rust/syn/mac.rs                           |  221 ++
 rust/syn/macros.rs                        |  179 +
 rust/syn/op.rs                            |  236 ++
 rust/syn/parse.rs                         | 1316 +++++++
 rust/syn/parse_macro_input.rs             |  181 +
 rust/syn/parse_quote.rs                   |  169 +
 rust/syn/pat.rs                           |  929 +++++
 rust/syn/path.rs                          |  856 +++++
 rust/syn/print.rs                         |   18 +
 rust/syn/punctuated.rs                    | 1070 ++++++
 rust/syn/reserved.rs                      |   46 +
 rust/syn/sealed.rs                        |    6 +
 rust/syn/span.rs                          |   69 +
 rust/syn/spanned.rs                       |  116 +
 rust/syn/stmt.rs                          |  351 ++
 rust/syn/thread.rs                        |   43 +
 rust/syn/token.rs                         | 1015 ++++++
 rust/syn/tt.rs                            |  109 +
 rust/syn/ty.rs                            | 1288 +++++++
 rust/syn/verbatim.rs                      |   35 +
 rust/syn/whitespace.rs                    |   67 +
 samples/rust/Kconfig                      |   28 +
 samples/rust/Makefile                     |    3 +
 samples/rust/local_data_format/de.rs      |  422 +++
 samples/rust/local_data_format/error.rs   |   73 +
 samples/rust/local_data_format/ser.rs     |  443 +++
 samples/rust/puzzle.rs                    |    4 +
 samples/rust/puzzle/error.rs              |   91 +
 samples/rust/puzzle/inode.rs              |  150 +
 samples/rust/puzzle/oci.rs                |   71 +
 samples/rust/puzzle/types.rs              |  389 +++
 samples/rust/puzzle/types/cbor_helpers.rs |   50 +
 samples/rust/puzzlefs.rs                  |  220 ++
 samples/rust/rust_fs.rs                   |  105 +
 samples/rust/rust_serde.rs                |  125 +
 scripts/Makefile.build                    |    4 +-
 160 files changed, 89204 insertions(+), 29 deletions(-)
 create mode 100644 arch/x86/configs/qemu-busybox-min.config
 create mode 100644 kernel/configs/qemu-busybox-min.config
 create mode 100644 rust/hex/error.rs
 create mode 100644 rust/hex/lib.rs
 create mode 100644 rust/hex/serde.rs
 create mode 100644 rust/kernel/cred.rs
 create mode 100644 rust/kernel/delay.rs
 create mode 100644 rust/kernel/driver.rs
 create mode 100644 rust/kernel/file.rs
 create mode 100644 rust/kernel/fs.rs
 create mode 100644 rust/kernel/fs/param.rs
 create mode 100644 rust/kernel/io_buffer.rs
 create mode 100644 rust/kernel/iov_iter.rs
 create mode 100644 rust/kernel/mm.rs
 create mode 100644 rust/kernel/mount.rs
 create mode 100644 rust/kernel/pages.rs
 create mode 100644 rust/kernel/test_serde.rs
 create mode 100644 rust/kernel/test_serde/de.rs
 create mode 100644 rust/kernel/test_serde/error.rs
 create mode 100644 rust/kernel/test_serde/ser.rs
 create mode 100644 rust/kernel/user_ptr.rs
 create mode 100644 rust/proc-macro2/detection.rs
 create mode 100644 rust/proc-macro2/fallback.rs
 create mode 100644 rust/proc-macro2/lib.rs
 create mode 100644 rust/proc-macro2/marker.rs
 create mode 100644 rust/proc-macro2/parse.rs
 create mode 100644 rust/proc-macro2/rcvec.rs
 create mode 100644 rust/proc-macro2/wrapper.rs
 create mode 100644 rust/quote/ext.rs
 create mode 100644 rust/quote/format.rs
 create mode 100644 rust/quote/ident_fragment.rs
 create mode 100644 rust/quote/lib.rs
 create mode 100644 rust/quote/runtime.rs
 create mode 100644 rust/quote/spanned.rs
 create mode 100644 rust/quote/to_tokens.rs
 create mode 100644 rust/serde/de/format.rs
 create mode 100644 rust/serde/de/ignored_any.rs
 create mode 100644 rust/serde/de/impls.rs
 create mode 100644 rust/serde/de/mod.rs
 create mode 100644 rust/serde/de/seed.rs
 create mode 100644 rust/serde/de/utf8.rs
 create mode 100644 rust/serde/de/value.rs
 create mode 100644 rust/serde/integer128.rs
 create mode 100644 rust/serde/lib.rs
 create mode 100644 rust/serde/macros.rs
 create mode 100644 rust/serde/private/de.rs
 create mode 100644 rust/serde/private/doc.rs
 create mode 100644 rust/serde/private/mod.rs
 create mode 100644 rust/serde/private/ser.rs
 create mode 100644 rust/serde/private/size_hint.rs
 create mode 100644 rust/serde/ser/fmt.rs
 create mode 100644 rust/serde/ser/impls.rs
 create mode 100644 rust/serde/ser/impossible.rs
 create mode 100644 rust/serde/ser/mod.rs
 create mode 100644 rust/serde/std_error.rs
 create mode 100644 rust/serde_cbor/de.rs
 create mode 100644 rust/serde_cbor/error.rs
 create mode 100644 rust/serde_cbor/lib.rs
 create mode 100644 rust/serde_cbor/read.rs
 create mode 100644 rust/serde_cbor/ser.rs
 create mode 100644 rust/serde_cbor/tags.rs
 create mode 100644 rust/serde_cbor/value/de.rs
 create mode 100644 rust/serde_cbor/value/mod.rs
 create mode 100644 rust/serde_cbor/value/ser.rs
 create mode 100644 rust/serde_cbor/write.rs
 create mode 100644 rust/serde_derive/bound.rs
 create mode 100644 rust/serde_derive/de.rs
 create mode 100644 rust/serde_derive/dummy.rs
 create mode 100644 rust/serde_derive/fragment.rs
 create mode 100644 rust/serde_derive/internals/ast.rs
 create mode 100644 rust/serde_derive/internals/attr.rs
 create mode 100644 rust/serde_derive/internals/case.rs
 create mode 100644 rust/serde_derive/internals/check.rs
 create mode 100644 rust/serde_derive/internals/ctxt.rs
 create mode 100644 rust/serde_derive/internals/mod.rs
 create mode 100644 rust/serde_derive/internals/receiver.rs
 create mode 100644 rust/serde_derive/internals/respan.rs
 create mode 100644 rust/serde_derive/internals/symbol.rs
 create mode 100644 rust/serde_derive/lib.rs
 create mode 100644 rust/serde_derive/pretend.rs
 create mode 100644 rust/serde_derive/ser.rs
 create mode 100644 rust/serde_derive/this.rs
 create mode 100644 rust/serde_derive/try.rs
 create mode 100644 rust/syn/attr.rs
 create mode 100644 rust/syn/await.rs
 create mode 100644 rust/syn/bigint.rs
 create mode 100644 rust/syn/buffer.rs
 create mode 100644 rust/syn/custom_keyword.rs
 create mode 100644 rust/syn/custom_punctuation.rs
 create mode 100644 rust/syn/data.rs
 create mode 100644 rust/syn/derive.rs
 create mode 100644 rust/syn/discouraged.rs
 create mode 100644 rust/syn/error.rs
 create mode 100644 rust/syn/export.rs
 create mode 100644 rust/syn/expr.rs
 create mode 100644 rust/syn/ext.rs
 create mode 100644 rust/syn/file.rs
 create mode 100644 rust/syn/gen/clone.rs
 create mode 100644 rust/syn/gen/debug.rs
 create mode 100644 rust/syn/gen/eq.rs
 create mode 100644 rust/syn/gen/fold.rs
 create mode 100644 rust/syn/gen/hash.rs
 create mode 100644 rust/syn/gen/visit.rs
 create mode 100644 rust/syn/gen/visit_mut.rs
 create mode 100644 rust/syn/gen_helper.rs
 create mode 100644 rust/syn/generics.rs
 create mode 100644 rust/syn/group.rs
 create mode 100644 rust/syn/ident.rs
 create mode 100644 rust/syn/item.rs
 create mode 100644 rust/syn/lib.rs
 create mode 100644 rust/syn/lifetime.rs
 create mode 100644 rust/syn/lit.rs
 create mode 100644 rust/syn/lookahead.rs
 create mode 100644 rust/syn/mac.rs
 create mode 100644 rust/syn/macros.rs
 create mode 100644 rust/syn/op.rs
 create mode 100644 rust/syn/parse.rs
 create mode 100644 rust/syn/parse_macro_input.rs
 create mode 100644 rust/syn/parse_quote.rs
 create mode 100644 rust/syn/pat.rs
 create mode 100644 rust/syn/path.rs
 create mode 100644 rust/syn/print.rs
 create mode 100644 rust/syn/punctuated.rs
 create mode 100644 rust/syn/reserved.rs
 create mode 100644 rust/syn/sealed.rs
 create mode 100644 rust/syn/span.rs
 create mode 100644 rust/syn/spanned.rs
 create mode 100644 rust/syn/stmt.rs
 create mode 100644 rust/syn/thread.rs
 create mode 100644 rust/syn/token.rs
 create mode 100644 rust/syn/tt.rs
 create mode 100644 rust/syn/ty.rs
 create mode 100644 rust/syn/verbatim.rs
 create mode 100644 rust/syn/whitespace.rs
 create mode 100644 samples/rust/local_data_format/de.rs
 create mode 100644 samples/rust/local_data_format/error.rs
 create mode 100644 samples/rust/local_data_format/ser.rs
 create mode 100644 samples/rust/puzzle.rs
 create mode 100644 samples/rust/puzzle/error.rs
 create mode 100644 samples/rust/puzzle/inode.rs
 create mode 100644 samples/rust/puzzle/oci.rs
 create mode 100644 samples/rust/puzzle/types.rs
 create mode 100644 samples/rust/puzzle/types/cbor_helpers.rs
 create mode 100644 samples/rust/puzzlefs.rs
 create mode 100644 samples/rust/rust_fs.rs
 create mode 100644 samples/rust/rust_serde.rs

-- 
2.40.1

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Miguel Ojeda]

On Fri, Jun 9, 2023 at 8:58 AM Ariel Miculas <amiculas@cisco.com> wrote:
>
> This is a proof of concept driver written for the PuzzleFS
> next-generation container filesystem [1]. I've included a short abstract
> about puzzlefs further below. This driver is based on the rust-next

Thanks for preparing this RFC Ariel!

I have left some procedural notes here and there.

Cheers,
Miguel

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Christian Brauner]

On Fri, Jun 09, 2023 at 09:29:58AM +0300, Ariel Miculas wrote:
> Hi all!
> 
> This is a proof of concept driver written for the PuzzleFS

Uhm, the PuzzleFS filesystem isn't actually sent to fsdevel? Yet I see
tons of patches in there that add wrappers to our core fs data
structures. I even see a ton of new files that all fall clearly into
fsdevel territory:

create mode 100644 rust/kernel/cred.rs
create mode 100644 rust/kernel/delay.rs
create mode 100644 rust/kernel/driver.rs
create mode 100644 rust/kernel/file.rs
create mode 100644 rust/kernel/fs.rs
create mode 100644 rust/kernel/fs/param.rs
create mode 100644 rust/kernel/io_buffer.rs
create mode 100644 rust/kernel/iov_iter.rs
create mode 100644 rust/kernel/mm.rs
create mode 100644 rust/kernel/mount.rs
create mode 100644 rust/kernel/pages.rs

There's also quite a lot of new mm/ in there, no?

Any wrappers and code for core fs should be maintained as part of fs.
Rust shouldn't become a way to avoid our reviews once you have a few
wrappers added somewhere.

> next-generation container filesystem [1]. I've included a short abstract
> about puzzlefs further below. This driver is based on the rust-next
> branch, on top of which I've backported the filesystem abstractions from
> Wedson Almeida Filho [2][3] and Miguel Ojeda's third-party crates
> support: proc-macro2, quote, syn, serde and serde_derive [4]. I've added
> the additional third-party crates serde_cbor[5] and hex [6]. Then I've
> adapted the user space puzzlefs code [1] so that the puzzlefs kernel
> module could present the directory hierarchy and implement the basic
> read functionality.
> For some additional context, puzzlefs was started by Tycho Andersen and
> it's the successor of atomfs. This FOSDEM presentation from 2019 [12]
> covers the rationale for a new oci image format and presents a higher
> level overview of our goals with puzzlefs.
> I've split the rest of the cover letter in following sections (using a
> markdown style):
> * Example: it describes a practical example of what was achieved
> * Limitations: it presents the existing limitations of this POC
> * Upstreaming steps: it describes the steps needed for upstreaming this
>   driver
> * Setup: it shows how to setup the necessary environment for testing the
>   puzzlefs driver
> * Puzzlefs abstract: it provides a short overview of puzzlefs
> 
> # Example
> An example is provided below:
> 
> ```
> ~ # cat /proc/filesystems | grep puzzlefs
> nodev   puzzlefs
> ~ # mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o image_manifest="2d
> 6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" none /mnt
> ~ # ls -lR /mnt/
> /mnt/:
> total 0
> drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-1
> drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-2
> drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-3
> drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-4
> -rw-r--r--    1 0        0                0 Jun  8 12:26 file1
> -rw-r--r--    1 0        0                0 Jun  8 12:26 file2
> 
> /mnt/dir-1:
> total 0
> 
> /mnt/dir-2:
> total 0
> 
> /mnt/dir-3:
> total 0
> 
> /mnt/dir-4:
> total 0
> ~ # cat /mnt/file2
> ana are mere bla bla bla
> ~ # wc /mnt/file1
>       202       202      5454 /mnt/file1
> ```
> 
> In this example, /home/puzzlefs_oci is the puzzlefs oci directory:
> ```
> ~ # ls -lR /home/puzzlefs_oci/
> /home/puzzlefs_oci/:
> total 8
> drwxr-xr-x    3 1000     1000             0 Jun  8 14:33 blobs
> -rw-r--r--    1 1000     1000           266 Jun  8 14:33 index.json
> -rw-r--r--    1 1000     1000            37 Jun  8 14:33 oci-layout
> 
> /home/puzzlefs_oci/blobs:
> total 0
> drwxr-xr-x    2 1000     1000             0 Jun  8 14:33 sha256
> 
> /home/puzzlefs_oci/blobs/sha256:
> total 16
> -rw-------    1 1000     1000            89 Jun  8 14:33 2d6602d678140540dc7e96de652a76a8b16eb
> -rw-------    1 1000     1000           925 Jun  8 14:33 4df03518eea406343dbb55046720f6a478881
> -rw-------    1 1000     1000          5479 Jun  8 14:33 d86a87b19bd9a2fec0d31687c1d669cdb59eb
> ```
> 
> `2d6602d678140540dc7e96de652a76a8b16eb` is the puzzlefs image manifest
> hash for the first_try tag:
> ```
> $ cat /tmp/oci-simple/index.json | jq .
> {
>   "schemaVersion": -1,
>   "manifests": [
>     {
>       "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b",
>       "size": 89,
>       "media_type": "application/vnd.puzzlefs.image.rootfs.v1",
>       "annotations": {
>         "org.opencontainers.image.ref.name": "first_try"
>       }
>     }
>   ],
>   "annotations": {}
> }
> ```
> 
> I will describe how to build a puzzlefs image in the `Setup` section, at
> step 5.
> 
> # Limitations
> One limitation is that the puzzlefs driver doesn't implement any lookup
> functionality and instead it inserts every directory entry into the
> dcache during init (see the `DCACHE_BASED` constant). This is similar to
> how the sample `rust_fs` driver works, but the goal is to implement
> proper lookup functions.  However, more filesystem abstractions need to
> be implemented before this can be achieved.
> 
> # Upstreaming steps
> Before the puzzlefs driver can be upstreamed, the following need to be
> merged:
> * Wedson's filesystem abstractions [3]
> * the necessary third-party crates [4] (with the preliminary discussion
> about whether this is desirable)
> 
> # Setup
> My setup is based on Wedson's tutorial [8]. Next, I will describe the
> necessary steps to build an initrd and run a custom kernel under qemu.
> 
> 1. Get the linux rust-next branch [9] and apply this patchset
> 
> 2. Follow the rust quickstart guide [10]
> 
> 3. configure and build the kernel
> ```
> $ make LLVM=1 allnoconfig qemu-busybox-min.config rust.config
> $ make LLVM=1 -j$(nproc)
> ```
> 
> 4. setup busybox
> ```
> git clone git://git.busybox.net/busybox
> cd busybox
> make menuconfig # enable 'Build static binary' config
> make
> make install
> ```
> This will create the `_install` directory with the rootfs inside it.
> 
> 5. create a home directory in the rootfs and copy a puzzlefs oci
> directory in home/puzzlefs_oci
> To create a puzzlefs oci directory:
> * download this custom puzzlefs repository [11] (it's custom because we
>   want to build an image without verity data)
> * run `make release`
> * create a simple filesystem structure with a few directories and files
>   (I've created one at ../test-puzzlefs/simple_rootfs)
> * build a puzzlefs oci image at
>   `~/work/busybox/_install/home/puzzlefs_oci` (replace this path with
>   your busybox path) with the tag `first_try`:
> ```
> $ target/release/puzzlefs build --omit-verity \
> ../test-puzzlefs/simple_rootfs ~/work/busybox/_install/home/puzzlefs_oci \
> first_try
> ```
> * get first_try's image manifest from index.json (inside `puzzlefs_oci`)
> ```
> $ cat index.json | jq . | grep digest
>       "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b",
> ```
> 
> 6. add the following 'init' script in the busybox rootfs (rootfs path
> defaults to `./_install'):
> ```
> #!/bin/sh
> mount -t devtmpfs none /dev
> mkdir -p /proc
> mount -t proc none /proc
> 
> ifconfig lo up
> udhcpc -i eth0
> 
> mkdir /mnt
> mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \
> image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" \
> none /mnt
> 
> setsid sh -c 'exec sh -l </dev/ttyS0 >/dev/ttyS0 2>&1'
> ```
> 
> Make sure to replace the `image_manifest` with your own digest. This
> init script will be passed to rdinit in the kernel command line.
> 
> 7. generate the initramfs
> 
> Assuming busybox is in `~/work/busybox`:
> ```
> cd ~/work/busybox/_install && find . | cpio -H newc -o | gzip > ../ramdisk.img
> ```
> This will generate a compressed ramdisk image in
> `~/work/busybox/ramdisk.img`.
> 
> 8. run with qemu (this assumes the linux tree is at '../linux' and busybox
> is at '../busybox'):
> ```
> qemu-system-x86_64 \
>     -accel kvm \
>     -cpu host \
>     -m 4G \
>     -initrd ../busybox/ramdisk.img \
>     -kernel ../linux/arch/x86/boot/bzImage \
>     -nographic \
>     -append 'console=ttyS0 nokaslr debug rdinit=/init' \
>     -nic user,model=rtl8139 \
>     -no-reboot
> ```
> 
> 9. Check whether puzzlefs has been successfully mounted
> ```
> ~ # mount | grep puzzlefs
> none on /mnt type puzzlefs (rw,relatime)
> ~ # ls /mnt/
> dir-1  dir-2  dir-3  dir-4  file1  file2
> ```
> 
> 
> # Puzzlefs abstract
> Puzzlefs [1] is a container filesystem designed to address the
> limitations of the existing OCI format. The main goals of the project
> are reduced duplication, reproducible image builds, direct mounting
> support and memory safety guarantees, some inspired by the OCIv2 design
> document [7].
> 
> Reduced duplication is achieved using the content defined chunking
> algorithm FastCDC. This implementation allows chunks to be shared among
> layers. Building a new layer starting from an existing one allows
> reusing most of the chunks.
> 
> Another goal of the project is reproducible image builds, which is
> achieved by defining a canonical representation of the image format.
> 
> Direct mounting support is a key feature of puzzlefs and, together with
> fs-verity, it provides data integrity. Currently, puzzlefs is
> implemented as a userspace filesystem (FUSE). A read-only kernel
> filesystem driver is underway.
> 
> Lastly, memory safety is critical to puzzlefs, leading to the decision
> to implement it in Rust. Another goal is to share the same code between
> user space and kernel space in order to provide one secure
> implementation.
> 
> 
> [1] https://github.com/anuvu/puzzlefs
> [2] https://github.com/wedsonaf/linux/tree/fs
> [3] https://github.com/Rust-for-Linux/linux/issues/1004
> [4] https://github.com/Rust-for-Linux/linux/pull/1007
> [5] https://docs.rs/serde_cbor/latest/serde_cbor/
> [6] https://docs.rs/hex/0.4.3/hex/
> [7] https://hackmd.io/@cyphar/ociv2-brainstorm
> [8] https://www.youtube.com/watch?v=tPs1uRqOnlk
> [9] https://github.com/Rust-for-Linux/linux/tree/rust-next
> [10] https://docs.kernel.org/rust/quick-start.html
> [11] https://github.com/ariel-miculas/puzzlefs/tree/support-no-verity-data
> [12] https://archive.fosdem.org/2019/schedule/event/containers_atomfs/
> 
> Ariel Miculas (58):
>   rust: kernel: add libraries required by the filesystem abstractions
>   rust: kernel: backport the delay module from the rust branch
>   rust: kernel: add container_of macro
>   rust: kernel: add offset_of macro
>   drop: Add crate::pr_warn declaration
>   rust: kernel: rename from_kernel_errno to from_errno
>   rust: kernel: Rename from_pointer to from_foreing and into_pointer to
>     into_foreign
>   rust: kernel: add count_paren_items macro, needed by define_fs_params
>     macro
>   rust: helpers: add missing rust helper 'alloc_pages'
>   kernel: configs: add qemu-busybox-min.config
>   rust: kernel: format the rust code
>   samples: puzzlefs: add initial puzzlefs sample, copied from rust_fs.rs
>   kernel: configs: enable rust samples in rust.config
>   Add SAMPLE_RUST_SERDE in rust.config
>   rust: kernel: fix compile errors after rebase to rust-next
>   rust: serde_cbor: import crate
>   rust: serde_cbor: add SPDX License Identifiers
>   rust: serde_cbor: add no_fp_fmt_parse support
>   rust: Kbuild: enable serde_cbor
>   samples: rust: add cbor serialize/deserialize example
>   rust: serde_cbor: add support for serde_cbor's from_slice method by
>     using a custom alloc_kernel feature
>   rust: serde: add support for deserializing Vec with kernel_alloc
>     feature
>   rust: file: Replace UnsafeCell with Opaque for File
>   rust: kernel: implement fmt::Debug for CString
>   samples: puzzlefs: rename RustFs to PuzzleFs
>   samples: puzzlefs: add basic deserializing support for the puzzlefs
>     metadata
>   rust: file: present the filesystem context to the open function
>   rust: kernel: add an abstraction over vfsmount to allow cloning a new
>     private mount
>   rust: file: add from_path, from_path_in_root_mnt and read_with_offset
>     methods to File
>   samples: puzzlefs: pass the Vfsmount structure from open to read and
>     return the contents of the data file inside /home/puzzlefs_oci
>   rust: file: move from_path, from_path_in_root_mnt and read_with_offset
>     methods to a RegularFile newtype
>   rust: file: ensure RegularFile can only create regular files
>   rust: file: add get_pos method to RegularFile
>   rust: file: add methods read_to_end, get_file_size and update_pos to
>     RegularFile
>   rust: file: define a minimal Read trait and implement it for
>     RegularFile
>   samples: puzzlefs: add cbor_get_array_size method
>   samples: puzzlefs: add KernelError to WireFormatError and implement
>     From conversion
>   samples: puzzlefs: implement new for MetadataBlob
>   samples: puzzlefs: build puzzlefs into the kernel, thus avoiding the
>     need to export rust symbols
>   rust: alloc: add try_clone for Vec<T>
>   rust: alloc: add from_iter_fallible for Vec<T>
>   samples: puzzlefs: implement to_errno and from_errno for
>     WireFormatError
>   samples: puzzlefs: add TryReserveError (and from conversion) to
>     WireFormatError
>   samples: puzzlefs: add higher level inode related functionality
>   samples: puzzlefs: populate the directory entries with the inodes from
>     the puzzlefs metadata file
>   rust: hex: import crate
>   rust: hex: add SPDX license identifiers
>   rust: Kbuild: enable `hex`
>   rust: hex: implement FromHex trait and hex::decode using a custom
>     kernel_alloc feature
>   rust: hex: add encode_hex_iter and encode_hex_upper_iter methods
>   rust: puzzlefs: add HexError to WireFormatError and implement the From
>     conversion
>   rust: puzzlefs: display the error value for
>     WireFormatError::KernelError
>   samples: puzzlefs: add Rootfs and Digest structs to types.rs
>   samples: puzzlefs: implement the conversion from WireFormatError to
>     kernel::error::Error
>   rust: puzzlefs: read the puzzlefs image manifest instead of an
>     individual metadata layer
>   rust: puzzlefs: rename PuzzleFs to PuzzleFsModule to avoid confusion
>     with the PuzzleFS struct
>   rust: puzzlefs: add support for reading files
>   rust: puzzlefs: add oci_root_dir and image_manifest filesystem
>     parameters
> 
> Miguel Ojeda (15):
>   rust: proc-macro2: import crate
>   rust: proc-macro2: add SPDX License Identifiers
>   rust: proc-macro2: remove `unicode_ident` dependency
>   rust: quote: import crate
>   rust: quote: add SPDX License Identifiers
>   rust: syn: import crate
>   rust: syn: add SPDX License Identifiers
>   rust: syn: remove `unicode-ident` dependency
>   rust: serde: import crate
>   rust: serde: add `no_fp_fmt_parse` support
>   rust: serde: add SPDX License Identifiers
>   rust: serde_derive: import crate
>   rust: serde_derive: add SPDX License Identifiers
>   rust: Kbuild: enable `proc-macro2`, `quote`, `syn`, `serde` and
>     `serde_derive`
>   rust: test `serde` support
> 
> Wedson Almeida Filho (7):
>   rust: add definitions for ref-counted inodes and dentries
>   rust: add ability to register a file system
>   rust: define fs context
>   rust: add support for file system parameters
>   rust: allow fs driver to initialise new superblocks
>   rust: add `module_fs` macro
>   WIP: rust: allow fs to be populated
> 
>  Makefile                                  |   14 +-
>  arch/x86/configs/qemu-busybox-min.config  |   11 +
>  kernel/configs/qemu-busybox-min.config    |   56 +
>  kernel/configs/rust.config                |   11 +
>  rust/.gitignore                           |    1 +
>  rust/Makefile                             |  232 +-
>  rust/alloc/vec/mod.rs                     |   48 +
>  rust/bindings/bindings_helper.h           |   14 +
>  rust/bindings/lib.rs                      |    5 +
>  rust/helpers.c                            |   76 +
>  rust/hex/error.rs                         |   78 +
>  rust/hex/lib.rs                           |  506 +++
>  rust/hex/serde.rs                         |  104 +
>  rust/kernel/cred.rs                       |   46 +
>  rust/kernel/delay.rs                      |  104 +
>  rust/kernel/driver.rs                     |   28 +
>  rust/kernel/error.rs                      |   52 +-
>  rust/kernel/file.rs                       | 1117 ++++++
>  rust/kernel/fs.rs                         | 1478 ++++++++
>  rust/kernel/fs/param.rs                   |  558 +++
>  rust/kernel/io_buffer.rs                  |  153 +
>  rust/kernel/iov_iter.rs                   |   81 +
>  rust/kernel/lib.rs                        |   83 +
>  rust/kernel/mm.rs                         |  149 +
>  rust/kernel/mount.rs                      |   66 +
>  rust/kernel/pages.rs                      |  144 +
>  rust/kernel/str.rs                        |    6 +
>  rust/kernel/test_serde.rs                 |   26 +
>  rust/kernel/test_serde/de.rs              |  439 +++
>  rust/kernel/test_serde/error.rs           |   73 +
>  rust/kernel/test_serde/ser.rs             |  466 +++
>  rust/kernel/user_ptr.rs                   |  175 +
>  rust/proc-macro2/detection.rs             |   77 +
>  rust/proc-macro2/fallback.rs              | 1004 ++++++
>  rust/proc-macro2/lib.rs                   | 1341 ++++++++
>  rust/proc-macro2/marker.rs                |   20 +
>  rust/proc-macro2/parse.rs                 |  874 +++++
>  rust/proc-macro2/rcvec.rs                 |  144 +
>  rust/proc-macro2/wrapper.rs               |  996 ++++++
>  rust/quote/ext.rs                         |  112 +
>  rust/quote/format.rs                      |  170 +
>  rust/quote/ident_fragment.rs              |   88 +
>  rust/quote/lib.rs                         | 1436 ++++++++
>  rust/quote/runtime.rs                     |  440 +++
>  rust/quote/spanned.rs                     |   45 +
>  rust/quote/to_tokens.rs                   |  211 ++
>  rust/serde/de/format.rs                   |   32 +
>  rust/serde/de/ignored_any.rs              |  246 ++
>  rust/serde/de/impls.rs                    | 2755 +++++++++++++++
>  rust/serde/de/mod.rs                      | 2313 +++++++++++++
>  rust/serde/de/seed.rs                     |   21 +
>  rust/serde/de/utf8.rs                     |   48 +
>  rust/serde/de/value.rs                    | 1718 ++++++++++
>  rust/serde/integer128.rs                  |   84 +
>  rust/serde/lib.rs                         |  351 ++
>  rust/serde/macros.rs                      |  238 ++
>  rust/serde/private/de.rs                  | 2997 ++++++++++++++++
>  rust/serde/private/doc.rs                 |  161 +
>  rust/serde/private/mod.rs                 |   52 +
>  rust/serde/private/ser.rs                 | 1316 +++++++
>  rust/serde/private/size_hint.rs           |   23 +
>  rust/serde/ser/fmt.rs                     |  180 +
>  rust/serde/ser/impls.rs                   |  987 ++++++
>  rust/serde/ser/impossible.rs              |  218 ++
>  rust/serde/ser/mod.rs                     | 1992 +++++++++++
>  rust/serde/std_error.rs                   |   50 +
>  rust/serde_cbor/de.rs                     | 1370 ++++++++
>  rust/serde_cbor/error.rs                  |  320 ++
>  rust/serde_cbor/lib.rs                    |  371 ++
>  rust/serde_cbor/read.rs                   |  647 ++++
>  rust/serde_cbor/ser.rs                    |  748 ++++
>  rust/serde_cbor/tags.rs                   |  224 ++
>  rust/serde_cbor/value/de.rs               |  168 +
>  rust/serde_cbor/value/mod.rs              |  158 +
>  rust/serde_cbor/value/ser.rs              |  447 +++
>  rust/serde_cbor/write.rs                  |  177 +
>  rust/serde_derive/bound.rs                |  408 +++
>  rust/serde_derive/de.rs                   | 3148 +++++++++++++++++
>  rust/serde_derive/dummy.rs                |   46 +
>  rust/serde_derive/fragment.rs             |   76 +
>  rust/serde_derive/internals/ast.rs        |  204 ++
>  rust/serde_derive/internals/attr.rs       | 1908 +++++++++++
>  rust/serde_derive/internals/case.rs       |  199 ++
>  rust/serde_derive/internals/check.rs      |  445 +++
>  rust/serde_derive/internals/ctxt.rs       |   64 +
>  rust/serde_derive/internals/mod.rs        |   30 +
>  rust/serde_derive/internals/receiver.rs   |  287 ++
>  rust/serde_derive/internals/respan.rs     |   18 +
>  rust/serde_derive/internals/symbol.rs     |   71 +
>  rust/serde_derive/lib.rs                  |  112 +
>  rust/serde_derive/pretend.rs              |  203 ++
>  rust/serde_derive/ser.rs                  | 1342 ++++++++
>  rust/serde_derive/this.rs                 |   34 +
>  rust/serde_derive/try.rs                  |   26 +
>  rust/syn/attr.rs                          |  664 ++++
>  rust/syn/await.rs                         |    4 +
>  rust/syn/bigint.rs                        |   68 +
>  rust/syn/buffer.rs                        |  400 +++
>  rust/syn/custom_keyword.rs                |  255 ++
>  rust/syn/custom_punctuation.rs            |  302 ++
>  rust/syn/data.rs                          |  495 +++
>  rust/syn/derive.rs                        |  276 ++
>  rust/syn/discouraged.rs                   |  196 ++
>  rust/syn/error.rs                         |  430 +++
>  rust/syn/export.rs                        |   41 +
>  rust/syn/expr.rs                          | 3560 +++++++++++++++++++
>  rust/syn/ext.rs                           |  141 +
>  rust/syn/file.rs                          |  127 +
>  rust/syn/gen/clone.rs                     | 2243 ++++++++++++
>  rust/syn/gen/debug.rs                     | 3044 +++++++++++++++++
>  rust/syn/gen/eq.rs                        | 2197 ++++++++++++
>  rust/syn/gen/fold.rs                      | 3343 ++++++++++++++++++
>  rust/syn/gen/hash.rs                      | 2871 ++++++++++++++++
>  rust/syn/gen/visit.rs                     | 3788 +++++++++++++++++++++
>  rust/syn/gen/visit_mut.rs                 | 3788 +++++++++++++++++++++
>  rust/syn/gen_helper.rs                    |  156 +
>  rust/syn/generics.rs                      | 1339 ++++++++
>  rust/syn/group.rs                         |  284 ++
>  rust/syn/ident.rs                         |  103 +
>  rust/syn/item.rs                          | 3315 ++++++++++++++++++
>  rust/syn/lib.rs                           |  985 ++++++
>  rust/syn/lifetime.rs                      |  156 +
>  rust/syn/lit.rs                           | 1602 +++++++++
>  rust/syn/lookahead.rs                     |  171 +
>  rust/syn/mac.rs                           |  221 ++
>  rust/syn/macros.rs                        |  179 +
>  rust/syn/op.rs                            |  236 ++
>  rust/syn/parse.rs                         | 1316 +++++++
>  rust/syn/parse_macro_input.rs             |  181 +
>  rust/syn/parse_quote.rs                   |  169 +
>  rust/syn/pat.rs                           |  929 +++++
>  rust/syn/path.rs                          |  856 +++++
>  rust/syn/print.rs                         |   18 +
>  rust/syn/punctuated.rs                    | 1070 ++++++
>  rust/syn/reserved.rs                      |   46 +
>  rust/syn/sealed.rs                        |    6 +
>  rust/syn/span.rs                          |   69 +
>  rust/syn/spanned.rs                       |  116 +
>  rust/syn/stmt.rs                          |  351 ++
>  rust/syn/thread.rs                        |   43 +
>  rust/syn/token.rs                         | 1015 ++++++
>  rust/syn/tt.rs                            |  109 +
>  rust/syn/ty.rs                            | 1288 +++++++
>  rust/syn/verbatim.rs                      |   35 +
>  rust/syn/whitespace.rs                    |   67 +
>  samples/rust/Kconfig                      |   28 +
>  samples/rust/Makefile                     |    3 +
>  samples/rust/local_data_format/de.rs      |  422 +++
>  samples/rust/local_data_format/error.rs   |   73 +
>  samples/rust/local_data_format/ser.rs     |  443 +++
>  samples/rust/puzzle.rs                    |    4 +
>  samples/rust/puzzle/error.rs              |   91 +
>  samples/rust/puzzle/inode.rs              |  150 +
>  samples/rust/puzzle/oci.rs                |   71 +
>  samples/rust/puzzle/types.rs              |  389 +++
>  samples/rust/puzzle/types/cbor_helpers.rs |   50 +
>  samples/rust/puzzlefs.rs                  |  220 ++
>  samples/rust/rust_fs.rs                   |  105 +
>  samples/rust/rust_serde.rs                |  125 +
>  scripts/Makefile.build                    |    4 +-
>  160 files changed, 89204 insertions(+), 29 deletions(-)
>  create mode 100644 arch/x86/configs/qemu-busybox-min.config
>  create mode 100644 kernel/configs/qemu-busybox-min.config
>  create mode 100644 rust/hex/error.rs
>  create mode 100644 rust/hex/lib.rs
>  create mode 100644 rust/hex/serde.rs
>  create mode 100644 rust/kernel/cred.rs
>  create mode 100644 rust/kernel/delay.rs
>  create mode 100644 rust/kernel/driver.rs
>  create mode 100644 rust/kernel/file.rs
>  create mode 100644 rust/kernel/fs.rs
>  create mode 100644 rust/kernel/fs/param.rs
>  create mode 100644 rust/kernel/io_buffer.rs
>  create mode 100644 rust/kernel/iov_iter.rs
>  create mode 100644 rust/kernel/mm.rs
>  create mode 100644 rust/kernel/mount.rs
>  create mode 100644 rust/kernel/pages.rs
>  create mode 100644 rust/kernel/test_serde.rs
>  create mode 100644 rust/kernel/test_serde/de.rs
>  create mode 100644 rust/kernel/test_serde/error.rs
>  create mode 100644 rust/kernel/test_serde/ser.rs
>  create mode 100644 rust/kernel/user_ptr.rs
>  create mode 100644 rust/proc-macro2/detection.rs
>  create mode 100644 rust/proc-macro2/fallback.rs
>  create mode 100644 rust/proc-macro2/lib.rs
>  create mode 100644 rust/proc-macro2/marker.rs
>  create mode 100644 rust/proc-macro2/parse.rs
>  create mode 100644 rust/proc-macro2/rcvec.rs
>  create mode 100644 rust/proc-macro2/wrapper.rs
>  create mode 100644 rust/quote/ext.rs
>  create mode 100644 rust/quote/format.rs
>  create mode 100644 rust/quote/ident_fragment.rs
>  create mode 100644 rust/quote/lib.rs
>  create mode 100644 rust/quote/runtime.rs
>  create mode 100644 rust/quote/spanned.rs
>  create mode 100644 rust/quote/to_tokens.rs
>  create mode 100644 rust/serde/de/format.rs
>  create mode 100644 rust/serde/de/ignored_any.rs
>  create mode 100644 rust/serde/de/impls.rs
>  create mode 100644 rust/serde/de/mod.rs
>  create mode 100644 rust/serde/de/seed.rs
>  create mode 100644 rust/serde/de/utf8.rs
>  create mode 100644 rust/serde/de/value.rs
>  create mode 100644 rust/serde/integer128.rs
>  create mode 100644 rust/serde/lib.rs
>  create mode 100644 rust/serde/macros.rs
>  create mode 100644 rust/serde/private/de.rs
>  create mode 100644 rust/serde/private/doc.rs
>  create mode 100644 rust/serde/private/mod.rs
>  create mode 100644 rust/serde/private/ser.rs
>  create mode 100644 rust/serde/private/size_hint.rs
>  create mode 100644 rust/serde/ser/fmt.rs
>  create mode 100644 rust/serde/ser/impls.rs
>  create mode 100644 rust/serde/ser/impossible.rs
>  create mode 100644 rust/serde/ser/mod.rs
>  create mode 100644 rust/serde/std_error.rs
>  create mode 100644 rust/serde_cbor/de.rs
>  create mode 100644 rust/serde_cbor/error.rs
>  create mode 100644 rust/serde_cbor/lib.rs
>  create mode 100644 rust/serde_cbor/read.rs
>  create mode 100644 rust/serde_cbor/ser.rs
>  create mode 100644 rust/serde_cbor/tags.rs
>  create mode 100644 rust/serde_cbor/value/de.rs
>  create mode 100644 rust/serde_cbor/value/mod.rs
>  create mode 100644 rust/serde_cbor/value/ser.rs
>  create mode 100644 rust/serde_cbor/write.rs
>  create mode 100644 rust/serde_derive/bound.rs
>  create mode 100644 rust/serde_derive/de.rs
>  create mode 100644 rust/serde_derive/dummy.rs
>  create mode 100644 rust/serde_derive/fragment.rs
>  create mode 100644 rust/serde_derive/internals/ast.rs
>  create mode 100644 rust/serde_derive/internals/attr.rs
>  create mode 100644 rust/serde_derive/internals/case.rs
>  create mode 100644 rust/serde_derive/internals/check.rs
>  create mode 100644 rust/serde_derive/internals/ctxt.rs
>  create mode 100644 rust/serde_derive/internals/mod.rs
>  create mode 100644 rust/serde_derive/internals/receiver.rs
>  create mode 100644 rust/serde_derive/internals/respan.rs
>  create mode 100644 rust/serde_derive/internals/symbol.rs
>  create mode 100644 rust/serde_derive/lib.rs
>  create mode 100644 rust/serde_derive/pretend.rs
>  create mode 100644 rust/serde_derive/ser.rs
>  create mode 100644 rust/serde_derive/this.rs
>  create mode 100644 rust/serde_derive/try.rs
>  create mode 100644 rust/syn/attr.rs
>  create mode 100644 rust/syn/await.rs
>  create mode 100644 rust/syn/bigint.rs
>  create mode 100644 rust/syn/buffer.rs
>  create mode 100644 rust/syn/custom_keyword.rs
>  create mode 100644 rust/syn/custom_punctuation.rs
>  create mode 100644 rust/syn/data.rs
>  create mode 100644 rust/syn/derive.rs
>  create mode 100644 rust/syn/discouraged.rs
>  create mode 100644 rust/syn/error.rs
>  create mode 100644 rust/syn/export.rs
>  create mode 100644 rust/syn/expr.rs
>  create mode 100644 rust/syn/ext.rs
>  create mode 100644 rust/syn/file.rs
>  create mode 100644 rust/syn/gen/clone.rs
>  create mode 100644 rust/syn/gen/debug.rs
>  create mode 100644 rust/syn/gen/eq.rs
>  create mode 100644 rust/syn/gen/fold.rs
>  create mode 100644 rust/syn/gen/hash.rs
>  create mode 100644 rust/syn/gen/visit.rs
>  create mode 100644 rust/syn/gen/visit_mut.rs
>  create mode 100644 rust/syn/gen_helper.rs
>  create mode 100644 rust/syn/generics.rs
>  create mode 100644 rust/syn/group.rs
>  create mode 100644 rust/syn/ident.rs
>  create mode 100644 rust/syn/item.rs
>  create mode 100644 rust/syn/lib.rs
>  create mode 100644 rust/syn/lifetime.rs
>  create mode 100644 rust/syn/lit.rs
>  create mode 100644 rust/syn/lookahead.rs
>  create mode 100644 rust/syn/mac.rs
>  create mode 100644 rust/syn/macros.rs
>  create mode 100644 rust/syn/op.rs
>  create mode 100644 rust/syn/parse.rs
>  create mode 100644 rust/syn/parse_macro_input.rs
>  create mode 100644 rust/syn/parse_quote.rs
>  create mode 100644 rust/syn/pat.rs
>  create mode 100644 rust/syn/path.rs
>  create mode 100644 rust/syn/print.rs
>  create mode 100644 rust/syn/punctuated.rs
>  create mode 100644 rust/syn/reserved.rs
>  create mode 100644 rust/syn/sealed.rs
>  create mode 100644 rust/syn/span.rs
>  create mode 100644 rust/syn/spanned.rs
>  create mode 100644 rust/syn/stmt.rs
>  create mode 100644 rust/syn/thread.rs
>  create mode 100644 rust/syn/token.rs
>  create mode 100644 rust/syn/tt.rs
>  create mode 100644 rust/syn/ty.rs
>  create mode 100644 rust/syn/verbatim.rs
>  create mode 100644 rust/syn/whitespace.rs
>  create mode 100644 samples/rust/local_data_format/de.rs
>  create mode 100644 samples/rust/local_data_format/error.rs
>  create mode 100644 samples/rust/local_data_format/ser.rs
>  create mode 100644 samples/rust/puzzle.rs
>  create mode 100644 samples/rust/puzzle/error.rs
>  create mode 100644 samples/rust/puzzle/inode.rs
>  create mode 100644 samples/rust/puzzle/oci.rs
>  create mode 100644 samples/rust/puzzle/types.rs
>  create mode 100644 samples/rust/puzzle/types/cbor_helpers.rs
>  create mode 100644 samples/rust/puzzlefs.rs
>  create mode 100644 samples/rust/rust_fs.rs
>  create mode 100644 samples/rust/rust_serde.rs
> 
> -- 
> 2.40.1
> 

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Ariel Miculas (amiculas)]

[-- Attachment #1: Type: text/plain, Size: 33334 bytes --]

Hello Christian,

I didn't send these patches to a wider audience because this is an initial prototype of the PuzzleFS driver, and it has a few prerequisites before it could be even considered for merging. First of all, the rust filesystem abstractions and their dependencies need to be upstreamed, then there needs to be a discussion regarding the inclusion of third-party crates in the linux kernel.

My plan was to send these patches to the rust-for-linux mailing list and then start a discussion with Miguel Ojeda regarding the upstreaming approach.
There are a lot of new files added in this patch series because I've included all the dependencies required so that my patches could be applied to the rust-next branch, but these dependencies will most likely need to be upstreamed separately.

It was never my intention to avoid your reviews, should I also send subsequent patches to linux-fsdevel, even if they're in the early stages of development?

Regards,
Ariel
________________________________
From: Christian Brauner <brauner@kernel.org>
Sent: Friday, June 9, 2023 1:36 PM
To: Ariel Miculas (amiculas) <amiculas@cisco.com>; linux-fsdevel@vger.kernel.org <linux-fsdevel@vger.kernel.org>
Cc: rust-for-linux@vger.kernel.org <rust-for-linux@vger.kernel.org>; linux-mm@kvack.org <linux-mm@kvack.org>
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

On Fri, Jun 09, 2023 at 09:29:58AM +0300, Ariel Miculas wrote:
> Hi all!
>
> This is a proof of concept driver written for the PuzzleFS

Uhm, the PuzzleFS filesystem isn't actually sent to fsdevel? Yet I see
tons of patches in there that add wrappers to our core fs data
structures. I even see a ton of new files that all fall clearly into
fsdevel territory:

create mode 100644 rust/kernel/cred.rs
create mode 100644 rust/kernel/delay.rs
create mode 100644 rust/kernel/driver.rs
create mode 100644 rust/kernel/file.rs
create mode 100644 rust/kernel/fs.rs
create mode 100644 rust/kernel/fs/param.rs
create mode 100644 rust/kernel/io_buffer.rs
create mode 100644 rust/kernel/iov_iter.rs
create mode 100644 rust/kernel/mm.rs
create mode 100644 rust/kernel/mount.rs
create mode 100644 rust/kernel/pages.rs

There's also quite a lot of new mm/ in there, no?

Any wrappers and code for core fs should be maintained as part of fs.
Rust shouldn't become a way to avoid our reviews once you have a few
wrappers added somewhere.

> next-generation container filesystem [1]. I've included a short abstract
> about puzzlefs further below. This driver is based on the rust-next
> branch, on top of which I've backported the filesystem abstractions from
> Wedson Almeida Filho [2][3] and Miguel Ojeda's third-party crates
> support: proc-macro2, quote, syn, serde and serde_derive [4]. I've added
> the additional third-party crates serde_cbor[5] and hex [6]. Then I've
> adapted the user space puzzlefs code [1] so that the puzzlefs kernel
> module could present the directory hierarchy and implement the basic
> read functionality.
> For some additional context, puzzlefs was started by Tycho Andersen and
> it's the successor of atomfs. This FOSDEM presentation from 2019 [12]
> covers the rationale for a new oci image format and presents a higher
> level overview of our goals with puzzlefs.
> I've split the rest of the cover letter in following sections (using a
> markdown style):
> * Example: it describes a practical example of what was achieved
> * Limitations: it presents the existing limitations of this POC
> * Upstreaming steps: it describes the steps needed for upstreaming this
>   driver
> * Setup: it shows how to setup the necessary environment for testing the
>   puzzlefs driver
> * Puzzlefs abstract: it provides a short overview of puzzlefs
>
> # Example
> An example is provided below:
>
> ```
> ~ # cat /proc/filesystems | grep puzzlefs
> nodev   puzzlefs
> ~ # mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o image_manifest="2d
> 6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" none /mnt
> ~ # ls -lR /mnt/
> /mnt/:
> total 0
> drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-1
> drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-2
> drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-3
> drwxr-xr-x    2 0        0                0 Jun  8 12:26 dir-4
> -rw-r--r--    1 0        0                0 Jun  8 12:26 file1
> -rw-r--r--    1 0        0                0 Jun  8 12:26 file2
>
> /mnt/dir-1:
> total 0
>
> /mnt/dir-2:
> total 0
>
> /mnt/dir-3:
> total 0
>
> /mnt/dir-4:
> total 0
> ~ # cat /mnt/file2
> ana are mere bla bla bla
> ~ # wc /mnt/file1
>       202       202      5454 /mnt/file1
> ```
>
> In this example, /home/puzzlefs_oci is the puzzlefs oci directory:
> ```
> ~ # ls -lR /home/puzzlefs_oci/
> /home/puzzlefs_oci/:
> total 8
> drwxr-xr-x    3 1000     1000             0 Jun  8 14:33 blobs
> -rw-r--r--    1 1000     1000           266 Jun  8 14:33 index.json
> -rw-r--r--    1 1000     1000            37 Jun  8 14:33 oci-layout
>
> /home/puzzlefs_oci/blobs:
> total 0
> drwxr-xr-x    2 1000     1000             0 Jun  8 14:33 sha256
>
> /home/puzzlefs_oci/blobs/sha256:
> total 16
> -rw-------    1 1000     1000            89 Jun  8 14:33 2d6602d678140540dc7e96de652a76a8b16eb
> -rw-------    1 1000     1000           925 Jun  8 14:33 4df03518eea406343dbb55046720f6a478881
> -rw-------    1 1000     1000          5479 Jun  8 14:33 d86a87b19bd9a2fec0d31687c1d669cdb59eb
> ```
>
> `2d6602d678140540dc7e96de652a76a8b16eb` is the puzzlefs image manifest
> hash for the first_try tag:
> ```
> $ cat /tmp/oci-simple/index.json | jq .
> {
>   "schemaVersion": -1,
>   "manifests": [
>     {
>       "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b",
>       "size": 89,
>       "media_type": "application/vnd.puzzlefs.image.rootfs.v1",
>       "annotations": {
>         "org.opencontainers.image.ref.name": "first_try"
>       }
>     }
>   ],
>   "annotations": {}
> }
> ```
>
> I will describe how to build a puzzlefs image in the `Setup` section, at
> step 5.
>
> # Limitations
> One limitation is that the puzzlefs driver doesn't implement any lookup
> functionality and instead it inserts every directory entry into the
> dcache during init (see the `DCACHE_BASED` constant). This is similar to
> how the sample `rust_fs` driver works, but the goal is to implement
> proper lookup functions.  However, more filesystem abstractions need to
> be implemented before this can be achieved.
>
> # Upstreaming steps
> Before the puzzlefs driver can be upstreamed, the following need to be
> merged:
> * Wedson's filesystem abstractions [3]
> * the necessary third-party crates [4] (with the preliminary discussion
> about whether this is desirable)
>
> # Setup
> My setup is based on Wedson's tutorial [8]. Next, I will describe the
> necessary steps to build an initrd and run a custom kernel under qemu.
>
> 1. Get the linux rust-next branch [9] and apply this patchset
>
> 2. Follow the rust quickstart guide [10]
>
> 3. configure and build the kernel
> ```
> $ make LLVM=1 allnoconfig qemu-busybox-min.config rust.config
> $ make LLVM=1 -j$(nproc)
> ```
>
> 4. setup busybox
> ```
> git clone git://git.busybox.net/busybox
> cd busybox
> make menuconfig # enable 'Build static binary' config
> make
> make install
> ```
> This will create the `_install` directory with the rootfs inside it.
>
> 5. create a home directory in the rootfs and copy a puzzlefs oci
> directory in home/puzzlefs_oci
> To create a puzzlefs oci directory:
> * download this custom puzzlefs repository [11] (it's custom because we
>   want to build an image without verity data)
> * run `make release`
> * create a simple filesystem structure with a few directories and files
>   (I've created one at ../test-puzzlefs/simple_rootfs)
> * build a puzzlefs oci image at
>   `~/work/busybox/_install/home/puzzlefs_oci` (replace this path with
>   your busybox path) with the tag `first_try`:
> ```
> $ target/release/puzzlefs build --omit-verity \
> ../test-puzzlefs/simple_rootfs ~/work/busybox/_install/home/puzzlefs_oci \
> first_try
> ```
> * get first_try's image manifest from index.json (inside `puzzlefs_oci`)
> ```
> $ cat index.json | jq . | grep digest
>       "digest": "sha256:2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b",
> ```
>
> 6. add the following 'init' script in the busybox rootfs (rootfs path
> defaults to `./_install'):
> ```
> #!/bin/sh
> mount -t devtmpfs none /dev
> mkdir -p /proc
> mount -t proc none /proc
>
> ifconfig lo up
> udhcpc -i eth0
>
> mkdir /mnt
> mount -t puzzlefs -o oci_root_dir="/home/puzzlefs_oci" -o \
> image_manifest="2d6602d678140540dc7e96de652a76a8b16e8aca190bae141297bcffdcae901b" \
> none /mnt
>
> setsid sh -c 'exec sh -l </dev/ttyS0 >/dev/ttyS0 2>&1'
> ```
>
> Make sure to replace the `image_manifest` with your own digest. This
> init script will be passed to rdinit in the kernel command line.
>
> 7. generate the initramfs
>
> Assuming busybox is in `~/work/busybox`:
> ```
> cd ~/work/busybox/_install && find . | cpio -H newc -o | gzip > ../ramdisk.img
> ```
> This will generate a compressed ramdisk image in
> `~/work/busybox/ramdisk.img`.
>
> 8. run with qemu (this assumes the linux tree is at '../linux' and busybox
> is at '../busybox'):
> ```
> qemu-system-x86_64 \
>     -accel kvm \
>     -cpu host \
>     -m 4G \
>     -initrd ../busybox/ramdisk.img \
>     -kernel ../linux/arch/x86/boot/bzImage \
>     -nographic \
>     -append 'console=ttyS0 nokaslr debug rdinit=/init' \
>     -nic user,model=rtl8139 \
>     -no-reboot
> ```
>
> 9. Check whether puzzlefs has been successfully mounted
> ```
> ~ # mount | grep puzzlefs
> none on /mnt type puzzlefs (rw,relatime)
> ~ # ls /mnt/
> dir-1  dir-2  dir-3  dir-4  file1  file2
> ```
>
>
> # Puzzlefs abstract
> Puzzlefs [1] is a container filesystem designed to address the
> limitations of the existing OCI format. The main goals of the project
> are reduced duplication, reproducible image builds, direct mounting
> support and memory safety guarantees, some inspired by the OCIv2 design
> document [7].
>
> Reduced duplication is achieved using the content defined chunking
> algorithm FastCDC. This implementation allows chunks to be shared among
> layers. Building a new layer starting from an existing one allows
> reusing most of the chunks.
>
> Another goal of the project is reproducible image builds, which is
> achieved by defining a canonical representation of the image format.
>
> Direct mounting support is a key feature of puzzlefs and, together with
> fs-verity, it provides data integrity. Currently, puzzlefs is
> implemented as a userspace filesystem (FUSE). A read-only kernel
> filesystem driver is underway.
>
> Lastly, memory safety is critical to puzzlefs, leading to the decision
> to implement it in Rust. Another goal is to share the same code between
> user space and kernel space in order to provide one secure
> implementation.
>
>
> [1] https://github.com/anuvu/puzzlefs
> [2] https://github.com/wedsonaf/linux/tree/fs
> [3] https://github.com/Rust-for-Linux/linux/issues/1004
> [4] https://github.com/Rust-for-Linux/linux/pull/1007
> [5] https://docs.rs/serde_cbor/latest/serde_cbor/
> [6] https://docs.rs/hex/0.4.3/hex/
> [7] https://hackmd.io/@cyphar/ociv2-brainstorm
> [8] https://www.youtube.com/watch?v=tPs1uRqOnlk
> [9] https://github.com/Rust-for-Linux/linux/tree/rust-next
> [10] https://docs.kernel.org/rust/quick-start.html
> [11] https://github.com/ariel-miculas/puzzlefs/tree/support-no-verity-data
> [12] https://archive.fosdem.org/2019/schedule/event/containers_atomfs/
>
> Ariel Miculas (58):
>   rust: kernel: add libraries required by the filesystem abstractions
>   rust: kernel: backport the delay module from the rust branch
>   rust: kernel: add container_of macro
>   rust: kernel: add offset_of macro
>   drop: Add crate::pr_warn declaration
>   rust: kernel: rename from_kernel_errno to from_errno
>   rust: kernel: Rename from_pointer to from_foreing and into_pointer to
>     into_foreign
>   rust: kernel: add count_paren_items macro, needed by define_fs_params
>     macro
>   rust: helpers: add missing rust helper 'alloc_pages'
>   kernel: configs: add qemu-busybox-min.config
>   rust: kernel: format the rust code
>   samples: puzzlefs: add initial puzzlefs sample, copied from rust_fs.rs
>   kernel: configs: enable rust samples in rust.config
>   Add SAMPLE_RUST_SERDE in rust.config
>   rust: kernel: fix compile errors after rebase to rust-next
>   rust: serde_cbor: import crate
>   rust: serde_cbor: add SPDX License Identifiers
>   rust: serde_cbor: add no_fp_fmt_parse support
>   rust: Kbuild: enable serde_cbor
>   samples: rust: add cbor serialize/deserialize example
>   rust: serde_cbor: add support for serde_cbor's from_slice method by
>     using a custom alloc_kernel feature
>   rust: serde: add support for deserializing Vec with kernel_alloc
>     feature
>   rust: file: Replace UnsafeCell with Opaque for File
>   rust: kernel: implement fmt::Debug for CString
>   samples: puzzlefs: rename RustFs to PuzzleFs
>   samples: puzzlefs: add basic deserializing support for the puzzlefs
>     metadata
>   rust: file: present the filesystem context to the open function
>   rust: kernel: add an abstraction over vfsmount to allow cloning a new
>     private mount
>   rust: file: add from_path, from_path_in_root_mnt and read_with_offset
>     methods to File
>   samples: puzzlefs: pass the Vfsmount structure from open to read and
>     return the contents of the data file inside /home/puzzlefs_oci
>   rust: file: move from_path, from_path_in_root_mnt and read_with_offset
>     methods to a RegularFile newtype
>   rust: file: ensure RegularFile can only create regular files
>   rust: file: add get_pos method to RegularFile
>   rust: file: add methods read_to_end, get_file_size and update_pos to
>     RegularFile
>   rust: file: define a minimal Read trait and implement it for
>     RegularFile
>   samples: puzzlefs: add cbor_get_array_size method
>   samples: puzzlefs: add KernelError to WireFormatError and implement
>     From conversion
>   samples: puzzlefs: implement new for MetadataBlob
>   samples: puzzlefs: build puzzlefs into the kernel, thus avoiding the
>     need to export rust symbols
>   rust: alloc: add try_clone for Vec<T>
>   rust: alloc: add from_iter_fallible for Vec<T>
>   samples: puzzlefs: implement to_errno and from_errno for
>     WireFormatError
>   samples: puzzlefs: add TryReserveError (and from conversion) to
>     WireFormatError
>   samples: puzzlefs: add higher level inode related functionality
>   samples: puzzlefs: populate the directory entries with the inodes from
>     the puzzlefs metadata file
>   rust: hex: import crate
>   rust: hex: add SPDX license identifiers
>   rust: Kbuild: enable `hex`
>   rust: hex: implement FromHex trait and hex::decode using a custom
>     kernel_alloc feature
>   rust: hex: add encode_hex_iter and encode_hex_upper_iter methods
>   rust: puzzlefs: add HexError to WireFormatError and implement the From
>     conversion
>   rust: puzzlefs: display the error value for
>     WireFormatError::KernelError
>   samples: puzzlefs: add Rootfs and Digest structs to types.rs
>   samples: puzzlefs: implement the conversion from WireFormatError to
>     kernel::error::Error
>   rust: puzzlefs: read the puzzlefs image manifest instead of an
>     individual metadata layer
>   rust: puzzlefs: rename PuzzleFs to PuzzleFsModule to avoid confusion
>     with the PuzzleFS struct
>   rust: puzzlefs: add support for reading files
>   rust: puzzlefs: add oci_root_dir and image_manifest filesystem
>     parameters
>
> Miguel Ojeda (15):
>   rust: proc-macro2: import crate
>   rust: proc-macro2: add SPDX License Identifiers
>   rust: proc-macro2: remove `unicode_ident` dependency
>   rust: quote: import crate
>   rust: quote: add SPDX License Identifiers
>   rust: syn: import crate
>   rust: syn: add SPDX License Identifiers
>   rust: syn: remove `unicode-ident` dependency
>   rust: serde: import crate
>   rust: serde: add `no_fp_fmt_parse` support
>   rust: serde: add SPDX License Identifiers
>   rust: serde_derive: import crate
>   rust: serde_derive: add SPDX License Identifiers
>   rust: Kbuild: enable `proc-macro2`, `quote`, `syn`, `serde` and
>     `serde_derive`
>   rust: test `serde` support
>
> Wedson Almeida Filho (7):
>   rust: add definitions for ref-counted inodes and dentries
>   rust: add ability to register a file system
>   rust: define fs context
>   rust: add support for file system parameters
>   rust: allow fs driver to initialise new superblocks
>   rust: add `module_fs` macro
>   WIP: rust: allow fs to be populated
>
>  Makefile                                  |   14 +-
>  arch/x86/configs/qemu-busybox-min.config  |   11 +
>  kernel/configs/qemu-busybox-min.config    |   56 +
>  kernel/configs/rust.config                |   11 +
>  rust/.gitignore                           |    1 +
>  rust/Makefile                             |  232 +-
>  rust/alloc/vec/mod.rs                     |   48 +
>  rust/bindings/bindings_helper.h           |   14 +
>  rust/bindings/lib.rs                      |    5 +
>  rust/helpers.c                            |   76 +
>  rust/hex/error.rs                         |   78 +
>  rust/hex/lib.rs                           |  506 +++
>  rust/hex/serde.rs                         |  104 +
>  rust/kernel/cred.rs                       |   46 +
>  rust/kernel/delay.rs                      |  104 +
>  rust/kernel/driver.rs                     |   28 +
>  rust/kernel/error.rs                      |   52 +-
>  rust/kernel/file.rs                       | 1117 ++++++
>  rust/kernel/fs.rs                         | 1478 ++++++++
>  rust/kernel/fs/param.rs                   |  558 +++
>  rust/kernel/io_buffer.rs                  |  153 +
>  rust/kernel/iov_iter.rs                   |   81 +
>  rust/kernel/lib.rs                        |   83 +
>  rust/kernel/mm.rs                         |  149 +
>  rust/kernel/mount.rs                      |   66 +
>  rust/kernel/pages.rs                      |  144 +
>  rust/kernel/str.rs                        |    6 +
>  rust/kernel/test_serde.rs                 |   26 +
>  rust/kernel/test_serde/de.rs              |  439 +++
>  rust/kernel/test_serde/error.rs           |   73 +
>  rust/kernel/test_serde/ser.rs             |  466 +++
>  rust/kernel/user_ptr.rs                   |  175 +
>  rust/proc-macro2/detection.rs             |   77 +
>  rust/proc-macro2/fallback.rs              | 1004 ++++++
>  rust/proc-macro2/lib.rs                   | 1341 ++++++++
>  rust/proc-macro2/marker.rs                |   20 +
>  rust/proc-macro2/parse.rs                 |  874 +++++
>  rust/proc-macro2/rcvec.rs                 |  144 +
>  rust/proc-macro2/wrapper.rs               |  996 ++++++
>  rust/quote/ext.rs                         |  112 +
>  rust/quote/format.rs                      |  170 +
>  rust/quote/ident_fragment.rs              |   88 +
>  rust/quote/lib.rs                         | 1436 ++++++++
>  rust/quote/runtime.rs                     |  440 +++
>  rust/quote/spanned.rs                     |   45 +
>  rust/quote/to_tokens.rs                   |  211 ++
>  rust/serde/de/format.rs                   |   32 +
>  rust/serde/de/ignored_any.rs              |  246 ++
>  rust/serde/de/impls.rs                    | 2755 +++++++++++++++
>  rust/serde/de/mod.rs                      | 2313 +++++++++++++
>  rust/serde/de/seed.rs                     |   21 +
>  rust/serde/de/utf8.rs                     |   48 +
>  rust/serde/de/value.rs                    | 1718 ++++++++++
>  rust/serde/integer128.rs                  |   84 +
>  rust/serde/lib.rs                         |  351 ++
>  rust/serde/macros.rs                      |  238 ++
>  rust/serde/private/de.rs                  | 2997 ++++++++++++++++
>  rust/serde/private/doc.rs                 |  161 +
>  rust/serde/private/mod.rs                 |   52 +
>  rust/serde/private/ser.rs                 | 1316 +++++++
>  rust/serde/private/size_hint.rs           |   23 +
>  rust/serde/ser/fmt.rs                     |  180 +
>  rust/serde/ser/impls.rs                   |  987 ++++++
>  rust/serde/ser/impossible.rs              |  218 ++
>  rust/serde/ser/mod.rs                     | 1992 +++++++++++
>  rust/serde/std_error.rs                   |   50 +
>  rust/serde_cbor/de.rs                     | 1370 ++++++++
>  rust/serde_cbor/error.rs                  |  320 ++
>  rust/serde_cbor/lib.rs                    |  371 ++
>  rust/serde_cbor/read.rs                   |  647 ++++
>  rust/serde_cbor/ser.rs                    |  748 ++++
>  rust/serde_cbor/tags.rs                   |  224 ++
>  rust/serde_cbor/value/de.rs               |  168 +
>  rust/serde_cbor/value/mod.rs              |  158 +
>  rust/serde_cbor/value/ser.rs              |  447 +++
>  rust/serde_cbor/write.rs                  |  177 +
>  rust/serde_derive/bound.rs                |  408 +++
>  rust/serde_derive/de.rs                   | 3148 +++++++++++++++++
>  rust/serde_derive/dummy.rs                |   46 +
>  rust/serde_derive/fragment.rs             |   76 +
>  rust/serde_derive/internals/ast.rs        |  204 ++
>  rust/serde_derive/internals/attr.rs       | 1908 +++++++++++
>  rust/serde_derive/internals/case.rs       |  199 ++
>  rust/serde_derive/internals/check.rs      |  445 +++
>  rust/serde_derive/internals/ctxt.rs       |   64 +
>  rust/serde_derive/internals/mod.rs        |   30 +
>  rust/serde_derive/internals/receiver.rs   |  287 ++
>  rust/serde_derive/internals/respan.rs     |   18 +
>  rust/serde_derive/internals/symbol.rs     |   71 +
>  rust/serde_derive/lib.rs                  |  112 +
>  rust/serde_derive/pretend.rs              |  203 ++
>  rust/serde_derive/ser.rs                  | 1342 ++++++++
>  rust/serde_derive/this.rs                 |   34 +
>  rust/serde_derive/try.rs                  |   26 +
>  rust/syn/attr.rs                          |  664 ++++
>  rust/syn/await.rs                         |    4 +
>  rust/syn/bigint.rs                        |   68 +
>  rust/syn/buffer.rs                        |  400 +++
>  rust/syn/custom_keyword.rs                |  255 ++
>  rust/syn/custom_punctuation.rs            |  302 ++
>  rust/syn/data.rs                          |  495 +++
>  rust/syn/derive.rs                        |  276 ++
>  rust/syn/discouraged.rs                   |  196 ++
>  rust/syn/error.rs                         |  430 +++
>  rust/syn/export.rs                        |   41 +
>  rust/syn/expr.rs                          | 3560 +++++++++++++++++++
>  rust/syn/ext.rs                           |  141 +
>  rust/syn/file.rs                          |  127 +
>  rust/syn/gen/clone.rs                     | 2243 ++++++++++++
>  rust/syn/gen/debug.rs                     | 3044 +++++++++++++++++
>  rust/syn/gen/eq.rs                        | 2197 ++++++++++++
>  rust/syn/gen/fold.rs                      | 3343 ++++++++++++++++++
>  rust/syn/gen/hash.rs                      | 2871 ++++++++++++++++
>  rust/syn/gen/visit.rs                     | 3788 +++++++++++++++++++++
>  rust/syn/gen/visit_mut.rs                 | 3788 +++++++++++++++++++++
>  rust/syn/gen_helper.rs                    |  156 +
>  rust/syn/generics.rs                      | 1339 ++++++++
>  rust/syn/group.rs                         |  284 ++
>  rust/syn/ident.rs                         |  103 +
>  rust/syn/item.rs                          | 3315 ++++++++++++++++++
>  rust/syn/lib.rs                           |  985 ++++++
>  rust/syn/lifetime.rs                      |  156 +
>  rust/syn/lit.rs                           | 1602 +++++++++
>  rust/syn/lookahead.rs                     |  171 +
>  rust/syn/mac.rs                           |  221 ++
>  rust/syn/macros.rs                        |  179 +
>  rust/syn/op.rs                            |  236 ++
>  rust/syn/parse.rs                         | 1316 +++++++
>  rust/syn/parse_macro_input.rs             |  181 +
>  rust/syn/parse_quote.rs                   |  169 +
>  rust/syn/pat.rs                           |  929 +++++
>  rust/syn/path.rs                          |  856 +++++
>  rust/syn/print.rs                         |   18 +
>  rust/syn/punctuated.rs                    | 1070 ++++++
>  rust/syn/reserved.rs                      |   46 +
>  rust/syn/sealed.rs                        |    6 +
>  rust/syn/span.rs                          |   69 +
>  rust/syn/spanned.rs                       |  116 +
>  rust/syn/stmt.rs                          |  351 ++
>  rust/syn/thread.rs                        |   43 +
>  rust/syn/token.rs                         | 1015 ++++++
>  rust/syn/tt.rs                            |  109 +
>  rust/syn/ty.rs                            | 1288 +++++++
>  rust/syn/verbatim.rs                      |   35 +
>  rust/syn/whitespace.rs                    |   67 +
>  samples/rust/Kconfig                      |   28 +
>  samples/rust/Makefile                     |    3 +
>  samples/rust/local_data_format/de.rs      |  422 +++
>  samples/rust/local_data_format/error.rs   |   73 +
>  samples/rust/local_data_format/ser.rs     |  443 +++
>  samples/rust/puzzle.rs                    |    4 +
>  samples/rust/puzzle/error.rs              |   91 +
>  samples/rust/puzzle/inode.rs              |  150 +
>  samples/rust/puzzle/oci.rs                |   71 +
>  samples/rust/puzzle/types.rs              |  389 +++
>  samples/rust/puzzle/types/cbor_helpers.rs |   50 +
>  samples/rust/puzzlefs.rs                  |  220 ++
>  samples/rust/rust_fs.rs                   |  105 +
>  samples/rust/rust_serde.rs                |  125 +
>  scripts/Makefile.build                    |    4 +-
>  160 files changed, 89204 insertions(+), 29 deletions(-)
>  create mode 100644 arch/x86/configs/qemu-busybox-min.config
>  create mode 100644 kernel/configs/qemu-busybox-min.config
>  create mode 100644 rust/hex/error.rs
>  create mode 100644 rust/hex/lib.rs
>  create mode 100644 rust/hex/serde.rs
>  create mode 100644 rust/kernel/cred.rs
>  create mode 100644 rust/kernel/delay.rs
>  create mode 100644 rust/kernel/driver.rs
>  create mode 100644 rust/kernel/file.rs
>  create mode 100644 rust/kernel/fs.rs
>  create mode 100644 rust/kernel/fs/param.rs
>  create mode 100644 rust/kernel/io_buffer.rs
>  create mode 100644 rust/kernel/iov_iter.rs
>  create mode 100644 rust/kernel/mm.rs
>  create mode 100644 rust/kernel/mount.rs
>  create mode 100644 rust/kernel/pages.rs
>  create mode 100644 rust/kernel/test_serde.rs
>  create mode 100644 rust/kernel/test_serde/de.rs
>  create mode 100644 rust/kernel/test_serde/error.rs
>  create mode 100644 rust/kernel/test_serde/ser.rs
>  create mode 100644 rust/kernel/user_ptr.rs
>  create mode 100644 rust/proc-macro2/detection.rs
>  create mode 100644 rust/proc-macro2/fallback.rs
>  create mode 100644 rust/proc-macro2/lib.rs
>  create mode 100644 rust/proc-macro2/marker.rs
>  create mode 100644 rust/proc-macro2/parse.rs
>  create mode 100644 rust/proc-macro2/rcvec.rs
>  create mode 100644 rust/proc-macro2/wrapper.rs
>  create mode 100644 rust/quote/ext.rs
>  create mode 100644 rust/quote/format.rs
>  create mode 100644 rust/quote/ident_fragment.rs
>  create mode 100644 rust/quote/lib.rs
>  create mode 100644 rust/quote/runtime.rs
>  create mode 100644 rust/quote/spanned.rs
>  create mode 100644 rust/quote/to_tokens.rs
>  create mode 100644 rust/serde/de/format.rs
>  create mode 100644 rust/serde/de/ignored_any.rs
>  create mode 100644 rust/serde/de/impls.rs
>  create mode 100644 rust/serde/de/mod.rs
>  create mode 100644 rust/serde/de/seed.rs
>  create mode 100644 rust/serde/de/utf8.rs
>  create mode 100644 rust/serde/de/value.rs
>  create mode 100644 rust/serde/integer128.rs
>  create mode 100644 rust/serde/lib.rs
>  create mode 100644 rust/serde/macros.rs
>  create mode 100644 rust/serde/private/de.rs
>  create mode 100644 rust/serde/private/doc.rs
>  create mode 100644 rust/serde/private/mod.rs
>  create mode 100644 rust/serde/private/ser.rs
>  create mode 100644 rust/serde/private/size_hint.rs
>  create mode 100644 rust/serde/ser/fmt.rs
>  create mode 100644 rust/serde/ser/impls.rs
>  create mode 100644 rust/serde/ser/impossible.rs
>  create mode 100644 rust/serde/ser/mod.rs
>  create mode 100644 rust/serde/std_error.rs
>  create mode 100644 rust/serde_cbor/de.rs
>  create mode 100644 rust/serde_cbor/error.rs
>  create mode 100644 rust/serde_cbor/lib.rs
>  create mode 100644 rust/serde_cbor/read.rs
>  create mode 100644 rust/serde_cbor/ser.rs
>  create mode 100644 rust/serde_cbor/tags.rs
>  create mode 100644 rust/serde_cbor/value/de.rs
>  create mode 100644 rust/serde_cbor/value/mod.rs
>  create mode 100644 rust/serde_cbor/value/ser.rs
>  create mode 100644 rust/serde_cbor/write.rs
>  create mode 100644 rust/serde_derive/bound.rs
>  create mode 100644 rust/serde_derive/de.rs
>  create mode 100644 rust/serde_derive/dummy.rs
>  create mode 100644 rust/serde_derive/fragment.rs
>  create mode 100644 rust/serde_derive/internals/ast.rs
>  create mode 100644 rust/serde_derive/internals/attr.rs
>  create mode 100644 rust/serde_derive/internals/case.rs
>  create mode 100644 rust/serde_derive/internals/check.rs
>  create mode 100644 rust/serde_derive/internals/ctxt.rs
>  create mode 100644 rust/serde_derive/internals/mod.rs
>  create mode 100644 rust/serde_derive/internals/receiver.rs
>  create mode 100644 rust/serde_derive/internals/respan.rs
>  create mode 100644 rust/serde_derive/internals/symbol.rs
>  create mode 100644 rust/serde_derive/lib.rs
>  create mode 100644 rust/serde_derive/pretend.rs
>  create mode 100644 rust/serde_derive/ser.rs
>  create mode 100644 rust/serde_derive/this.rs
>  create mode 100644 rust/serde_derive/try.rs
>  create mode 100644 rust/syn/attr.rs
>  create mode 100644 rust/syn/await.rs
>  create mode 100644 rust/syn/bigint.rs
>  create mode 100644 rust/syn/buffer.rs
>  create mode 100644 rust/syn/custom_keyword.rs
>  create mode 100644 rust/syn/custom_punctuation.rs
>  create mode 100644 rust/syn/data.rs
>  create mode 100644 rust/syn/derive.rs
>  create mode 100644 rust/syn/discouraged.rs
>  create mode 100644 rust/syn/error.rs
>  create mode 100644 rust/syn/export.rs
>  create mode 100644 rust/syn/expr.rs
>  create mode 100644 rust/syn/ext.rs
>  create mode 100644 rust/syn/file.rs
>  create mode 100644 rust/syn/gen/clone.rs
>  create mode 100644 rust/syn/gen/debug.rs
>  create mode 100644 rust/syn/gen/eq.rs
>  create mode 100644 rust/syn/gen/fold.rs
>  create mode 100644 rust/syn/gen/hash.rs
>  create mode 100644 rust/syn/gen/visit.rs
>  create mode 100644 rust/syn/gen/visit_mut.rs
>  create mode 100644 rust/syn/gen_helper.rs
>  create mode 100644 rust/syn/generics.rs
>  create mode 100644 rust/syn/group.rs
>  create mode 100644 rust/syn/ident.rs
>  create mode 100644 rust/syn/item.rs
>  create mode 100644 rust/syn/lib.rs
>  create mode 100644 rust/syn/lifetime.rs
>  create mode 100644 rust/syn/lit.rs
>  create mode 100644 rust/syn/lookahead.rs
>  create mode 100644 rust/syn/mac.rs
>  create mode 100644 rust/syn/macros.rs
>  create mode 100644 rust/syn/op.rs
>  create mode 100644 rust/syn/parse.rs
>  create mode 100644 rust/syn/parse_macro_input.rs
>  create mode 100644 rust/syn/parse_quote.rs
>  create mode 100644 rust/syn/pat.rs
>  create mode 100644 rust/syn/path.rs
>  create mode 100644 rust/syn/print.rs
>  create mode 100644 rust/syn/punctuated.rs
>  create mode 100644 rust/syn/reserved.rs
>  create mode 100644 rust/syn/sealed.rs
>  create mode 100644 rust/syn/span.rs
>  create mode 100644 rust/syn/spanned.rs
>  create mode 100644 rust/syn/stmt.rs
>  create mode 100644 rust/syn/thread.rs
>  create mode 100644 rust/syn/token.rs
>  create mode 100644 rust/syn/tt.rs
>  create mode 100644 rust/syn/ty.rs
>  create mode 100644 rust/syn/verbatim.rs
>  create mode 100644 rust/syn/whitespace.rs
>  create mode 100644 samples/rust/local_data_format/de.rs
>  create mode 100644 samples/rust/local_data_format/error.rs
>  create mode 100644 samples/rust/local_data_format/ser.rs
>  create mode 100644 samples/rust/puzzle.rs
>  create mode 100644 samples/rust/puzzle/error.rs
>  create mode 100644 samples/rust/puzzle/inode.rs
>  create mode 100644 samples/rust/puzzle/oci.rs
>  create mode 100644 samples/rust/puzzle/types.rs
>  create mode 100644 samples/rust/puzzle/types/cbor_helpers.rs
>  create mode 100644 samples/rust/puzzlefs.rs
>  create mode 100644 samples/rust/rust_fs.rs
>  create mode 100644 samples/rust/rust_serde.rs
>
> --
> 2.40.1
>

[-- Attachment #2: Type: text/html, Size: 61754 bytes --]

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Christian Brauner]

On Fri, Jun 09, 2023 at 11:22:12AM +0000, Ariel Miculas (amiculas) wrote:
> Hello Christian,
> 
> I didn't send these patches to a wider audience because this is an
> initial prototype of the PuzzleFS driver, and it has a few
> prerequisites before it could be even considered for merging. First of
> all, the rust filesystem abstractions and their dependencies need to
> be upstreamed, then there needs to be a discussion regarding the

Yes.

> inclusion of third-party crates in the linux kernel.

> 
> My plan was to send these patches to the rust-for-linux mailing list and then start a discussion with Miguel Ojeda regarding the upstreaming approach.
> There are a lot of new files added in this patch series because I've included all the dependencies required so that my patches could be applied to the rust-next branch, but these dependencies will most likely need to be upstreamed separately.
> 
> It was never my intention to avoid your reviews, should I also send
> subsequent patches to linux-fsdevel, even if they're in the early
> stages of development?

Yeah, I think that would be great.

Because the series you sent here touches on a lot of things in terms of
infrastructure alone. That work could very well be rather interesting
independent of PuzzleFS. We might just want to get enough infrastructure
to start porting a tiny existing fs (binderfs or something similar
small) to Rust to see how feasible this is and to wet our appetite for
bigger changes such as accepting a new filesystem driver completely
written in Rust.

But aside from the infrastructure discussion:

This is yet another filesystem for solving the container image problem
in the kernel with the addition of yet another filesystem. We just went
through this excercise with another filesystem. So I'd expect some
reluctance here. Tbh, the container world keeps sending us filesystems
at an alarming rate. That's two within a few months and that leaves a
rather disorganized impression.

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Ariel Miculas (amiculas)]

Adding Serge Hallyn for visibility.

Regards,
Ariel

From: Christian Brauner <brauner@kernel.org>
Sent: Friday, June 9, 2023 2:45 PM
To: Ariel Miculas (amiculas) <amiculas@cisco.com>
Cc: linux-fsdevel@vger.kernel.org <linux-fsdevel@vger.kernel.org>; rust-for-linux@vger.kernel.org <rust-for-linux@vger.kernel.org>; linux-mm@kvack.org <linux-mm@kvack.org>
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver 
 
On Fri, Jun 09, 2023 at 11:22:12AM +0000, Ariel Miculas (amiculas) wrote:
> Hello Christian,
> 
> I didn't send these patches to a wider audience because this is an
> initial prototype of the PuzzleFS driver, and it has a few
> prerequisites before it could be even considered for merging. First of
> all, the rust filesystem abstractions and their dependencies need to
> be upstreamed, then there needs to be a discussion regarding the

Yes.

> inclusion of third-party crates in the linux kernel.

> 
> My plan was to send these patches to the rust-for-linux mailing list and then start a discussion with Miguel Ojeda regarding the upstreaming approach.
> There are a lot of new files added in this patch series because I've included all the dependencies required so that my patches could be applied to the rust-next branch, but these dependencies will most likely need to be upstreamed separately.
> 
> It was never my intention to avoid your reviews, should I also send
> subsequent patches to linux-fsdevel, even if they're in the early
> stages of development?

Yeah, I think that would be great.

Because the series you sent here touches on a lot of things in terms of
infrastructure alone. That work could very well be rather interesting
independent of PuzzleFS. We might just want to get enough infrastructure
to start porting a tiny existing fs (binderfs or something similar
small) to Rust to see how feasible this is and to wet our appetite for
bigger changes such as accepting a new filesystem driver completely
written in Rust.

But aside from the infrastructure discussion:

This is yet another filesystem for solving the container image problem
in the kernel with the addition of yet another filesystem. We just went
through this excercise with another filesystem. So I'd expect some
reluctance here. Tbh, the container world keeps sending us filesystems
at an alarming rate. That's two within a few months and that leaves a
rather disorganized impression.

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Gao Xiang]

On 2023/6/9 20:03, Ariel Miculas (amiculas) wrote:

...

> 
> But aside from the infrastructure discussion:
> 
> This is yet another filesystem for solving the container image problem
> in the kernel with the addition of yet another filesystem. We just went
> through this excercise with another filesystem. So I'd expect some
> reluctance here. Tbh, the container world keeps sending us filesystems
> at an alarming rate. That's two within a few months and that leaves a
> rather disorganized impression.

Just a head up.  Since Rust kernel infrastructure is too premature,
it's impossible to handle page cache / iomap and many useful stuffs.
In the long term, at least (someday) after Rust infrastructure is
mature, I will implement EROFS ino Rust as a try as well.

As for chunk CDC, I don't see it's hard (since we already have some
CDC approach since Linux v6.1) but as an effective disk filesystem
for performance, EROFS on-disk data is all block-aligned to match
storage and page cache alignment.  If it's really needed, I could
update a more complete (but ineffective and slow) index version to
implement unaligned extents (both for decoded and encoded sides).
Yet I really think the main purpose of a kernel filesystem is to
make full use of kernel infrastructure for performance (like page
cache handling) otherwise a FUSE approach is enough.

Finally, as for OCI container image stuffs, I'd like to avoid
saying this topic anymore on the list (too tired about this).  I've
seen _three_ in-kernel approaches already before this one and I tend
to avoid listing the complete names (including FUSE alternatives)
here.  I really suggest if you guys could sit down and plan at least
a complete OCI standard for the next image format (even you don't
want to reuse any exist filesystem for whatever reasons).

Thanks,
Gao Xiang

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Miguel Ojeda]

On Fri, Jun 9, 2023 at 1:48 PM Christian Brauner <brauner@kernel.org> wrote:
>
> Because the series you sent here touches on a lot of things in terms of
> infrastructure alone. That work could very well be rather interesting
> independent of PuzzleFS. We might just want to get enough infrastructure
> to start porting a tiny existing fs (binderfs or something similar
> small) to Rust to see how feasible this is and to wet our appetite for
> bigger changes such as accepting a new filesystem driver completely
> written in Rust.

That would be great, thanks Christian! (Cc'ing Alice for binderfs -- I
think Rust Binder is keeping binderfs in C for the moment, but if you
are willing to try things, they are probably interested :)

Ariel: sorry, we crossed messages; I didn't receive your message at
[1], the rust-for-linux list probably dropped it due to the included
HTML.

[1] https://lore.kernel.org/linux-mm/CH0PR11MB529981313ED5A1F815350E41CD51A@CH0PR11MB5299.namprd11.prod.outlook.com/

Cheers,
Miguel

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Ariel Miculas (amiculas)]

Sorry about that, it seems like I need to switch to plain text mode for every reply in outlook, which is annoying.

Regards,
Ariel


From: Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>
Sent: Friday, June 9, 2023 3:07 PM
To: Christian Brauner <brauner@kernel.org>
Cc: Ariel Miculas (amiculas) <amiculas@cisco.com>; linux-fsdevel@vger.kernel.org <linux-fsdevel@vger.kernel.org>; rust-for-linux@vger.kernel.org <rust-for-linux@vger.kernel.org>; linux-mm@kvack.org <linux-mm@kvack.org>; Alice Ryhl <aliceryhl@google.com>
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver 
 
On Fri, Jun 9, 2023 at 1:48 PM Christian Brauner <brauner@kernel.org> wrote:
>
> Because the series you sent here touches on a lot of things in terms of
> infrastructure alone. That work could very well be rather interesting
> independent of PuzzleFS. We might just want to get enough infrastructure
> to start porting a tiny existing fs (binderfs or something similar
> small) to Rust to see how feasible this is and to wet our appetite for
> bigger changes such as accepting a new filesystem driver completely
> written in Rust.

That would be great, thanks Christian! (Cc'ing Alice for binderfs -- I
think Rust Binder is keeping binderfs in C for the moment, but if you
are willing to try things, they are probably interested :)

Ariel: sorry, we crossed messages; I didn't receive your message at
[1], the rust-for-linux list probably dropped it due to the included
HTML.

[1] https://lore.kernel.org/linux-mm/CH0PR11MB529981313ED5A1F815350E41CD51A@CH0PR11MB5299.namprd11.prod.outlook.com/

Cheers,
Miguel

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Greg KH]

On Fri, Jun 09, 2023 at 12:11:14PM +0000, Ariel Miculas (amiculas) wrote:
> Sorry about that, it seems like I need to switch to plain text mode for every reply in outlook, which is annoying.

You should also turn off the top-posting mode, as that's not good
etiquette on the kernel mailing lists for obvious reasons :)

thanks,

greg k-h

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Alice Ryhl]

Miguel Ojeda writes:
> That would be great, thanks Christian! (Cc'ing Alice for binderfs -- I
> think Rust Binder is keeping binderfs in C for the moment, but if you
> are willing to try things, they are probably interested :)

Yeah, Rust binder already needs bindings to a lot of other parts of the
kernel, so I decided to not rewrite the binderfs part for now to cut
down on the number of subsystems I would need to upstream bindings for.
Upstreaming bindings has proved to be a lot of work.

If someone else wants to upstream the filesystem bindings that binderfs
would need, then we can certainly look into using them in Rust binder.

Alice

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Colin Walters]

On Fri, Jun 9, 2023, at 7:45 AM, Christian Brauner wrote:
>
> Because the series you sent here touches on a lot of things in terms of
> infrastructure alone. That work could very well be rather interesting
> independent of PuzzleFS. We might just want to get enough infrastructure
> to start porting a tiny existing fs (binderfs or something similar
> small) to Rust to see how feasible this is and to wet our appetite for
> bigger changes such as accepting a new filesystem driver completely
> written in Rust.

(Not a kernel developer, but this argument makes sense to me)

> But aside from the infrastructure discussion:
>
> This is yet another filesystem for solving the container image problem
> in the kernel with the addition of yet another filesystem. We just went
> through this excercise with another filesystem. So I'd expect some
> reluctance here. Tbh, the container world keeps sending us filesystems
> at an alarming rate. That's two within a few months and that leaves a
> rather disorganized impression.

I am sure you are aware there's not some "container world" monoculture, there are many organizations, people and companies here with some healthy co-opetition but also some duplication inherent from that.

That said at a practical level, Ariel in the https://github.com/containers GH organization we're kind of a "big tent" place.  A subset of the organization is very heavily Rust oriented now (certainly the parts I touch) and briefly skimming the puzzlefs code, there are definitely some bits of code we could consider sharing in userspace.  Actually though since this isn't releated to the in-kernel discussion I'll file an issue on Github and we can discuss there.

But there is definitely a subset of the discussion that Christian is referring to here that is about the intersections/overlap with the composefs approach that is relevant for this list.  Maybe we could try to collaborate on an unbiased "puzzlefs vs composefs" document?  (What's in https://github.com/anuvu/puzzlefs/tree/master/doc is a bit sparse right now)

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Christian Brauner]

On Fri, Jun 09, 2023 at 08:20:30AM -0400, Colin Walters wrote:
> 
> 
> On Fri, Jun 9, 2023, at 7:45 AM, Christian Brauner wrote:
> >
> > Because the series you sent here touches on a lot of things in terms of
> > infrastructure alone. That work could very well be rather interesting
> > independent of PuzzleFS. We might just want to get enough infrastructure
> > to start porting a tiny existing fs (binderfs or something similar
> > small) to Rust to see how feasible this is and to wet our appetite for
> > bigger changes such as accepting a new filesystem driver completely
> > written in Rust.
> 
> (Not a kernel developer, but this argument makes sense to me)
> 
> > But aside from the infrastructure discussion:
> >
> > This is yet another filesystem for solving the container image problem
> > in the kernel with the addition of yet another filesystem. We just went
> > through this excercise with another filesystem. So I'd expect some
> > reluctance here. Tbh, the container world keeps sending us filesystems
> > at an alarming rate. That's two within a few months and that leaves a
> > rather disorganized impression.
> 
> I am sure you are aware there's not some "container world"
> monoculture, there are many organizations, people and companies here

That submission here explicitly references OCI v2. Composefs explicitly
advertises 100% compatibility with OCI. So, there's a set of OCI specs
including runtime and image. As far as I'm concerned you're all one
container world under the OCI umbrella.

We're not going to push multiple filesystems into the kernel that all do
slightly different things but all serve the OCI container world and use
some spec as an argument to move stuff into the kernel.

The OCI initiative is hailed as unifying the container ecosystem. Hence,
we can expect coordination.

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Serge Hallyn]

On Fri, Jun 09, 2023 at 02:42:47PM +0200, Christian Brauner wrote:
> On Fri, Jun 09, 2023 at 08:20:30AM -0400, Colin Walters wrote:
> > 
> > 
> > On Fri, Jun 9, 2023, at 7:45 AM, Christian Brauner wrote:
> > >
> > > Because the series you sent here touches on a lot of things in terms of
> > > infrastructure alone. That work could very well be rather interesting
> > > independent of PuzzleFS. We might just want to get enough infrastructure
> > > to start porting a tiny existing fs (binderfs or something similar
> > > small) to Rust to see how feasible this is and to wet our appetite for
> > > bigger changes such as accepting a new filesystem driver completely
> > > written in Rust.
> > 
> > (Not a kernel developer, but this argument makes sense to me)
> > 
> > > But aside from the infrastructure discussion:
> > >
> > > This is yet another filesystem for solving the container image problem
> > > in the kernel with the addition of yet another filesystem. We just went
> > > through this excercise with another filesystem. So I'd expect some
> > > reluctance here. Tbh, the container world keeps sending us filesystems
> > > at an alarming rate. That's two within a few months and that leaves a
> > > rather disorganized impression.
> > 
> > I am sure you are aware there's not some "container world"
> > monoculture, there are many organizations, people and companies here
> 
> That submission here explicitly references OCI v2. Composefs explicitly
> advertises 100% compatibility with OCI. So, there's a set of OCI specs

OCI v2 doesn't currently exist :)  There were many design goals for
it, and near as I can tell, after everyone discussed those together,
everyone went off to work on implementing the bits the needed - which
is a right and proper step before coming back and comparing notes
about what went well, etc.

The two main things where puzzlefs is experimenting are the use of
content defined chunking (CDC), and being written in rust (and,
especially, to have the same rust code base be used for the in kernel
driver, the fuse mounter, the builder, and the userspace extractor).
(Well, and reproducible images through a canonical representation,
but...)

It requires a POC in order to really determine whether the CDC will
be worth it, or will have pitfalls.  So far, it looks very promising.
Adding that functionality to composefs one day could be cool.

Likewise, it requires a user in order to push on the infrastructure
required to support a full filesystem in rust in the kernel.  But that
really isn't something we can "add to composefs".  :)

The main goal of this posting, then, was to show the infrastructure
pieces and work on that with the community.  We're definitely not
(currently :) asking for puzzlefs to be included.  However, it is
our (business and personal) justification for the rest of the work.

> including runtime and image. As far as I'm concerned you're all one
> container world under the OCI umbrella.

One big happy family!

> We're not going to push multiple filesystems into the kernel that all do
> slightly different things but all serve the OCI container world and use
> some spec as an argument to move stuff into the kernel.
> 
> The OCI initiative is hailed as unifying the container ecosystem. Hence,
> we can expect coordination.

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Ariel Miculas (amiculas)]

A "puzzlefs vs composefs" document sounds like a good idea. The documentation in puzzlefs is a little outdated and could be improved.
Feel free to create a github issue and tag me in there.

PS: as soon as I figure out how to turn off the top-posting mode, I'll do it.

Regards,
Ariel

________________________________________
From: Colin Walters <walters@verbum.org>
Sent: Friday, June 9, 2023 3:20 PM
To: Christian Brauner; Ariel Miculas (amiculas)
Cc: linux-fsdevel@vger.kernel.org; rust-for-linux@vger.kernel.org; linux-mm@kvack.org
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver



On Fri, Jun 9, 2023, at 7:45 AM, Christian Brauner wrote:
>
> Because the series you sent here touches on a lot of things in terms of
> infrastructure alone. That work could very well be rather interesting
> independent of PuzzleFS. We might just want to get enough infrastructure
> to start porting a tiny existing fs (binderfs or something similar
> small) to Rust to see how feasible this is and to wet our appetite for
> bigger changes such as accepting a new filesystem driver completely
> written in Rust.

(Not a kernel developer, but this argument makes sense to me)

> But aside from the infrastructure discussion:
>
> This is yet another filesystem for solving the container image problem
> in the kernel with the addition of yet another filesystem. We just went
> through this excercise with another filesystem. So I'd expect some
> reluctance here. Tbh, the container world keeps sending us filesystems
> at an alarming rate. That's two within a few months and that leaves a
> rather disorganized impression.

I am sure you are aware there's not some "container world" monoculture, there are many organizations, people and companies here with some healthy co-opetition but also some duplication inherent from that.

That said at a practical level, Ariel in the https://github.com/containers GH organization we're kind of a "big tent" place.  A subset of the organization is very heavily Rust oriented now (certainly the parts I touch) and briefly skimming the puzzlefs code, there are definitely some bits of code we could consider sharing in userspace.  Actually though since this isn't releated to the in-kernel discussion I'll file an issue on Github and we can discuss there.

But there is definitely a subset of the discussion that Christian is referring to here that is about the intersections/overlap with the composefs approach that is relevant for this list.  Maybe we could try to collaborate on an unbiased "puzzlefs vs composefs" document?  (What's in https://github.com/anuvu/puzzlefs/tree/master/doc is a bit sparse right now)

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Trilok Soni]

On 6/9/2023 6:45 AM, Ariel Miculas (amiculas) wrote:
> A "puzzlefs vs composefs" document sounds like a good idea. The documentation in puzzlefs is a little outdated and could be improved.
> Feel free to create a github issue and tag me in there.
> 
> PS: as soon as I figure out how to turn off the top-posting mode, I'll do it.
> 

Let me know as well if you could do w/ Outlook :). Switch to other email 
clients if possible.

---Trilok Soni

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Ariel Miculas (amiculas)]

I could switch to my personal gmail, but last time Miguel Ojeda asked me to use my cisco email when I send commits signed off by amiculas@cisco.com.
If this is not a hard requirement, then I could switch.

Regards,
Ariel

________________________________________
From: Trilok Soni <quic_tsoni@quicinc.com>
Sent: Friday, June 9, 2023 8:10 PM
To: Ariel Miculas (amiculas); Colin Walters; Christian Brauner
Cc: linux-fsdevel@vger.kernel.org; rust-for-linux@vger.kernel.org; linux-mm@kvack.org
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

On 6/9/2023 6:45 AM, Ariel Miculas (amiculas) wrote:
> A "puzzlefs vs composefs" document sounds like a good idea. The documentation in puzzlefs is a little outdated and could be improved.
> Feel free to create a github issue and tag me in there.
>
> PS: as soon as I figure out how to turn off the top-posting mode, I'll do it.
>

Let me know as well if you could do w/ Outlook :). Switch to other email
clients if possible.

---Trilok Soni

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Miguel Ojeda]

On Fri, Jun 9, 2023 at 7:25 PM Ariel Miculas (amiculas)
<amiculas@cisco.com> wrote:
>
> I could switch to my personal gmail, but last time Miguel Ojeda asked me to use my cisco email when I send commits signed off by amiculas@cisco.com.
> If this is not a hard requirement, then I could switch.

For patches, yeah, that is ideal, so that it matches the Git author / `From:`.

But for the other emails, you could use your personal address, if that
makes things easier.

Hope that helps!

Cheers,
Miguel

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[James Bottomley]

On Fri, 2023-06-09 at 19:41 +0200, Miguel Ojeda wrote:
> On Fri, Jun 9, 2023 at 7:25 PM Ariel Miculas (amiculas)
> <amiculas@cisco.com> wrote:
> > 
> > I could switch to my personal gmail, but last time Miguel Ojeda
> > asked me to use my cisco email when I send commits signed off by
> > amiculas@cisco.com. If this is not a hard requirement, then I could
> > switch.
> 
> For patches, yeah, that is ideal, so that it matches the Git author /
> `From:`.
> 
> But for the other emails, you could use your personal address, if
> that makes things easier.

It's still not a requirement, though.  You can send from your gmail
account and still have

From: Ariel Miculas <amiculas@cisco.com>

As the first line (separated from the commit message by a blank line),
which git am (or b4) will pick up as the author email.  This behaviour
is specifically for people who want the author to be their corporate
email address, but have failed to persuade corporate IT to make it
possible.

Regards,

James

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Miguel Ojeda]

On Fri, Jun 9, 2023 at 8:49 PM James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
>
> On Fri, 2023-06-09 at 19:41 +0200, Miguel Ojeda wrote:
> > For patches, yeah, that is ideal, so that it matches the Git author /
> > `From:`.
>
> It's still not a requirement, though.  You can send from your gmail
> account and still have

Yeah, that is what I said "ideal". When Ariel sent the first patch, he
didn't use the `From:` tag within the email body, but attributed it to
Cisco, and I had not seen any commit/email from his Cisco address yet,
so I asked him if he could use his corporate address.

Cheers,
Miguel

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Ariel Miculas]

Yes, but then how can you be sure that amiculas@cisco.com is the real
author of the commit? I think that's why Miguel Ojeda asked me to send
them from my business email, otherwise some random gmail account could
claim that he is "Ariel Miculas", so he's entitled to sign-off as
amiculas@cisco.com.

Regards,
Ariel

On Fri, Jun 9, 2023 at 10:03 PM James Bottomley
<James.Bottomley@hansenpartnership.com> wrote:
>
> On Fri, 2023-06-09 at 19:41 +0200, Miguel Ojeda wrote:
> > On Fri, Jun 9, 2023 at 7:25 PM Ariel Miculas (amiculas)
> > <amiculas@cisco.com> wrote:
> > >
> > > I could switch to my personal gmail, but last time Miguel Ojeda
> > > asked me to use my cisco email when I send commits signed off by
> > > amiculas@cisco.com. If this is not a hard requirement, then I could
> > > switch.
> >
> > For patches, yeah, that is ideal, so that it matches the Git author /
> > `From:`.
> >
> > But for the other emails, you could use your personal address, if
> > that makes things easier.
>
> It's still not a requirement, though.  You can send from your gmail
> account and still have
>
> From: Ariel Miculas <amiculas@cisco.com>
>
> As the first line (separated from the commit message by a blank line),
> which git am (or b4) will pick up as the author email.  This behaviour
> is specifically for people who want the author to be their corporate
> email address, but have failed to persuade corporate IT to make it
> possible.
>
> Regards,
>
> James
>

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[James Bottomley]

On Fri, 2023-06-09 at 22:11 +0300, Ariel Miculas wrote:
> Yes, but then how can you be sure that amiculas@cisco.com is the real
> author of the commit? I think that's why Miguel Ojeda asked me to
> send them from my business email, otherwise some random gmail account
> could claim that he is "Ariel Miculas", so he's entitled to sign-off
> as amiculas@cisco.com.

Because it's a public list and the real one would observe and repudiate
...

The DCO is an attestation which we basically believe absent any proof
to the contrary.  It's also exhausting and an incredible amount of
pointless admin not to believe anything until it's proved, which is why
we don't.

James

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Miguel Ojeda]

On Fri, Jun 9, 2023 at 9:11 PM Ariel Miculas <ariel.miculas@gmail.com> wrote:
>
> Yes, but then how can you be sure that amiculas@cisco.com is the real
> author of the commit? I think that's why Miguel Ojeda asked me to send
> them from my business email, otherwise some random gmail account could
> claim that he is "Ariel Miculas", so he's entitled to sign-off as
> amiculas@cisco.com.

Yeah, that was the main thing; and by asking that, if you managed to
send that original patch from the corporate one, then there was no
need for the `From:` to begin with :)

Cheers,
Miguel

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[James Bottomley]

On Fri, 2023-06-09 at 17:16 +0000, Ariel Miculas (amiculas) wrote:
> I could switch to my personal gmail, but last time Miguel Ojeda asked
> me to use my cisco email when I send commits signed off by
> amiculas@cisco.com.
> If this is not a hard requirement, then I could switch.

For sending patches, you can simply use git-send-email.  All you need
to point it at is the outgoing email server (which should be a config
setting in whatever tool you are using now).  We have a (reasonably) up
to date document with some recommendations:

https://www.kernel.org/doc/html/latest/process/email-clients.html

I've successfully used evolution with an exchange server for many
years, but the interface isn't to everyone's taste and Mozilla
Thunderbird is also known to connect to it.  Basic outlook has proven
impossible to configure correctly (which is why it doesn't have an
entry).

Regards,

James

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Ariel Miculas (amiculas)]

I did use git send-email for sending this patch series, but I cannot find any setting in the Outlook web client for disabling "top posting" when replying to emails:
https://answers.microsoft.com/en-us/outlook_com/forum/all/eliminate-top-posting/5e1e5729-30f8-41e9-84cb-fb5e81229c7c

Regards,
Ariel

________________________________________
From: James Bottomley <James.Bottomley@HansenPartnership.com>
Sent: Friday, June 9, 2023 9:43 PM
To: Ariel Miculas (amiculas); Trilok Soni; Colin Walters; Christian Brauner
Cc: linux-fsdevel@vger.kernel.org; rust-for-linux@vger.kernel.org; linux-mm@kvack.org
Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

On Fri, 2023-06-09 at 17:16 +0000, Ariel Miculas (amiculas) wrote:
> I could switch to my personal gmail, but last time Miguel Ojeda asked
> me to use my cisco email when I send commits signed off by
> amiculas@cisco.com.
> If this is not a hard requirement, then I could switch.

For sending patches, you can simply use git-send-email.  All you need
to point it at is the outgoing email server (which should be a config
setting in whatever tool you are using now).  We have a (reasonably) up
to date document with some recommendations:

https://www.kernel.org/doc/html/latest/process/email-clients.html

I've successfully used evolution with an exchange server for many
years, but the interface isn't to everyone's taste and Mozilla
Thunderbird is also known to connect to it.  Basic outlook has proven
impossible to configure correctly (which is why it doesn't have an
entry).

Regards,

James

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Ariel Miculas]

Now if I could figure out how to disable "top posting" in gmail...

Regards,
Ariel

On Fri, Jun 9, 2023 at 10:06 PM Ariel Miculas (amiculas)
<amiculas@cisco.com> wrote:
>
> I did use git send-email for sending this patch series, but I cannot find any setting in the Outlook web client for disabling "top posting" when replying to emails:
> https://answers.microsoft.com/en-us/outlook_com/forum/all/eliminate-top-posting/5e1e5729-30f8-41e9-84cb-fb5e81229c7c
>
> Regards,
> Ariel
>
> ________________________________________
> From: James Bottomley <James.Bottomley@HansenPartnership.com>
> Sent: Friday, June 9, 2023 9:43 PM
> To: Ariel Miculas (amiculas); Trilok Soni; Colin Walters; Christian Brauner
> Cc: linux-fsdevel@vger.kernel.org; rust-for-linux@vger.kernel.org; linux-mm@kvack.org
> Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver
>
> On Fri, 2023-06-09 at 17:16 +0000, Ariel Miculas (amiculas) wrote:
> > I could switch to my personal gmail, but last time Miguel Ojeda asked
> > me to use my cisco email when I send commits signed off by
> > amiculas@cisco.com.
> > If this is not a hard requirement, then I could switch.
>
> For sending patches, you can simply use git-send-email.  All you need
> to point it at is the outgoing email server (which should be a config
> setting in whatever tool you are using now).  We have a (reasonably) up
> to date document with some recommendations:
>
> https://www.kernel.org/doc/html/latest/process/email-clients.html
>
> I've successfully used evolution with an exchange server for many
> years, but the interface isn't to everyone's taste and Mozilla
> Thunderbird is also known to connect to it.  Basic outlook has proven
> impossible to configure correctly (which is why it doesn't have an
> entry).
>
> Regards,
>
> James
>

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Trilok Soni]

On 6/9/2023 12:20 PM, Ariel Miculas wrote:
> Now if I could figure out how to disable "top posting" in gmail...
> 
> Regards,
> Ariel
> 
> On Fri, Jun 9, 2023 at 10:06 PM Ariel Miculas (amiculas)
> <amiculas@cisco.com> wrote:
>>
>> I did use git send-email for sending this patch series, but I cannot find any setting in the Outlook web client for disabling "top posting" when replying to emails:
>> https://answers.microsoft.com/en-us/outlook_com/forum/all/eliminate-top-posting/5e1e5729-30f8-41e9-84cb-fb5e81229c7c
>>
>> Regards,
>> Ariel
>>
>> ________________________________________
>> From: James Bottomley <James.Bottomley@HansenPartnership.com>
>> Sent: Friday, June 9, 2023 9:43 PM
>> To: Ariel Miculas (amiculas); Trilok Soni; Colin Walters; Christian Brauner
>> Cc: linux-fsdevel@vger.kernel.org; rust-for-linux@vger.kernel.org; linux-mm@kvack.org
>> Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver
>>
>> On Fri, 2023-06-09 at 17:16 +0000, Ariel Miculas (amiculas) wrote:
>>> I could switch to my personal gmail, but last time Miguel Ojeda asked
>>> me to use my cisco email when I send commits signed off by
>>> amiculas@cisco.com.
>>> If this is not a hard requirement, then I could switch.
>>
>> For sending patches, you can simply use git-send-email.  All you need
>> to point it at is the outgoing email server (which should be a config
>> setting in whatever tool you are using now).  We have a (reasonably) up
>> to date document with some recommendations:
>>
>> https://www.kernel.org/doc/html/latest/process/email-clients.html
>>
>> I've successfully used evolution with an exchange server for many
>> years, but the interface isn't to everyone's taste and Mozilla
>> Thunderbird is also known to connect to it.  Basic outlook has proven
>> impossible to configure correctly (which is why it doesn't have an
>> entry).

One of the big reasons why I have quic_tsoni at quicinc dot com 
(Thunderbird/Mutt/whateveryoulike except Outlook) on top of tsoni at 
quicinc dot com (Outlook) to make sure that we comply w/ upstream
guidelines. Two email IDs are hard to manage but it gives
the good separation and freedom.

---Trilok Soni

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Alice Ryhl]

On 6/9/23 20:59, Ariel Miculas (amiculas) wrote:
> I did use git send-email for sending this patch series, but I cannot find any setting in the Outlook web client for disabling "top posting" when replying to emails:
> https://answers.microsoft.com/en-us/outlook_com/forum/all/eliminate-top-posting/5e1e5729-30f8-41e9-84cb-fb5e81229c7c
> 
> Regards,
> Ariel

You can also use git-send-email for sending replies. Just make a .txt 
file with this format:

Subject: <subject goes here>

<your reply after an empty line>

The lore.kernel.org site generates the appropriate git-send-email 
command for sending the reply, so you can copy-paste that to send it.

I send most of my replies like that.

> From: James Bottomley <James.Bottomley@HansenPartnership.com>
> Sent: Friday, June 9, 2023 9:43 PM
> To: Ariel Miculas (amiculas); Trilok Soni; Colin Walters; Christian Brauner
> Cc: linux-fsdevel@vger.kernel.org; rust-for-linux@vger.kernel.org; linux-mm@kvack.org
> Subject: Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver
> 
> On Fri, 2023-06-09 at 17:16 +0000, Ariel Miculas (amiculas) wrote:
>> I could switch to my personal gmail, but last time Miguel Ojeda asked
>> me to use my cisco email when I send commits signed off by
>> amiculas@cisco.com.
>> If this is not a hard requirement, then I could switch.
> 
> For sending patches, you can simply use git-send-email.  All you need
> to point it at is the outgoing email server (which should be a config
> setting in whatever tool you are using now).  We have a (reasonably) up
> to date document with some recommendations:
> 
> https://www.kernel.org/doc/html/latest/process/email-clients.html
> 
> I've successfully used evolution with an exchange server for many
> years, but the interface isn't to everyone's taste and Mozilla
> Thunderbird is also known to connect to it.  Basic outlook has proven
> impossible to configure correctly (which is why it doesn't have an
> entry).
> 
> Regards,
> 
> James
> 

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Miguel Ojeda]

On Fri, Jun 9, 2023 at 1:06 PM Christian Brauner <brauner@kernel.org> wrote:
>
> Any wrappers and code for core fs should be maintained as part of fs.
> Rust shouldn't become a way to avoid our reviews once you have a few
> wrappers added somewhere.

Definitely and, to be clear, we are strict about it (e.g.
https://rust-for-linux.com/contributing#the-rust-subsystem).

In fact, we appreciate maintainers that are willing to take patches
through their tree and take ownership of code too (e.g. KUnit and DRM
are doing so already).

I imagine Ariel sent the RFC as a way to announce his work early on,
especially given how some patches were split, the lack of commit
messages and tags, etc.

In other cases, we have reviewed patches privately first to iron out
this sort of thing, but I wasn't aware of this series coming (I knew
Ariel was working on puzzlefs and that he wanted to submit it
eventually, but not that an RFC was ready).

Also, some of the code here comes from the `rust` branch, but some of
it may not be ready yet for upstreaming.

Cheers,
Miguel

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Kent Overstreet]

On Fri, Jun 09, 2023 at 12:36:29PM +0200, Christian Brauner wrote:
> On Fri, Jun 09, 2023 at 09:29:58AM +0300, Ariel Miculas wrote:
> > Hi all!
> > 
> > This is a proof of concept driver written for the PuzzleFS
> 
> Uhm, the PuzzleFS filesystem isn't actually sent to fsdevel? Yet I see
> tons of patches in there that add wrappers to our core fs data
> structures. I even see a ton of new files that all fall clearly into
> fsdevel territory:
> 
> create mode 100644 rust/kernel/cred.rs
> create mode 100644 rust/kernel/delay.rs
> create mode 100644 rust/kernel/driver.rs
> create mode 100644 rust/kernel/file.rs
> create mode 100644 rust/kernel/fs.rs
> create mode 100644 rust/kernel/fs/param.rs
> create mode 100644 rust/kernel/io_buffer.rs
> create mode 100644 rust/kernel/iov_iter.rs
> create mode 100644 rust/kernel/mm.rs
> create mode 100644 rust/kernel/mount.rs
> create mode 100644 rust/kernel/pages.rs
> 
> There's also quite a lot of new mm/ in there, no?
> 
> Any wrappers and code for core fs should be maintained as part of fs.
> Rust shouldn't become a way to avoid our reviews once you have a few
> wrappers added somewhere.

I'm of the opinion that Rust wrappers should generally live in the same
directories as the code they're wrapping. The Rust wrappers are unsafe
code that is intimately coupled with the C code they're creating safe
interfaces for, and the organizational structure should reflect that.

This'll be particularly important when mixing Rust and C in the same
module, as I intend to do in bcachefs in the not-too-distant future.

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Miguel Ojeda]

On Sat, Jun 10, 2023 at 2:12 AM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> I'm of the opinion that Rust wrappers should generally live in the same
> directories as the code they're wrapping. The Rust wrappers are unsafe
> code that is intimately coupled with the C code they're creating safe
> interfaces for, and the organizational structure should reflect that.

+1, that is the plan, which also means `MAINTAINERS` does not need
extra `F` fields etc.

Cheers,
Miguel

Re: [RFC PATCH 00/80] Rust PuzzleFS filesystem driver

[Kent Overstreet]

On Fri, Jun 09, 2023 at 09:29:58AM +0300, Ariel Miculas wrote:
> Hi all!
> 
> This is a proof of concept driver written for the PuzzleFS
> next-generation container filesystem [1].

I can't find the link to the git tree that corresponds to the patch
series you just posted. Could you make sure that is front and center
when submitting large patch series? I find it a lot easier to get my
head around a large body of code by fetching into my local repository.