[PATCH] rust: macros: add `paste!` proc macro

[PATCH] rust: macros: add `paste!` proc macro

[Gary Guo]

This macro provides a flexible way to concatenated identifiers together
and it allows the resulting identifier to be used to declare new items,
which `concat_idents!` does not allow. It also allows identifiers to be
transformed before concatenated.

The `concat_idents!` example

    let x_1 = 42;
    let x_2 = concat_idents!(x, _1);
    assert!(x_1 == x_2);

can be written with `paste!` macro like this:

    let x_1 = 42;
    let x_2 = paste!([<x _1>]);
    assert!(x_1 == x_2);

However `paste!` macro is more flexible because it can be used to create
a new variable:

    let x_1 = 42;
    paste!(let [<x _2>] = [<x _1>];);
    assert!(x_1 == x_2);

While this is not possible with `concat_idents!`.

This macro is similar to the `paste!` crate [1], but this is a fresh
implementation to avoid vendoring large amount of code directly. Also, I
have augmented it to provide a way to specify span of the resulting
token, allowing precise control.

For example, this code is broken because the variable is declared inside
the macro, so Rust macro hygiene rules prevents access from the outside:

    macro_rules! m {
        ($id: ident) => {
            // The resulting token has hygiene of the macro.
            paste!(let [<$id>] = 1;)
        }
    }

    m!(a);
    let _ = a;

In this versionn of `paste!` macro I added a `span` modifier to allow
this:

    macro_rules! m {
        ($id: ident) => {
            // The resulting token has hygiene of `$id`.
            paste!(let [<$id:span>] = 1;)
        }
    }

    m!(a);
    let _ = a;

Link: http://docs.rs/paste/ [1]
Signed-off-by: Gary Guo <gary@garyguo.net>
---
 rust/macros/lib.rs   | 97 ++++++++++++++++++++++++++++++++++++++++++++
 rust/macros/paste.rs | 94 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 191 insertions(+)
 create mode 100644 rust/macros/paste.rs

diff --git a/rust/macros/lib.rs b/rust/macros/lib.rs
index 3fc74cb4ea19..b4bc44c27bd4 100644
--- a/rust/macros/lib.rs
+++ b/rust/macros/lib.rs
@@ -7,6 +7,7 @@
 mod concat_idents;
 mod helpers;
 mod module;
+mod paste;
 mod pin_data;
 mod pinned_drop;
 mod vtable;
@@ -246,3 +247,99 @@ pub fn pin_data(inner: TokenStream, item: TokenStream) -> TokenStream {
 pub fn pinned_drop(args: TokenStream, input: TokenStream) -> TokenStream {
     pinned_drop::pinned_drop(args, input)
 }
+
+/// Paste identifiers together.
+///
+/// Within the `paste!` macro, identifiers inside `[<` and `>]` are concatenated together to form a
+/// single identifier.
+///
+/// This is similar to the [`paste`] crate, but with pasting feature limited to identifiers
+/// (literals, lifetimes and documentation strings are not supported). There is a difference in
+/// supported modifiers as well.
+///
+/// # Example
+///
+/// ```ignore
+/// use kernel::macro::paste;
+///
+/// macro_rules! pub_no_prefix {
+///     ($prefix:ident, $($newname:ident),+) => {
+///         paste! {
+///             $(pub(crate) const $newname: u32 = [<$prefix $newname>];)+
+///         }
+///     };
+/// }
+///
+/// pub_no_prefix!(
+///     binder_driver_return_protocol_,
+///     BR_OK,
+///     BR_ERROR,
+///     BR_TRANSACTION,
+///     BR_REPLY,
+///     BR_DEAD_REPLY,
+///     BR_TRANSACTION_COMPLETE,
+///     BR_INCREFS,
+///     BR_ACQUIRE,
+///     BR_RELEASE,
+///     BR_DECREFS,
+///     BR_NOOP,
+///     BR_SPAWN_LOOPER,
+///     BR_DEAD_BINDER,
+///     BR_CLEAR_DEATH_NOTIFICATION_DONE,
+///     BR_FAILED_REPLY
+/// );
+///
+/// assert_eq!(BR_OK, binder_driver_return_protocol_BR_OK);
+/// ```
+///
+/// # Modifiers
+///
+/// For each identifier, it is possible to attach one or multiple modifiers to
+/// it.
+///
+/// Currently supported modifiers are:
+/// * `span`: change the span of concatenated identifier to the span of the specified token. By
+/// default the span of the `[< >]` group is used.
+/// * `lower`: change the identifier to lower case.
+/// * `upper`: change the identifier to upper case.
+///
+/// ```ignore
+/// use kernel::macro::paste;
+///
+/// macro_rules! pub_no_prefix {
+///     ($prefix:ident, $($newname:ident),+) => {
+///         kernel::macros::paste! {
+///             $(pub(crate) const fn [<$newname:lower:span>]: u32 = [<$prefix $newname:span>];)+
+///         }
+///     };
+/// }
+///
+/// pub_no_prefix!(
+///     binder_driver_return_protocol_,
+///     BR_OK,
+///     BR_ERROR,
+///     BR_TRANSACTION,
+///     BR_REPLY,
+///     BR_DEAD_REPLY,
+///     BR_TRANSACTION_COMPLETE,
+///     BR_INCREFS,
+///     BR_ACQUIRE,
+///     BR_RELEASE,
+///     BR_DECREFS,
+///     BR_NOOP,
+///     BR_SPAWN_LOOPER,
+///     BR_DEAD_BINDER,
+///     BR_CLEAR_DEATH_NOTIFICATION_DONE,
+///     BR_FAILED_REPLY
+/// );
+///
+/// assert_eq!(br_ok(), binder_driver_return_protocol_BR_OK);
+/// ```
+///
+/// [`paste`]: https://docs.rs/paste/
+#[proc_macro]
+pub fn paste(input: TokenStream) -> TokenStream {
+    let mut tokens = input.into_iter().collect();
+    paste::expand(&mut tokens);
+    tokens.into_iter().collect()
+}
diff --git a/rust/macros/paste.rs b/rust/macros/paste.rs
new file mode 100644
index 000000000000..42fde0930b05
--- /dev/null
+++ b/rust/macros/paste.rs
@@ -0,0 +1,94 @@
+use proc_macro::{Delimiter, Group, Ident, Spacing, Span, TokenTree};
+
+fn concat(tokens: &[TokenTree], group_span: Span) -> TokenTree {
+    let mut tokens = tokens.iter();
+    let mut segments = Vec::new();
+    let mut span = None;
+    loop {
+        match tokens.next() {
+            None => break,
+            Some(TokenTree::Literal(lit)) => segments.push((lit.to_string(), lit.span())),
+            Some(TokenTree::Ident(ident)) => {
+                let mut value = ident.to_string();
+                if value.starts_with("r#") {
+                    value.replace_range(0..2, "");
+                }
+                segments.push((value, ident.span()));
+            }
+            Some(TokenTree::Punct(p)) if p.as_char() == ':' => {
+                let Some(TokenTree::Ident(ident)) = tokens.next() else {
+                    panic!("expected identifier as modifier");
+                };
+
+                let (mut value, sp) = segments.pop().expect("expected identifier before modifier");
+                match ident.to_string().as_str() {
+                    // Set the overall span of concatenated token as current span
+                    "span" => {
+                        assert!(
+                            span.is_none(),
+                            "span modifier should only appear at most once"
+                        );
+                        span = Some(sp);
+                    }
+                    "lower" => value = value.to_lowercase(),
+                    "upper" => value = value.to_uppercase(),
+                    v => panic!("unknown modifier `{v}`"),
+                };
+                segments.push((value, sp));
+            }
+            _ => panic!("unexpected token in paste segments"),
+        };
+    }
+
+    let pasted: String = segments.into_iter().map(|x| x.0).collect();
+    TokenTree::Ident(Ident::new(&pasted, span.unwrap_or(group_span)))
+}
+
+pub(crate) fn expand(tokens: &mut Vec<TokenTree>) {
+    for token in tokens.iter_mut() {
+        if let TokenTree::Group(group) = token {
+            let delimiter = group.delimiter();
+            let span = group.span();
+            let mut stream: Vec<_> = group.stream().into_iter().collect();
+            // Find groups that looks like `[< A B C D >]`
+            if delimiter == Delimiter::Bracket
+                && stream.len() >= 3
+                && matches!(&stream[0], TokenTree::Punct(p) if p.as_char() == '<')
+                && matches!(&stream[stream.len() - 1], TokenTree::Punct(p) if p.as_char() == '>')
+            {
+                // Replace the group with concatenated token
+                *token = concat(&stream[1..stream.len() - 1], span);
+            } else {
+                // Recursively expand tokens inside the group
+                expand(&mut stream);
+                let mut group = Group::new(delimiter, stream.into_iter().collect());
+                group.set_span(span);
+                *token = TokenTree::Group(group);
+            }
+        }
+    }
+
+    // Path segments cannot contain invisible delimiter group, so remove them if any.
+    for i in (0..tokens.len().saturating_sub(3)).rev() {
+        // Looking for a double colon
+        if matches!(
+            (&tokens[i + 1], &tokens[i + 2]),
+            (TokenTree::Punct(a), TokenTree::Punct(b))
+                if a.as_char() == ':' && a.spacing() == Spacing::Joint && b.as_char() == ':'
+        ) {
+            match &tokens[i + 3] {
+                TokenTree::Group(group) if group.delimiter() == Delimiter::None => {
+                    tokens.splice(i + 3..i + 4, group.stream());
+                }
+                _ => (),
+            }
+
+            match &tokens[i] {
+                TokenTree::Group(group) if group.delimiter() == Delimiter::None => {
+                    tokens.splice(i..i + 1, group.stream());
+                }
+                _ => (),
+            }
+        }
+    }
+}
-- 
2.34.1

Re: [PATCH] rust: macros: add `paste!` proc macro

[Benno Lossin]

On 28.06.23 19:11, Gary Guo wrote:
> This macro provides a flexible way to concatenated identifiers together
> and it allows the resulting identifier to be used to declare new items,
> which `concat_idents!` does not allow. It also allows identifiers to be
> transformed before concatenated.
> 
> The `concat_idents!` example
> 
>      let x_1 = 42;
>      let x_2 = concat_idents!(x, _1);
>      assert!(x_1 == x_2);
> 
> can be written with `paste!` macro like this:
> 
>      let x_1 = 42;
>      let x_2 = paste!([<x _1>]);
>      assert!(x_1 == x_2);
> 
> However `paste!` macro is more flexible because it can be used to create
> a new variable:
> 
>      let x_1 = 42;
>      paste!(let [<x _2>] = [<x _1>];);
>      assert!(x_1 == x_2);
> 
> While this is not possible with `concat_idents!`.
> 
> This macro is similar to the `paste!` crate [1], but this is a fresh
> implementation to avoid vendoring large amount of code directly. Also, I
> have augmented it to provide a way to specify span of the resulting
> token, allowing precise control.
> 
> For example, this code is broken because the variable is declared inside
> the macro, so Rust macro hygiene rules prevents access from the outside:
> 
>      macro_rules! m {
>          ($id: ident) => {
>              // The resulting token has hygiene of the macro.
>              paste!(let [<$id>] = 1;)
>          }
>      }
> 
>      m!(a);
>      let _ = a;
> 
> In this versionn of `paste!` macro I added a `span` modifier to allow
> this:
> 
>      macro_rules! m {
>          ($id: ident) => {
>              // The resulting token has hygiene of `$id`.
>              paste!(let [<$id:span>] = 1;)
>          }
>      }
> 
>      m!(a);
>      let _ = a;
> 
> Link: http://docs.rs/paste/ [1]
> Signed-off-by: Gary Guo <gary@garyguo.net>

Reviewed-by: Benno Lossin <benno.lossin@proton.me>

> ---
>   rust/macros/lib.rs   | 97 ++++++++++++++++++++++++++++++++++++++++++++
>   rust/macros/paste.rs | 94 ++++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 191 insertions(+)
>   create mode 100644 rust/macros/paste.rs
> 
> diff --git a/rust/macros/lib.rs b/rust/macros/lib.rs
> index 3fc74cb4ea19..b4bc44c27bd4 100644
> --- a/rust/macros/lib.rs
> +++ b/rust/macros/lib.rs
> @@ -7,6 +7,7 @@
>   mod concat_idents;
>   mod helpers;
>   mod module;
> +mod paste;
>   mod pin_data;
>   mod pinned_drop;
>   mod vtable;
> @@ -246,3 +247,99 @@ pub fn pin_data(inner: TokenStream, item: TokenStream) -> TokenStream {
>   pub fn pinned_drop(args: TokenStream, input: TokenStream) -> TokenStream {
>       pinned_drop::pinned_drop(args, input)
>   }
> +
> +/// Paste identifiers together.
> +///
> +/// Within the `paste!` macro, identifiers inside `[<` and `>]` are concatenated together to form a
> +/// single identifier.
> +///
> +/// This is similar to the [`paste`] crate, but with pasting feature limited to identifiers
> +/// (literals, lifetimes and documentation strings are not supported). There is a difference in
> +/// supported modifiers as well.
> +///
> +/// # Example
> +///
> +/// ```ignore
> +/// use kernel::macro::paste;
> +///
> +/// macro_rules! pub_no_prefix {
> +///     ($prefix:ident, $($newname:ident),+) => {
> +///         paste! {
> +///             $(pub(crate) const $newname: u32 = [<$prefix $newname>];)+
> +///         }
> +///     };
> +/// }
> +///
> +/// pub_no_prefix!(
> +///     binder_driver_return_protocol_,
> +///     BR_OK,
> +///     BR_ERROR,
> +///     BR_TRANSACTION,
> +///     BR_REPLY,
> +///     BR_DEAD_REPLY,
> +///     BR_TRANSACTION_COMPLETE,
> +///     BR_INCREFS,
> +///     BR_ACQUIRE,
> +///     BR_RELEASE,
> +///     BR_DECREFS,
> +///     BR_NOOP,
> +///     BR_SPAWN_LOOPER,
> +///     BR_DEAD_BINDER,
> +///     BR_CLEAR_DEATH_NOTIFICATION_DONE,
> +///     BR_FAILED_REPLY
> +/// );
> +///
> +/// assert_eq!(BR_OK, binder_driver_return_protocol_BR_OK);
> +/// ```
> +///
> +/// # Modifiers
> +///
> +/// For each identifier, it is possible to attach one or multiple modifiers to
> +/// it.
> +///
> +/// Currently supported modifiers are:
> +/// * `span`: change the span of concatenated identifier to the span of the specified token. By
> +/// default the span of the `[< >]` group is used.
> +/// * `lower`: change the identifier to lower case.
> +/// * `upper`: change the identifier to upper case.
> +///
> +/// ```ignore
> +/// use kernel::macro::paste;
> +///
> +/// macro_rules! pub_no_prefix {
> +///     ($prefix:ident, $($newname:ident),+) => {
> +///         kernel::macros::paste! {
> +///             $(pub(crate) const fn [<$newname:lower:span>]: u32 = [<$prefix $newname:span>];)+
> +///         }
> +///     };
> +/// }
> +///
> +/// pub_no_prefix!(
> +///     binder_driver_return_protocol_,
> +///     BR_OK,
> +///     BR_ERROR,
> +///     BR_TRANSACTION,
> +///     BR_REPLY,
> +///     BR_DEAD_REPLY,
> +///     BR_TRANSACTION_COMPLETE,
> +///     BR_INCREFS,
> +///     BR_ACQUIRE,
> +///     BR_RELEASE,
> +///     BR_DECREFS,
> +///     BR_NOOP,
> +///     BR_SPAWN_LOOPER,
> +///     BR_DEAD_BINDER,
> +///     BR_CLEAR_DEATH_NOTIFICATION_DONE,
> +///     BR_FAILED_REPLY
> +/// );
> +///
> +/// assert_eq!(br_ok(), binder_driver_return_protocol_BR_OK);
> +/// ```
> +///
> +/// [`paste`]: https://docs.rs/paste/
> +#[proc_macro]
> +pub fn paste(input: TokenStream) -> TokenStream {
> +    let mut tokens = input.into_iter().collect();
> +    paste::expand(&mut tokens);
> +    tokens.into_iter().collect()
> +}
> diff --git a/rust/macros/paste.rs b/rust/macros/paste.rs
> new file mode 100644
> index 000000000000..42fde0930b05
> --- /dev/null
> +++ b/rust/macros/paste.rs
> @@ -0,0 +1,94 @@
> +use proc_macro::{Delimiter, Group, Ident, Spacing, Span, TokenTree};
> +
> +fn concat(tokens: &[TokenTree], group_span: Span) -> TokenTree {
> +    let mut tokens = tokens.iter();
> +    let mut segments = Vec::new();
> +    let mut span = None;
> +    loop {
> +        match tokens.next() {
> +            None => break,
> +            Some(TokenTree::Literal(lit)) => segments.push((lit.to_string(), lit.span())),
> +            Some(TokenTree::Ident(ident)) => {
> +                let mut value = ident.to_string();
> +                if value.starts_with("r#") {
> +                    value.replace_range(0..2, "");
> +                }
> +                segments.push((value, ident.span()));
> +            }
> +            Some(TokenTree::Punct(p)) if p.as_char() == ':' => {
> +                let Some(TokenTree::Ident(ident)) = tokens.next() else {
> +                    panic!("expected identifier as modifier");
> +                };
> +
> +                let (mut value, sp) = segments.pop().expect("expected identifier before modifier");
> +                match ident.to_string().as_str() {
> +                    // Set the overall span of concatenated token as current span
> +                    "span" => {
> +                        assert!(
> +                            span.is_none(),
> +                            "span modifier should only appear at most once"
> +                        );
> +                        span = Some(sp);
> +                    }
> +                    "lower" => value = value.to_lowercase(),
> +                    "upper" => value = value.to_uppercase(),
> +                    v => panic!("unknown modifier `{v}`"),
> +                };
> +                segments.push((value, sp));
> +            }
> +            _ => panic!("unexpected token in paste segments"),
> +        };
> +    }
> +
> +    let pasted: String = segments.into_iter().map(|x| x.0).collect();
> +    TokenTree::Ident(Ident::new(&pasted, span.unwrap_or(group_span)))
> +}
> +
> +pub(crate) fn expand(tokens: &mut Vec<TokenTree>) {
> +    for token in tokens.iter_mut() {
> +        if let TokenTree::Group(group) = token {
> +            let delimiter = group.delimiter();
> +            let span = group.span();
> +            let mut stream: Vec<_> = group.stream().into_iter().collect();
> +            // Find groups that looks like `[< A B C D >]`
> +            if delimiter == Delimiter::Bracket
> +                && stream.len() >= 3
> +                && matches!(&stream[0], TokenTree::Punct(p) if p.as_char() == '<')
> +                && matches!(&stream[stream.len() - 1], TokenTree::Punct(p) if p.as_char() == '>')
> +            {
> +                // Replace the group with concatenated token
> +                *token = concat(&stream[1..stream.len() - 1], span);
> +            } else {
> +                // Recursively expand tokens inside the group
> +                expand(&mut stream);
> +                let mut group = Group::new(delimiter, stream.into_iter().collect());
> +                group.set_span(span);
> +                *token = TokenTree::Group(group);
> +            }
> +        }
> +    }
> +
> +    // Path segments cannot contain invisible delimiter group, so remove them if any.
> +    for i in (0..tokens.len().saturating_sub(3)).rev() {
> +        // Looking for a double colon
> +        if matches!(
> +            (&tokens[i + 1], &tokens[i + 2]),
> +            (TokenTree::Punct(a), TokenTree::Punct(b))
> +                if a.as_char() == ':' && a.spacing() == Spacing::Joint && b.as_char() == ':'
> +        ) {
> +            match &tokens[i + 3] {
> +                TokenTree::Group(group) if group.delimiter() == Delimiter::None => {
> +                    tokens.splice(i + 3..i + 4, group.stream());
> +                }
> +                _ => (),
> +            }
> +
> +            match &tokens[i] {
> +                TokenTree::Group(group) if group.delimiter() == Delimiter::None => {
> +                    tokens.splice(i..i + 1, group.stream());
> +                }
> +                _ => (),
> +            }
> +        }
> +    }
> +}
> --
> 2.34.1
>

Re: [PATCH] rust: macros: add `paste!` proc macro

[Björn Roy Baron]

On Wednesday, June 28th, 2023 at 19:11, Gary Guo <gary@garyguo.net> wrote:

> This macro provides a flexible way to concatenated identifiers together
> and it allows the resulting identifier to be used to declare new items,
> which `concat_idents!` does not allow. It also allows identifiers to be
> transformed before concatenated.
> 
> The `concat_idents!` example
> 
>     let x_1 = 42;
>     let x_2 = concat_idents!(x, _1);
>     assert!(x_1 == x_2);
> 
> can be written with `paste!` macro like this:
> 
>     let x_1 = 42;
>     let x_2 = paste!([<x _1>]);
>     assert!(x_1 == x_2);
> 
> However `paste!` macro is more flexible because it can be used to create
> a new variable:
> 
>     let x_1 = 42;
>     paste!(let [<x _2>] = [<x _1>];);
>     assert!(x_1 == x_2);
> 
> While this is not possible with `concat_idents!`.
> 
> This macro is similar to the `paste!` crate [1], but this is a fresh
> implementation to avoid vendoring large amount of code directly. Also, I
> have augmented it to provide a way to specify span of the resulting
> token, allowing precise control.
> 
> For example, this code is broken because the variable is declared inside
> the macro, so Rust macro hygiene rules prevents access from the outside:
> 
>     macro_rules! m {
>         ($id: ident) => {
>             // The resulting token has hygiene of the macro.
>             paste!(let [<$id>] = 1;)
>         }
>     }
> 
>     m!(a);
>     let _ = a;
> 
> In this versionn of `paste!` macro I added a `span` modifier to allow

*version

> this:
> 
>     macro_rules! m {
>         ($id: ident) => {
>             // The resulting token has hygiene of `$id`.
>             paste!(let [<$id:span>] = 1;)
>         }
>     }
> 
>     m!(a);
>     let _ = a;
> 
> Link: http://docs.rs/paste/ [1]
> Signed-off-by: Gary Guo <gary@garyguo.net>

With the typo above fixed:

Reviewed-by: Björn Roy Baron <bjorn3_gh@protonmail.com>

I have also got a minor suggestion below, but I'm ok with keeping it as is.

> ---
>  rust/macros/lib.rs   | 97 ++++++++++++++++++++++++++++++++++++++++++++
>  rust/macros/paste.rs | 94 ++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 191 insertions(+)
>  create mode 100644 rust/macros/paste.rs
> 
> diff --git a/rust/macros/lib.rs b/rust/macros/lib.rs
> index 3fc74cb4ea19..b4bc44c27bd4 100644
> --- a/rust/macros/lib.rs
> +++ b/rust/macros/lib.rs
> @@ -7,6 +7,7 @@
>  mod concat_idents;
>  mod helpers;
>  mod module;
> +mod paste;
>  mod pin_data;
>  mod pinned_drop;
>  mod vtable;
> @@ -246,3 +247,99 @@ pub fn pin_data(inner: TokenStream, item: TokenStream) -> TokenStream {
>  pub fn pinned_drop(args: TokenStream, input: TokenStream) -> TokenStream {
>      pinned_drop::pinned_drop(args, input)
>  }
> +
> +/// Paste identifiers together.
> +///
> +/// Within the `paste!` macro, identifiers inside `[<` and `>]` are concatenated together to form a
> +/// single identifier.
> +///
> +/// This is similar to the [`paste`] crate, but with pasting feature limited to identifiers
> +/// (literals, lifetimes and documentation strings are not supported). There is a difference in
> +/// supported modifiers as well.
> +///
> +/// # Example
> +///
> +/// ```ignore
> +/// use kernel::macro::paste;
> +///
> +/// macro_rules! pub_no_prefix {
> +///     ($prefix:ident, $($newname:ident),+) => {
> +///         paste! {
> +///             $(pub(crate) const $newname: u32 = [<$prefix $newname>];)+
> +///         }
> +///     };
> +/// }
> +///
> +/// pub_no_prefix!(
> +///     binder_driver_return_protocol_,
> +///     BR_OK,
> +///     BR_ERROR,
> +///     BR_TRANSACTION,
> +///     BR_REPLY,
> +///     BR_DEAD_REPLY,
> +///     BR_TRANSACTION_COMPLETE,
> +///     BR_INCREFS,
> +///     BR_ACQUIRE,
> +///     BR_RELEASE,
> +///     BR_DECREFS,
> +///     BR_NOOP,
> +///     BR_SPAWN_LOOPER,
> +///     BR_DEAD_BINDER,
> +///     BR_CLEAR_DEATH_NOTIFICATION_DONE,
> +///     BR_FAILED_REPLY
> +/// );
> +///
> +/// assert_eq!(BR_OK, binder_driver_return_protocol_BR_OK);
> +/// ```
> +///
> +/// # Modifiers
> +///
> +/// For each identifier, it is possible to attach one or multiple modifiers to
> +/// it.
> +///
> +/// Currently supported modifiers are:
> +/// * `span`: change the span of concatenated identifier to the span of the specified token. By
> +/// default the span of the `[< >]` group is used.
> +/// * `lower`: change the identifier to lower case.
> +/// * `upper`: change the identifier to upper case.
> +///
> +/// ```ignore
> +/// use kernel::macro::paste;
> +///
> +/// macro_rules! pub_no_prefix {
> +///     ($prefix:ident, $($newname:ident),+) => {
> +///         kernel::macros::paste! {
> +///             $(pub(crate) const fn [<$newname:lower:span>]: u32 = [<$prefix $newname:span>];)+

Having multiple : when using multiple flags feels a bit weird to me. Maybe
use $newname:lower+span, $newname:lower,span or something like that? If you
prefer the current syntax that is fine with me too though.

> +///         }
> +///     };
> +/// }
> +///
> +/// pub_no_prefix!(
> +///     binder_driver_return_protocol_,
> +///     BR_OK,
> +///     BR_ERROR,
> +///     BR_TRANSACTION,
> +///     BR_REPLY,
> +///     BR_DEAD_REPLY,
> +///     BR_TRANSACTION_COMPLETE,
> +///     BR_INCREFS,
> +///     BR_ACQUIRE,
> +///     BR_RELEASE,
> +///     BR_DECREFS,
> +///     BR_NOOP,
> +///     BR_SPAWN_LOOPER,
> +///     BR_DEAD_BINDER,
> +///     BR_CLEAR_DEATH_NOTIFICATION_DONE,
> +///     BR_FAILED_REPLY
> +/// );
> +///
> +/// assert_eq!(br_ok(), binder_driver_return_protocol_BR_OK);
> +/// ```
> +///
> +/// [`paste`]: https://docs.rs/paste/
> +#[proc_macro]
> +pub fn paste(input: TokenStream) -> TokenStream {
> +    let mut tokens = input.into_iter().collect();
> +    paste::expand(&mut tokens);
> +    tokens.into_iter().collect()
> +}
> diff --git a/rust/macros/paste.rs b/rust/macros/paste.rs
> new file mode 100644
> index 000000000000..42fde0930b05
> --- /dev/null
> +++ b/rust/macros/paste.rs
> @@ -0,0 +1,94 @@
> +use proc_macro::{Delimiter, Group, Ident, Spacing, Span, TokenTree};
> +
> +fn concat(tokens: &[TokenTree], group_span: Span) -> TokenTree {
> +    let mut tokens = tokens.iter();
> +    let mut segments = Vec::new();
> +    let mut span = None;
> +    loop {
> +        match tokens.next() {
> +            None => break,
> +            Some(TokenTree::Literal(lit)) => segments.push((lit.to_string(), lit.span())),
> +            Some(TokenTree::Ident(ident)) => {
> +                let mut value = ident.to_string();
> +                if value.starts_with("r#") {
> +                    value.replace_range(0..2, "");
> +                }
> +                segments.push((value, ident.span()));
> +            }
> +            Some(TokenTree::Punct(p)) if p.as_char() == ':' => {
> +                let Some(TokenTree::Ident(ident)) = tokens.next() else {
> +                    panic!("expected identifier as modifier");
> +                };
> +
> +                let (mut value, sp) = segments.pop().expect("expected identifier before modifier");
> +                match ident.to_string().as_str() {
> +                    // Set the overall span of concatenated token as current span
> +                    "span" => {
> +                        assert!(
> +                            span.is_none(),
> +                            "span modifier should only appear at most once"
> +                        );
> +                        span = Some(sp);
> +                    }
> +                    "lower" => value = value.to_lowercase(),
> +                    "upper" => value = value.to_uppercase(),
> +                    v => panic!("unknown modifier `{v}`"),
> +                };
> +                segments.push((value, sp));
> +            }
> +            _ => panic!("unexpected token in paste segments"),
> +        };
> +    }
> +
> +    let pasted: String = segments.into_iter().map(|x| x.0).collect();
> +    TokenTree::Ident(Ident::new(&pasted, span.unwrap_or(group_span)))
> +}
> +
> +pub(crate) fn expand(tokens: &mut Vec<TokenTree>) {
> +    for token in tokens.iter_mut() {
> +        if let TokenTree::Group(group) = token {
> +            let delimiter = group.delimiter();
> +            let span = group.span();
> +            let mut stream: Vec<_> = group.stream().into_iter().collect();
> +            // Find groups that looks like `[< A B C D >]`
> +            if delimiter == Delimiter::Bracket
> +                && stream.len() >= 3
> +                && matches!(&stream[0], TokenTree::Punct(p) if p.as_char() == '<')
> +                && matches!(&stream[stream.len() - 1], TokenTree::Punct(p) if p.as_char() == '>')
> +            {
> +                // Replace the group with concatenated token
> +                *token = concat(&stream[1..stream.len() - 1], span);
> +            } else {
> +                // Recursively expand tokens inside the group
> +                expand(&mut stream);
> +                let mut group = Group::new(delimiter, stream.into_iter().collect());
> +                group.set_span(span);
> +                *token = TokenTree::Group(group);
> +            }
> +        }
> +    }
> +
> +    // Path segments cannot contain invisible delimiter group, so remove them if any.
> +    for i in (0..tokens.len().saturating_sub(3)).rev() {
> +        // Looking for a double colon
> +        if matches!(
> +            (&tokens[i + 1], &tokens[i + 2]),
> +            (TokenTree::Punct(a), TokenTree::Punct(b))
> +                if a.as_char() == ':' && a.spacing() == Spacing::Joint && b.as_char() == ':'
> +        ) {
> +            match &tokens[i + 3] {
> +                TokenTree::Group(group) if group.delimiter() == Delimiter::None => {
> +                    tokens.splice(i + 3..i + 4, group.stream());
> +                }
> +                _ => (),
> +            }
> +
> +            match &tokens[i] {
> +                TokenTree::Group(group) if group.delimiter() == Delimiter::None => {
> +                    tokens.splice(i..i + 1, group.stream());
> +                }
> +                _ => (),
> +            }
> +        }
> +    }
> +}
> -- 
> 2.34.1

Re: [PATCH] rust: macros: add `paste!` proc macro

[Martin Rodriguez Reboredo]

On 6/28/23 14:11, Gary Guo wrote:
> This macro provides a flexible way to concatenated identifiers together
> and it allows the resulting identifier to be used to declare new items,
> which `concat_idents!` does not allow. It also allows identifiers to be
> transformed before concatenated.
> 
> The `concat_idents!` example
> 
>      let x_1 = 42;
>      let x_2 = concat_idents!(x, _1);
>      assert!(x_1 == x_2);
> 
> can be written with `paste!` macro like this:
> 
>      let x_1 = 42;
>      let x_2 = paste!([<x _1>]);
>      assert!(x_1 == x_2);
> 
> However `paste!` macro is more flexible because it can be used to create
> a new variable:
> 
>      let x_1 = 42;
>      paste!(let [<x _2>] = [<x _1>];);
>      assert!(x_1 == x_2);
> 
> While this is not possible with `concat_idents!`.
> 
> This macro is similar to the `paste!` crate [1], but this is a fresh
> implementation to avoid vendoring large amount of code directly. Also, I
> have augmented it to provide a way to specify span of the resulting
> token, allowing precise control.
> 
> For example, this code is broken because the variable is declared inside
> the macro, so Rust macro hygiene rules prevents access from the outside:
> 
>      macro_rules! m {
>          ($id: ident) => {
>              // The resulting token has hygiene of the macro.
>              paste!(let [<$id>] = 1;)
>          }
>      }
> 
>      m!(a);
>      let _ = a;
> 
> In this versionn of `paste!` macro I added a `span` modifier to allow
> this:
> 
>      macro_rules! m {
>          ($id: ident) => {
>              // The resulting token has hygiene of `$id`.
>              paste!(let [<$id:span>] = 1;)
>          }
>      }
> 
>      m!(a);
>      let _ = a;
> 
> Link: http://docs.rs/paste/ [1]
> Signed-off-by: Gary Guo <gary@garyguo.net>
> ---
> [...]

Reviewed-by: Martin Rodriguez Reboredo <yakoyoku@gmail.com>

Re: [PATCH] rust: macros: add `paste!` proc macro

[Alice Ryhl]

Gary Guo <gary@garyguo.net> writes:
> This macro provides a flexible way to concatenated identifiers together
> and it allows the resulting identifier to be used to declare new items,
> which `concat_idents!` does not allow. It also allows identifiers to be
> transformed before concatenated.
> 
> The `concat_idents!` example
> 
>     let x_1 = 42;
>     let x_2 = concat_idents!(x, _1);
>     assert!(x_1 == x_2);
> 
> can be written with `paste!` macro like this:
> 
>     let x_1 = 42;
>     let x_2 = paste!([<x _1>]);
>     assert!(x_1 == x_2);
> 
> However `paste!` macro is more flexible because it can be used to create
> a new variable:
> 
>     let x_1 = 42;
>     paste!(let [<x _2>] = [<x _1>];);
>     assert!(x_1 == x_2);
> 
> While this is not possible with `concat_idents!`.
> 
> This macro is similar to the `paste!` crate [1], but this is a fresh
> implementation to avoid vendoring large amount of code directly. Also, I
> have augmented it to provide a way to specify span of the resulting
> token, allowing precise control.
> 
> For example, this code is broken because the variable is declared inside
> the macro, so Rust macro hygiene rules prevents access from the outside:
> 
>     macro_rules! m {
>         ($id: ident) => {
>             // The resulting token has hygiene of the macro.
>             paste!(let [<$id>] = 1;)
>         }
>     }
> 
>     m!(a);
>     let _ = a;
> 
> In this versionn of `paste!` macro I added a `span` modifier to allow
> this:
> 
>     macro_rules! m {
>         ($id: ident) => {
>             // The resulting token has hygiene of `$id`.
>             paste!(let [<$id:span>] = 1;)
>         }
>     }
> 
>     m!(a);
>     let _ = a;
> 
> Link: http://docs.rs/paste/ [1]
> Signed-off-by: Gary Guo <gary@garyguo.net>

Reviewed-by: Alice Ryhl <aliceryhl@google.com>

Re: [PATCH] rust: macros: add `paste!` proc macro

[Miguel Ojeda]

On Wed, Jun 28, 2023 at 7:12 PM Gary Guo <gary@garyguo.net> wrote:
>
> diff --git a/rust/macros/paste.rs b/rust/macros/paste.rs
> new file mode 100644
> index 000000000000..42fde0930b05
> --- /dev/null
> +++ b/rust/macros/paste.rs
> @@ -0,0 +1,94 @@
> +use proc_macro::{Delimiter, Group, Ident, Spacing, Span, TokenTree};

We need the license identifier to apply this -- is it our usual one?

    // SPDX-License-Identifier: GPL-2.0

If so, for the record, could you please confirm it is? Then I will add
it on my side and apply it -- thanks!

Cheers,
Miguel

Re: [PATCH] rust: macros: add `paste!` proc macro

[Gary Guo]

On Wed, 9 Aug 2023 23:45:34 +0200
Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> wrote:

> On Wed, Jun 28, 2023 at 7:12 PM Gary Guo <gary@garyguo.net> wrote:
> >
> > diff --git a/rust/macros/paste.rs b/rust/macros/paste.rs
> > new file mode 100644
> > index 000000000000..42fde0930b05
> > --- /dev/null
> > +++ b/rust/macros/paste.rs
> > @@ -0,0 +1,94 @@
> > +use proc_macro::{Delimiter, Group, Ident, Spacing, Span, TokenTree};  
> 
> We need the license identifier to apply this -- is it our usual one?
> 
>     // SPDX-License-Identifier: GPL-2.0
> 
> If so, for the record, could you please confirm it is? Then I will add
> it on my side and apply it -- thanks!
> 
> Cheers,
> Miguel

Hi Miguel,

Sorry I forgot to add license comments!

All my kernel contributions are permissively licensed if possible, so I
am fine with this being either MIT or GPL-2.0 (I think GPL-2.0 is
deprecated in as a SPDX license identifier and it should be
GPL-2.0-only going forward, though).

Given this is non-kernel specific generic code, I think it might worth
following the convention of the paste and pin-init code and make it
`Apache-2.0 OR MIT`? This would also make it the same license as the
`paste` crate (although we don't have to keep the same license as this
is a different implementation).

I'll leave the final decision to you.

Best,
Gary

Re: [PATCH] rust: macros: add `paste!` proc macro

[Miguel Ojeda]

On Thu, Aug 10, 2023 at 12:02 AM Gary Guo <gary@garyguo.net> wrote:
>
> Sorry I forgot to add license comments!

No problem at all :)

> All my kernel contributions are permissively licensed if possible, so I
> am fine with this being either MIT or GPL-2.0 (I think GPL-2.0 is
> deprecated in as a SPDX license identifier and it should be
> GPL-2.0-only going forward, though).

Yeah, the 3.0 version of the SPDX license list deprecated `GPL-2.0`,
but the kernel still allows `GPL-2.0` and lists it first. I recall
thinking about this before the initial merge, and I think I went with
the original form because the main `COPYING` file still uses that.
After that I am just keeping it consistent, though I am not sure when
the kernel will migrate.

> Given this is non-kernel specific generic code, I think it might worth
> following the convention of the paste and pin-init code and make it
> `Apache-2.0 OR MIT`? This would also make it the same license as the
> `paste` crate (although we don't have to keep the same license as this
> is a different implementation).
>
> I'll leave the final decision to you.

Since you prefer it and it makes sense that someone may want to use it
(`concat` and `expand`) elsewhere, let's go with that. I will add:

    // SPDX-License-Identifier: Apache-2.0 OR MIT

then. Thanks for the very quick reply!

Cheers,
Miguel

Re: [PATCH] rust: macros: add `paste!` proc macro

[Miguel Ojeda]

On Wed, Jun 28, 2023 at 7:12 PM Gary Guo <gary@garyguo.net> wrote:
>
> This macro provides a flexible way to concatenated identifiers together
> and it allows the resulting identifier to be used to declare new items,
> which `concat_idents!` does not allow. It also allows identifiers to be
> transformed before concatenated.
>
> The `concat_idents!` example
>
>     let x_1 = 42;
>     let x_2 = concat_idents!(x, _1);
>     assert!(x_1 == x_2);
>
> can be written with `paste!` macro like this:
>
>     let x_1 = 42;
>     let x_2 = paste!([<x _1>]);
>     assert!(x_1 == x_2);
>
> However `paste!` macro is more flexible because it can be used to create
> a new variable:
>
>     let x_1 = 42;
>     paste!(let [<x _2>] = [<x _1>];);
>     assert!(x_1 == x_2);
>
> While this is not possible with `concat_idents!`.
>
> This macro is similar to the `paste!` crate [1], but this is a fresh
> implementation to avoid vendoring large amount of code directly. Also, I
> have augmented it to provide a way to specify span of the resulting
> token, allowing precise control.
>
> For example, this code is broken because the variable is declared inside
> the macro, so Rust macro hygiene rules prevents access from the outside:
>
>     macro_rules! m {
>         ($id: ident) => {
>             // The resulting token has hygiene of the macro.
>             paste!(let [<$id>] = 1;)
>         }
>     }
>
>     m!(a);
>     let _ = a;
>
> In this versionn of `paste!` macro I added a `span` modifier to allow
> this:
>
>     macro_rules! m {
>         ($id: ident) => {
>             // The resulting token has hygiene of `$id`.
>             paste!(let [<$id:span>] = 1;)
>         }
>     }
>
>     m!(a);
>     let _ = a;
>
> Link: http://docs.rs/paste/ [1]
> Signed-off-by: Gary Guo <gary@garyguo.net>

Applied to `rust-next` with the SPDX license identifier added as
discussed and the typo fixed -- thanks everyone!

Cheers,
Miguel

Re: [PATCH] rust: macros: add `paste!` proc macro

[Greg KH]

On Thu, Aug 10, 2023 at 12:29:06AM +0200, Miguel Ojeda wrote:
> > All my kernel contributions are permissively licensed if possible, so I
> > am fine with this being either MIT or GPL-2.0 (I think GPL-2.0 is
> > deprecated in as a SPDX license identifier and it should be
> > GPL-2.0-only going forward, though).
> 
> Yeah, the 3.0 version of the SPDX license list deprecated `GPL-2.0`,
> but the kernel still allows `GPL-2.0` and lists it first. I recall
> thinking about this before the initial merge, and I think I went with
> the original form because the main `COPYING` file still uses that.
> After that I am just keeping it consistent, though I am not sure when
> the kernel will migrate.

The kernel will migrate when we have converted all files in the tree to
SPDX and can worry about things like the SPDX version level.  We have a
ways to go still...

> > Given this is non-kernel specific generic code, I think it might worth
> > following the convention of the paste and pin-init code and make it
> > `Apache-2.0 OR MIT`? This would also make it the same license as the
> > `paste` crate (although we don't have to keep the same license as this
> > is a different implementation).
> >
> > I'll leave the final decision to you.
> 
> Since you prefer it and it makes sense that someone may want to use it
> (`concat` and `expand`) elsewhere, let's go with that. I will add:
> 
>     // SPDX-License-Identifier: Apache-2.0 OR MIT

Be VERY careful with dual licenses please, and especially non-GPL ones
in the kernel tree.  It gets tricky very very quickly and you need to
know what you are doing.  So much so that I really want to see a lawyer
sign off on such a thing so that everyone involved understands the
issues that this requires.

Otherwise please, just default to GPL-2.0 for kernel code, unless you
have real reasons why it can't be that way, as remember, the overall
license of the codebase is that.

thanks,

greg k-h

Re: [PATCH] rust: macros: add `paste!` proc macro

[Miguel Ojeda]

On Thu, Aug 10, 2023 at 7:08 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> The kernel will migrate when we have converted all files in the tree to
> SPDX and can worry about things like the SPDX version level.  We have a
> ways to go still...

I see, thanks!

> Be VERY careful with dual licenses please, and especially non-GPL ones
> in the kernel tree.  It gets tricky very very quickly and you need to
> know what you are doing.  So much so that I really want to see a lawyer
> sign off on such a thing so that everyone involved understands the
> issues that this requires.

It is the common one used in Rust projects, which we are using for
other bits too, e.g. vendoring the `alloc` standard library.

Since these couple functions are essentially a compiler plugin (a proc
macro) that is useful in userspace and other contexts too, Gary wanted
to use that license (he contributes the other kernel code under
GPL-2.0). For instance, he may possibly want to put those functions in
crates.io or similar, I imagine (like the linked crate this replaces
as a simplification).

He is also OK with GPL-2.0, so we can just do that here, of course.
But I am mentioning the above because, if this one is problematic,
then perhaps we should revisit again `rust/alloc`, our `std_vendor.rs`
files and the `pinned-init` library (which all use the same dual
license).

> Otherwise please, just default to GPL-2.0 for kernel code, unless you
> have real reasons why it can't be that way, as remember, the overall
> license of the codebase is that.

That is our default, definitely. I OK'd these two functions for the
reasons above only.

Thanks for keeping an eye on our list, by the way.

Cheers,
Miguel

Re: [PATCH] rust: macros: add `paste!` proc macro

[Greg KH]

On Thu, Aug 10, 2023 at 09:56:18AM +0200, Miguel Ojeda wrote:
> On Thu, Aug 10, 2023 at 7:08 AM Greg KH <gregkh@linuxfoundation.org> wrote:
> >
> > The kernel will migrate when we have converted all files in the tree to
> > SPDX and can worry about things like the SPDX version level.  We have a
> > ways to go still...
> 
> I see, thanks!
> 
> > Be VERY careful with dual licenses please, and especially non-GPL ones
> > in the kernel tree.  It gets tricky very very quickly and you need to
> > know what you are doing.  So much so that I really want to see a lawyer
> > sign off on such a thing so that everyone involved understands the
> > issues that this requires.
> 
> It is the common one used in Rust projects, which we are using for
> other bits too, e.g. vendoring the `alloc` standard library.
> 
> Since these couple functions are essentially a compiler plugin (a proc
> macro) that is useful in userspace and other contexts too, Gary wanted
> to use that license (he contributes the other kernel code under
> GPL-2.0). For instance, he may possibly want to put those functions in
> crates.io or similar, I imagine (like the linked crate this replaces
> as a simplification).

If he, as the copyright owner, wants to take the code and do anything
else with it, under any other license, they can.  There's nothing
preventing them from doing that, a dual license is not needed (as long
as you don't take any changes that anyone else made under a different
license.)

Which is one of the main reasons dual license isn't really needed, if
the author wants the code to go somewhere else also, they are free to do
so as they own the copyright.

So please think carefully about mixing licenses like this, it's almost
never needed, and keeping the files with multiple licenses is a major
pain to handle over time.

good luck!

greg k-h

Re: [PATCH] rust: macros: add `paste!` proc macro

[Miguel Ojeda]

On Thu, Aug 10, 2023 at 5:46 PM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> If he, as the copyright owner, wants to take the code and do anything
> else with it, under any other license, they can.  There's nothing
> preventing them from doing that, a dual license is not needed (as long
> as you don't take any changes that anyone else made under a different
> license.)

Yeah, definitely. I imagine some developers have their reasons, e.g.
they may prefer to have the same one everywhere so that people reading
the code anywhere know it is more permissively licensed elsewhere (and
that dual license is popular for Rust projects).

> Which is one of the main reasons dual license isn't really needed, if
> the author wants the code to go somewhere else also, they are free to do
> so as they own the copyright.
>
> So please think carefully about mixing licenses like this, it's almost
> never needed, and keeping the files with multiple licenses is a major
> pain to handle over time.
>
> good luck!

Let's keep things simple then. I will replace it with `GPL-2.0` since
Gary was OK with that one too.

Thanks!

Cheers,
Miguel