Xtables2 Netlink spec

Xtables2 Netlink spec

[Jan Engelhardt]

By request of Pablo, I am posting the Xtables2 Netlink interface 
specification for review. Additionally, further documentation and 
toolchain around it is available through the temporary project page at

	http://jengelh.medozas.de/projects/xtables/

which currently includes

 * User Documentation Chapter 1: Architectural Differences

 * Developer Documentation Part 1: Netlink interface (WIP)
   This is copied below to facilitate inline replies

 * Runnable Linux source tree

 * Runnable userspace library (libnetfilter_xtables)
   with small test-and-debug program

--8<--

Netlink interface

1 General use

1.1 Socket

Xtables2 is usable through a Netlink socket of type 
NETLINK_XTABLES. No intermediate subsystem like nfnetlink is 
used, because the kernel's nfnetlink parser does not make all 
attributes available to (in-kernel) nfnetlink users.

#include <sys/socket.h>
#include <linux/netlink.h>
#define NETFILTER_XTABLES 21

nfxt_socket = socket(AF_NETLINK, SOCK_RAW, NETFILTER_XTABLES);

The NETLINK_XTABLES constant is defined in linux/netlink.h with 
the value 21.

1.2 Message format

All messages transmitted over the Netlink socket are to have the 
base struct nlmsghdr header, followed by a version tag to allow 
for the flexibility of data following it:

struct xtnetlink_genhdr {
        uint32_t version;
};

The version member is always 0 in the current implementation.

Following the genhdr can be any number of standard Netlink 
attributes (struct nlattr plus their payload).

Often, a logical tree structure is used to describe something, 
such as for example tables of chains of rules:

filter
 \__ INPUT
 |    \__ some rule
 \__ FORWARD
 |    \__ rule2
 |    \__ rule3
 |    \__ rule4
 \__ OUTPUT
      \__ rule5
      \__ rule6

For this document, child objects are always “nested” within a 
parent object, irrespective of the serialized encoding.

There are different ways to encode such a tree structure into a 
serialized stream. In many Netlink protocols, children attributes 
are encapsulated (a. k. a. “nested”, though we will avoid this 
term to avoid double-use) and treated as a whole as a parent's 
opaque data. We will call this format “Encapsulated Encoding”.

To encode an attribute's length, struct nlattr only has a 16-bit 
field, which means the attribute header plus payload is limited 
to 64 KB. This is easily exceedable with the encapsulated 
encoding as chains are collected rules in a chain, for example. 
The problem is aggreviated by the kernel's Netlink handler only 
allocating skbs a page size worth, which in the worst case means 
that the usable payload for attributes is around 3600 bytes only. 
In light of xt_u32's private data block being 1984 bytes already, 
that means that you won't be able to fit two -m u32 invocations 
nested in a single rule into a dump.

The Xtables2 Netlink protocol however encodes each node as a 
standalone attribute, to be called Flat Encoding, that is 
appended (a. k. a. “chained”) to the data stream. This makes it 
possible to split requests and dumps at a finer level than 
encapsulation would. Above all, it gets extensions the guarantee 
to have data blocks of a minimum guaranteed size.

Since Netlink messages do have a 32-bit quantity to store the 
message length, rulesets of roughly up to 4 GB are possibile, 
which is currently regarded as sufficient. The largest (and 
meaningful) rulesets seen to date in the industry weighed in at 
approximately 150 MB.

Whereas attribute nesting automatically provided for boundaries, 
this is realized using a dummy attribute in the chained approach. 
Certain attributes can start such a flattened nesting, and 
NFXTA_STOP terminates it.

2 Attributes

The meaning of attributes depends upon the nesting level in which 
they appear. Their type however remains the same, such that a 
single Netlink attribute validation policy object (struct 
nla_policy) is sufficient.

A table of all known attributes:


+--------+-----------------+---------------+----------------+
| Value  | Mnemonic        |    C type     | NLA type       |
+--------+-----------------+---------------+----------------+
+--------+-----------------+---------------+----------------+
|   1    | NFXTA_STOP      |               | NLA_FLAG       |
+--------+-----------------+---------------+----------------+
|   2    | NFXTA_ERRNO     |     int       | NLA_U32        |
+--------+-----------------+---------------+----------------+
|   3    | NFXTA_NAME      |   char []     | NLA_NUL_STRING |
+--------+-----------------+---------------+----------------+
|   4    | NFXTA_CHAIN     |               | NLA_FLAG       |
+--------+-----------------+---------------+----------------+
|   5    | NFXTA_HOOKNUM   | unsigned int  | NLA_U32        |
+--------+-----------------+---------------+----------------+
|   6    | NFXTA_PRIORITY  |     int       | NLA_U32        |
+--------+-----------------+---------------+----------------+
|   7    | NFXTA_NFPROTO   |   uint8_t     | NLA_U32        |
+--------+-----------------+---------------+----------------+
|        | NFXTA_RULE      |               | NLA_FLAG       |
+--------+-----------------+---------------+----------------+
|        | NFXTA_OFFSET    | unsigned int  | NLA_U32        |
+--------+-----------------+---------------+----------------+
|        | NFXTA_LENGTH    |    size_t     | NLA_U32        |
+--------+-----------------+---------------+----------------+
|        | NFXTA_VERDICT   | unsigned int  | NLA_U32        |
+--------+-----------------+---------------+----------------+
|        | NFXTA_MATCH     |               | NLA_FLAG       |
+--------+-----------------+---------------+----------------+
|        | NFXTA_DATA      |               | NLA_BINARY     |
+--------+-----------------+---------------+----------------+
|        | NFXTA_TARGET    |               | NLA_FLAG       |
+--------+-----------------+---------------+----------------+
|        | NFXTA_JUMP      |   char []     | NLA_NUL_STRING |
+--------+-----------------+---------------+----------------+
|        | NFXTA_GOTO      |   char []     | NLA_NUL_STRING |
+--------+-----------------+---------------+----------------+
|        | NFXTA_REVISION  |   uint8_t     | NLA_U32        |
+--------+-----------------+---------------+----------------+
|        | NFXTA_SIZE      |    size_t     | NLA_U32        |
+--------+-----------------+---------------+----------------+
|        | NFXTA_HOOKMASK  | unsigned int  | NLA_U32        |
+--------+-----------------+---------------+----------------+


The kernel ignores attributes with value 0 during validation, so 
it was left unused.

2.1 Nest level terminator<sub:nfxta_stop>

This attribute serves to denote the end of a nesting level as 
introduced by NFXTA_CHAIN, NFXTA_RULE, NFXTA_MATCH or 
NFXTA_TARGET. It has no data portion.

+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4                   | nla_type = NFXTA_STOP         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

2.2 Dump error code<sub:nfxta_errno>

Once a NLM_F_MULTI dump operation has been started, for example 
with the NFXTM_CHAIN_DUMP request, Netlink kernel users must 
always end it successfully with NLMSG_DONE. To convey an error 
during the dump, Xtables2 will emit a NFXTA_ERRNO attribute into 
the stream (if it can), emit no further attributes for the 
request, and cause the dump to stop.

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 8                   | nla_type = NFXTA_ERRNO        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| int errno;                                                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

2.3 Match extension<sub:nfxta_match>

Invocation of a match is represented using the NFXTA_MATCH 
attribute which starts a nest level. A match attribute must 
contain two attributes:

• NFXTA_NAME: the name of the target extension

• NFXTA_DATA: data private to this instance of the extension

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4                   | nla_type = NFXTA_MATCH        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4 + payload         | nla_type = NFXTA_NAME         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.                                                               .
. name of the extension, e.g. "hashlimit"                       .
.                                                               .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4 + payload         | nla_type = NFXTA_DATA         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.                                                               .
. e.g. struct xt_hashlimit_info                                 .
.                                                               .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4                   | nla_type = NFXTA_STOP         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

2.4 Target extension<sub:nfxta_target>

Invocation of a match is represented using the NFXTA_TARGET 
attribute which starts a nest level. A target attribute must 
contain two attributes:

• NFXTA_NAME: the name of the target extension

• NFXTA_DATA: data private to this instance of the extension

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4                   | nla_type = NFXTA_TARGET       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4 + payload         | nla_type = NFXTA_NAME         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.                                                               .
. name of the extension, e.g. "TCPMSS"                          .
.                                                               .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4 + payload         | nla_type = NFXTA_DATA         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.                                                               .
. e.g. struct xt_tcpmss_info                                    .
.                                                               .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4                   | nla_type = NFXTA_STOP         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

2.5 Rule<sub:nfxta_rule>

A rule is started using the NFXTA_RULE attribute, which starts a 
nest level, and is ended with an NFXTA_STOP attribute. Rules can 
contain:

• Zero or more match extensions (NFXTA_MATCH..NFXTA_STOP).

• Zero or more target extensions (NFXTA_TARGET..NFXTA_STOP).

• Zero or one NFXTA_VERDICT attribute that specifies the rule's 
  verdict as data, which can either be NF_ACCEPT or NF_DROP. 
  (Non-normative notes: The supplied verdict is executed if no 
  target has reached a verdict on its own. Omission of the 
  verdict attribute counts as XT_CONTINUE.)

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4                   | nla_type = NFXTA_RULE         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.                                                               .
. matches, targets, verdict                                     .
.                                                               .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4                   | nla_type = NFXTA_STOP         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

2.6 Chain<sub:nfxta_chain>

A chain is started using the NFXTA_CHAIN attribute, which starts 
a nest level, and is ended with an NFXTA_STOP attribute. Chains 
can contain:

• Zero or one of this group of three (= specify all three, or 
  none at all), specifying that this chain is a base chain 
  hooking in at some point:

  – One NFXTA_HOOKNUM attribute for giving a hook number. This is 
    (unfortunately) dependent on the chosen nfproto, so it is 
    either NF_INET_*, NF_BR_* or NF_ARP_*.

  – One NFXTA_PRIORITY attribute.

  – One NFXTA_NFPROTO attribute that is NFPROTO_*.

• Zero or more rules (NFXTA_RULE..NFXTA_STOP).

Example of a fully populated chain:

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4                   | nla_type = NFXTA_CHAIN        |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 8                   | nla_type = NFXTA_HOOKNUM      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| hook number (0..7)                                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 8                   | nla_type = NFXTA_PRIORITY     |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| priority (-2147483648..2147483647)                            |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 8                   | nla_type = NFXTA_NFPROTO      |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nfproto value (2=ipv4, 3=arp, 7=bridge, 10=ipv6, 12=decnet)   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.                                                               .
. rules                                                         .
.                                                               .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = 4                   | nla_type = NFXTA_STOP         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

3 Message types

3.1 IDENTIFYNFXTM_IDENTIFY: Identification

First and foremost a debug command. And to get something 
(table/chain-independent) that users can glare at (they love 
doing that).

Request:

• nlmsg_type = NFXTM_IDENTIFY;

Response:

• An NFXTA_NAMENFXTA_NAME attribute contains the name and version 
  of the implementation/patchset.

• Zero or more attributes of type NFXTA_MATCH, terminated by 
  NFXTA_STOP, giving meta information about the loaded match 
  extensions. Per available match, a group of three attributes 
  follows:

  – One NFXTA_NAME attribute for the name of the extension

  – One NFXTA_REVISION attribute to denote the version of the 
    extension's parameter protocol

  – One NFXTA_SIZE attribute for the size of its per-instance 
    data block

• Zero or more attributes of type NFXTA_TARGET, terminated by 
  NFXTA_STOP, giving meta information about the loaded and 
  available target extensions:

  – same attributes as with NFXTA_MATCH above

3.2 CHAIN_NEWNFXTM_CHAIN_NEW: Create new chain

Request:

• nlmsg_type = NFXTM_CHAIN_NEW;

• NFXTA_NAME attribute carrying the name of the new chain.

• Zero or one of this group of three:

  – NFXTA_HOOKNUM

  – NFXTA_PRIORITY

  – NFXTA_NFPROTO

Response:

• Standard ACK.

Remarks:

Right now, a chain can only be promoted to a base chain during 
creation (as far as the userspace view goes; when the kernel 
exactly installs the nf_hook_ops is not of concern to userspace), 
and it can only be demoted by deleting it. Should a 
NFXTM_CHAIN_PROMOTE be split off the NFXTM_CHAIN_NEW 
functionality?

3.3 CHAIN_DELNFXTM_CHAIN_DEL: Delete a chain

Request:

• nlmsg_type = NFXTM_CHAIN_DEL;

• NFXTA_NAME attribute carrying the name of the chain to delete

Response:

• Standard ACK.

3.4 CHAIN_MOVENFXTM_CHAIN_MOVE: Rename a chain

Request:

• nlmsg_type = NFXTM_CHAIN_MOVE;

• Two NFXTA_NAME attributes (order is important):

  – First one specifies the current name of the chain

  – Second one specifies the new name of the chain

0                   1                   2                   3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nlmsg_len = at least 24                                       |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nlmsg_type = NFXTM_CHAIN_MOVE | nlmsg_flags = NLM_F_REQUEST   |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nlmsg_seq = whatever                                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nlmsg_pid = whatever                                          |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = at least 4          | nla_type = NFXTA_NAME         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.                                                               .
. old name                                                      .
.                                                               .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| nla_len = at least 4          | nla_type = NFXTA_NAME         |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
.                                                               .
. new name                                                      .
.                                                               .
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

3.5 CHAIN_DUMPNFXTM_CHAIN_DUMP: Chain dump

Request:

• nlmsg_type = NFXTM_CHAIN_DUMP;

• NFXTA_NAMENFXTA_NAME attribute specifying the name of the chain 
  to dump

Response:

• Zero or one of this group of three:

  – NFXTA_HOOKNUMNFXTA_HOOKNUM, NFXTA_PRIORITYNFXTA_PRIORITY, 
    NFXTA_NFPROTONFXTA_NFPROTO.

• Zero or more NFXTA_RULE attributes as per section [sub:nfxta_rule]
  .

Errors:

• If an error occurs during dump, an NFXTA_ERRNO attribute is 
  emitted into the stream and the dump will immediately terminate 
  with a standard NLMSG_DONE message. No NFXTA_STOP attributes 
  will be emitted if the dump stopped in the middle of a nesting 
  level.

3.6 TABLE_DUMPNFXTM_TABLE_DUMP: Table dump

Returns an atomic snapshot of the table.

Request:

• nlmsg_type = NFXTM_TABLE_DUMP;

Response:

• Zero or more NFXTA_CHAINNFXTA_CHAIN attributes as described in 
  section [sub:nfxta_chain].

3.7 CHAIN_SPLICENFXTM_CHAIN_SPLICE: Add/delete rules

The NFXTM_CHAIN_SPLICE request does a bulk deletion of zero or 
more consecutive rules, followed by a bulk insertion of zero or 
more consecutive rules, all done in an atomic fashion. It 
operates similar to Perl's splice function on arrays. The request 
message needs to have at least the first three attributes.

Request:

• NFXTA_NAMENFXTA_NAME: Name of the chain to modify.

• NFXTA_OFFSETNFXTA_OFFSET: Index of entry where operation should 
  start.

• NFXTA_LENGTHNFXTA_LENGTH: Number of entries starting from 
  offset that should be removed. May be zero or more.

• Zero or more NFXTA_RULENFXTA_RULE as per section [sub:nfxta_rule]
  .

Response:

• Standard ACK.

• Desired: detailed error code and origin of error (result of 
  running ->check in extensions)

3.8 TABLE_REPLACENFXTM_TABLE_REPLACE

Atomic exchange of an entire table.

Request:

• nlmsg_type = NFXTM_TABLE_REPLACE;

• Zero or more NFXTA_CHAINNFXTA_CHAIN attributes as per section [sub:nfxta_chain]
  .

Response:

• Standard ACK.

• Desired: detailed error code and origin of error (result of 
  running ->check in extensions)

Re: Xtables2 Netlink spec

[Jozsef Kadlecsik]

Hi Jan,

On Wed, 24 Nov 2010, Jan Engelhardt wrote:

> By request of Pablo, I am posting the Xtables2 Netlink interface 
> specification for review. Additionally, further documentation and 
> toolchain around it is available through the temporary project page at
> 
> 	http://jengelh.medozas.de/projects/xtables/
> 
> which currently includes
> 
>  * User Documentation Chapter 1: Architectural Differences
> 
>  * Developer Documentation Part 1: Netlink interface (WIP)
>    This is copied below to facilitate inline replies
> 
>  * Runnable Linux source tree
> 
>  * Runnable userspace library (libnetfilter_xtables)
>    with small test-and-debug program
[...]

Please add fine-grained error reporting to the protocol: in my opinion the 
main shortcoming of the current kernel-userspace xtables protocol is the 
lack of the proper error reporting. I mean, the new protocol should be 
able to carry back which rule caused the error, in the rule whether it was 
a general kind of error (ENOMEM), or a table, chain, match or target error 
and exactly what was the error at table/chain/match/target level.

Say, the TCPMSS target should be able to report back that it cannot be 
used outside of FORWARD, OUTPUT and POSTROUTING. Or that the rule must 
match TCP SYN packets.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Xtables2 Netlink spec

[Pablo Neira Ayuso]

On 26/11/10 20:01, Jozsef Kadlecsik wrote:
> Hi Jan,
> 
> On Wed, 24 Nov 2010, Jan Engelhardt wrote:
> 
>> By request of Pablo, I am posting the Xtables2 Netlink interface 
>> specification for review. Additionally, further documentation and 
>> toolchain around it is available through the temporary project page at
>>
>> 	http://jengelh.medozas.de/projects/xtables/
>>
>> which currently includes
>>
>>  * User Documentation Chapter 1: Architectural Differences
>>
>>  * Developer Documentation Part 1: Netlink interface (WIP)
>>    This is copied below to facilitate inline replies
>>
>>  * Runnable Linux source tree
>>
>>  * Runnable userspace library (libnetfilter_xtables)
>>    with small test-and-debug program
> [...]
> 
> Please add fine-grained error reporting to the protocol: in my opinion the 
> main shortcoming of the current kernel-userspace xtables protocol is the 
> lack of the proper error reporting. I mean, the new protocol should be 
> able to carry back which rule caused the error, in the rule whether it was 
> a general kind of error (ENOMEM), or a table, chain, match or target error 
> and exactly what was the error at table/chain/match/target level.
> 
> Say, the TCPMSS target should be able to report back that it cannot be 
> used outside of FORWARD, OUTPUT and POSTROUTING. Or that the rule must 
> match TCP SYN packets.

If we follow the one message per rule basis, you can put several
messages into one batch with different sequence numbers. Thus, you can
know what message in the batch has triggered the error and the reason.

Re: Xtables2 Netlink spec

[Jan Engelhardt]

On Thursday 2010-12-09 13:08, Pablo Neira Ayuso wrote:
>
>If we follow the one message per rule basis, you can put several
>messages into one batch with different sequence numbers. Thus, you can
>know what message in the batch has triggered the error and the reason.

/* The unwritten laws of netlink */

Normally, the sequence number of a response message is simply
the one from the request message. But in a dump where there
can be multiple messages, do they all share the sequence number?

Must the response sequence numbers match at all, or is it like TCP
where each side has its own set?

BTW, can response messages - all those leading up to NLMSG_DONE -
have different nlmsg_type, or not? Does the nlmsg_type need
to match the request type?

Jan

Re: Xtables2 Netlink spec

[James Nurmi]

On Mon, Dec 13, 2010 at 6:01 PM, Jan Engelhardt <jengelh@medozas.de> wrote:
>
> On Thursday 2010-12-09 13:08, Pablo Neira Ayuso wrote:
>>
>>If we follow the one message per rule basis, you can put several
>>messages into one batch with different sequence numbers. Thus, you can
>>know what message in the batch has triggered the error and the reason.
>
> /* The unwritten laws of netlink */
>
> Normally, the sequence number of a response message is simply
> the one from the request message. But in a dump where there
> can be multiple messages, do they all share the sequence number?
>
> Must the response sequence numbers match at all, or is it like TCP
> where each side has its own set?
>
> BTW, can response messages - all those leading up to NLMSG_DONE -
> have different nlmsg_type, or not? Does the nlmsg_type need
> to match the request type?
>

There's an upper layer netlink flag set (NLM_F_MULTI , NLMSG_DONE,
etc) that's usually used in NETLINK_ROUTE, etc.

Essentially, you set the MULTI flag on multi-part messages, and DONE
on the last one, clients are expected to collect and respond;

In those cases, the sequence number does NOT change intra-messageset
in my experience, instead it is assumed they may be received out of
order.

> Jan
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Xtables2 Netlink spec

[Jan Engelhardt]

On Tuesday 2010-12-14 03:16, James Nurmi wrote:
>On Mon, Dec 13, 2010 at 6:01 PM, Jan Engelhardt <jengelh@medozas.de> wrote:
>>
>> Normally, the sequence number of a response message is simply
>> the one from the request message. But in a dump where there
>> can be multiple messages, do they all share the sequence number?
>
>In those cases, the sequence number does NOT change intra-messageset
>in my experience, instead it is assumed they may be received out of
>order.

That would defeat the purpose (as seen from over here) of multipart
messages in the first place.

Re: Xtables2 Netlink spec

[Pablo Neira Ayuso]

On 14/12/10 03:01, Jan Engelhardt wrote:
> 
> On Thursday 2010-12-09 13:08, Pablo Neira Ayuso wrote:
>>
>> If we follow the one message per rule basis, you can put several
>> messages into one batch with different sequence numbers. Thus, you can
>> know what message in the batch has triggered the error and the reason.
> 
> /* The unwritten laws of netlink */
>
> Normally, the sequence number of a response message is simply
> the one from the request message. But in a dump where there
> can be multiple messages, do they all share the sequence number?

Yes, because they are the result of one request. Responses use the
sequence number of the original request. Thus, you can identify what
messages come as part of what requests.

> Must the response sequence numbers match at all, or is it like TCP
> where each side has its own set?

They have to match.

> BTW, can response messages - all those leading up to NLMSG_DONE -
> have different nlmsg_type, or not?

They all have the same type.

> Does the nlmsg_type need to match the request type?

No.

Re: Xtables2 Netlink spec

[Thomas Graf]

On Wed, Dec 15, 2010 at 02:54:26PM +0100, Pablo Neira Ayuso wrote:
> > BTW, can response messages - all those leading up to NLMSG_DONE -
> > have different nlmsg_type, or not?
> 
> They all have the same type.

This is not a MUST. It is perfectly legal to f.e.:

 -> FOO_GET (seq=1, NLM_F_REQUEST)
 <- FOO_DEL (seq=1, NLM_F_MULTI)
 <- FOO_ADD (seq=1, NLM_F_MULTI)
 <- NLMSG_DONE (seq=1)

Re: Xtables2 Netlink spec

[Jan Engelhardt]

On Thursday 2010-12-16 15:05, Thomas Graf wrote:

>On Wed, Dec 15, 2010 at 02:54:26PM +0100, Pablo Neira Ayuso wrote:
>> > BTW, can response messages - all those leading up to NLMSG_DONE -
>> > have different nlmsg_type, or not?
>> 
>> They all have the same type.
>
>This is not a MUST. It is perfectly legal to f.e.:
>
> -> FOO_GET (seq=1, NLM_F_REQUEST)
> <- FOO_DEL (seq=1, NLM_F_MULTI)
> <- FOO_ADD (seq=1, NLM_F_MULTI)
> <- NLMSG_DONE (seq=1)

Oh great, now the confusion is complete. One person says this, another 
says something else. Best of all, the Netlink RFC leaves it unspecified, 
so it's all hearsay, beliefs and Perl5-style ("Source acts as normative 
reference") referencing. I guess we are doomed until the original 
Netlink3549 authors step up and tell us their intentions.

As I see it, we need a discussion to specify what is to be done with 
unspecified parts, with 3549 as an origin.

Re: Xtables2 Netlink spec

[Thomas Graf]

On Thu, Dec 16, 2010 at 03:22:07PM +0100, Jan Engelhardt wrote:
> On Thursday 2010-12-16 15:05, Thomas Graf wrote:
> >
> > -> FOO_GET (seq=1, NLM_F_REQUEST)
> > <- FOO_DEL (seq=1, NLM_F_MULTI)
> > <- FOO_ADD (seq=1, NLM_F_MULTI)
> > <- NLMSG_DONE (seq=1)
> 
> Oh great, now the confusion is complete. One person says this, another 
> says something else. Best of all, the Netlink RFC leaves it unspecified, 
> so it's all hearsay, beliefs and Perl5-style ("Source acts as normative 
> reference") referencing. I guess we are doomed until the original 
> Netlink3549 authors step up and tell us their intentions.
> 
> As I see it, we need a discussion to specify what is to be done with 
> unspecified parts, with 3549 as an origin.

The RFC was not written prior to the implementation but after it has
been around for a while.

The current netlink code implementation defines the standard. It is the
standard because we have not been breaking it and will never do.

Netlink is very open minded, it does not care if individual protocols
define their own semantics. Most will never make use of the above but
it is perfectly legal to do so.

NLM_F_MULTI && NLMSG_DONE is simply a way to have the receiver
continue recieving and parsing. The flag states "Wait, be patient,
my reply consists of multiple messages" and nothing more.

Re: Xtables2 Netlink spec

[Jan Engelhardt]

On Friday 2010-12-17 08:25, Thomas Graf wrote:
>On Thu, Dec 16, 2010 at 03:22:07PM +0100, Jan Engelhardt wrote:
>> 
>>Oh great, now the confusion is complete. One person says this,
>>another says something else. Best of all, the Netlink RFC leaves it
>>unspecified, so it's all hearsay, beliefs and Perl5-style ("Source
>>acts as normative reference") referencing. I guess we are doomed
>>until the original Netlink3549 authors step up and tell us their
>>intentions.
>> 
>>As I see it, we need a discussion to specify what is to be done
>>with unspecified parts, with 3549 as an origin.
>
>The current netlink code implementation defines the standard. It is
>the standard because we have not been breaking it and will never do.
>
>Netlink is very open minded, it does not care if individual
>protocols define their own semantics.

So in fact, it does allow for preservation of attribute order and 
support for multiple attributes appearing with the same type, since that 
is part of my subprotocol anyway, right?

Cf. http://marc.info/?l=netfilter-devel&m=129068531114996&w=2

Re: Xtables2 Netlink spec

[Pablo Neira Ayuso]

On 17/12/10 10:35, Jan Engelhardt wrote:
> 
> On Friday 2010-12-17 08:25, Thomas Graf wrote:
>> On Thu, Dec 16, 2010 at 03:22:07PM +0100, Jan Engelhardt wrote:
>>>
>>> Oh great, now the confusion is complete. One person says this,
>>> another says something else. Best of all, the Netlink RFC leaves it
>>> unspecified, so it's all hearsay, beliefs and Perl5-style ("Source
>>> acts as normative reference") referencing. I guess we are doomed
>>> until the original Netlink3549 authors step up and tell us their
>>> intentions.
>>>
>>> As I see it, we need a discussion to specify what is to be done
>>> with unspecified parts, with 3549 as an origin.
>>
>> The current netlink code implementation defines the standard. It is
>> the standard because we have not been breaking it and will never do.
>>
>> Netlink is very open minded, it does not care if individual
>> protocols define their own semantics.
> 
> So in fact, it does allow for preservation of attribute order and 
> support for multiple attributes appearing with the same type, since that 
> is part of my subprotocol anyway, right?
> 
> Cf. http://marc.info/?l=netfilter-devel&m=129068531114996&w=2

As Thomas said, Netlink whatever protocol upon it, but I already told
you: making assumptions on the order of the attributes is not a good
practise because you'll have to stick to a certain message layout.

That's the opposite to what we aim which is to provide protocols that
can be easily extended in the future. If you assume that we cannot
change the attribute ordering, that's a rule that everybody will have to
live with forever.

Again, it will be valid, yes, but it's a poorly designed protocol.

Moreover, the reason why you want that attribute trailer is because of
the supposed-to-be limitations that you're trying to avoid. And, again,
I have to tell you that, avoiding the limitation by introducing
assumptions in the protocol is not a good idea.

Re: Xtables2 Netlink spec

[Pablo Neira Ayuso]

On 16/12/10 15:05, Thomas Graf wrote:
> On Wed, Dec 15, 2010 at 02:54:26PM +0100, Pablo Neira Ayuso wrote:
>>> BTW, can response messages - all those leading up to NLMSG_DONE -
>>> have different nlmsg_type, or not?
>>
>> They all have the same type.
> 
> This is not a MUST. It is perfectly legal to f.e.:
> 
>  -> FOO_GET (seq=1, NLM_F_REQUEST)
>  <- FOO_DEL (seq=1, NLM_F_MULTI)
>  <- FOO_ADD (seq=1, NLM_F_MULTI)
>  <- NLMSG_DONE (seq=1)

What realistic situation will require this?

Re: Xtables2 Netlink spec

[Jan Engelhardt]

On Friday 2010-12-17 10:55, Pablo Neira Ayuso wrote:

>On 16/12/10 15:05, Thomas Graf wrote:
>> On Wed, Dec 15, 2010 at 02:54:26PM +0100, Pablo Neira Ayuso wrote:
>>>> BTW, can response messages - all those leading up to NLMSG_DONE -
>>>> have different nlmsg_type, or not?
>>>
>>> They all have the same type.
>> 
>> This is not a MUST. It is perfectly legal to f.e.:
>> 
>>  -> FOO_GET (seq=1, NLM_F_REQUEST)
>>  <- FOO_DEL (seq=1, NLM_F_MULTI)
>>  <- FOO_ADD (seq=1, NLM_F_MULTI)
>>  <- NLMSG_DONE (seq=1)
>
>What realistic situation will require this?

This does:

-> NFXTM_CHAIN_DUMP<NFXTA_NAME>
<- NFXTM_RULE_START<>
<- NFXTM_EMATCH<NFXTA_NAME,NFXTA_REVISION,NFXTA_DATA>
<- NFXTM_EMATCH<NFXTA_NAME,NFXTA_REVISION,NFXTA_DATA>
<- NFXTM_ETARGET<NFXTA_NAME,NFXTA_REVISION,NFXTA_DATA>
<- NFXTM_ETARGET<NFXTA_NAME,NFXTA_REVISION,NFXTA_DATA>
<- NFXTM_RULE_END<>
<- NFXTM_RULE_START<>
<- NFXTM_ETARGET<NFXTA_VERDICT>
<- NFXTM_RULE_END<>
<- NLMSG_DONE

This is 9 messages with answers related to the ruleset.

If only a single nlmsg_type was possible for NLM_F_MULTI replies,
this is probably how things would have looked:

-> CHAIN_DUMP<NFXTA_NAME>
<- CHAIN_DUMP<NFXTA_RULE_START>
<- CHAIN_DUMP<NFXTA_MATCH_START>
<- CHAIN_DUMP<NFXTA_NAME><NFXTA_REVISION><NFXTA_DATA>
<- CHAIN_DUMP<NFXTA_MATCH_END>
<- CHAIN_DUMP<NFXTA_MATCH_START>
<- CHAIN_DUMP<NFXTA_NAME><NFXTA_REVISION><NFXTA_DATA>
<- CHAIN_DUMP<NFXTA_MATCH_END>
<- CHAIN_DUMP<NFXTA_TARGET_START>
<- CHAIN_DUMP<NFXTA_NAME><NFXTA_REVISION><NFXTA_DATA>
<- CHAIN_DUMP<NFXTA_TARGET_END>
<- CHAIN_DUMP<NFXTA_TARGET_START>
<- CHAIN_DUMP<NFXTA_NAME><NFXTA_REVISION><NFXTA_DATA>
<- CHAIN_DUMP<NFXTA_TARGET_END>
<- CHAIN_DUMP<NFXTA_RULE_END>
<- CHAIN_DUMP<NFXTA_RULE_START>
<- CHAIN_DUMP<NFXTA_TARGET_START>
<- CHAIN_DUMP<NFXTA_VERDICT>
<- CHAIN_DUMP<NFXTA_TARGET_END>
<- CHAIN_DUMP<NFXTA_RULE_END>
<- NLMSG_DONE

This requires more forth-and-back between userspace and the kernel:
19 messages instead. Using multiple nlmsg_type seems a good thing to
exploit.

Re: Xtables2 Netlink spec

[Jan Engelhardt]

On Friday 2010-11-26 20:01, Jozsef Kadlecsik wrote:
>On Wed, 24 Nov 2010, Jan Engelhardt wrote:
>
>> By request of Pablo, I am posting the Xtables2 Netlink interface 
>> specification for review. Additionally, further documentation and 
>> toolchain around it is available through the temporary project page at
>> 
>> 	http://jengelh.medozas.de/projects/xtables/
>>  * Runnable userspace library (libnetfilter_xtables)
>>    with small test-and-debug program
>[...]
>
>Please add fine-grained error reporting to the protocol: in my opinion the 
>main shortcoming of the current kernel-userspace xtables protocol is the 
>lack of the proper error reporting. I mean, the new protocol should be 
>able to carry back which rule caused the error, in the rule whether it was 
>a general kind of error (ENOMEM), or a table, chain, match or target error 
>and exactly what was the error at table/chain/match/target level.

That should not be a problem. However, what do we do with the general
kind of error that is overly general? A.k.a. the dreaded EINVAL.

Say a user requested jumping to a chain, but did not give a chain name.
Normally that would be EINVAL, but EINVAL is overused already.

What would you like? Comprehensive error numbers (sort of like the
ones Windows is said to use) aka. NFXTE_NO_CHAIN_NAME_GIVEN,
or a human-readable string; or something else?

>Say, the TCPMSS target should be able to report back that it cannot be 
>used outside of FORWARD, OUTPUT and POSTROUTING.

NFXTE_HOOKMASK_NOT_ADHERED or string?

>Or that the rule must match TCP SYN packets.

TCPMSS doing a rule-search for -p tcp is pretty ugly (it must understand
the data structures, and that is sort of a backwards shot); Patrick
once suggested IIRC that TCPMSS should just silently skip non-SYNs.

Maybe both error numbers, and providing extensions with the
possibility to send strings? It is impossible to provision error
numbers for out-of-tree extensions, so having a way for an extension
to return some NFXTA_ERRSTR "One of my parameters is wrong!" seems
required at least.

Re: Xtables2 Netlink spec

[Jozsef Kadlecsik]

On Wed, 15 Dec 2010, Jan Engelhardt wrote:

> On Friday 2010-11-26 20:01, Jozsef Kadlecsik wrote:
> >On Wed, 24 Nov 2010, Jan Engelhardt wrote:
> >
> >> By request of Pablo, I am posting the Xtables2 Netlink interface 
> >> specification for review. Additionally, further documentation and 
> >> toolchain around it is available through the temporary project page at
> >> 
> >> 	http://jengelh.medozas.de/projects/xtables/
> >>  * Runnable userspace library (libnetfilter_xtables)
> >>    with small test-and-debug program
> >[...]
> >
> >Please add fine-grained error reporting to the protocol: in my opinion the 
> >main shortcoming of the current kernel-userspace xtables protocol is the 
> >lack of the proper error reporting. I mean, the new protocol should be 
> >able to carry back which rule caused the error, in the rule whether it was 
> >a general kind of error (ENOMEM), or a table, chain, match or target error 
> >and exactly what was the error at table/chain/match/target level.
> 
> That should not be a problem. However, what do we do with the general
> kind of error that is overly general? A.k.a. the dreaded EINVAL.
> 
> Say a user requested jumping to a chain, but did not give a chain name.
> Normally that would be EINVAL, but EINVAL is overused already.

When I wrote general error I meant the ones where there is no point (or 
cannot be) to specify the nature of the error exactly. Like in the 
example, ENOMEM: it's needles to report which new data field could not be 
allocated. But if the specified chain does not exists, that must not be 
masked by a general EINVAL. The user must be alerted that the chain with 
the given name does not exist.
 
> What would you like? Comprehensive error numbers (sort of like the
> ones Windows is said to use) aka. NFXTE_NO_CHAIN_NAME_GIVEN,
> or a human-readable string; or something else?

Yes, use comprehensive error codes. And it's the responsibility of the 
userspace tool to translate them to proper error messages.
 
> >Say, the TCPMSS target should be able to report back that it cannot be 
> >used outside of FORWARD, OUTPUT and POSTROUTING.
> 
> NFXTE_HOOKMASK_NOT_ADHERED or string?

The former, i.e. error code.
 
> >Or that the rule must match TCP SYN packets.
> 
> TCPMSS doing a rule-search for -p tcp is pretty ugly (it must understand
> the data structures, and that is sort of a backwards shot); Patrick
> once suggested IIRC that TCPMSS should just silently skip non-SYNs.

For the clarity of the rules I'd prefer the current solution, i.e. check 
and enforce that the rule matches TCP SYN packets. If we relax it, next 
time someone could complain why TCPMSS is restricted to FORWARD, OUTPUT, 
POSTROUTING, why can't the system simply skip the target when called at 
non-appropriate hooks.

But I just picked TCPMSS target for errors which currently expressed 
in EINVAL.
 
> Maybe both error numbers, and providing extensions with the
> possibility to send strings? It is impossible to provision error
> numbers for out-of-tree extensions, so having a way for an extension
> to return some NFXTA_ERRSTR "One of my parameters is wrong!" seems
> required at least.

I have got a three-level error coding in my mind: general, standard error 
codes (ENOMEM, EPERM, etc.), general netfilter specific ones (like 
NFXTE_HOOKMASK_NOT_ADHERED) and table/match/target specific ones.

But I do realize that it's much easier (and therefore quite tempting) to 
construct the full error message in kernel space and just send it back.
However, that'd make quite hard to support internationalization.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Xtables2 Netlink spec

[Jesper Dangaard Brouer]

Cc.ed Thomas Graf (tgraf@redhat.com), Thomas presented some interesting 
ideas on netlink error-codes and strings during NetConf 2010, see:

  http://vger.kernel.org/netconf2010.html
  http://vger.kernel.org/netconf2010_slides/tgraf_netconf10.odp


On Wed, 15 Dec 2010, Jozsef Kadlecsik wrote:
> On Wed, 15 Dec 2010, Jan Engelhardt wrote:
>> On Friday 2010-11-26 20:01, Jozsef Kadlecsik wrote:
>>> On Wed, 24 Nov 2010, Jan Engelhardt wrote:
>>>
>>>> By request of Pablo, I am posting the Xtables2 Netlink interface
>>>> specification for review. Additionally, further documentation and
>>>> toolchain around it is available through the temporary project page at
>>>>
>>>> 	http://jengelh.medozas.de/projects/xtables/
>>>>  * Runnable userspace library (libnetfilter_xtables)
>>>>    with small test-and-debug program
>>> [...]
>>>
>>> Please add fine-grained error reporting to the protocol: in my opinion the
>>> main shortcoming of the current kernel-userspace xtables protocol is the
>>> lack of the proper error reporting. I mean, the new protocol should be
>>> able to carry back which rule caused the error, in the rule whether it was
>>> a general kind of error (ENOMEM), or a table, chain, match or target error
>>> and exactly what was the error at table/chain/match/target level.
>>
>> That should not be a problem. However, what do we do with the general
>> kind of error that is overly general? A.k.a. the dreaded EINVAL.
>>
>> Say a user requested jumping to a chain, but did not give a chain name.
>> Normally that would be EINVAL, but EINVAL is overused already.
>
> When I wrote general error I meant the ones where there is no point (or
> cannot be) to specify the nature of the error exactly. Like in the
> example, ENOMEM: it's needles to report which new data field could not be
> allocated. But if the specified chain does not exists, that must not be
> masked by a general EINVAL. The user must be alerted that the chain with
> the given name does not exist.
>
>> What would you like? Comprehensive error numbers (sort of like the
>> ones Windows is said to use) aka. NFXTE_NO_CHAIN_NAME_GIVEN,
>> or a human-readable string; or something else?
>
> Yes, use comprehensive error codes. And it's the responsibility of the
> userspace tool to translate them to proper error messages.
>
>>> Say, the TCPMSS target should be able to report back that it cannot be
>>> used outside of FORWARD, OUTPUT and POSTROUTING.
>>
>> NFXTE_HOOKMASK_NOT_ADHERED or string?
>
> The former, i.e. error code.
>
>>> Or that the rule must match TCP SYN packets.
>>
>> TCPMSS doing a rule-search for -p tcp is pretty ugly (it must understand
>> the data structures, and that is sort of a backwards shot); Patrick
>> once suggested IIRC that TCPMSS should just silently skip non-SYNs.
>
> For the clarity of the rules I'd prefer the current solution, i.e. check
> and enforce that the rule matches TCP SYN packets. If we relax it, next
> time someone could complain why TCPMSS is restricted to FORWARD, OUTPUT,
> POSTROUTING, why can't the system simply skip the target when called at
> non-appropriate hooks.
>
> But I just picked TCPMSS target for errors which currently expressed
> in EINVAL.
>
>> Maybe both error numbers, and providing extensions with the
>> possibility to send strings? It is impossible to provision error
>> numbers for out-of-tree extensions, so having a way for an extension
>> to return some NFXTA_ERRSTR "One of my parameters is wrong!" seems
>> required at least.

I like this.

> I have got a three-level error coding in my mind: general, standard error
> codes (ENOMEM, EPERM, etc.), general netfilter specific ones (like
> NFXTE_HOOKMASK_NOT_ADHERED) and table/match/target specific ones.
>
> But I do realize that it's much easier (and therefore quite tempting) to
> construct the full error message in kernel space and just send it back.
> However, that'd make quite hard to support internationalization.

To support internationalization, we could just add an error-number-code 
in front of the constructed error message?

I'm a fan of the full error message system from the kernel, becuase its 
much easier to maintain, as we don't need to update iptables userspace 
each time we introduce a new error code/message.


Cheers,
   Jesper Brouer

--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen
Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------

Error reporting in Netlink (Re: Xtables2 Netlink spec)

[Jan Engelhardt]

On Thursday 2010-12-16 10:57, Jesper Dangaard Brouer wrote:

>Cc.ed Thomas Graf (tgraf@redhat.com), Thomas presented some interesting 
>ideas on netlink error-codes and strings during NetConf 2010, see:
>
>http://vger.kernel.org/netconf2010.html 
>http://vger.kernel.org/netconf2010_slides/tgraf_netconf10.odp

The idea is appending an error string is ok for Netlink as a protocol
(specification-wise), but the size constraints of the skbuffs in the
Linux may make its practical implementation a little harder. "Half of
the packet" is already used for the original request message, and
cramming an extra error string may bust the space.
It also does not look very netlinky to not use nlattrs ;-)


On Wed, 15 Dec 2010, Jozsef Kadlecsik came about with:
>
>>I have got a three-level error coding in my mind: general, standard 
>>error codes (ENOMEM, EPERM, etc.), general netfilter specific ones 
>>(like NFXTE_HOOKMASK_NOT_ADHERED) and table/match/target specific 
>>ones.
>>
>>But I do realize that it's much easier (and therefore quite tempting) 
>>to construct the full error message in kernel space and just send it 
>>back. However, that'd make quite hard to support internationalization.

It's not like those strings change all that much.


>To support internationalization, we could just add an error-number-code 
>in front of the constructed error message?

Buying us what? If you change the string, but the gettext lookup is 
based upon a number, you will get an outdated translation, which is not 
nice either. IMHO: Better an English message than an inaccurate message.

Re: Error reporting in Netlink (Re: Xtables2 Netlink spec)

[Thomas Graf]

On Thu, 2010-12-16 at 13:51 +0100, Jan Engelhardt wrote: 
> On Thursday 2010-12-16 10:57, Jesper Dangaard Brouer wrote:
> 
> >Cc.ed Thomas Graf (tgraf@redhat.com), Thomas presented some interesting 
> >ideas on netlink error-codes and strings during NetConf 2010, see:
> >
> >http://vger.kernel.org/netconf2010.html 
> >http://vger.kernel.org/netconf2010_slides/tgraf_netconf10.odp
> 
> The idea is appending an error string is ok for Netlink as a protocol
> (specification-wise), but the size constraints of the skbuffs in the
> Linux may make its practical implementation a little harder. "Half of
> the packet" is already used for the original request message, and
> cramming an extra error string may bust the space.
> It also does not look very netlinky to not use nlattrs ;-)

Why not use nlattr to encode the error string? It would make error
messages easier to extend in the future. At some point we might want
to add an offset field which points into the original netlink message
describing the attribute which caused the failure.

> On Wed, 15 Dec 2010, Jozsef Kadlecsik came about with:
> >
> >>I have got a three-level error coding in my mind: general, standard 
> >>error codes (ENOMEM, EPERM, etc.), general netfilter specific ones 
> >>(like NFXTE_HOOKMASK_NOT_ADHERED) and table/match/target specific 
> >>ones.
> >>
> >>But I do realize that it's much easier (and therefore quite tempting) 
> >>to construct the full error message in kernel space and just send it 
> >>back. However, that'd make quite hard to support internationalization.

Thinking of netlink protocols in general and netfilter in specific,
maintaining a list of reserved error codes for each subsystem/target/
module will result in an unbearable pain if the error codes are not
separated into individual namespaces for each module.

That would in turn require each module to define a unique number or
some form of namespace resolution mechanism which does not help to keep
things simple.

This is the main reason why I advocate the use of error strings.

> It's not like those strings change all that much.
> 
> 
> >To support internationalization, we could just add an error-number-code 
> >in front of the constructed error message?
> 
> Buying us what? If you change the string, but the gettext lookup is 
> based upon a number, you will get an outdated translation, which is not 
> nice either. IMHO: Better an English message than an inaccurate message.

Do we *really* need internationalization for error messages on this
level? Primarily userspace should be in charge of checking for all kinds
of erroneous user input and produce meaningful context based,
translatable error messages. Error messages produced by the kernel
should be the exception and not a substitute for proper userspace error
handling.

Re: Error reporting in Netlink (Re: Xtables2 Netlink spec)

[Jan Engelhardt]

On Thursday 2010-12-16 14:43, Thomas Graf wrote:
>
>>It also does not look very netlinky to not use nlattrs
>
>Why not use nlattr to encode the error string? It would make error
>messages easier to extend in the future. At some point we might want
>to add an offset field which points into the original netlink
>message describing the attribute which caused the failure.

Is that a yes or a no?


>>>>But I do realize that it's much easier (and therefore quite
>>>>tempting) to construct the full error message in kernel space and
>>>>just send it back. However, that'd make quite hard to support
>>>>internationalization.
>
>Thinking of netlink protocols in general and netfilter in specific,
>maintaining a list of reserved error codes for each
>subsystem/target/ module will result in an unbearable pain if the
>error codes are not separated into individual namespaces for each
>module.

I did not plan on giving extensions a numeric namespace; here it's
largely strings only.


>> It's not like those strings change all that much.
>> 
>> >To support internationalization, we could just add an error-number-code 
>> >in front of the constructed error message?
>> 
>> Buying us what? If you change the string, but the gettext lookup is 
>> based upon a number, you will get an outdated translation, which is not 
>> nice either. IMHO: Better an English message than an inaccurate message.
>
>Do we *really* need internationalization for error messages on this
>level? Primarily userspace should be in charge of checking for all kinds
>of erroneous user input and produce meaningful context based,
>translatable error messages.

Let's see, why does iproute2 not just do that, then? Because some
things only the kernel knows about, so even if the parameters are
correct from the userspace side of view, the kernel may reject the
request nevertheless.

Re: Error reporting in Netlink (Re: Xtables2 Netlink spec)

[Thomas Graf]

On Thu, Dec 16, 2010 at 02:51:07PM +0100, Jan Engelhardt wrote:
> >Why not use nlattr to encode the error string? It would make error
> >messages easier to extend in the future. At some point we might want
> >to add an offset field which points into the original netlink
> >message describing the attribute which caused the failure.
> 
> Is that a yes or a no?

The proposed solution at netconf involved appending the error string
directly. Inspired by your comment I realizezd that encoding the
error string as nlattr allow for additional attributes would be a
better implementation.

As for size limitations, even though most netlink protocols do it, I
don't see the point in appending the whole request message in a error
message. The header would be completely sufficient for all request/reply
based protocols. It is no problem for userspace to keep a copy of the
last request sent.

> Let's see, why does iproute2 not just do that, then? Because some
> things only the kernel knows about, so even if the parameters are
> correct from the userspace side of view, the kernel may reject the
> request nevertheless.

You are not the first to come up with this but it is still a pretty
lazy excuse. iproute2 could do a lot better than it does today and
it has been improved a lot over time. The main reason for the current
situation is the atomic nature of routing netlink requests handlers.
Checking for errors in an atomic context where no interface disappears
and no route can be removed while the request is being processed simply
makes checking for errors a lot easier.

This does not mean that userspace should have a card blanche for
sending bogus combinations of netlink attributes and expect the kernel
to always come up with a perfect verbose error message.

The kernel can still send ENODEV to indicate the specified network
device does not exist but it should only cover the case where the
interface disappeared between the userspace checking for its existance
and the request being processed.

Re: Error reporting in Netlink (Re: Xtables2 Netlink spec)

[Pablo Neira Ayuso]

On 16/12/10 15:19, Thomas Graf wrote:
> On Thu, Dec 16, 2010 at 02:51:07PM +0100, Jan Engelhardt wrote:
>>> Why not use nlattr to encode the error string? It would make error
>>> messages easier to extend in the future. At some point we might want
>>> to add an offset field which points into the original netlink
>>> message describing the attribute which caused the failure.
>>
>> Is that a yes or a no?
> 
> The proposed solution at netconf involved appending the error string
> directly. Inspired by your comment I realizezd that encoding the
> error string as nlattr allow for additional attributes would be a
> better implementation.

Indeed, I'd prefer adding an extra netlink header with a new type like
NLM_ERROR2 followed by attributes like the string (or an int) that
contain the error.

This also can be added with breaking existing apps and libraries IIRC.

> As for size limitations, even though most netlink protocols do it, I
> don't see the point in appending the whole request message in a error
> message. The header would be completely sufficient for all request/reply
> based protocols. It is no problem for userspace to keep a copy of the
> last request sent.

At least, we require the netlink header of the request message in error
messages. This is useful for message batches that are sent to
kernel-space, to know which one triggered the error.

Re: Error reporting in Netlink (Re: Xtables2 Netlink spec)

[Jozsef Kadlecsik]

On Thu, 16 Dec 2010, Thomas Graf wrote:

> On Thu, 2010-12-16 at 13:51 +0100, Jan Engelhardt wrote: 
> > On Thursday 2010-12-16 10:57, Jesper Dangaard Brouer wrote:
> > 
> > >Cc.ed Thomas Graf (tgraf@redhat.com), Thomas presented some interesting 
> > >ideas on netlink error-codes and strings during NetConf 2010, see:
> > >
> > >http://vger.kernel.org/netconf2010.html 
> > >http://vger.kernel.org/netconf2010_slides/tgraf_netconf10.odp
> > 
> > The idea is appending an error string is ok for Netlink as a protocol
> > (specification-wise), but the size constraints of the skbuffs in the
> > Linux may make its practical implementation a little harder. "Half of
> > the packet" is already used for the original request message, and
> > cramming an extra error string may bust the space.
> > It also does not look very netlinky to not use nlattrs ;-)
> 
> Why not use nlattr to encode the error string? It would make error
> messages easier to extend in the future. At some point we might want
> to add an offset field which points into the original netlink message
> describing the attribute which caused the failure.

I use full netlink messages with respect the buffers, taking into account 
the size of struct nlmsgerr in case of an error. If there was an error 
string, how much space should be reserved for that?

> > On Wed, 15 Dec 2010, Jozsef Kadlecsik came about with:
> > >
> > >>I have got a three-level error coding in my mind: general, standard 
> > >>error codes (ENOMEM, EPERM, etc.), general netfilter specific ones 
> > >>(like NFXTE_HOOKMASK_NOT_ADHERED) and table/match/target specific 
> > >>ones.
> > >>
> > >>But I do realize that it's much easier (and therefore quite tempting) 
> > >>to construct the full error message in kernel space and just send it 
> > >>back. However, that'd make quite hard to support internationalization.
> 
> Thinking of netlink protocols in general and netfilter in specific,
> maintaining a list of reserved error codes for each subsystem/target/
> module will result in an unbearable pain if the error codes are not
> separated into individual namespaces for each module.
> 
> That would in turn require each module to define a unique number or
> some form of namespace resolution mechanism which does not help to keep
> things simple.
> 
> This is the main reason why I advocate the use of error strings.

But why error codes looks complicated? Netlink already supports it and 
it's simple to separate them:
	errcode < 256	generic errors
256 <=	errcode < 512	generic netfilter specific errors
512 <=	errcode		table/match/module specific errors

An error pointer is required which points to the table/match/module, which
triggered the error.
 
> > It's not like those strings change all that much.
> > 
> > 
> > >To support internationalization, we could just add an error-number-code 
> > >in front of the constructed error message?
> > 
> > Buying us what? If you change the string, but the gettext lookup is 
> > based upon a number, you will get an outdated translation, which is not 
> > nice either. IMHO: Better an English message than an inaccurate message.
> 
> Do we *really* need internationalization for error messages on this
> level? Primarily userspace should be in charge of checking for all kinds
> of erroneous user input and produce meaningful context based,
> translatable error messages. Error messages produced by the kernel
> should be the exception and not a substitute for proper userspace error
> handling.

Netlink based protocol is the path to internationalization. Netlink based 
protol leads to usable library. Library leads to gui. Gui leads to 
internationalization. ;-)

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@mail.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Error reporting in Netlink (Re: Xtables2 Netlink spec)

[Jan Engelhardt]

On Thursday 2010-12-16 15:47, Jozsef Kadlecsik wrote:
>> 
>> Thinking of netlink protocols in general and netfilter in specific,
>> maintaining a list of reserved error codes for each subsystem/target/
>> module will result in an unbearable pain if the error codes are not
>> separated into individual namespaces for each module.
>> 
>> That would in turn require each module to define a unique number or
>> some form of namespace resolution mechanism which does not help to keep
>> things simple.
>> 
>> This is the main reason why I advocate the use of error strings.
>
>But why error codes looks complicated? Netlink already supports it and 
>it's simple to separate them:
>	errcode < 256	generic errors
>256 <=	errcode < 512	generic netfilter specific errors
>512 <=	errcode		table/match/module specific errors

Actually, <4096 is reserved for the generic system errors (Exxx).

The specific issue here however is that you cannot easily delegate
the space >=512 to modules, so it's probably best if we don't try.

>>Do we *really* need internationalization for error messages on this
>>level? Primarily userspace should be in charge of checking for all kinds
>>of erroneous user input and produce meaningful context based,
>>translatable error messages. Error messages produced by the kernel
>>should be the exception and not a substitute for proper userspace error
>>handling.
>
>Netlink based protocol is the path to internationalization. Netlink based 
>protol leads to usable library. Library leads to gui. Gui leads to 
>internationalization. ;-)

On the contrary:
"Users leads to guis, guis lead to requests, requests lead to libraries,
libraries lead to netlink.
Doesn't make sense? Ask SUN!"

Re: Error reporting in Netlink (Re: Xtables2 Netlink spec)

[Patrick McHardy]

Am 16.12.2010 14:43, schrieb Thomas Graf:
> Thinking of netlink protocols in general and netfilter in specific,
> maintaining a list of reserved error codes for each subsystem/target/
> module will result in an unbearable pain if the error codes are not
> separated into individual namespaces for each module.
> 
> That would in turn require each module to define a unique number or
> some form of namespace resolution mechanism which does not help to keep
> things simple.
> 
> This is the main reason why I advocate the use of error strings.

I completely disagree. As I said previously, userspace has to have
knowledge of the kernel interpretation anyways.

We already have libc calls which define complex errors like:

stdtod(): if val == HUGE_VAL && errno == ERANGE: positive overflow

I see no reason why we can't define combination of attributes
and errno values for netlink messages.

Something like:

[IFLA_VLAN_ID] == NULL && errno == EINVAL: missing attribute
[IFLA_VLAN_LINK] && errno == ENODEV: lower link does not exist

and so on.

Re: Error reporting in Netlink (Re: Xtables2 Netlink spec)

[Thomas Graf]

On Fri, 2010-12-17 at 00:31 +0100, Patrick McHardy wrote: 
> I completely disagree. As I said previously, userspace has to have
> knowledge of the kernel interpretation anyways.
> 
> We already have libc calls which define complex errors like:
> 
> stdtod(): if val == HUGE_VAL && errno == ERANGE: positive overflow
> 
> I see no reason why we can't define combination of attributes
> and errno values for netlink messages.
> 
> Something like:
> 
> [IFLA_VLAN_ID] == NULL && errno == EINVAL: missing attribute
> [IFLA_VLAN_LINK] && errno == ENODEV: lower link does not exist
> 
> and so on.

Using strings would still involve including an errno as we currently
do and as I pointed out in my other mail I am also very positive
about including an offset pointer or a attribute type to specify the
attribute that caused the failure.

Maybe I am thinking too much about other netlink protocols where
errors often occur due to complicated combinations of missing attributes
and specific attribute values having special meanings. Those cases
would benefit a lot from error strings.

I really don't want to see:

return nl_errstring(ENOMEM, "Out of memory");

I am aiming at a verbose error or status string which acts as an
additional helping point in case of complicated errors. I really
don't want to replace the method of using errno to report errors
in general.

I thought about using attrs to specify the error and the reason I
did choose strings was that when we introduce new errors in the
kernel we have to update all applications. Which is no problem if
everyone uses libraries, so it is probably trivial for netfilter but
will be less trivial for netlink users as a group.

That said, I completely understand your point of view a well.

Re: Error reporting in Netlink (Re: Xtables2 Netlink spec)

[Patrick McHardy]

Am 16.12.2010 13:51, schrieb Jan Engelhardt:
> On Thursday 2010-12-16 10:57, Jesper Dangaard Brouer wrote:
> 
>> Cc.ed Thomas Graf (tgraf@redhat.com), Thomas presented some interesting 
>> ideas on netlink error-codes and strings during NetConf 2010, see:
>>
>> http://vger.kernel.org/netconf2010.html 
>> http://vger.kernel.org/netconf2010_slides/tgraf_netconf10.odp
> 
> The idea is appending an error string is ok for Netlink as a protocol
> (specification-wise), but the size constraints of the skbuffs in the
> Linux may make its practical implementation a little harder. "Half of
> the packet" is already used for the original request message, and
> cramming an extra error string may bust the space.
> It also does not look very netlinky to not use nlattrs ;-)

I agree, error strings don't look like a viable solution to me,
they are basically impossible to interpret by an application,
you run into localization issues and so on.

I'd suggest to return an errno value and the attribute causing
the error, possibly also the value (we append the original message
anyways, but in case of lists it might be hard to locate the specific
attribute). The harder cases are when a combination of multiple
attributes are responsible for the error, but still, the application
has to understand the kernel interpretation anyways, so I'd simply
return the errno and all attributes responsible. Leave interpretation
up to userspace.

> On Wed, 15 Dec 2010, Jozsef Kadlecsik came about with:
>>
>>> I have got a three-level error coding in my mind: general, standard 
>>> error codes (ENOMEM, EPERM, etc.), general netfilter specific ones 
>>> (like NFXTE_HOOKMASK_NOT_ADHERED) and table/match/target specific 
>>> ones.
>>>
>>> But I do realize that it's much easier (and therefore quite tempting) 
>>> to construct the full error message in kernel space and just send it 
>>> back. However, that'd make quite hard to support internationalization.
> 
> It's not like those strings change all that much.
> 
> 
>> To support internationalization, we could just add an error-number-code 
>> in front of the constructed error message?
> 
> Buying us what? If you change the string, but the gettext lookup is 
> based upon a number, you will get an outdated translation, which is not 
> nice either. IMHO: Better an English message than an inaccurate message.

Forget about strings, any error returned by the kernel should not only
be suitable for interpretation by a human, but also by an application.

Re: Error reporting in Netlink (Re: Xtables2 Netlink spec)

[Pablo Neira Ayuso]

On 17/12/10 00:23, Patrick McHardy wrote:
> Am 16.12.2010 13:51, schrieb Jan Engelhardt:
>> On Thursday 2010-12-16 10:57, Jesper Dangaard Brouer wrote:
>>
>>> Cc.ed Thomas Graf (tgraf@redhat.com), Thomas presented some interesting 
>>> ideas on netlink error-codes and strings during NetConf 2010, see:
>>>
>>> http://vger.kernel.org/netconf2010.html 
>>> http://vger.kernel.org/netconf2010_slides/tgraf_netconf10.odp
>>
>> The idea is appending an error string is ok for Netlink as a protocol
>> (specification-wise), but the size constraints of the skbuffs in the
>> Linux may make its practical implementation a little harder. "Half of
>> the packet" is already used for the original request message, and
>> cramming an extra error string may bust the space.
>> It also does not look very netlinky to not use nlattrs ;-)
> 
> I agree, error strings don't look like a viable solution to me,
> they are basically impossible to interpret by an application,
> you run into localization issues and so on.
> 
> I'd suggest to return an errno value and the attribute causing
> the error, possibly also the value (we append the original message
> anyways, but in case of lists it might be hard to locate the specific
> attribute). The harder cases are when a combination of multiple
> attributes are responsible for the error, but still, the application
> has to understand the kernel interpretation anyways, so I'd simply
> return the errno and all attributes responsible. Leave interpretation
> up to userspace.

I'd also prefer an int. We can define the meaning of the error numbers
in the protocol header file.