OVS VXLAN decap rule has full match on TTL for the outer headers?

OVS VXLAN decap rule has full match on TTL for the outer headers?

[Or Gerlitz]

Hi Joe/Jesse,

We've noticed that VXLAN decap rules set by OVS in the below trivial 
VXLAN config contain full match on TTL=64 for the outer headers, can you 
explain the reasoning behind it? is that justa typo in dumping the flow?

I also noticed that on my systems (upstream kernel 4.3.0-rc6+, veth 
emulating a VM network 192.168.52/24 and host network 192.168.31/24, ovs 
user-space 2.3.2) something is broken in the encap rule reporting, 
traffic goes fine (below)

I tried downgrading the ovs 2.0.90 and ovs-dpctl dump-flows crashes, the 
core dump (...) doesn't say much.

Is there a kernel patch that can assist here? if not, what user-space 
version you recommend to make that dumping work
better?

Or.

# ovs-vsctl show
0ea2d6c6-93d0-4e5d-ad99-d47213bb0bf1
     Bridge ovs-tun
         Port ovs-tun
             Interface ovs-tun
                 type: internal
         Port "vxlan0"
             Interface "vxlan0"
                 type: vxlan
                 options: {dst_port="4789", key="98", 
remote_ip="192.168.31.18"}
         Port "veth1"
             Interface "veth1"
     ovs_version: "2.3.2"

# ovs-dpctl show
system@ovs-system:
         lookups: hit:41456 missed:1364 lost:15
         flows: 4
         masks: hit:54475 total:6 hit/pkt:1.27
         port 0: ovs-system (internal)
         port 1: ovs-tun (internal)
         port 2: vxlan_sys_4789 (vxlan: df_default=false, ttl=0)
         port 3: veth1

# ovs-dpctl dump-flows

decap rule:

recirc_id(0),skb_priority(0),tunnel(tun_id=0x62,src=192.168.31.18,dst=192.168.31.17,tos=0x0,ttl=64,flags(key)),in_port(2),skb_mark(0),eth(src=9e:1e:90:87:27:1a,dst=ce:57:32:ec:06:1a),eth_type(0x0800),ipv4(src=192.168.52.18/0.0.0.0,dst=192.168.52.17/0.0.0.0,proto=1/0,tos=0/0,ttl=64/0,frag=no/0xff), 
packets:3, bytes:294, used:0.012s, actions:3

encap rule:

recirc_id(0),skb_priority(0),in_port(3),eth(src=ce:57:32:ec:06:1a,dst=9e:1e:90:87:27:1a),eth_type(0x0800),ipv4(src=192.168.52.17/0.0.0.0,dst=192.168.52.18/0.0.0.0,proto=1/0,tos=0/0x3,ttl=64/0,frag=no/0xff), 
packets:2, bytes:196, used:0.012s, actions:set(unspec(bad key length 8, 
expected -1)(00 00 00 00 00 00 00 62)),2


[root@r-dcs54 vxlan]# tcpdump -nnei veth0 icmp -c 2
16:41:06.037788 0e:35:5a:35:13:5c > 9e:1e:90:87:27:1a, ethertype IPv4 
(0x0800), length 98: 192.168.52.17 > 192.168.52.18: ICMP echo request, 
id 5946, seq 1566, length 64
16:41:06.037903 9e:1e:90:87:27:1a > 0e:35:5a:35:13:5c, ethertype IPv4 
(0x0800), length 98: 192.168.52.18 > 192.168.52.17: ICMP echo reply, id 
5946, seq 1566, length 64


[root@r-dcs54 vxlan]# tcpdump -nnei eth3 udp -c 2
16:40:56.037061 00:02:c9:e9:bf:32 > f4:52:14:01:da:82, ethertype IPv4 
(0x0800), length 148: 192.168.31.17.51757 > 192.168.31.18.4789: UDP, 
length 106
16:40:56.037121 f4:52:14:01:da:82 > 00:02:c9:e9:bf:32, ethertype IPv4 
(0x0800), length 148: 192.168.31.18.53633 > 192.168.31.17.4789: UDP, 
length 106

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Jesse Gross]

On Wed, Nov 11, 2015 at 6:47 AM, Or Gerlitz <ogerlitz@mellanox.com> wrote:
> Hi Joe/Jesse,
>
> We've noticed that VXLAN decap rules set by OVS in the below trivial VXLAN
> config contain full match on TTL=64 for the outer headers, can you explain
> the reasoning behind it? is that justa typo in dumping the flow?

Looking at the code, this does indeed seem to be the case. To the best
of my knowledge, there was no particular reasoning behind it - it
simply was never an issue so the wildcard generation wasn't improved.

> I also noticed that on my systems (upstream kernel 4.3.0-rc6+, veth
> emulating a VM network 192.168.52/24 and host network 192.168.31/24, ovs
> user-space 2.3.2) something is broken in the encap rule reporting, traffic
> goes fine (below)

That is pretty strange, I have never seen that before. It seems that
the key to be set is being reported as UNSPEC, which is weird if
traffic is flowing normally. I guess you could try a newer version of
userspace but I'm not aware of any patches that went in that would
have fixed a bug in this area.

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Or Gerlitz]

On Thu, Nov 12, 2015 at 12:44 AM, Jesse Gross <jesse@kernel.org> wrote:
> On Wed, Nov 11, 2015 at 6:47 AM, Or Gerlitz <ogerlitz@mellanox.com> wrote:
>> Hi Joe/Jesse,
>>
>> We've noticed that VXLAN decap rules set by OVS in the below trivial VXLAN
>> config contain full match on TTL=64 for the outer headers, can you explain
>> the reasoning behind it? is that justa typo in dumping the flow?
>
> Looking at the code, this does indeed seem to be the case. To the best
> of my knowledge, there was no particular reasoning behind it - it
> simply was never an issue so the wildcard generation wasn't improved.

so all of the multi-million vxlan packets that were ever decapsulatd by OVS
arrived with TTL=X=64, this is impossible... when tunneled traffic goes
through routers, they change TTL of the outer packet, right?

so if the was a router between my hosts, the packet would arrive with TTL-63, we
still miss something here.

08:09:32.704276 00:02:c9:e9:bf:32 > f4:52:14:01:da:82, ethertype IPv4
(0x0800), length 148: (tos 0x0, ttl 64, id 31703, offset 0, flags
[DF], proto UDP (17), length 134)
    192.168.31.17.51757 > 192.168.31.18.4789: [no cksum] UDP, length 106
08:09:32.704323 f4:52:14:01:da:82 > 00:02:c9:e9:bf:32, ethertype IPv4
(0x0800), length 148: (tos 0x0, ttl 64, id 45301, offset 0, flags
[DF], proto UDP (17), length 134)
    192.168.31.18.53633 > 192.168.31.17.4789: [no cksum] UDP, length 106


>> I also noticed that on my systems (upstream kernel 4.3.0-rc6+, veth
>> emulating a VM network 192.168.52/24 and host network 192.168.31/24, ovs
>> user-space 2.3.2) something is broken in the encap rule reporting, traffic
>> goes fine (below)

> That is pretty strange, I have never seen that before. It seems that
> the key to be set is being reported as UNSPEC, which is weird if
> traffic is flowing normally. I guess you could try a newer version of
> userspace but I'm not aware of any patches that went in that would
> have fixed a bug in this area.

Seems as clear regression here w.r.t inter-operability between OVS user/kernel,
user space 2.3.2 which is listed in the web site as the latest release
on the LTS branch reports junk,
user space 2.0.90 crashes...

looking in the kernel logs when I did the runs with 2.3.2 I do see
this prints few times

openvswitch: netlink: Key type 62 is out of range max 26

and later the below "Dropping previously announced user features" warning

So I now moved to 2.4.0, and things aren't much better... can you give
a quick try on
your systems for upstream kernel against upstream OVS w.r.t to simple
VXLAN config?

Or.


[root@r-dcs54 ~]# ovs-vsctl show
0ea2d6c6-93d0-4e5d-ad99-d47213bb0bf1
    Bridge ovs-tun
        Port ovs-tun
            Interface ovs-tun
                type: internal
        Port "vxlan0"
            Interface "vxlan0"
                type: vxlan
                options: {dst_port="4789", key="98", remote_ip="192.168.31.18"}
        Port "veth1"
            Interface "veth1"
    ovs_version: "2.4.0"

[root@r-dcs54 ~]# ovs-dpctl show
system@ovs-system:
        lookups: hit:53786 missed:1835 lost:22
        flows: 4
        masks: hit:99881 total:14 hit/pkt:1.80
        port 0: ovs-system (internal)
        port 1: ovs-tun (internal)
        port 2: vxlan_sys_4789 (vxlan)
        port 3: veth1

[root@r-dcs54 ~]# ovs-dpctl dump-flows
recirc_id(0),in_port(3),eth(src=02:d3:6e:35:59:35,dst=9e:1e:90:87:27:1a),eth_type(0x0800),ipv4(tos=0/0x3,frag=no),
packets:111, bytes:10878, used:0.569s, actions:set(unspec(bad key
length 8, expected 0)(00 00 00 00 00 00 00 62)),2

recirc_id(0),tunnel(tun_id=0x62,src=192.168.31.18,dst=192.168.31.17,ttl=64,flags(-df-csum+key)),in_port(2),skb_mark(0),eth(src=9e:1e:90:87:27:1a,dst=02:d3:6e:35:59:35),eth_type(0x0800),ipv4(frag=no),
packets:111, bytes:10878, used:0.569s, actions:3

recirc_id(0),in_port(3),eth(src=02:d3:6e:35:59:35,dst=9e:1e:90:87:27:1a),eth_type(0x0806),
packets:0, bytes:0, used:never, actions:set(unspec(bad key length 8,
expected 0)(00 00 00 00 00 00 00 62)),2

recirc_id(0),tunnel(tun_id=0x62,src=192.168.31.18,dst=192.168.31.17,ttl=64,flags(-df-csum+key)),in_port(2),skb_mark(0),eth(src=9e:1e:90:87:27:1a,dst=02:d3:6e:35:59:35),eth_type(0x0806),
packets:0, bytes:0, used:never, actions:3

[root@r-dcs54 ~]# tcpdump -nnei veth1 icmp -c 2
08:32:34.636523 02:d3:6e:35:59:35 > 9e:1e:90:87:27:1a, ethertype IPv4
(0x0800), length 98: 192.168.52.17 > 192.168.52.18: ICMP echo request,
id 61217, seq 128, length 64
08:32:34.636642 9e:1e:90:87:27:1a > 02:d3:6e:35:59:35, ethertype IPv4
(0x0800), length 98: 192.168.52.18 > 192.168.52.17: ICMP echo reply,
id 61217, seq 128, length 64

[root@r-dcs54 ~]# tcpdump -nnei eth3 udp -c 2
08:32:24.636522 00:02:c9:e9:bf:32 > f4:52:14:01:da:82, ethertype IPv4
(0x0800), length 148: 192.168.31.17.51757 > 192.168.31.18.4789: UDP,
length 106
08:32:24.636575 f4:52:14:01:da:82 > 00:02:c9:e9:bf:32, ethertype IPv4
(0x0800), length 148: 192.168.31.18.53633 > 192.168.31.17.4789: UDP,
length 106


Or.


------------[ cut here ]------------
WARNING: CPU: 3 PID: 16953 at net/openvswitch/datapath.c:1518
ovs_dp_cmd_new+0x1e0/0x3b8 [openvswitch]()
Dropping previously announced user features
Modules linked in: vport_vxlan openvswitch nf_conntrack crc32c_generic
libcrc32c veth netconsole nfsv3 nfs_acl auth_rpcgss oid_registry nfsv4
nfs lockd grace autofs4
 sunrpc rdma_ucm rdma_cm iw_cm ib_ipoib ib_cm ib_uverbs ib_umad
mlx4_en mlx4_ib vxlan udp_tunnel ip6_udp_tunnel ib_sa ib_mad ib_core
ib_addr ipv6 mlx4_core dm_mirror
 dm_region_hash dm_log dm_mod fjes igb dca ptp pps_core sg joydev
microcode rng_core pcspkr ehci_pci ehci_hcd shpchp button sr_mod ext4
jbd2 crc16 usb_storage floppy
 sd_mod ata_generic ata_piix libata scsi_mod uhci_hcd radeon ttm hwmon
[last unloaded: mst_pci]
CPU: 3 PID: 16953 Comm: ovs-vswitchd Tainted: G           O    4.3.0-rc6+ #553
Hardware name: Supermicro X7DWU/X7DWU, BIOS  1.1 04/30/2008
 00000000000005ee ffff88022506b968 ffffffff81216d8b ffffffffa0518068
 ffff88022506b9b8 ffff88022506b9a8 ffffffff81047324 ffff88022506b9a8
 ffffffffa050d8f5 ffff8802144baa80 ffff8802146f0e80 ffff88022506bae8
Call Trace:
 [<ffffffff81216d8b>] dump_stack+0x4f/0x68
 [<ffffffff81047324>] warn_slowpath_common+0xa1/0xbb
 [<ffffffffa050d8f5>] ? ovs_dp_cmd_new+0x1e0/0x3b8 [openvswitch]
 [<ffffffff810473e1>] warn_slowpath_fmt+0x46/0x5f
 [<ffffffffa050d6ef>] ? lookup_datapath.clone.3+0x130/0x156 [openvswitch]
 [<ffffffffa050d8f5>] ovs_dp_cmd_new+0x1e0/0x3b8 [openvswitch]
 [<ffffffff81239705>] ? nla_parse+0x4f/0xec
 [<ffffffff8141d3d0>] genl_family_rcv_msg+0x272/0x2ce
 [<ffffffff81418779>] ? __netlink_alloc_skb+0x1a/0x1a
 [<ffffffff8141c90b>] ? genl_rcv+0x1d/0x3c
 [<ffffffff8141d497>] genl_rcv_msg+0x6b/0x8e
 [<ffffffff8141d42c>] ? genl_family_rcv_msg+0x2ce/0x2ce
 [<ffffffff8141c077>] netlink_rcv_skb+0x43/0x9a
 [<ffffffff8141c90b>] ? genl_rcv+0x1d/0x3c
 [<ffffffff8141c91a>] genl_rcv+0x2c/0x3c
 [<ffffffff8141b625>] netlink_unicast+0x108/0x196
 [<ffffffff81121c36>] ? __might_fault+0x44/0x94
 [<ffffffff8141bc9d>] netlink_sendmsg+0x2da/0x329
 [<ffffffff813d5cf4>] sock_sendmsg+0x31/0x41
 [<ffffffff813d5e4b>] ___sys_sendmsg+0x147/0x1c9
 [<ffffffff8141bcb0>] ? netlink_sendmsg+0x2ed/0x329
 [<ffffffff813d6118>] ? SyS_sendto+0xe2/0x10a
 [<ffffffff81121c7f>] ? __might_fault+0x8d/0x94
 [<ffffffff8116d9b0>] ? __fget_light+0x56/0x81
 [<ffffffff813d4926>] ? move_addr_to_user+0x83/0xa8
 [<ffffffff813d6005>] __sys_sendmsg+0x42/0x63
 [<ffffffff813d6034>] SyS_sendmsg+0xe/0x10
 [<ffffffff814aadd7>] entry_SYSCALL_64_fastpath+0x12/0x6f
---[ end trace f1594c69e4486257 ]---

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Joe Stringer]

On 11 November 2015 at 22:34, Or Gerlitz <gerlitz.or@gmail.com> wrote:
> On Thu, Nov 12, 2015 at 12:44 AM, Jesse Gross <jesse@kernel.org> wrote:
>> On Wed, Nov 11, 2015 at 6:47 AM, Or Gerlitz <ogerlitz@mellanox.com> wrote:
>>> Hi Joe/Jesse,
>>>
>>> We've noticed that VXLAN decap rules set by OVS in the below trivial VXLAN
>>> config contain full match on TTL=64 for the outer headers, can you explain
>>> the reasoning behind it? is that justa typo in dumping the flow?
>>
>> Looking at the code, this does indeed seem to be the case. To the best
>> of my knowledge, there was no particular reasoning behind it - it
>> simply was never an issue so the wildcard generation wasn't improved.
>
> so all of the multi-million vxlan packets that were ever decapsulatd by OVS
> arrived with TTL=X=64, this is impossible... when tunneled traffic goes
> through routers, they change TTL of the outer packet, right?
>
> so if the was a router between my hosts, the packet would arrive with TTL-63, we
> still miss something here.

I don't follow the logic. You observed one flow which matched on
TTL=64, therefore all vxlan packets terminated at OVS have TTL=64?

If OVS received packets with different TTLs, they would miss and
ovs-vswitchd would generate flows to match that traffic too. If that
becomes an issue, presumably the wildcard generation can be improved.

> 08:09:32.704276 00:02:c9:e9:bf:32 > f4:52:14:01:da:82, ethertype IPv4
> (0x0800), length 148: (tos 0x0, ttl 64, id 31703, offset 0, flags
> [DF], proto UDP (17), length 134)
>     192.168.31.17.51757 > 192.168.31.18.4789: [no cksum] UDP, length 106
> 08:09:32.704323 f4:52:14:01:da:82 > 00:02:c9:e9:bf:32, ethertype IPv4
> (0x0800), length 148: (tos 0x0, ttl 64, id 45301, offset 0, flags
> [DF], proto UDP (17), length 134)
>     192.168.31.18.53633 > 192.168.31.17.4789: [no cksum] UDP, length 106
>
>
>>> I also noticed that on my systems (upstream kernel 4.3.0-rc6+, veth
>>> emulating a VM network 192.168.52/24 and host network 192.168.31/24, ovs
>>> user-space 2.3.2) something is broken in the encap rule reporting, traffic
>>> goes fine (below)
>
>> That is pretty strange, I have never seen that before. It seems that
>> the key to be set is being reported as UNSPEC, which is weird if
>> traffic is flowing normally. I guess you could try a newer version of
>> userspace but I'm not aware of any patches that went in that would
>> have fixed a bug in this area.
>
> Seems as clear regression here w.r.t inter-operability between OVS user/kernel,
> user space 2.3.2 which is listed in the web site as the latest release
> on the LTS branch reports junk,
> user space 2.0.90 crashes...

I agree that this UNSPEC issue on v2.3 could do with a bit of a closer
look. I'll see if I can find some time for it. Alternatively if you're
willing and have bandwith, I'd be curious if it's related to the
masked set field feature introduced in Linux-4.0.

OVS 2.0.90 isn't an actual version number, it typically means master
at some point between v2.0 and v2.1 (unknown), and furthermore it
sounds like that issue is fixed in v2.3 so upgrading is the best
option.

> looking in the kernel logs when I did the runs with 2.3.2 I do see
> this prints few times
>
> openvswitch: netlink: Key type 62 is out of range max 26

This particular issue looks like the experimental MPLS support in
OVS2.3.x which was later reshuffled when it was merged upstream. At
that time there was no way to silently probe for features, so the MPLS
feature probe ends up causing a log message like this when you pair
OVS 2.3.x with any kernel module that didn't also have the same
experimental MPLS support (which, I'm not sure if there's any official
version released anywhere). It is harmless.

> and later the below "Dropping previously announced user features" warning

In this case it looks like you created the datapath using a newer
version of the userspace utilities, then without deleting the
datapath, attempted to reuse the datapath with an older version of the
userspace utilities. This is fine, but it warns you because it drops
particular user features which the newer userspace supported (because
the older userspace doesn't support them). Sure, it's not the most
graceful, but it doesn't look fatal in and of itself. Comment from the
code below for context:

/* An outdated user space instance that does not understand
* the concept of user_features has attempted to create a new
* datapath and is likely to reuse it. Drop all user features.
*/

> So I now moved to 2.4.0, and things aren't much better... can you give
> a quick try on
> your systems for upstream kernel against upstream OVS w.r.t to simple
> VXLAN config?

What do you mean by "not much better"? Do you mean that you still
observe one of the above three issues, or you see a different issue?
In particular I'd be curious if you observe the UNSPEC issue.

FWIW I've been regularly running a very trivial vxlan test with the
upstream (master) OVS and at least v4.3, and I haven't noticed
anything particularly unusual. That said I haven't been looking
closely for strange behaviour either.

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Or Gerlitz]

On Fri, Nov 13, 2015 at 10:14 AM, Joe Stringer <joestringer@nicira.com> wrote:

> I don't follow the logic. You observed one flow which matched on
> TTL=64, therefore all vxlan packets terminated at OVS have TTL=64?

> If OVS received packets with different TTLs, they would miss and
> ovs-vswitchd would generate flows to match that traffic too.

ok, that makes things a bit better, but (see next)

> If that becomes an issue, presumably the wildcard generation can be improved.

is there a deep reason for vlxan "learned flows" to actually match w
or w.o wild cards on TTLs?? for non-tunneled flow I don't see  this
happening.


> I agree that this UNSPEC issue on v2.3 could do with a bit of a closer
> look. I'll see if I can find some time for it. Alternatively if you're
> willing and have bandwith, I'd be curious if it's related to the
> masked set field feature introduced in Linux-4.0.

so what would you suggest here? run with 3.19 or earlier?


> In this case it looks like you created the datapath using a newer
> version of the userspace utilities, then without deleting the
> datapath, attempted to reuse the datapath with an older version of the
> userspace utilities. This is fine, but it warns you because it drops
> particular user features which the newer userspace supported (because
> the older userspace doesn't support them). Sure, it's not the most
> graceful, but it doesn't look fatal in and of itself. Comment from the
> code below for context:
>
> /* An outdated user space instance that does not understand
> * the concept of user_features has attempted to create a new
> * datapath and is likely to reuse it. Drop all user features.
> */

thanks for the clarification/s, yes, I tried few user-space versions
one after the other, possibly w.o deleting the datapath

>> So I now moved to 2.4.0, and things aren't much better... can you give
>> a quick try on
>> your systems for upstream kernel against upstream OVS w.r.t to simple
>> VXLAN config?

> What do you mean by "not much better"? Do you mean that you still
> observe one of the above three issues, or you see a different issue?
> In particular I'd be curious if you observe the UNSPEC issue.

I mean this is printed by ovs-dpctl dump-flows for the encap rule

recirc_id(0),in_port(3),eth(src=02:d3:6e:35:59:35,dst=9e:1e:90:87:27:1a),eth_type(0x0800),ipv4(tos=0/0x3,frag=no),
packets:111, bytes:10878, used:0.569s, actions:set(unspec(bad key
length 8, expected 0)(00 00 00 00 00 00 00 62)),2

Or.

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Joe Stringer]

On 13 November 2015 at 06:46, Or Gerlitz <gerlitz.or@gmail.com> wrote:
> On Fri, Nov 13, 2015 at 10:14 AM, Joe Stringer <joestringer@nicira.com> wrote:
>
>> I don't follow the logic. You observed one flow which matched on
>> TTL=64, therefore all vxlan packets terminated at OVS have TTL=64?
>
>> If OVS received packets with different TTLs, they would miss and
>> ovs-vswitchd would generate flows to match that traffic too.
>
> ok, that makes things a bit better, but (see next)
>
>> If that becomes an issue, presumably the wildcard generation can be improved.
>
> is there a deep reason for vlxan "learned flows" to actually match w
> or w.o wild cards on TTLs?? for non-tunneled flow I don't see  this
> happening.

No deep reason I'm aware of.

>> I agree that this UNSPEC issue on v2.3 could do with a bit of a closer
>> look. I'll see if I can find some time for it. Alternatively if you're
>> willing and have bandwith, I'd be curious if it's related to the
>> masked set field feature introduced in Linux-4.0.
>
> so what would you suggest here? run with 3.19 or earlier?

On second thought, I think if it were related to that then userspace
v2.4 would not exhibit the problem. I'd have to dig in to see why it
occurs.

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Or Gerlitz]

On Sat, Nov 14, 2015 at 8:45 AM, Joe Stringer <joestringer@nicira.com> wrote:
> On 13 November 2015 at 06:46, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>> On Fri, Nov 13, 2015 at 10:14 AM, Joe Stringer <joestringer@nicira.com> wrote:

>>> I agree that this UNSPEC issue on v2.3 could do with a bit of a closer
>>> look. I'll see if I can find some time for it. Alternatively if you're
>>> willing and have bandwith, I'd be curious if it's related to the
>>> masked set field feature introduced in Linux-4.0.

>> so what would you suggest here? run with 3.19 or earlier?

> On second thought, I think if it were related to that then userspace
> v2.4 would not exhibit the problem. I'd have to dig in to see why it
> occurs.

Any findings?

Or.

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Or Gerlitz]

On Thu, Nov 19, 2015 at 5:40 PM, Or Gerlitz <gerlitz.or@gmail.com> wrote:
> On Sat, Nov 14, 2015 at 8:45 AM, Joe Stringer <joestringer@nicira.com> wrote:
>> On 13 November 2015 at 06:46, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>> On Fri, Nov 13, 2015 at 10:14 AM, Joe Stringer <joestringer@nicira.com> wrote:
>
>>>> I agree that this UNSPEC issue on v2.3 could do with a bit of a closer
>>>> look. I'll see if I can find some time for it. Alternatively if you're
>>>> willing and have bandwith, I'd be curious if it's related to the
>>>> masked set field feature introduced in Linux-4.0.
>
>>> so what would you suggest here? run with 3.19 or earlier?
>
>> On second thought, I think if it were related to that then userspace
>> v2.4 would not exhibit the problem. I'd have to dig in to see why it
>> occurs.
>
> Any findings?

Hi Joe/Jesse,

Apart from Haggai's note on the TTL, I am pinging you here for the 2nd
time... can you please let me know how
this goes on your end?

Or.

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Joe Stringer]

On 29 November 2015 at 05:37, Or Gerlitz <gerlitz.or@gmail.com> wrote:
> On Thu, Nov 19, 2015 at 5:40 PM, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>> On Sat, Nov 14, 2015 at 8:45 AM, Joe Stringer <joestringer@nicira.com> wrote:
>>> On 13 November 2015 at 06:46, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>>> On Fri, Nov 13, 2015 at 10:14 AM, Joe Stringer <joestringer@nicira.com> wrote:
>>
>>>>> I agree that this UNSPEC issue on v2.3 could do with a bit of a closer
>>>>> look. I'll see if I can find some time for it. Alternatively if you're
>>>>> willing and have bandwith, I'd be curious if it's related to the
>>>>> masked set field feature introduced in Linux-4.0.
>>
>>>> so what would you suggest here? run with 3.19 or earlier?
>>
>>> On second thought, I think if it were related to that then userspace
>>> v2.4 would not exhibit the problem. I'd have to dig in to see why it
>>> occurs.
>>
>> Any findings?
>
> Hi Joe/Jesse,
>
> Apart from Haggai's note on the TTL, I am pinging you here for the 2nd
> time... can you please let me know how
> this goes on your end?

Apologies for the delayed response, we haven't found anything
interesting yet although we've mostly looked at plain set-field
actions with a combination of kernel/userspace versions. I plan to
carve out some time later this week to take another look.

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Joe Stringer]

On 2 December 2015 at 10:01, Joe Stringer <joe@ovn.org> wrote:
> On 29 November 2015 at 05:37, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>> On Thu, Nov 19, 2015 at 5:40 PM, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>> On Sat, Nov 14, 2015 at 8:45 AM, Joe Stringer <joestringer@nicira.com> wrote:
>>>> On 13 November 2015 at 06:46, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>>>> On Fri, Nov 13, 2015 at 10:14 AM, Joe Stringer <joestringer@nicira.com> wrote:
>>>
>>>>>> I agree that this UNSPEC issue on v2.3 could do with a bit of a closer
>>>>>> look. I'll see if I can find some time for it. Alternatively if you're
>>>>>> willing and have bandwith, I'd be curious if it's related to the
>>>>>> masked set field feature introduced in Linux-4.0.
>>>
>>>>> so what would you suggest here? run with 3.19 or earlier?
>>>
>>>> On second thought, I think if it were related to that then userspace
>>>> v2.4 would not exhibit the problem. I'd have to dig in to see why it
>>>> occurs.
>>>
>>> Any findings?
>>
>> Hi Joe/Jesse,
>>
>> Apart from Haggai's note on the TTL, I am pinging you here for the 2nd
>> time... can you please let me know how
>> this goes on your end?
>
> Apologies for the delayed response, we haven't found anything
> interesting yet although we've mostly looked at plain set-field
> actions with a combination of kernel/userspace versions. I plan to
> carve out some time later this week to take another look.

(resending due to teething issues with new email and plain-text, sorry
for the spam)

As far as the mask, I briefly discussed this with Jarno and it seems
like it could be something as simple as zeroing the ip_ttl mask in
tnl_wc_init().

As far as the UNSPEC issue, I couldn't reproduce it. Are you sure that
there weren't multiple versions of OVS installed, and while the OVS
running was v2.3, your ovs-dpctl was an older version? I was running
latest OVS branch-2.3 (c2e761f5b152).

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Or Gerlitz]

On Tue, Dec 8, 2015 at 9:20 PM, Joe Stringer <joe@ovn.org> wrote:
> On 2 December 2015 at 10:01, Joe Stringer <joe@ovn.org> wrote:
>> On 29 November 2015 at 05:37, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>> On Thu, Nov 19, 2015 at 5:40 PM, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>>> On Sat, Nov 14, 2015 at 8:45 AM, Joe Stringer <joestringer@nicira.com> wrote:
>>>>> On 13 November 2015 at 06:46, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>>>>> On Fri, Nov 13, 2015 at 10:14 AM, Joe Stringer <joestringer@nicira.com> wrote:
>>>>
>>>>>>> I agree that this UNSPEC issue on v2.3 could do with a bit of a closer
>>>>>>> look. I'll see if I can find some time for it. Alternatively if you're
>>>>>>> willing and have bandwith, I'd be curious if it's related to the
>>>>>>> masked set field feature introduced in Linux-4.0.
>>>>
>>>>>> so what would you suggest here? run with 3.19 or earlier?
>>>>
>>>>> On second thought, I think if it were related to that then userspace
>>>>> v2.4 would not exhibit the problem. I'd have to dig in to see why it
>>>>> occurs.
>>>>
>>>> Any findings?
>>>
>>> Hi Joe/Jesse,
>>>
>>> Apart from Haggai's note on the TTL, I am pinging you here for the 2nd
>>> time... can you please let me know how
>>> this goes on your end?

>> Apologies for the delayed response, we haven't found anything
>> interesting yet although we've mostly looked at plain set-field
>> actions with a combination of kernel/userspace versions. I plan to
>> carve out some time later this week to take another look.

> (resending due to teething issues with new email and plain-text, sorry
> for the spam)

> As far as the mask, I briefly discussed this with Jarno and it seems
> like it could be something as simple as zeroing the ip_ttl mask in
> tnl_wc_init().

to make sure I follow, will that have the consequence that we (user +
kernel) will practically not be testing the ttl for these flows?

> As far as the UNSPEC issue, I couldn't reproduce it. Are you sure that
> there weren't multiple versions of OVS installed, and while the OVS
> running was v2.3, your ovs-dpctl was an older version? I was running
> latest OVS branch-2.3 (c2e761f5b152).

OK, for the sake of double checking, I will do over-all cleanup and
re-check, but will be able to do that only next week.

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Joe Stringer]

On 8 December 2015 at 13:23, Or Gerlitz <gerlitz.or@gmail.com> wrote:
> On Tue, Dec 8, 2015 at 9:20 PM, Joe Stringer <joe@ovn.org> wrote:
>> On 2 December 2015 at 10:01, Joe Stringer <joe@ovn.org> wrote:
>>> On 29 November 2015 at 05:37, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>>> On Thu, Nov 19, 2015 at 5:40 PM, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>>>> On Sat, Nov 14, 2015 at 8:45 AM, Joe Stringer <joestringer@nicira.com> wrote:
>>>>>> On 13 November 2015 at 06:46, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>>>>>> On Fri, Nov 13, 2015 at 10:14 AM, Joe Stringer <joestringer@nicira.com> wrote:
>>>>>
>>>>>>>> I agree that this UNSPEC issue on v2.3 could do with a bit of a closer
>>>>>>>> look. I'll see if I can find some time for it. Alternatively if you're
>>>>>>>> willing and have bandwith, I'd be curious if it's related to the
>>>>>>>> masked set field feature introduced in Linux-4.0.
>>>>>
>>>>>>> so what would you suggest here? run with 3.19 or earlier?
>>>>>
>>>>>> On second thought, I think if it were related to that then userspace
>>>>>> v2.4 would not exhibit the problem. I'd have to dig in to see why it
>>>>>> occurs.
>>>>>
>>>>> Any findings?
>>>>
>>>> Hi Joe/Jesse,
>>>>
>>>> Apart from Haggai's note on the TTL, I am pinging you here for the 2nd
>>>> time... can you please let me know how
>>>> this goes on your end?
>
>>> Apologies for the delayed response, we haven't found anything
>>> interesting yet although we've mostly looked at plain set-field
>>> actions with a combination of kernel/userspace versions. I plan to
>>> carve out some time later this week to take another look.
>
>> (resending due to teething issues with new email and plain-text, sorry
>> for the spam)
>
>> As far as the mask, I briefly discussed this with Jarno and it seems
>> like it could be something as simple as zeroing the ip_ttl mask in
>> tnl_wc_init().
>
> to make sure I follow, will that have the consequence that we (user +
> kernel) will practically not be testing the ttl for these flows?

Yes, it would cause userspace to 'wildcard' the field so the kernel
flows that are installed will ignore it during lookup.

>> As far as the UNSPEC issue, I couldn't reproduce it. Are you sure that
>> there weren't multiple versions of OVS installed, and while the OVS
>> running was v2.3, your ovs-dpctl was an older version? I was running
>> latest OVS branch-2.3 (c2e761f5b152).
>
> OK, for the sake of double checking, I will do over-all cleanup and
> re-check, but will be able to do that only next week.

Thanks,
Joe

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Or Gerlitz]

On Wed, Dec 9, 2015 at 2:22 AM, Joe Stringer <joe@ovn.org> wrote:
> On 8 December 2015 at 13:23, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>> On Tue, Dec 8, 2015 at 9:20 PM, Joe Stringer <joe@ovn.org> wrote:

>>>> Apologies for the delayed response, we haven't found anything
>>>> interesting yet although we've mostly looked at plain set-field
>>>> actions with a combination of kernel/userspace versions. I plan to
>>>> carve out some time later this week to take another look.

>>> (resending due to teething issues with new email and plain-text, sorry
>>> for the spam)

>>> As far as the mask, I briefly discussed this with Jarno and it seems
>>> like it could be something as simple as zeroing the ip_ttl mask in
>>> tnl_wc_init().

>> to make sure I follow, will that have the consequence that we (user +
>> kernel) will practically not be testing the ttl for these flows?

> Yes, it would cause userspace to 'wildcard' the field so the kernel
> flows that are installed will ignore it during lookup.

Cool, any chance this is gonna fit into your schedule to meet 4.4? if
not, for 4.5?

Also, can the patch be made simple/small enough to go into -stable as well?

Or.

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Joe Stringer]

On 10 December 2015 at 13:06, Or Gerlitz <gerlitz.or@gmail.com> wrote:
> On Wed, Dec 9, 2015 at 2:22 AM, Joe Stringer <joe@ovn.org> wrote:
>> On 8 December 2015 at 13:23, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>> On Tue, Dec 8, 2015 at 9:20 PM, Joe Stringer <joe@ovn.org> wrote:
>
>>>>> Apologies for the delayed response, we haven't found anything
>>>>> interesting yet although we've mostly looked at plain set-field
>>>>> actions with a combination of kernel/userspace versions. I plan to
>>>>> carve out some time later this week to take another look.
>
>>>> (resending due to teething issues with new email and plain-text, sorry
>>>> for the spam)
>
>>>> As far as the mask, I briefly discussed this with Jarno and it seems
>>>> like it could be something as simple as zeroing the ip_ttl mask in
>>>> tnl_wc_init().
>
>>> to make sure I follow, will that have the consequence that we (user +
>>> kernel) will practically not be testing the ttl for these flows?
>
>> Yes, it would cause userspace to 'wildcard' the field so the kernel
>> flows that are installed will ignore it during lookup.
>
> Cool, any chance this is gonna fit into your schedule to meet 4.4? if
> not, for 4.5?
>
> Also, can the patch be made simple/small enough to go into -stable as well?

It's a userspace change.

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Or Gerlitz]

On Thu, Dec 10, 2015 at 11:23 PM, Joe Stringer <joe@ovn.org> wrote:
> On 10 December 2015 at 13:06, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>> On Wed, Dec 9, 2015 at 2:22 AM, Joe Stringer <joe@ovn.org> wrote:

>>>>> As far as the mask, I briefly discussed this with Jarno and it seems
>>>>> like it could be something as simple as zeroing the ip_ttl mask in
>>>>> tnl_wc_init().

>>>> to make sure I follow, will that have the consequence that we (user +
>>>> kernel) will practically not be testing the ttl for these flows?

>>> Yes, it would cause userspace to 'wildcard' the field so the kernel
>>> flows that are installed will ignore it during lookup.

>> Cool, any chance this is gonna fit into your schedule to meet 4.4? if
>> not, for 4.5?
>> Also, can the patch be made simple/small enough to go into -stable as well?

> It's a userspace change.


mmm, in a downstream post of this thread [1] Haggai pointed to you
that there's code in the OVS kernel path that that rejects new tunnel
flows if they don't have the TTL mask set, so he's wrong? where?

Or.

[1] http://marc.info/?l=linux-netdev&m=144880328121156&w=2

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Joe Stringer]

On 10 December 2015 at 13:43, Or Gerlitz <gerlitz.or@gmail.com> wrote:
> On Thu, Dec 10, 2015 at 11:23 PM, Joe Stringer <joe@ovn.org> wrote:
>> On 10 December 2015 at 13:06, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>> On Wed, Dec 9, 2015 at 2:22 AM, Joe Stringer <joe@ovn.org> wrote:
>
>>>>>> As far as the mask, I briefly discussed this with Jarno and it seems
>>>>>> like it could be something as simple as zeroing the ip_ttl mask in
>>>>>> tnl_wc_init().
>
>>>>> to make sure I follow, will that have the consequence that we (user +
>>>>> kernel) will practically not be testing the ttl for these flows?
>
>>>> Yes, it would cause userspace to 'wildcard' the field so the kernel
>>>> flows that are installed will ignore it during lookup.
>
>>> Cool, any chance this is gonna fit into your schedule to meet 4.4? if
>>> not, for 4.5?
>>> Also, can the patch be made simple/small enough to go into -stable as well?
>
>> It's a userspace change.
>
>
> mmm, in a downstream post of this thread [1] Haggai pointed to you
> that there's code in the OVS kernel path that that rejects new tunnel
> flows if they don't have the TTL mask set, so he's wrong? where?
>
> Or.
>
> [1] http://marc.info/?l=linux-netdev&m=144880328121156&w=2

The rejection is within an if statement called "if (!is_mask)", so it
seems to me like it is enforcing the flow key to specify a TTL value
(any), and doesn't care what the mask does.

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Haggai Eran]

On 14/11/2015 08:45, Joe Stringer wrote:
> On 13 November 2015 at 06:46, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>> > On Fri, Nov 13, 2015 at 10:14 AM, Joe Stringer <joestringer@nicira.com> wrote:
>> >
>>> >> I don't follow the logic. You observed one flow which matched on
>>> >> TTL=64, therefore all vxlan packets terminated at OVS have TTL=64?
>> >
>>> >> If OVS received packets with different TTLs, they would miss and
>>> >> ovs-vswitchd would generate flows to match that traffic too.
>> >
>> > ok, that makes things a bit better, but (see next)
>> >
>>> >> If that becomes an issue, presumably the wildcard generation can be improved.
>> >
>> > is there a deep reason for vlxan "learned flows" to actually match w
>> > or w.o wild cards on TTLs?? for non-tunneled flow I don't see  this
>> > happening.
> No deep reason I'm aware of.

Hi,

We looked into the OVS kernel module, and apparently there's a check
that rejects new tunnel flows if they don't have the TTL mask set [1].

I was able to trace it to this commit [2] on the OVS tree, but I don't
quite understand why the check was added. There was some discussion
about the patch on the mailing list [3] that hints this was about
catching zero TTL, but it has too little context for me to understand.

I'm adding the author and reviewer of the patch, perhaps they can help
explain this requirement.

Regards,
Haggai

[1]
http://lxr.free-electrons.com/source/net/openvswitch/flow_netlink.c?v=4.3#L660
[2] datapath: More flexible kernel/userspace tunneling attribute.
    9b405f1aa8d175dc63ad3ffe5d0fe05d5ee09162
[3] http://openvswitch.org/pipermail/dev/2013-January/024573.html

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Joe Stringer]

On 29 November 2015 at 05:06, Haggai Eran <haggaie@mellanox.com> wrote:
> On 14/11/2015 08:45, Joe Stringer wrote:
>> On 13 November 2015 at 06:46, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>> > On Fri, Nov 13, 2015 at 10:14 AM, Joe Stringer <joestringer@nicira.com> wrote:
>>> >
>>>> >> I don't follow the logic. You observed one flow which matched on
>>>> >> TTL=64, therefore all vxlan packets terminated at OVS have TTL=64?
>>> >
>>>> >> If OVS received packets with different TTLs, they would miss and
>>>> >> ovs-vswitchd would generate flows to match that traffic too.
>>> >
>>> > ok, that makes things a bit better, but (see next)
>>> >
>>>> >> If that becomes an issue, presumably the wildcard generation can be improved.
>>> >
>>> > is there a deep reason for vlxan "learned flows" to actually match w
>>> > or w.o wild cards on TTLs?? for non-tunneled flow I don't see  this
>>> > happening.
>> No deep reason I'm aware of.
>
> Hi,
>
> We looked into the OVS kernel module, and apparently there's a check
> that rejects new tunnel flows if they don't have the TTL mask set [1].
>
> I was able to trace it to this commit [2] on the OVS tree, but I don't
> quite understand why the check was added. There was some discussion
> about the patch on the mailing list [3] that hints this was about
> catching zero TTL, but it has too little context for me to understand.
>
> I'm adding the author and reviewer of the patch, perhaps they can help
> explain this requirement.

I'm not the author or reviewer, but it seems like this is an attempt
to prevent flows from matching TTL=0 then proceeding to forward the
frame. You could still match a non-zero TTL with a wildcarded mask,
but it wouldn't be possible for a single flow to match all (non-zero)
TTL values.

Re: OVS VXLAN decap rule has full match on TTL for the outer headers?

[Jesse Gross]

On Wed, Dec 2, 2015 at 9:52 AM, Joe Stringer <joe@ovn.org> wrote:
> On 29 November 2015 at 05:06, Haggai Eran <haggaie@mellanox.com> wrote:
>> On 14/11/2015 08:45, Joe Stringer wrote:
>>> On 13 November 2015 at 06:46, Or Gerlitz <gerlitz.or@gmail.com> wrote:
>>>> > On Fri, Nov 13, 2015 at 10:14 AM, Joe Stringer <joestringer@nicira.com> wrote:
>>>> >
>>>>> >> I don't follow the logic. You observed one flow which matched on
>>>>> >> TTL=64, therefore all vxlan packets terminated at OVS have TTL=64?
>>>> >
>>>>> >> If OVS received packets with different TTLs, they would miss and
>>>>> >> ovs-vswitchd would generate flows to match that traffic too.
>>>> >
>>>> > ok, that makes things a bit better, but (see next)
>>>> >
>>>>> >> If that becomes an issue, presumably the wildcard generation can be improved.
>>>> >
>>>> > is there a deep reason for vlxan "learned flows" to actually match w
>>>> > or w.o wild cards on TTLs?? for non-tunneled flow I don't see  this
>>>> > happening.
>>> No deep reason I'm aware of.
>>
>> Hi,
>>
>> We looked into the OVS kernel module, and apparently there's a check
>> that rejects new tunnel flows if they don't have the TTL mask set [1].
>>
>> I was able to trace it to this commit [2] on the OVS tree, but I don't
>> quite understand why the check was added. There was some discussion
>> about the patch on the mailing list [3] that hints this was about
>> catching zero TTL, but it has too little context for me to understand.
>>
>> I'm adding the author and reviewer of the patch, perhaps they can help
>> explain this requirement.
>
> I'm not the author or reviewer, but it seems like this is an attempt
> to prevent flows from matching TTL=0 then proceeding to forward the
> frame. You could still match a non-zero TTL with a wildcarded mask,
> but it wouldn't be possible for a single flow to match all (non-zero)
> TTL values.

You can match on TTL=0 as long as you do it explicitly. The assumption
though is that this isn't what most people want to do if they don't
specify any value so it would be bad default.

However, it looks like while this was primarily targeted at the key,
the check also bled over into the mask translation as well, preventing
it from being wildcarded through omission. It's still not clear to me
that this makes any difference in the real world though.