Re: [Networking] ERSPAN decapsulation drops DHCP unicast packets

Re: [Networking] ERSPAN decapsulation drops DHCP unicast packets

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 4212 bytes --]

On Sat, Aug 26, 2023 at 09:55:30PM +0200, Volodymyr Litovka wrote:
> Hi colleagues,
> 
> I'm trying to catch and process (in 3rd party analytics app) DHCP packets
> from ERSPAN session, but cannot do this due to absence of DHCP unicast
> packets after decapsulation.
> 
> The model is pretty simple: there is PHY interface (enp2s0) which receive
> ERSPAN traffic and erspan-type interface to get decapsulated packets
> (inspan, created using command "ip link add inspan type erspan seq key 10
> local 10.171.165.65 erspan_ver 1", where 10.171.165.65 is ERSPAN target).
> Then I'm going to rewrite headers in the proper ways (nftable's netdev
> family) and forward packets to the pool of workers.
> 
> Having this, I'm expecting everything, which is encapsulated inside ERSPAN,
> on 'inspan' interface. And there is _almost_ everything except DHCP unicast
> packets - tcpdump shows about 1kps on this interface of decapsulated
> packets, but no DHCP unicast (see below traces).
> 
> To avoid any interactions, I removed and disabled everything that can catch
> DHCP in userspace - systemd-networkd, netplan, dhcp-client. There is no DHCP
> server and ifupdown - for test purposes, I'm bringing networking manually.
> Apparmor disabled as well. Kernel (Linux 5.19.0-42-generic
> #43~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC) compiled without CONFIG_IP_PNP
> (according to /boot/config-5.19.0-42-generic). Nothing in userspace listens
> on UDP/68 and UDP/67:

Can you reproduce this on latest mainline?

> 
> # netstat -tunlpa
> Active Internet connections (servers and established)
> Proto Recv-Q Send-Q Local Address           Foreign Address        
> State       PID/Program name
> tcp        0      0 0.0.0.0:22 0.0.0.0:*               LISTEN      544/sshd:
> /usr/sbin
> tcp6       0      0 :::22 :::*                    LISTEN      544/sshd:
> /usr/sbin
> 
> I have no ideas, why this is happening. Decapsulation itself works, but
> particular kind of packets get lost.
> 
> I will appreciate if anyone can help me understand where is the bug - in my
> configuration or somewhere inside the kernel?
> 
> Evidence of traffic presence/absence is below.
> 
> Thank you.
> 
> Encapsulated ERSPAN session (udp and port 67/68) contains lot of different
> kinds of DHCP packets:
> 
> # tcpdump -s0 -w- -i enp2s0 'proto gre and ether[73:1]=17 and
> (ether[84:2]=67 or ether[84:2]=68)' | tshark -r- -l
>  [ ... ]
>     7   0.001942  0.0.0.0 → 255.255.255.255 DHCP 392 DHCP Discover -
> Transaction ID 0x25c096fc
>     8   0.003432  z.z.z.z → a.a.a.a         DHCP 418 DHCP ACK      -
> Transaction ID 0x5515126a
>     9   0.005170  m.m.m.m → z.z.z.z         DHCP 435 DHCP Discover -
> Transaction ID 0xa7b7
>    10   0.005171  m.m.m.m → z.z.z.z         DHCP 435 DHCP Discover -
> Transaction ID 0xa7b7
>    11   0.015399  n.n.n.n → z.z.z.z         DHCP 690 DHCP Request  -
> Transaction ID 0x54955233
>    12   0.025537  z.z.z.z → n.n.n.n         DHCP 420 DHCP ACK      -
> Transaction ID 0x54955233
>    13   0.030313  z.z.z.z → m.m.m.m         DHCP 413 DHCP Offer    -
> Transaction ID 0xa7b7
> 
> but decapsulated traffic (which I'm seeing on inspan interface) contains
> just the following:
> 
> # tcpdump -i inspan 'port 67 or port 68'
> listening on inspan, link-type EN10MB (Ethernet), snapshot length 262144
> bytes
> 17:23:36.540721 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
> 17:23:39.760036 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
> 17:23:44.135711 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
> 17:23:52.008504 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
> 

What hardware?

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [Networking] ERSPAN decapsulation drops DHCP unicast packets

[Volodymyr Litovka]

Hi Bagas,

this tested on:

- 5.19.0-42 on Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection
   -- this is host hardware
- 6.2.0-32 on Virtio network device (under KVM 6.2 on host hardware above)
- 6.5.0-060500rc7 on Virtio network device (under KVM 6.2 on host 
hardware above)

Result is the same for all cases.

Thank you.

On 8/27/23 04:07, Bagas Sanjaya wrote:
> On Sat, Aug 26, 2023 at 09:55:30PM +0200, Volodymyr Litovka wrote:
>> Hi colleagues,
>>
>> I'm trying to catch and process (in 3rd party analytics app) DHCP packets
>> from ERSPAN session, but cannot do this due to absence of DHCP unicast
>> packets after decapsulation.
>>
>> The model is pretty simple: there is PHY interface (enp2s0) which receive
>> ERSPAN traffic and erspan-type interface to get decapsulated packets
>> (inspan, created using command "ip link add inspan type erspan seq key 10
>> local 10.171.165.65 erspan_ver 1", where 10.171.165.65 is ERSPAN target).
>> Then I'm going to rewrite headers in the proper ways (nftable's netdev
>> family) and forward packets to the pool of workers.
>>
>> Having this, I'm expecting everything, which is encapsulated inside ERSPAN,
>> on 'inspan' interface. And there is _almost_ everything except DHCP unicast
>> packets - tcpdump shows about 1kps on this interface of decapsulated
>> packets, but no DHCP unicast (see below traces).
>>
>> To avoid any interactions, I removed and disabled everything that can catch
>> DHCP in userspace - systemd-networkd, netplan, dhcp-client. There is no DHCP
>> server and ifupdown - for test purposes, I'm bringing networking manually.
>> Apparmor disabled as well. Kernel (Linux 5.19.0-42-generic
>> #43~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC) compiled without CONFIG_IP_PNP
>> (according to /boot/config-5.19.0-42-generic). Nothing in userspace listens
>> on UDP/68 and UDP/67:
> Can you reproduce this on latest mainline?
>
>> # netstat -tunlpa
>> Active Internet connections (servers and established)
>> Proto Recv-Q Send-Q Local Address           Foreign Address
>> State       PID/Program name
>> tcp        0      0 0.0.0.0:22 0.0.0.0:*               LISTEN      544/sshd:
>> /usr/sbin
>> tcp6       0      0 :::22 :::*                    LISTEN      544/sshd:
>> /usr/sbin
>>
>> I have no ideas, why this is happening. Decapsulation itself works, but
>> particular kind of packets get lost.
>>
>> I will appreciate if anyone can help me understand where is the bug - in my
>> configuration or somewhere inside the kernel?
>>
>> Evidence of traffic presence/absence is below.
>>
>> Thank you.
>>
>> Encapsulated ERSPAN session (udp and port 67/68) contains lot of different
>> kinds of DHCP packets:
>>
>> # tcpdump -s0 -w- -i enp2s0 'proto gre and ether[73:1]=17 and
>> (ether[84:2]=67 or ether[84:2]=68)' | tshark -r- -l
>>   [ ... ]
>>      7   0.001942  0.0.0.0 → 255.255.255.255 DHCP 392 DHCP Discover -
>> Transaction ID 0x25c096fc
>>      8   0.003432  z.z.z.z → a.a.a.a         DHCP 418 DHCP ACK      -
>> Transaction ID 0x5515126a
>>      9   0.005170  m.m.m.m → z.z.z.z         DHCP 435 DHCP Discover -
>> Transaction ID 0xa7b7
>>     10   0.005171  m.m.m.m → z.z.z.z         DHCP 435 DHCP Discover -
>> Transaction ID 0xa7b7
>>     11   0.015399  n.n.n.n → z.z.z.z         DHCP 690 DHCP Request  -
>> Transaction ID 0x54955233
>>     12   0.025537  z.z.z.z → n.n.n.n         DHCP 420 DHCP ACK      -
>> Transaction ID 0x54955233
>>     13   0.030313  z.z.z.z → m.m.m.m         DHCP 413 DHCP Offer    -
>> Transaction ID 0xa7b7
>>
>> but decapsulated traffic (which I'm seeing on inspan interface) contains
>> just the following:
>>
>> # tcpdump -i inspan 'port 67 or port 68'
>> listening on inspan, link-type EN10MB (Ethernet), snapshot length 262144
>> bytes
>> 17:23:36.540721 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
>> 17:23:39.760036 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
>> 17:23:44.135711 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
>> 17:23:52.008504 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
>>
> What hardware?
>
-- 
Volodymyr Litovka
   "Vision without Execution is Hallucination." -- Thomas Edison

Re: [Networking] ERSPAN decapsulation drops DHCP unicast packets

[Volodymyr Litovka]

Hi colleagues,

sorry bothering you, but can anyone shed light on this issue? This stops 
me and I will be glad to hear, where I'm wrong and/or where try to look 
into the problem.

Thank you very much.

On 8/27/23 10:34, Volodymyr Litovka wrote:
> Hi Bagas,
>
> this tested on:
>
> - 5.19.0-42 on Intel 82599ES 10-Gigabit SFI/SFP+ Network Connection
>   -- this is host hardware
> - 6.2.0-32 on Virtio network device (under KVM 6.2 on host hardware 
> above)
> - 6.5.0-060500rc7 on Virtio network device (under KVM 6.2 on host 
> hardware above)
>
> Result is the same for all cases.
>
> Thank you.
>
> On 8/27/23 04:07, Bagas Sanjaya wrote:
>> On Sat, Aug 26, 2023 at 09:55:30PM +0200, Volodymyr Litovka wrote:
>>> Hi colleagues,
>>>
>>> I'm trying to catch and process (in 3rd party analytics app) DHCP 
>>> packets
>>> from ERSPAN session, but cannot do this due to absence of DHCP unicast
>>> packets after decapsulation.
>>>
>>> The model is pretty simple: there is PHY interface (enp2s0) which 
>>> receive
>>> ERSPAN traffic and erspan-type interface to get decapsulated packets
>>> (inspan, created using command "ip link add inspan type erspan seq 
>>> key 10
>>> local 10.171.165.65 erspan_ver 1", where 10.171.165.65 is ERSPAN 
>>> target).
>>> Then I'm going to rewrite headers in the proper ways (nftable's netdev
>>> family) and forward packets to the pool of workers.
>>>
>>> Having this, I'm expecting everything, which is encapsulated inside 
>>> ERSPAN,
>>> on 'inspan' interface. And there is _almost_ everything except DHCP 
>>> unicast
>>> packets - tcpdump shows about 1kps on this interface of decapsulated
>>> packets, but no DHCP unicast (see below traces).
>>>
>>> To avoid any interactions, I removed and disabled everything that 
>>> can catch
>>> DHCP in userspace - systemd-networkd, netplan, dhcp-client. There is 
>>> no DHCP
>>> server and ifupdown - for test purposes, I'm bringing networking 
>>> manually.
>>> Apparmor disabled as well. Kernel (Linux 5.19.0-42-generic
>>> #43~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC) compiled without CONFIG_IP_PNP
>>> (according to /boot/config-5.19.0-42-generic). Nothing in userspace 
>>> listens
>>> on UDP/68 and UDP/67:
>> Can you reproduce this on latest mainline?
>>
>>> # netstat -tunlpa
>>> Active Internet connections (servers and established)
>>> Proto Recv-Q Send-Q Local Address           Foreign Address
>>> State       PID/Program name
>>> tcp        0      0 0.0.0.0:22 0.0.0.0:* LISTEN      544/sshd:
>>> /usr/sbin
>>> tcp6       0      0 :::22 :::*                    LISTEN 544/sshd:
>>> /usr/sbin
>>>
>>> I have no ideas, why this is happening. Decapsulation itself works, but
>>> particular kind of packets get lost.
>>>
>>> I will appreciate if anyone can help me understand where is the bug 
>>> - in my
>>> configuration or somewhere inside the kernel?
>>>
>>> Evidence of traffic presence/absence is below.
>>>
>>> Thank you.
>>>
>>> Encapsulated ERSPAN session (udp and port 67/68) contains lot of 
>>> different
>>> kinds of DHCP packets:
>>>
>>> # tcpdump -s0 -w- -i enp2s0 'proto gre and ether[73:1]=17 and
>>> (ether[84:2]=67 or ether[84:2]=68)' | tshark -r- -l
>>>   [ ... ]
>>>      7   0.001942  0.0.0.0 → 255.255.255.255 DHCP 392 DHCP Discover -
>>> Transaction ID 0x25c096fc
>>>      8   0.003432  z.z.z.z → a.a.a.a         DHCP 418 DHCP ACK      -
>>> Transaction ID 0x5515126a
>>>      9   0.005170  m.m.m.m → z.z.z.z         DHCP 435 DHCP Discover -
>>> Transaction ID 0xa7b7
>>>     10   0.005171  m.m.m.m → z.z.z.z         DHCP 435 DHCP Discover -
>>> Transaction ID 0xa7b7
>>>     11   0.015399  n.n.n.n → z.z.z.z         DHCP 690 DHCP Request  -
>>> Transaction ID 0x54955233
>>>     12   0.025537  z.z.z.z → n.n.n.n         DHCP 420 DHCP ACK      -
>>> Transaction ID 0x54955233
>>>     13   0.030313  z.z.z.z → m.m.m.m         DHCP 413 DHCP Offer    -
>>> Transaction ID 0xa7b7
>>>
>>> but decapsulated traffic (which I'm seeing on inspan interface) 
>>> contains
>>> just the following:
>>>
>>> # tcpdump -i inspan 'port 67 or port 68'
>>> listening on inspan, link-type EN10MB (Ethernet), snapshot length 
>>> 262144
>>> bytes
>>> 17:23:36.540721 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>>> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
>>> 17:23:39.760036 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>>> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
>>> 17:23:44.135711 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>>> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
>>> 17:23:52.008504 IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP,
>>> Request from 00:1a:64:33:8d:fa (oui Unknown), length 300
>>>
>> What hardware?
>>
-- 
Volodymyr Litovka
   "Vision without Execution is Hallucination." -- Thomas Edison