Logging SNAT'ed connections

Logging SNAT'ed connections

[Jan Kasprzak]

Hi all,

how do you log SNAT'ed connections for further forensic purposes,
when somebody complains about your public IP address and you want to be
able to find a local user behind NAT?

I used to run something like

conntrack --event --buffer-size 1600000 -o id,timestamp --src-nat >> $LOGFILE

which did what I want. But now I tried to run the same on AlmaLinux9 host,
and it seems that --src-nat filter does not get applied - it shows
even connections which are not SNAT'ed at all, because they are for my
own IP address:

[1780936563.343940]	    [NEW] icmp     1 30 src=128.1.227.61 dst=xx.yy.zz.13 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.13 dst=128.1.227.61 type=0 code=0 id=8724 id=2525400839
[1780936563.343994]	 [UPDATE] icmp     1 30 src=128.1.227.61 dst=xx.yy.zz.13 type=8 code=0 id=8724 src=xx.yy.zz.13 dst=128.1.227.61 type=0 code=0 id=8724 id=2525400839
[1780936565.281121]	    [NEW] icmp     1 30 src=128.1.227.61 dst=xx.yy.zz.11 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.11 dst=128.1.227.61 type=0 code=0 id=8724 id=2574615308
[1780936565.281167]	 [UPDATE] icmp     1 30 src=128.1.227.61 dst=xx.yy.zz.11 type=8 code=0 id=8724 src=xx.yy.zz.11 dst=128.1.227.61 type=0 code=0 id=8724 id=2574615308

Here xx.yy.zz.11 and .13 are my public IP addresses. Why do these connections
show up in the conntrack --event --src-nat output? Thanks!

-Yenya

-- 
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| https://www.fi.muni.cz/~kas/                        GPG: 4096R/A45477D5 |
    I don't like Python; its lack of inline, anonymous, multi-statement
    functions makes me sad.                                --Eric Wastl

Re: Logging SNAT'ed connections

[Kerin Millar]

On Mon, 8 Jun 2026, at 5:38 PM, Jan Kasprzak wrote:
> Hi all,
>
> how do you log SNAT'ed connections for further forensic purposes,
> when somebody complains about your public IP address and you want to be
> able to find a local user behind NAT?
>
> I used to run something like
>
> conntrack --event --buffer-size 1600000 -o id,timestamp --src-nat >> $LOGFILE
>
> which did what I want. But now I tried to run the same on AlmaLinux9 host,
> and it seems that --src-nat filter does not get applied - it shows
> even connections which are not SNAT'ed at all, because they are for my
> own IP address:
>
> [1780936563.343940]	    [NEW] icmp     1 30 src=128.1.227.61 
> dst=xx.yy.zz.13 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.13 
> dst=128.1.227.61 type=0 code=0 id=8724 id=2525400839
> [1780936563.343994]	 [UPDATE] icmp     1 30 src=128.1.227.61 
> dst=xx.yy.zz.13 type=8 code=0 id=8724 src=xx.yy.zz.13 dst=128.1.227.61 
> type=0 code=0 id=8724 id=2525400839
> [1780936565.281121]	    [NEW] icmp     1 30 src=128.1.227.61 
> dst=xx.yy.zz.11 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.11 
> dst=128.1.227.61 type=0 code=0 id=8724 id=2574615308
> [1780936565.281167]	 [UPDATE] icmp     1 30 src=128.1.227.61 
> dst=xx.yy.zz.11 type=8 code=0 id=8724 src=xx.yy.zz.11 dst=128.1.227.61 
> type=0 code=0 id=8724 id=2574615308
>
> Here xx.yy.zz.11 and .13 are my public IP addresses. Why do these connections
> show up in the conntrack --event --src-nat output? Thanks!

Unfortunately, it appears to be a bug affecting ICMP. Possibly here:

https://git.netfilter.org/libnetfilter_conntrack/tree/src/conntrack/objopt.c?id=7e5397b9167bdd7597be809b8f088ff333e1ad31#n189

For now, I would suggest filtering with something else.

conntrack -E -o id,timestamp |
perl -ne 'print if /\bsrc=(\S+).*\bdst=(\S+)/ && $1 ne $2'

-- 
Kerin Millar

Re: Logging SNAT'ed connections

[Jan Kasprzak]

	Hi, Kerin,

Kerin Millar wrote:
> Unfortunately, it appears to be a bug affecting ICMP. Possibly here:
> 
> https://git.netfilter.org/libnetfilter_conntrack/tree/src/conntrack/objopt.c?id=7e5397b9167bdd7597be809b8f088ff333e1ad31#n189
> 
> For now, I would suggest filtering with something else.
> 
> conntrack -E -o id,timestamp |
> perl -ne 'print if /\bsrc=(\S+).*\bdst=(\S+)/ && $1 ne $2'

Oh, I did not expect it to be a bug.

Anyway, you probably mean src= in both cases. Using backreferences,
I did it this way:

perl -nE 'print if /\bsrc=(\S+)\s.*\bsrc=(?!\1\s)/'

Thanks!

-Yenya

-- 
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| https://www.fi.muni.cz/~kas/                        GPG: 4096R/A45477D5 |
    I don't like Python; its lack of inline, anonymous, multi-statement
    functions makes me sad.                                --Eric Wastl

Re: Logging SNAT'ed connections

[Kerin Millar]

On Mon, 8 Jun 2026, at 9:11 PM, Jan Kasprzak wrote:
> Hi, Kerin,
>
> Kerin Millar wrote:
>> Unfortunately, it appears to be a bug affecting ICMP. Possibly here:
>> 
>> https://git.netfilter.org/libnetfilter_conntrack/tree/src/conntrack/objopt.c?id=7e5397b9167bdd7597be809b8f088ff333e1ad31#n189
>> 
>> For now, I would suggest filtering with something else.
>> 
>> conntrack -E -o id,timestamp |
>> perl -ne 'print if /\bsrc=(\S+).*\bdst=(\S+)/ && $1 ne $2'
>
> Oh, I did not expect it to be a bug.
>
> Anyway, you probably mean src= in both cases. Using backreferences,
> I did it this way:
>
> perl -nE 'print if /\bsrc=(\S+)\s.*\bsrc=(?!\1\s)/'

The idea I had in mind was to compare orig.src against reply.dst. Where orig.src != reply.dst, NAT applies. Your approach compares orig.src to reply.src. Those will differ for most flows, whether they were subjected to NAT or not.

I would tend not to use backreferences in that way because it becomes quite easy to make a mistake.

$ printf 'x=123.45 y=123.45\n' | perl -nE 'say "matched: $1" if /x=(\S+).*\by=(?!\1\s)/'
matched: 123.4

--
Kerin Millar

Re: Logging SNAT'ed connections

[Jan Kasprzak]

Kerin Millar wrote:
> On Mon, 8 Jun 2026, at 9:11 PM, Jan Kasprzak wrote:
> > Hi, Kerin,
> >
> > Kerin Millar wrote:
> >> Unfortunately, it appears to be a bug affecting ICMP. Possibly here:
> >> 
> >> https://git.netfilter.org/libnetfilter_conntrack/tree/src/conntrack/objopt.c?id=7e5397b9167bdd7597be809b8f088ff333e1ad31#n189
> >> 
> >> For now, I would suggest filtering with something else.
> >> 
> >> conntrack -E -o id,timestamp |
> >> perl -ne 'print if /\bsrc=(\S+).*\bdst=(\S+)/ && $1 ne $2'
> >
> > Oh, I did not expect it to be a bug.
> >
> > Anyway, you probably mean src= in both cases. Using backreferences,
> > I did it this way:
> >
> > perl -nE 'print if /\bsrc=(\S+)\s.*\bsrc=(?!\1\s)/'
> 
> The idea I had in mind was to compare orig.src against reply.dst. Where orig.src != reply.dst, NAT applies. Your approach compares orig.src to reply.src. Those will differ for most flows, whether they were subjected to NAT or not.

OK, understood. Thanks for clarification.

> I would tend not to use backreferences in that way because it becomes quite easy to make a mistake.
> 
> $ printf 'x=123.45 y=123.45\n' | perl -nE 'say "matched: $1" if /x=(\S+).*\by=(?!\1\s)/'
> matched: 123.4

This is why I had \s after the first group: (\S+)\s.* instead of (\S+).*
But yes, if we are trying to match the first src= against the second dst=,
we would hav to avoid .* to match the first dst= instead. Something like
this:

perl -nE 'print if /\bsrc=(\S+)\s.*\bdst=.*\bdst=(?!\1\s)/'

-Yenya

-- 
| Jan "Yenya" Kasprzak <kas at {fi.muni.cz - work | yenya.net - private}> |
| https://www.fi.muni.cz/~kas/                        GPG: 4096R/A45477D5 |
    I don't like Python; its lack of inline, anonymous, multi-statement
    functions makes me sad.                                --Eric Wastl

Re: Logging SNAT'ed connections

[Pablo Neira Ayuso]

On Mon, Jun 08, 2026 at 06:53:30PM +0100, Kerin Millar wrote:
> On Mon, 8 Jun 2026, at 5:38 PM, Jan Kasprzak wrote:
> > Hi all,
> >
> > how do you log SNAT'ed connections for further forensic purposes,
> > when somebody complains about your public IP address and you want to be
> > able to find a local user behind NAT?
> >
> > I used to run something like
> >
> > conntrack --event --buffer-size 1600000 -o id,timestamp --src-nat >> $LOGFILE
> >
> > which did what I want. But now I tried to run the same on AlmaLinux9 host,
> > and it seems that --src-nat filter does not get applied - it shows
> > even connections which are not SNAT'ed at all, because they are for my
> > own IP address:
> >
> > [1780936563.343940]	    [NEW] icmp     1 30 src=128.1.227.61 
> > dst=xx.yy.zz.13 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.13 
> > dst=128.1.227.61 type=0 code=0 id=8724 id=2525400839
> > [1780936563.343994]	 [UPDATE] icmp     1 30 src=128.1.227.61 
> > dst=xx.yy.zz.13 type=8 code=0 id=8724 src=xx.yy.zz.13 dst=128.1.227.61 
> > type=0 code=0 id=8724 id=2525400839
> > [1780936565.281121]	    [NEW] icmp     1 30 src=128.1.227.61 
> > dst=xx.yy.zz.11 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.11 
> > dst=128.1.227.61 type=0 code=0 id=8724 id=2574615308
> > [1780936565.281167]	 [UPDATE] icmp     1 30 src=128.1.227.61 
> > dst=xx.yy.zz.11 type=8 code=0 id=8724 src=xx.yy.zz.11 dst=128.1.227.61 
> > type=0 code=0 id=8724 id=2574615308
> >
> > Here xx.yy.zz.11 and .13 are my public IP addresses. Why do these connections
> > show up in the conntrack --event --src-nat output? Thanks!
> 
> Unfortunately, it appears to be a bug affecting ICMP. Possibly here:
> 
> https://git.netfilter.org/libnetfilter_conntrack/tree/src/conntrack/objopt.c?id=7e5397b9167bdd7597be809b8f088ff333e1ad31#n189

Maybe then this helps?

https://patchwork.ozlabs.org/project/netfilter-devel/patch/20260610103039.167819-1-pablo@netfilter.org/

Re: Logging SNAT'ed connections

[Pablo Neira Ayuso]

On Mon, Jun 08, 2026 at 06:38:54PM +0200, Jan Kasprzak wrote:
> Hi all,
> 
> how do you log SNAT'ed connections for further forensic purposes,
> when somebody complains about your public IP address and you want to be
> able to find a local user behind NAT?
> 
> I used to run something like
> 
> conntrack --event --buffer-size 1600000 -o id,timestamp --src-nat >> $LOGFILE
> 
> which did what I want. But now I tried to run the same on AlmaLinux9 host,
> and it seems that --src-nat filter does not get applied - it shows
> even connections which are not SNAT'ed at all, because they are for my
> own IP address:
> 
> [1780936563.343940]	    [NEW] icmp     1 30 src=128.1.227.61 dst=xx.yy.zz.13 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.13 dst=128.1.227.61 type=0 code=0 id=8724 id=2525400839
> [1780936563.343994]	 [UPDATE] icmp     1 30 src=128.1.227.61 dst=xx.yy.zz.13 type=8 code=0 id=8724 src=xx.yy.zz.13 dst=128.1.227.61 type=0 code=0 id=8724 id=2525400839
> [1780936565.281121]	    [NEW] icmp     1 30 src=128.1.227.61 dst=xx.yy.zz.11 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.11 dst=128.1.227.61 type=0 code=0 id=8724 id=2574615308
> [1780936565.281167]	 [UPDATE] icmp     1 30 src=128.1.227.61 dst=xx.yy.zz.11 type=8 code=0 id=8724 src=xx.yy.zz.11 dst=128.1.227.61 type=0 code=0 id=8724 id=2574615308
> 
> Here xx.yy.zz.11 and .13 are my public IP addresses. Why do these connections
> show up in the conntrack --event --src-nat output? Thanks!

Could you check if this fixes the issue for you?

https://patchwork.ozlabs.org/project/netfilter-devel/patch/20260610103039.167819-1-pablo@netfilter.org/

Thanks.

Re: Logging SNAT'ed connections

[Kerin Millar]

On Wed, 10 Jun 2026, at 12:18 PM, Pablo Neira Ayuso wrote:
> On Mon, Jun 08, 2026 at 06:53:30PM +0100, Kerin Millar wrote:
>> On Mon, 8 Jun 2026, at 5:38 PM, Jan Kasprzak wrote:
>> > Hi all,
>> >
>> > how do you log SNAT'ed connections for further forensic purposes,
>> > when somebody complains about your public IP address and you want to be
>> > able to find a local user behind NAT?
>> >
>> > I used to run something like
>> >
>> > conntrack --event --buffer-size 1600000 -o id,timestamp --src-nat >> $LOGFILE
>> >
>> > which did what I want. But now I tried to run the same on AlmaLinux9 host,
>> > and it seems that --src-nat filter does not get applied - it shows
>> > even connections which are not SNAT'ed at all, because they are for my
>> > own IP address:
>> >
>> > [1780936563.343940]	    [NEW] icmp     1 30 src=128.1.227.61 
>> > dst=xx.yy.zz.13 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.13 
>> > dst=128.1.227.61 type=0 code=0 id=8724 id=2525400839
>> > [1780936563.343994]	 [UPDATE] icmp     1 30 src=128.1.227.61 
>> > dst=xx.yy.zz.13 type=8 code=0 id=8724 src=xx.yy.zz.13 dst=128.1.227.61 
>> > type=0 code=0 id=8724 id=2525400839
>> > [1780936565.281121]	    [NEW] icmp     1 30 src=128.1.227.61 
>> > dst=xx.yy.zz.11 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.11 
>> > dst=128.1.227.61 type=0 code=0 id=8724 id=2574615308
>> > [1780936565.281167]	 [UPDATE] icmp     1 30 src=128.1.227.61 
>> > dst=xx.yy.zz.11 type=8 code=0 id=8724 src=xx.yy.zz.11 dst=128.1.227.61 
>> > type=0 code=0 id=8724 id=2574615308
>> >
>> > Here xx.yy.zz.11 and .13 are my public IP addresses. Why do these connections
>> > show up in the conntrack --event --src-nat output? Thanks!
>> 
>> Unfortunately, it appears to be a bug affecting ICMP. Possibly here:
>> 
>> https://git.netfilter.org/libnetfilter_conntrack/tree/src/conntrack/objopt.c?id=7e5397b9167bdd7597be809b8f088ff333e1ad31#n189
>
> Maybe then this helps?
>
> https://patchwork.ozlabs.org/project/netfilter-devel/patch/20260610103039.167819-1-pablo@netfilter.org/

I have not yet tried it but shouldn't the helper function be checking against the protonum field? Also, the referenced commit appears unrelated to the fix. I think it should refer to b4c3a23c884c.

--
Kerin Millar

Re: Logging SNAT'ed connections

[Pablo Neira Ayuso]

On Wed, Jun 10, 2026 at 01:04:54PM +0100, Kerin Millar wrote:
> On Wed, 10 Jun 2026, at 12:18 PM, Pablo Neira Ayuso wrote:
> > On Mon, Jun 08, 2026 at 06:53:30PM +0100, Kerin Millar wrote:
> >> On Mon, 8 Jun 2026, at 5:38 PM, Jan Kasprzak wrote:
> >> > Hi all,
> >> >
> >> > how do you log SNAT'ed connections for further forensic purposes,
> >> > when somebody complains about your public IP address and you want to be
> >> > able to find a local user behind NAT?
> >> >
> >> > I used to run something like
> >> >
> >> > conntrack --event --buffer-size 1600000 -o id,timestamp --src-nat >> $LOGFILE
> >> >
> >> > which did what I want. But now I tried to run the same on AlmaLinux9 host,
> >> > and it seems that --src-nat filter does not get applied - it shows
> >> > even connections which are not SNAT'ed at all, because they are for my
> >> > own IP address:
> >> >
> >> > [1780936563.343940]	    [NEW] icmp     1 30 src=128.1.227.61 
> >> > dst=xx.yy.zz.13 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.13 
> >> > dst=128.1.227.61 type=0 code=0 id=8724 id=2525400839
> >> > [1780936563.343994]	 [UPDATE] icmp     1 30 src=128.1.227.61 
> >> > dst=xx.yy.zz.13 type=8 code=0 id=8724 src=xx.yy.zz.13 dst=128.1.227.61 
> >> > type=0 code=0 id=8724 id=2525400839
> >> > [1780936565.281121]	    [NEW] icmp     1 30 src=128.1.227.61 
> >> > dst=xx.yy.zz.11 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.11 
> >> > dst=128.1.227.61 type=0 code=0 id=8724 id=2574615308
> >> > [1780936565.281167]	 [UPDATE] icmp     1 30 src=128.1.227.61 
> >> > dst=xx.yy.zz.11 type=8 code=0 id=8724 src=xx.yy.zz.11 dst=128.1.227.61 
> >> > type=0 code=0 id=8724 id=2574615308
> >> >
> >> > Here xx.yy.zz.11 and .13 are my public IP addresses. Why do these connections
> >> > show up in the conntrack --event --src-nat output? Thanks!
> >> 
> >> Unfortunately, it appears to be a bug affecting ICMP. Possibly here:
> >> 
> >> https://git.netfilter.org/libnetfilter_conntrack/tree/src/conntrack/objopt.c?id=7e5397b9167bdd7597be809b8f088ff333e1ad31#n189
> >
> > Maybe then this helps?
> >
> > https://patchwork.ozlabs.org/project/netfilter-devel/patch/20260610103039.167819-1-pablo@netfilter.org/
> 
> I have not yet tried it but shouldn't the helper function be checking against the protonum field? Also, the referenced commit appears unrelated to the fix. I think it should refer to b4c3a23c884c.

Thanks for reviewing, v2 is here:

https://patchwork.ozlabs.org/project/netfilter-devel/patch/20260610151735.192168-1-pablo@netfilter.org/

Re: Logging SNAT'ed connections

[Kerin Millar]

On Wed, 10 Jun 2026, at 4:31 PM, Pablo Neira Ayuso wrote:
> On Wed, Jun 10, 2026 at 01:04:54PM +0100, Kerin Millar wrote:
>> On Wed, 10 Jun 2026, at 12:18 PM, Pablo Neira Ayuso wrote:
>> > On Mon, Jun 08, 2026 at 06:53:30PM +0100, Kerin Millar wrote:
>> >> On Mon, 8 Jun 2026, at 5:38 PM, Jan Kasprzak wrote:
>> >> > Hi all,
>> >> >
>> >> > how do you log SNAT'ed connections for further forensic purposes,
>> >> > when somebody complains about your public IP address and you want to be
>> >> > able to find a local user behind NAT?
>> >> >
>> >> > I used to run something like
>> >> >
>> >> > conntrack --event --buffer-size 1600000 -o id,timestamp --src-nat >> $LOGFILE
>> >> >
>> >> > which did what I want. But now I tried to run the same on AlmaLinux9 host,
>> >> > and it seems that --src-nat filter does not get applied - it shows
>> >> > even connections which are not SNAT'ed at all, because they are for my
>> >> > own IP address:
>> >> >
>> >> > [1780936563.343940]	    [NEW] icmp     1 30 src=128.1.227.61 
>> >> > dst=xx.yy.zz.13 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.13 
>> >> > dst=128.1.227.61 type=0 code=0 id=8724 id=2525400839
>> >> > [1780936563.343994]	 [UPDATE] icmp     1 30 src=128.1.227.61 
>> >> > dst=xx.yy.zz.13 type=8 code=0 id=8724 src=xx.yy.zz.13 dst=128.1.227.61 
>> >> > type=0 code=0 id=8724 id=2525400839
>> >> > [1780936565.281121]	    [NEW] icmp     1 30 src=128.1.227.61 
>> >> > dst=xx.yy.zz.11 type=8 code=0 id=8724 [UNREPLIED] src=xx.yy.zz.11 
>> >> > dst=128.1.227.61 type=0 code=0 id=8724 id=2574615308
>> >> > [1780936565.281167]	 [UPDATE] icmp     1 30 src=128.1.227.61 
>> >> > dst=xx.yy.zz.11 type=8 code=0 id=8724 src=xx.yy.zz.11 dst=128.1.227.61 
>> >> > type=0 code=0 id=8724 id=2574615308
>> >> >
>> >> > Here xx.yy.zz.11 and .13 are my public IP addresses. Why do these connections
>> >> > show up in the conntrack --event --src-nat output? Thanks!
>> >> 
>> >> Unfortunately, it appears to be a bug affecting ICMP. Possibly here:
>> >> 
>> >> https://git.netfilter.org/libnetfilter_conntrack/tree/src/conntrack/objopt.c?id=7e5397b9167bdd7597be809b8f088ff333e1ad31#n189
>> >
>> > Maybe then this helps?
>> >
>> > https://patchwork.ozlabs.org/project/netfilter-devel/patch/20260610103039.167819-1-pablo@netfilter.org/
>> 
>> I have not yet tried it but shouldn't the helper function be checking against the protonum field? Also, the referenced commit appears unrelated to the fix. I think it should refer to b4c3a23c884c.
>
> Thanks for reviewing, v2 is here:
>
> https://patchwork.ozlabs.org/project/netfilter-devel/patch/20260610151735.192168-1-pablo@netfilter.org/

Thanks. I'll try v2 out here.

--
Kerin Millar