Why does ipv6 enabled interfere with ipv4 SNAT?

Why does ipv6 enabled interfere with ipv4 SNAT?

[Whit Blauvelt]

I've just been setting up a replacement for an old server happily running
Netfilter with SNAT for years. The new one - using Ubuntu Server 7.10 - was
refusing to SNAT or MASQ, despite running the same long-proven Netfilter
rules. After checking that everything was getting set right in
/proc/sys/net/ipv4/conf (it was), I for some reason took a look in
/proc/sys/net/ipv6/conf. (Ubuntu insists on running ipv6 by default.) There
was an anomoly: The new machine has 6 NICs, and while ipv4/conf had eth0
through eth5, ipv6/conf only had them through eth3 - just the first 4. As it
happens the Internet-facing NICs for this box are eth4 and eth5.

At first I looked for a way to populate the ipv6 proc area. But not finding
that, and really having no use for ipv6 anyway - stupid to leave stuff on on
a server when it's not being used right? - I finally went to
/etc/modprobe.d/arch/i386 and added a line, "alias net-pf-10 off", which on
rebooting blocks the ipv6 kernel module from loading. Now, with no other
change, SNAT works as well as ever.

But I'd still like to understand what the principle here is. Are there two
independent ipv6-related bugs - one that prevents its /proc space being
populated beyond 4 NICs, and a totally different one that breaks ipv4
Netfilter SNAT? Or was the first bug what broke SNAT? What is ipv4 Netfilter
doing being vulnerable to ipv6 problems anyway? I know there's Netfilter
stuff for ipv6; is it somehow required to invoke that somehow if ipv6 is
enabled in order for the ipv4 Netfilter code to work right with SNAT? 

Finally, is the Ubuntu Server project being boneheaded to even have ipv6
enabled by default, if it's still this far from being ready for prime time?
Servers should be rock-solid at basic functions like SNAT out-of-the-box,
IMHO.

Thanks for any explanations. As I say, things work fine once I figure out
what to disable. But frankly it took me half the day to track down what the
problem was. 

Regards,
Whit

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Jan Engelhardt]

On Tuesday 2008-03-25 02:28, Whit Blauvelt wrote:
> I for some reason took a look in
> /proc/sys/net/ipv6/conf. (Ubuntu insists on running ipv6 by default.) There
> was an anomoly: The new machine has 6 NICs, and while ipv4/conf had eth0
> through eth5, ipv6/conf only had them through eth3 - just the first 4.

Just what _do_ you actually have in /proc/sys/net/ipv6/conf?
Just 4 entries seems a bit spartanic, since there are also
the "all" and "default" entries:

# ls /proc/sys/net/ipv6/conf/
all default lo rtl0 sis0 tun0 vmnet1

NAT-out device is sis0. Even if I add in a number of dummies,
all remains normal:

all      dummy0  dummy2  dummy4  dummy6  dummy8  lo    sis0  vmnet1
default  dummy1  dummy3  dummy5  dummy7  dummy9  rtl0  tun0

It still works with opensuse plus 2.6.23. Well, I suggest you
try a stock kernel.

> Finally, is the Ubuntu Server project being boneheaded to even have ipv6
> enabled by default, if it's still this far from being ready for prime time?

I would not say IPv6 was not ready.

> Servers should be rock-solid at basic functions like SNAT out-of-the-box,
> IMHO.

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Whit Blauvelt]

On Tue, Mar 25, 2008 at 02:58:25AM +0100, Jan Engelhardt wrote:

> Just what _do_ you actually have in /proc/sys/net/ipv6/conf?
> Just 4 entries seems a bit spartanic, since there are also
> the "all" and "default" entries:

I was mentioning where the mismatch was. Yeah the other stuff's there.

/proc/sys/net/ipv6/conf ends up with only:

   all default eth0 eth1 eth2 eth3 lo

while /proc/sys/net/ipv4/conf ends up (correctly) with:

   all default eth0 eth1 eth2 eth3 eth4 eth5 lo

Now, I don't know just which process is (not) doing the populating there,
but that's consistently where it ends up with ipv6 enabled and 6 NICs in the
box.

> # ls /proc/sys/net/ipv6/conf/
> all default lo rtl0 sis0 tun0 vmnet1
>
> NAT-out device is sis0. Even if I add in a number of dummies,
> all remains normal:
>
> all      dummy0  dummy2  dummy4  dummy6  dummy8  lo    sis0  vmnet1
> default  dummy1  dummy3  dummy5  dummy7  dummy9  rtl0  tun0

Not sure what you're thinking there. My problem wasn't with there being too
many devices, but with two of the devices I actually have not being
represented - and with Netfilter not doing _ipv4_ SNAT on account of
something with _ipv6_. Why should Netfilter ipv4 code even _care_ about
what's right or not with ipv6? Do you have any knowledge about
interdependency there? 

> It still works with opensuse plus 2.6.23. Well, I suggest you
> try a stock kernel.

It is a stock kernel, if by that you mean a stock distro kernel -
Ubuntu's latest, 2.6.22-14-server. 

> I would not say IPv6 was not ready.

I have no idea if the failure to fully populate the ipv6 proc eth? devices
was Debian-specific, Ubuntu-specific, a shortcoming of the ifupdown suite
both use, or a bug in the kernel itself. But it seems wrong on the face of
it for ipv4 Netfilter SNAT code to depend on ipv6 in any way. Yet something
about ipv6 on Ubuntu - possibly the failure to set NIC devices beyond eth3
up properly in the ipv6 /proc space - breaks Netfilter ipv4 snat. I'm still
hoping to understand why, in part because there's obviously a serious bug
_somewhere_, and it would be nice to report it to the right place.

Best,
Whit

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Jan Engelhardt]

On Tuesday 2008-03-25 03:44, Whit Blauvelt wrote:
>
> Not sure what you're thinking there. My problem wasn't with there being too
> many devices, but with two of the devices I actually have not being
> represented - and with Netfilter not doing _ipv4_ SNAT on account of
> something with _ipv6_.

How does it break? Do the counters increase in the nat table at all?
Do the chain and/or rule counters increase if you add the same rule
without action? (I.e.:

 	-t nat -A POSTROUTING -o eth4 -m whatever
 	-t nat -A POSTROUTING -o eth4 -m whatever -j SNAT --to xyz

>> It still works with opensuse plus 2.6.23. Well, I suggest you
>> try a stock kernel.
>
> It is a stock kernel, if by that you mean a stock distro kernel -
> Ubuntu's latest, 2.6.22-14-server.

Stock vanilla kernel.org kernel.

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Whit Blauvelt]

On Tue, Mar 25, 2008 at 03:57:49AM +0100, Jan Engelhardt wrote:

> How does it break? Do the counters increase in the nat table at all?
> Do the chain and/or rule counters increase if you add the same rule
> without action? (I.e.:
>
> 	-t nat -A POSTROUTING -o eth4 -m whatever
> 	-t nat -A POSTROUTING -o eth4 -m whatever -j SNAT --to xyz

No the counters don't increase. Again, ipv4 Netfilter SNAT does not work if
ipv6 is enabled on the system. It works perfectly if ipv6 is disabled - no
other changes. Do you have any theory about that, at all? It may well be
downstream of the failure to handle /proc assignments correctly for > 4 NICs
on the ipv6 side - understandably most people aren't network admins on my
level running boxes with > 4 NICs so that wouldn't bite that often. That
wouldn't make it not a bug, though.

>> It is a stock kernel, if by that you mean a stock distro kernel -
>> Ubuntu's latest, 2.6.22-14-server.

> Stock vanilla kernel.org kernel.

Why would I want to do that? If you read my orginal post closely, I have no
need for ipv6. It's fine with me to run without it - which works perfectly.
But what I want to do is understand where the fine bug is ... perhaps to
report it to those responsible. Now, it looks pretty obviously like there's
a serious bug in Netfilter here, because how else could anything about the
state of the ipv6 configuration affect the success of ipv4 Netfilter SNAT?

Now, if you can explain, theoretically, why I'm wrong about that, why the
state of the ipv6 configuration should be critical to ipv4 Netfilter SNAT
operation, I'm quite curious. But if you have no idea why the ipv4 Netfilter
SNAT rule fails if-and-only-if ipv6 operation is also enabled on the system,
then please let's see if someone who knows the innards of this stuff better
than either of us can help illuminate the puzzle.

Whit

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Jozsef Kadlecsik]

On Mon, 24 Mar 2008, Whit Blauvelt wrote:

> I was mentioning where the mismatch was. Yeah the other stuff's there.
> 
> /proc/sys/net/ipv6/conf ends up with only:
> 
>    all default eth0 eth1 eth2 eth3 lo
> 
> while /proc/sys/net/ipv4/conf ends up (correctly) with:
> 
>    all default eth0 eth1 eth2 eth3 eth4 eth5 lo

If you have got ethernet devices with MTU smaller than 1500, those won't 
support IPv6 and therefore don't show up in /proc/sys/net/ipv6/conf.

> Not sure what you're thinking there. My problem wasn't with there being too
> many devices, but with two of the devices I actually have not being
> represented - and with Netfilter not doing _ipv4_ SNAT on account of
> something with _ipv6_. Why should Netfilter ipv4 code even _care_ about
> what's right or not with ipv6? Do you have any knowledge about
> interdependency there? 
> 
> > It still works with opensuse plus 2.6.23. Well, I suggest you
> > try a stock kernel.
> 
> It is a stock kernel, if by that you mean a stock distro kernel -
> Ubuntu's latest, 2.6.22-14-server. 

Please check which modules are loaded in. Probably IPv4 NAT related 
modules are not auto-loaded. Also, is there any error message when you 
issue 'iptables -t nat ...' commands?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Whit Blauvelt]

On Tue, Mar 25, 2008 at 12:03:58PM +0100, Jozsef Kadlecsik wrote:

> > /proc/sys/net/ipv6/conf ends up with only:
> > 
> >    all default eth0 eth1 eth2 eth3 lo
> > 
> > while /proc/sys/net/ipv4/conf ends up (correctly) with:
> > 
> >    all default eth0 eth1 eth2 eth3 eth4 eth5 lo
> 
> If you have got ethernet devices with MTU smaller than 1500, those won't 
> support IPv6 and therefore don't show up in /proc/sys/net/ipv6/conf.

Interesting. But there is no difference between the MTU settings of eth2,
eth3, eth4 and eth5 - in fact all four ports are on the same physical card.
All the eth? devices are running at default of 1500. I've checked.

> Please check which modules are loaded in. Probably IPv4 NAT related 
> modules are not auto-loaded. Also, is there any error message when you 
> issue 'iptables -t nat ...' commands?

Josef, the set of iptables modules is precisely the same whether or not ipv6
is enabled on the system (yes, I've looked). The only difference is whether
the ipv6 module is loaded. Blocking the ipv6 module from loading at boot
both prevents (of course) the /proc/sys/net/ipv6 space from being populated.
It also fully restores SNAT functionality.

There are no error messages from "iptables -t nat" commands. The nat rules
show up precisely the same in the resulting tables - they just don't count
any packets traversing them if ipv6 is also enabled. I haven't done packet
inspection to see what's happening, but the problem remains:

With _everything else_ unchanged, simply letting the ipv6 module load at
boot results in:

1) eth4 and eth5 not showing up as they should for ipv6 in
   /proc/sys/net/ipv6/conf

and (whether related or a separate bug)

2) ipv4 Netfilter SNAT (in POSTROUTING in this case) becoming a black hole

Boxes behind the firewall can still ping the firewall. The firewall can
still ping the world. But just with the ipv6 module loaded, the boxes behind
the firewall cannot ping the world. Clearly ipv6 is not properly
provisioning its space. But why should ipv6's failing have any effect at all
on ipv4 functionality? 

For me, not loading the ipv6 module is a totally fine workaround. It's
useless cruft in my context anyway. But even though my problem is fixed for
now, it will be a problem for other people (maybe only those with > 4
ethernet devices?) when the ipv6 transition finally happens. I remember in
the early 90s when we were expecting it by about 1998.

So in the spirit of open source development, I'm looking for for someone
with better knowledge of the underlying theory to help identify where the
bug here is. Then the responsible developers can have a chance to fix it
before a lot of other sysadmins also end up wasting many hours because of
the non-obviousness of having to look at ipv6 stuff to find why ipv4 SNAT
becomes a black hole.

Regards,
Whit

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Patrick McHardy]

Whit Blauvelt wrote:
>
> Josef, the set of iptables modules is precisely the same whether or not ipv6
> is enabled on the system (yes, I've looked). The only difference is whether
> the ipv6 module is loaded. Blocking the ipv6 module from loading at boot
> both prevents (of course) the /proc/sys/net/ipv6 space from being populated.
> It also fully restores SNAT functionality.
> 
> There are no error messages from "iptables -t nat" commands. The nat rules
> show up precisely the same in the resulting tables - they just don't count
> any packets traversing them if ipv6 is also enabled. I haven't done packet
> inspection to see what's happening, but the problem remains:
> 
> With _everything else_ unchanged, simply letting the ipv6 module load at
> boot results in:
> 
> 1) eth4 and eth5 not showing up as they should for ipv6 in
>    /proc/sys/net/ipv6/conf
> 
> and (whether related or a separate bug)
> 
> 2) ipv4 Netfilter SNAT (in POSTROUTING in this case) becoming a black hole
> 
> Boxes behind the firewall can still ping the firewall. The firewall can
> still ping the world. But just with the ipv6 module loaded, the boxes behind
> the firewall cannot ping the world. Clearly ipv6 is not properly
> provisioning its space. But why should ipv6's failing have any effect at all
> on ipv4 functionality? 
> 
> For me, not loading the ipv6 module is a totally fine workaround. It's
> useless cruft in my context anyway. But even though my problem is fixed for
> now, it will be a problem for other people (maybe only those with > 4
> ethernet devices?) when the ipv6 transition finally happens. I remember in
> the early 90s when we were expecting it by about 1998.
> 
> So in the spirit of open source development, I'm looking for for someone
> with better knowledge of the underlying theory to help identify where the
> bug here is. Then the responsible developers can have a chance to fix it
> before a lot of other sysadmins also end up wasting many hours because of
> the non-obviousness of having to look at ipv6 stuff to find why ipv4 SNAT
> becomes a black hole.

Please post the list of modules loaded and the output of
/proc/net/nf_conntrack.

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Jozsef Kadlecsik]

On Tue, 25 Mar 2008, Whit Blauvelt wrote:

> > If you have got ethernet devices with MTU smaller than 1500, those won't 
> > support IPv6 and therefore don't show up in /proc/sys/net/ipv6/conf.
> 
> Interesting. But there is no difference between the MTU settings of eth2,
> eth3, eth4 and eth5 - in fact all four ports are on the same physical card.
> All the eth? devices are running at default of 1500. I've checked.

OK. That was a possibility which could have explained the difference in 
the interfaces.
 
> > Please check which modules are loaded in. Probably IPv4 NAT related 
> > modules are not auto-loaded. Also, is there any error message when you 
> > issue 'iptables -t nat ...' commands?
> 
> Josef, the set of iptables modules is precisely the same whether or not ipv6
> is enabled on the system (yes, I've looked). 

But which netfilter modules are loaded in? Again, there can be some 
auto-loading problem which we cannot figure out without the list of the 
modules. And as Patrick suggested: do you see the connections in 
/proc/net/nf_conntrack? Or you have got /proc/net/ip_conntrack instead?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Pascal Hambourg]

Jozsef Kadlecsik a écrit :
> 
> If you have got ethernet devices with MTU smaller than 1500, those won't 
> support IPv6 and therefore don't show up in /proc/sys/net/ipv6/conf.

Don't you mean smaller than 1280 ?

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Jozsef Kadlecsik]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 583 bytes --]

On Wed, 26 Mar 2008, Pascal Hambourg wrote:

> Jozsef Kadlecsik a écrit :
> > 
> > If you have got ethernet devices with MTU smaller than 1500, those won't
> > support IPv6 and therefore don't show up in /proc/sys/net/ipv6/conf.
> 
> Don't you mean smaller than 1280 ?

You are right, the exact boundary is 1280.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Whit Blauvelt]

On Tue, Mar 25, 2008 at 04:53:19PM +0100, Patrick McHardy wrote:

> Please post the list of modules loaded and the output of
> /proc/net/nf_conntrack.

First here is the list by the system in question, working once the ipv6
module is blocked from loading at boot. Next is the list from a system with
identical hardware and near-identical configuration (same firewall rules),
but with ipv6 loading - and which also has only 4 of the 6 NICs showing up
in the ipv6 proc conf space, and also has NAT (in this case DNAT is what I
tested) failing - also where the NICs on the Internet side of things are
those coincidentally not showing up with proc ipv6 conf settings. 

As to the output of /proc/net/nf_conntrack, you just want to see anything,
or under specific load? I'm not going to just publicly post the raw data -
although both systems have some there - since IPs can identify my client and
their clients, which would violate confidentiality.

Okay, the fixed system:

Module                  Size  Used by
drbd                  208136  2 
cn                      9632  1 drbd
parport_pc             37668  0 
lp                     12452  0 
parport                37448  2 parport_pc,lp
loop                   19076  0 
sg                     36380  0 
sr_mod                 17700  0 
cdrom                  37408  1 sr_mod
ata_generic             8580  0 
usbhid                 29664  0 
hid                    28928  1 usbhid
pcspkr                  4224  0 
psmouse                39952  0 
serio_raw               8068  0 
shpchp                 34580  0 
pci_hotplug            32576  1 shpchp
evdev                  11136  0 
ipt_TOS                 3200  16 
ipt_REJECT              5760  2 
xt_state                3456  372 
nf_nat_ftp              4352  0 
nf_conntrack_ftp       11136  1 nf_nat_ftp
xt_limit                3584  3 
xt_tcpudp               4224  616 
ipt_LOG                 7552  2 
iptable_mangle          3840  1 
iptable_nat             8708  1 
nf_nat                 20012  2 nf_nat_ftp,iptable_nat
nf_conntrack_ipv4      19724  374 iptable_nat
nf_conntrack           65160  6 xt_state,nf_nat_ftp,nf_conntrack_ftp,iptable_nat,nf_nat,nf_conntrack_ipv4
nfnetlink               6936  3 nf_nat,nf_conntrack_ipv4,nf_conntrack
iptable_filter          3968  1 
ip_tables              13924  3 iptable_mangle,iptable_nat,iptable_filter
x_tables               16260  8 ipt_TOS,ipt_REJECT,xt_state,xt_limit,xt_tcpudp,ipt_LOG,iptable_nat,ip_tables
ext3                  133640  4 
jbd                    60456  1 ext3
mbcache                 9732  1 ext3
ata_piix               17540  0 
libata                125296  2 ata_generic,ata_piix
ehci_hcd               36748  0 
bnx2                  157208  0 
e1000                 126656  0 
uhci_hcd               26640  0 
usbcore               138760  4 usbhid,ehci_hcd,uhci_hcd
cciss                  61700  7 
scsi_mod              146828  4 sg,sr_mod,libata,cciss
dm_mirror              24320  0 
dm_snapshot            18980  0 
dm_mod                 58816  10 dm_mirror,dm_snapshot
thermal                14344  0 
processor              32072  1 thermal
fan                     5764  0 
fuse                   47124  1 
apparmor               40600  0 
commoncap               8320  1 apparmor

Here's the list from a nearly identical sytem that's still got the ipv6
module loading, and that's also failing at both populating the proc ipv6
space fully (same thing - just four of the 6 NICs) and also failing at NAT
(in this case DNAT was what I tried):

Module                  Size  Used by
ipt_TOS                 3200  16 
ipt_REJECT              5760  2 
nf_nat_ftp              4352  0 
nf_conntrack_ftp       11136  1 nf_nat_ftp
xt_limit                3584  3 
xt_state                3456  92 
xt_tcpudp               4224  266 
ipt_LOG                 7552  2 
iptable_mangle          3840  1 
iptable_nat             8708  1 
nf_nat                 20012  2 nf_nat_ftp,iptable_nat
nf_conntrack_ipv4      19724  94 iptable_nat
nf_conntrack           65160  6 nf_nat_ftp,nf_conntrack_ftp,xt_state,iptable_nat,nf_nat,nf_conntrack_ipv4
nfnetlink               6936  3 nf_nat,nf_conntrack_ipv4,nf_conntrack
iptable_filter          3968  1 
ip_tables              13924  3 iptable_mangle,iptable_nat,iptable_filter
x_tables               16260  8 ipt_TOS,ipt_REJECT,xt_limit,xt_state,xt_tcpudp,ipt_LOG,iptable_nat,ip_tables
drbd                  208136  1 
cn                      9632  1 drbd
ipv6                  278916  30 
parport_pc             37668  0 
af_packet              24840  2 
lp                     12452  0 
parport                37448  2 parport_pc,lp
loop                   19076  0 
serio_raw               8068  0 
pcspkr                  4224  0 
psmouse                39952  0 
shpchp                 34580  0 
pci_hotplug            32576  1 shpchp
evdev                  11136  0 
sg                     36380  0 
sr_mod                 17700  0 
cdrom                  37408  1 sr_mod
usbhid                 29664  0 
hid                    28928  1 usbhid
ata_piix               17540  0 
ext3                  133640  2 
jbd                    60456  1 ext3
mbcache                 9732  1 ext3
ehci_hcd               36748  0 
ata_generic             8580  0 
libata                125296  2 ata_piix,ata_generic
uhci_hcd               26640  0 
usbcore               138760  4 usbhid,ehci_hcd,uhci_hcd
e1000                 126656  0 
bnx2                  157208  0 
cciss                  61700  6 
scsi_mod              146828  4 sg,sr_mod,libata,cciss
dm_mirror              24320  0 
dm_snapshot            18980  0 
dm_mod                 58816  10 dm_mirror,dm_snapshot
thermal                14344  0 
processor              32072  1 thermal
fan                     5764  0 
fuse                   47124  1 
apparmor               40600  0 
commoncap               8320  1 apparmor

- Whit

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Whit Blauvelt]

On Wed, Mar 26, 2008 at 10:45:29AM +0100, Jozsef Kadlecsik wrote:

> But which netfilter modules are loaded in? Again, there can be some 
> auto-loading problem which we cannot figure out without the list of the 
> modules. And as Patrick suggested: do you see the connections in 
> /proc/net/nf_conntrack? Or you have got /proc/net/ip_conntrack instead?

I've just posted the modules list. What should I be looking for between
/proc/net/nf_conntrack vs /proc/net/ip_conntrack? Anything that matches the
internal or external IP? What should be there so I can check for anomoly,
since I don't want to just put raw data to the list.

Thanks,
Whit

Re: Why does ipv6 enabled interfere with ipv4 SNAT?

[Patrick McHardy]

Whit Blauvelt wrote:
> On Tue, Mar 25, 2008 at 04:53:19PM +0100, Patrick McHardy wrote:
> 
>> Please post the list of modules loaded and the output of
>> /proc/net/nf_conntrack.
> 
> First here is the list by the system in question, working once the ipv6
> module is blocked from loading at boot. Next is the list from a system with
> identical hardware and near-identical configuration (same firewall rules),
> but with ipv6 loading - and which also has only 4 of the 6 NICs showing up
> in the ipv6 proc conf space, and also has NAT (in this case DNAT is what I
> tested) failing - also where the NICs on the Internet side of things are
> those coincidentally not showing up with proc ipv6 conf settings. 

The firewall rules appear to be different between the two systems,
the first one has a lot more references to the IPv4 conntrack module.

> As to the output of /proc/net/nf_conntrack, you just want to see anything,
> or under specific load? I'm not going to just publicly post the raw data -
> although both systems have some there - since IPs can identify my client and
> their clients, which would violate confidentiality.

I'm mainly interested in one or more of the conntrack entries that
should get NATed but don't. One entry should be enough, feel free
to replace IPs as long as similar IPs still are similar.

> 
> Okay, the fixed system:
> 
> Module                  Size  Used by
> ...
 > iptable_nat             8708  1
 > nf_nat                 20012  2 nf_nat_ftp,iptable_nat
 > nf_conntrack_ipv4      19724  374 iptable_nat
> 
> Here's the list from a nearly identical sytem that's still got the ipv6
> module loading, and that's also failing at both populating the proc ipv6
> space fully (same thing - just four of the 6 NICs) and also failing at NAT
> (in this case DNAT was what I tried):
> 
> iptable_nat             8708  1 
> nf_nat                 20012  2 nf_nat_ftp,iptable_nat
> nf_conntrack_ipv4      19724  94 iptable_nat

Could you figure out whats causing the different amount of references
to nf_conntrack_ipv4? "-m state" rules, "-m conntrack" etc. take
references, maybe something fails during load?