How to properly log and view nftables activity?

Question

On Debian 10 buster I am having problems with docker containers unable to ping the docker host or even docker bridge interface, but able to reach the internet.
Allowing access as in related questions here, doesn't fix it in my case. Seems iptables/nftables related, and I can probably figure out what to do, if I could first figure out how to log the errors.

I put in the log rules in both DOCKER-USER and INPUT, with likes of nft insert rule ip filter DOCKER-USER counter log but they all show 0 packets logged.

/var/log/kern.log doesn't show any firewall related info, and neither does journalctl -k.

How is the new way to view firewall activity with this nftables system?

nft list ip table filter

table ip filter {
    chain INPUT {
        type filter hook input priority 0; policy drop;
        ct state invalid counter packets 80 bytes 3200 drop
        iifname "vif*" meta l4proto udp udp dport 68 counter packets 0 bytes 0 drop
        ct state related,established counter packets 9479197 bytes 17035404271 accept
        iifname "vif*" meta l4proto icmp counter packets 0 bytes 0 accept
        iifname "lo" counter packets 9167 bytes 477120 accept
        iifname "vif*" counter packets 0 bytes 0 reject with icmp type host-prohibited
        counter packets 28575 bytes 1717278 drop
        counter packets 0 bytes 0 log
        counter packets 0 bytes 0 log
        iifname "docker0" counter packets 0 bytes 0 accept
    }

    chain FORWARD {
        type filter hook forward priority 0; policy drop;
        counter packets 880249 bytes 851779418 jump DOCKER-ISOLATION-STAGE-1
        oifname "br-cc7b89b40bee" ct state related,established counter packets 7586 bytes 14719677 accept
        oifname "br-cc7b89b40bee" counter packets 0 bytes 0 jump DOCKER
        iifname "br-cc7b89b40bee" oifname != "br-cc7b89b40bee" counter packets 5312 bytes 2458488 accept
        iifname "br-cc7b89b40bee" oifname "br-cc7b89b40bee" counter packets 0 bytes 0 accept
        oifname "br-d41d1510d330" ct state related,established counter packets 8330 bytes 7303256 accept
        oifname "br-d41d1510d330" counter packets 0 bytes 0 jump DOCKER
        iifname "br-d41d1510d330" oifname != "br-d41d1510d330" counter packets 7750 bytes 7569465 accept
        iifname "br-d41d1510d330" oifname "br-d41d1510d330" counter packets 0 bytes 0 accept
        oifname "br-79fccb9a0478" ct state related,established counter packets 11828 bytes 474832 accept
        oifname "br-79fccb9a0478" counter packets 11796 bytes 707760 jump DOCKER
        iifname "br-79fccb9a0478" oifname != "br-79fccb9a0478" counter packets 7 bytes 526 accept
        iifname "br-79fccb9a0478" oifname "br-79fccb9a0478" counter packets 11796 bytes 707760 accept
        counter packets 1756295 bytes 1727495359 jump DOCKER-USER
        oifname "docker0" ct state related,established counter packets 1010328 bytes 1597833795 accept
        oifname "docker0" counter packets 0 bytes 0 jump DOCKER
        iifname "docker0" oifname != "docker0" counter packets 284235 bytes 16037499 accept
        iifname "docker0" oifname "docker0" counter packets 0 bytes 0 accept
        ct state invalid counter packets 0 bytes 0 drop
        ct state related,established counter packets 0 bytes 0 accept
        counter packets 0 bytes 0 jump QBS-FORWARD
        iifname "vif*" oifname "vif*" counter packets 0 bytes 0 drop
        iifname "vif*" counter packets 0 bytes 0 accept
        counter packets 0 bytes 0 drop
    }

    chain OUTPUT {
        type filter hook output priority 0; policy accept;
    }

    chain QBS-FORWARD {
    }

    chain DOCKER {
    }

    chain DOCKER-ISOLATION-STAGE-1 {
        iifname "br-cc7b89b40bee" oifname != "br-cc7b89b40bee" counter packets 5312 bytes 2458488 jump DOCKER-ISOLATION-STAGE-2
        iifname "br-d41d1510d330" oifname != "br-d41d1510d330" counter packets 7750 bytes 7569465 jump DOCKER-ISOLATION-STAGE-2
        iifname "br-79fccb9a0478" oifname != "br-79fccb9a0478" counter packets 7 bytes 526 jump DOCKER-ISOLATION-STAGE-2
        iifname "docker0" oifname != "docker0" counter packets 590138 bytes 34612496 jump DOCKER-ISOLATION-STAGE-2
        counter packets 1808904 bytes 1760729363 return
    }

    chain DOCKER-ISOLATION-STAGE-2 {
        oifname "br-cc7b89b40bee" counter packets 0 bytes 0 drop
        oifname "br-d41d1510d330" counter packets 0 bytes 0 drop
        oifname "br-79fccb9a0478" counter packets 0 bytes 0 drop
        oifname "docker0" counter packets 0 bytes 0 drop
        counter packets 644929 bytes 74784737 return
    }

    chain DOCKER-USER {
        counter packets 0 bytes 0 log
        iifname "docker0" counter packets 305903 bytes 18574997 accept
        counter packets 1450392 bytes 1708920362 return
    }
}

Solution

You can use nftrace to trace packet flows. It's very verbose but doesn't go to kernel logs but instead is distributed over multicast netlink socket (ie if nothing listens to them, traces just go to "/dev/null").

If you really want to trace everything, trace from prerouting and output at a low priority. Better use a separate table, because what you are displaying with nft list ip table filter is actually iptables-over-nftables with the compatibility xt match layer API and shouldn't be tampered with (but can safely be used along traces). Also you should know there are probably other tables for iptables, like the nat table.

So, with a ruleset from the file traceall.nft loaded with nft -f traceall.nft:

table ip traceall
delete table ip traceall

table ip traceall {
    chain prerouting {
        type filter hook prerouting priority -350; policy accept;
        meta nftrace set 1
    }

    chain output {
        type filter hook output priority -350; policy accept;
        meta nftrace set 1
    }
}

You can now follow these (very verbose) IPv4 traces with:

nft monitor trace

This would even work the same if doing this inside a container (which is usually not the case for log targets).

You can activate these traces elsewhere, or put conditions before activating them, and also deactivate them (meta nftrace set 0) again in a rule in a later priority to avoid tracing all hooks/chains. Following this schematic will help understand the order of events: Packet flow in Netfilter and General Networking.

If choosing to use the equivalent -j TRACE target in iptables, consult also the man for xtables-monitor, because iptables-over-nftables changes its behaviour (compared to iptables-legacy).

---

While I answered OP's question, here are wild guesses about both issues and log issues:

• if Docker itself is running within a container, logs might not be available. They can be made available to the host, and to all containers allowed to query the kernel messages, with sysctl -w net.netfilter.nf_log_all_netns=1, simply because kernel messages don't have namespace instances.

• the counter at the log rule in ip filter INPUT is zero, while the counter at the previous rule with a drop statement is not. That means the log rule is made too late: after drop. The log rule (or rather iptables's -j LOG) should be inserted before the final drop statement, not appended after where it will never be reached.

• The only INPUT rule about Docker is iifname "docker0" counter packets 0 bytes 0 accept. If the containers are not on the default Docker network, there's no rule allowing them to reach the host.

  Try adding a rule for testing this. Be sure the result is inserted before the drop rule. Use iptables, avoid adding a rule with nftables that could be incompatible with iptables-over-nftables:

  iptables -I INPUT 8 -i "br-*" -j ACCEPT