nmap raw packet privileges not working ("operation not permitted", even as root)

Question

Why am I getting the "operation not permitted" with nmap - even when executed as root?

$ sudo nmap 192.168.1.2

Starting Nmap 7.12 ( https://nmap.org ) at 2017-01-13 02:12 CST
sendto in send_ip_packet_sd: sendto(5, packet, 44, 0, 192.168.1.2, 16) => Operation not permitted
Offending packet: TCP 192.168.1.1:53769 > 192.168.1.2:2099 S ttl=47 id=47294 iplen=44  seq=2821662280 win=1024 <mss 1460>
...
Omitting future Sendto error messages now that 10 have been shown.  Use -d2 if you really want to see them.

This is not an iptables issue - my OUTPUT chain is wide open:

$ sudo iptables -L OUTPUT
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Now, I do have a few different interfaces here, with VLANs and a bridge. This is working on some interfaces and not others:

• br0: Bridge over eth0 (untagged) and vbox0 (using VirtualBox), has IP 192.168.1.1 -> Not working (above).
  • For kicks, removing vbox0 from the bridge doesn't fix anything.
• eth0.2: VLAN 2, with IP 192.168.2.1. Executing nmap on addresses in this subnet works as expected -> working.
  • This seems significant, as this goes out the same physical NIC as eth0 (above).
• vbox1: Has IP 192.16.3.1. Executing nmap on addresses in this subnet works as expected -> working.

This is a physical workstation - not being operated under any virtualization or container that might impose additional restrictions here.

Bridge:

$ brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.0015171970fc       no              eth0
                                                        vbox0

Granted, I can work-around this by using a less-privileged TCP connect scan (-sT) rather than the default TCP SYN scan (-sS) - but this doesn't explain why this is happening.

Are there any known limitations here with the Ethernet bridging, or anything else I should be looking at?

Edits (2017-01-14):

• Attempting to replicate in a clean VM (2 vCPU on an i7 physical system). Even after setting up the bridge, unable to reproduce.
• Disabling all Ethernet offloading options (using ethtool) does nothing to help.
• Running the latest compiled-from-source Nmap 7.40 does nothing to help.
• This appears to be a kernel issue? http://seclists.org/nmap-dev/2016/q4/131 . Not sure why I couldn't reproduce on the VM, despite the same versions. Possibly also hardware/driver-specific...
  • This looks to be an issue with the iptable_nat module in the current 4.8.x kernels: https://bugzilla.redhat.com/show_bug.cgi?id=1402695
• The scan still runs. This only seems to impact the start of the scan - for which I remain concerned as I may be missing results.
  • It says that all messages after the first 10 have been omitted. However, even after repeating with the -d2 as prompted, I still see only 10. (Could be a bug in itself, however.)
  • If I repeat the scan for a given port as listed (e.g. with -p 2099 for the example shown above), it scans successfully for that port - so it isn't as if certain ports are blocked.
• Running with --max-parallelism=1 drastically reduces the occurrence.
  • Setting to 50 doesn't seem to help.
  • Setting to 30 seems to work about half the time for a single host - but still eventually starts failing for a subnet scan.
  • Progressively lower values lengthen the time it takes into a subnet scan for any failures to be observed - but even using 1 fails eventually.
  • This does not appear to be a parallelism issue within nmap itself. Running multiple concurrent nmap scans with parallel and --max-parallelism=1 re-increases the occurrence of the issue.

Host info: Ubuntu 16.10, kernel 4.8.0-34-generic #36-Ubuntu. Intel(R) Core(TM) i7-2600S, 32 GB RAM.

Solution

This looks to be an issue with the iptable_nat module in the current 4.8.x Linux kernels (< 4.8.16), as per https://bugzilla.redhat.com/show_bug.cgi?id=1402695.

The 4.9 kernel includes a "real" fix - but as for Ubuntu, I am guessing we'll have to wait for Ubuntu 17.04 (Zesty Zapus) to see this officially. (4.9 will be included there, as per the release notes).

As for Ubuntu 16.10 (Yakkety Yak), it looks like a fixed 4.8.16 kernel is pending release per https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1654584, which includes the following fixes:

Revert "netfilter: nat: convert nat bysrc hash to rhashtable"
Revert "netfilter: move nat hlist_head to nf_conn"

I updated to this kernel using http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8.16/ and the instructions at https://wiki.ubuntu.com/Kernel/MainlineBuilds. (I trust it will be further upgraded as usual as additional updates follow.) This not only resolved my issue, but resulted in a massive scanning performance improvement.

Comments

[bonsaiviking]

Thanks for tracking this down! I had lost track of it, but we (Nmap) did have users reporting it back in February. Always nice when a bug turns out to be someone else's problem and they fix it :)