What's the best way to nmap thousands of subnets in parallel from a script?

Question

To inventory a port in part of a Class-A network, I scan that as a few thousand Class-C networks using nmap. I use parallel to run 32 subnet scan jobs at once.

A minimized version of the script:

while read subnet; do
    echo nmap -Pn -p"$tcpport" "$subnet" >> /tmp/nmap_parallel.list;
done < $subnets
parallel -j32 < /tmp/nmap_parallel.list
wait
echo "Subnet scan for port $tcpport complete."

Problem with this approach is the script stops at parallel.

Is there a better way to use parallel from a script?

Solution 1

Can't you send the commands to the background with & in order to process the others simultaneously? Something like below;

#!/bin/bash
ports="20 21 22 25 80 443"
for p in $ports
do
nmap -Pn -p$p 10.0.1.0/24 &
done

Solution 2

Nmap has built-in parallelism that should be able to handle scanning a Class-A network in a single command. In fact, because of Nmap's network status monitoring and feedback mechanisms, it is usually better to run just one instance of Nmap at a time. The bottleneck for Nmap is not the processor, so running multiple instances with parallel is not going to help. Instead, Nmap will send many probes at once and wait for responses. As new responses come in, new probes can be sent out. If Nmap gets a response for every probe, it increases the number of outstanding probes (parallelism) it sends. When it detects a packet drop, it decreases this number (as well as some other timing-related variables).

This adaptive timing behavior is discussed at length in the official Nmap Network Scanning book, and is based on public algorithms used in TCP.

You may be able to speed up your scan by adjusting some timing options and eliminating scan phases that do not matter to you. On the simple end, you can try -T4 to increase several timing-related settings at once, without exceeding the capability of a high-speed link. You can also try adding -n to skip the reverse-DNS name lookup phase, since you may not be interested in those results.

You have already used the -Pn flag to skip the host discovery phase; if you are only scanning one port, this may be a good idea, but it may also result in confusing output and slower scan times, since Nmap must assume that every host is up and do a real port scan. Remember the adaptive timing algorithms? They have slightly different behavior when doing host discovery that may result in faster scan times. If you don't like the default host discovery probes, you can tune them yourself. If I am scanning for port 22, I can use that as a host discovery probe with -PS22, which means my output will only show hosts with that port open or closed (not firewalled and not down). If you stick with -Pn, you should probably also use the --open option to only show hosts with your chosen ports open, otherwise you will have a lot of output to slog through.

Comments

[bonsaiviking]

Often, people will suggest that Nmap use threads or some other CPU parallelism mechanism to speed up scans. This article is a fantastic illustration of why that doesn't really matter in a network scanner.

[user3126740]

Thanks for the detailed answer. I simplified the script above, in practice I output to an XML file, convert to JSON and ingest JSON into a database.

[user3126740]

Let me expand on the above comment - If I simply use -iL for the subnet list and use a 1-liner job, the output file would be huge and wouldn't be generated for a half-day. I also see a huge performance difference between 'parallel -j4 < nmap_command_list' and 'parallel -j32 < nmap_command_list' most likely due to generating the output file (-oX subnet_port.xml). I also use the --open flag.

[james-see]

This is really a nice simple approach that allows the kernel and cpu to handle the queuing and availability of cores / threads. Always KISS - Keep It Simple Stupid. Thank you.