Solution to route/proxy SNMP Traps (or Netflow, generic UDP, etc) for network monitoring?

Question

I'm implementing a network monitoring solution for a very large network (approximately 5000 network devices). We'd like to have all devices on our network send SNMP traps to a single box (technically this will probably be an HA pair of boxes) and then have that box pass the SNMP traps on to the real processing boxes. This will allow us to have multiple back-end boxes handling traps, and to distribute load among those back end boxes.

One key feature that we need is the ability to forward the traps to a specific box depending on the source address of the trap. Any suggestions for the best way to handle this?

Among the things we've considered are:

• Using snmptrapd to accept the traps, and have it pass them off to a custom written perl handler script to rewrite the trap and send it to the proper processing box
• Using some sort of load balancing software running on a Linux box to handle this (having some difficulty finding many load balancing programs that will handle UDP)
• Using a Load Balancing Appliance (F5, etc)
• Using IPTables on a Linux box to route the SNMP traps with NATing

We've currently implemented and are testing the last solution, with a Linux box with IPTables configured to receive the traps, and then depending on the source address of the trap, rewrite it with a destination nat (DNAT) so the packet gets sent to the proper server. For example:

# Range: 10.0.0.0/19       Site: abc01    Destination: foo01
iptables -t nat -A PREROUTING -p udp --dport 162 -s 10.0.0.0/19 -j DNAT --to-destination 10.1.2.3
# Range: 10.0.33.0/21       Site: abc01    Destination: foo01
iptables -t nat -A PREROUTING -p udp --dport 162 -s 10.0.33.0/21 -j DNAT --to-destination 10.1.2.3
# Range: 10.1.0.0/16       Site: xyz01    Destination: bar01
iptables -t nat -A PREROUTING -p udp --dport 162 -s 10.1.0.0/16 -j DNAT --to-destination 10.3.2.1

This should work with excellent efficiency for basic trap routing, but it leaves us completely limited to what we can mach and filter on with IPTables, so we're concerned about flexibility for the future.

Another feature that we'd really like, but isn't quite a "must have" is the ability to duplicate or mirror the UDP packets. Being able to take one incoming trap and route it to multiple destinations would be very useful.

Has anyone tried any of the possible solutions above for SNMP traps (or Netflow, general UDP, etc) load balancing? Or can anyone think of any other alternatives to solve this?

Solution 1

A co-worker just showed me samplicator. This tool looks to be just about a perfect solution what I was looking for. From the tool's website:

This simple program listens for UDP datagrams on a network port, and sends copies of these datagrams on to a set of destinations. Optionally, it can perform sampling, i.e. rather than forwarding every packet, forward only 1 in N. Another option is that it can "spoof" the IP source address, so that the copies appear to come from the original source, rather than the relay. Currently only supports IPv4.

It can been used to distribute e.g. Netflow packets, SNMP traps (but not informs), or Syslog messages to multiple receivers.

Solution 2

I would go implementing the solution myself, as I don't know if you will find something as specific as you want.

I would use a high-level language like ruby to implement the balance rules and even the trap listener. For instance, using this libraries seems easy.

Listen to traps:

m = SNMP::TrapListener.new(:Port => 1062, :Community => 'public') do |manager|
  manager.on_trap_default { |trap| p trap }
end
m.join

You should add the balance logic in the on_trap_default block.

Send traps:

Manager.open(:Version => :SNMPv1) do |snmp|
  snmp.trap_v1(
    "enterprises.9",
    "10.1.2.3",
    :enterpriseSpecific,
    42,
    12345,
    [VarBind.new("1.3.6.1.2.3.4", Integer.new(1))])
end

To build the daemon you could use the daemon-kit ruby gem.

If you keep it simple and define good objects you can maintain the software with not much effort.

Comments

[Admin]

net-snmp's snmptrapd handles trap forwarding.

[demonkoryu]

I appreciate the answer, but honestly if I build something myself, it'll be based around Net-SNMP's snmptrapd, and implemented in Perl, as snmptrapd has built-in support for accepting traps and calling Perl modules to handle them. That keeps it simpler and much better supported (we have a dozen guys who can handle basic Perl, and one guy who's (barely) toyed with Ruby).

[demonkoryu]

Actually, we very much don't want the load spread randomly. We want all traps from a given subnet to hit the same machine so we can correlate events to specific sites. Right now my IPTables rules set the DNAT destination based on the source of the trap.

[pQd]

@Christopher Cashell - then alternatively to your solution you can use u32 netfilter module to 'hash' destination server based on src ip address. eg take last 2 bits of src ip address and spread load to 4 snmp 'consumers'. netfilter.org/documentation/HOWTO/…

[pQd]

@Christopher Cashell stearns.org/doc/iptables-u32.v0.1.html is nice tutorial for u32 match. alternativly - look at "linux virtual server" project - they can do load balancing for udp packets based on src/dst ip as well.

[Aleksandar Ivanisevic]

I understood you right, probably you didn't understand me ;) JMS is used as a transport because modern brokers have all those nice failover, persistance and balancing features. You can POST to a URL, send an email, SOAP, whatever works. UDP was never built to be reliable or balancable since it has no concept of data stream or flow control. You'll just be screwed on the long run trying to make UDP do what it was not designed to do.

[demonkoryu]

I appreciate the suggestion, but I really have absolutely no intention or interest in building my own enterprise level network monitoring system. There are plenty of them already available, and implementing one with the feature set and scalability that we require would need a team of a dozen programmers for 2-4 years. It's not feasible or desirable. That leaves me with interacting with existing systems, and that leaves me dealing with a lot of SNMP over UDP.