Secure IPTables rules for Corosync
Solution 1
By default, Corosync uses IP multicast to communicate between nodes:
mcastaddr: 226.94.1.1
mcastport: 5405
Either configure your firewall to allow multicast traffic:
# iptables -A INPUT -p igmp -j ACCEPT
# iptables -A INPUT -m addrtype --dst-type MULTICAST -j ACCEPT
# iptables -A INPUT -p udp -m state --state NEW -m multiport --dports 5404,5405 -j ACCEPT
or switch to unicast.
Solution 2
When using multicast communication (the default for Corosync), IGMP traffic should be allowed and the Corosync packets can be allowed with rules that are much more specific than in the other answer. The following rules are sufficient (assuming that the OUTPUT
chain does not block any traffic):
iptables -A INPUT -p igmp -i $corosync_interface -j ACCEPT
for src_addr in $ip_addr_self $ip_addr_partner1 $ip_addr_partner2; do
iptables -A INPUT -i $corosync_interface -s $src_addr -d $ip_addr_self \
-p udp --source-port $(($corosync_port - 1)) \
--destination-port $corosync_port -j ACCEPT
iptables -A INPUT -i $corosync_interface -s $src_addr -d $ip_addr_mcast \
-p udp --source-port $(($corosync_port - 1)) \
--destination-port $corosync_port -j ACCEPT
done
In this example, the following variables are assumed to be defined:
-
$corosync_interface
: the network interface which is used by Corosync -
$ip_addr_self
: the IP address to which Corosync binds locally (specified asbindnetaddr
incorosync.conf
) -
$ip_addr_partner1
,$ip_addr_partner2
: the IP addresses of the other Corosync nodes - more can be added if the cluster has more than three nodes. -
$ip_addr_mcast
: the multicast address used for Corosync (specified asmcastaddr
incorosync.conf
) -
$corosync_port
: the (destination) port used by Corosync (specified asmcastport
incorosync.conf
)
On one node, where the interface used by Corosync is a port that is a member of an Open vSwitch bridge, some of the multicast packets where received on the interface of the bridge instead of the one which had the IP address. Therefore, I had to add an additional rule that allowed multicast packets on this interface:
for src_addr in $ip_addr_self $ip_addr_partner1 $ip_addr_partner2; do
iptables -A INPUT -i $bridge_interface -s $src_addr -d $ip_addr_mcast -p udp --source-port $(($corosync_port - 1)) --destination-port $corosync_port -j ACCEPT
done
If the OUTPUT
chain does not accept the packets by default, rules need to be added that allow IGMP traffic and that allow sending Corosync packets:
iptables -A OUTPUT -p igmp -o $corosync_interface -j ACCEPT
for dst_addr in $ip_addr_self $ip_addr_mcast $ip_addr_partner1 $ip_addr_partner2; do
iptables -A OUTPUT o $corosync_interface -s $ip_addr_self -d $dst_addr \
-p udp --source-port $(($corosync_port - 1)) \
--destination-port $corosync_port -j ACCEPT
done
Related videos on Youtube
jetboy
Updated on September 18, 2022Comments
-
jetboy almost 2 years
I have two HA load balancers (
hollywood
andwolfman
) running Corosync and Pacemaker. Theeth1
interfaces are connected to the WAN, and theeth0
interfaces to the LAN, using a virtual IP as the gateway for the back end servers. Theeth1
IP ofhollywood
isxxx.xxx.195.45
, and theeth1
IP ofwolfman
isxxx.xxx.195.46
. Thebindnetaddr
in Corosync isxxx.xxx.195.32
, the same as the WAN's network address, and the Corosync port is the default5405
.The relevant IP tables rules on both servers are:
*filter --flush :INPUT DROP --append INPUT --protocol udp --destination-port 5404 --jump ACCEPT --append INPUT --protocol udp --destination-port 5405 --jump ACCEPT
This setup seems to work fine, but initially I added
--in-interface eth1
and--source xxx.xxx.195.46
towolfman
, and--source xxx.xxx.195.45
tohollywood
. Most of the time this seemed to work, but rebooting the passive balancer sometimes killed communication between the load balancers, writing these errors to syslog:[TOTEM ] Totem is unable to form a cluster because of an operating system or network fault. The most common cause of this message is that the local firewall is configured improperly.
So it seems that either my simplistic belief that all the Corosync traffic is directly between the two load balancers over
eth1
is wrong, or that something else is causing a problem.I'd like to lock port
5404/5405
down in IPTables to just the cluster. What do I need to do to make this happen?Edit:
corosync.conf
as requested. This is all default Ubuntu other than thebindnetaddr
.# Please read the openais.conf.5 manual page totem { version: 2 # How long before declaring a token lost (ms) token: 3000 # How many token retransmits before forming a new configuration token_retransmits_before_loss_const: 10 # How long to wait for join messages in the membership protocol (ms) join: 60 # How long to wait for consensus to be achieved before starting a new round of membership configuration (ms) consensus: 3600 # Turn off the virtual synchrony filter vsftype: none # Number of messages that may be sent by one processor on receipt of the token max_messages: 20 # Limit generated nodeids to 31-bits (positive signed integers) clear_node_high_bit: yes # Disable encryption secauth: off # How many threads to use for encryption/decryption threads: 0 # Optionally assign a fixed node id (integer) # nodeid: 1234 # This specifies the mode of redundant ring, which may be none, active, or passive. rrp_mode: none interface { # The following values need to be set based on your environment ringnumber: 0 bindnetaddr: xxx.xxx.195.32 mcastaddr: 226.94.1.1 mcastport: 5405 } } amf { mode: disabled } service { # Load the Pacemaker Cluster Resource Manager ver: 0 name: pacemaker } aisexec { user: root group: root } logging { fileline: off to_stderr: yes to_logfile: no to_syslog: yes syslog_facility: daemon debug: off timestamp: on logger_subsys { subsys: AMF debug: off tags: enter|leave|trace1|trace2|trace3|trace4|trace6 } }
-
Greg Petersen almost 12 yearsAre you running multicast? Could you please post the content of
corosync.conf
? -
jetboy almost 12 yearscorosync.conf as requested.
-
-
Jacek Konieczny over 11 yearsThe UDP rule provided in the question would often be enough to accept the multicast traffic. I don't think accepting IGMP is needed here – more important is that IGMP can be sent out, so any switch in between can set up multicast forwarding. And accepting every multicast, as you suggest is not needed, so probably not a good idea, especially when a rule for specific UDP ports follows.
-
BatchyX over 11 years@JacekKonieczny: If you don't acccept IGMP, then you may violate the IGMP protocol, as you will send unecessary IGMP packets even if another host already sent membership reports for the same group.
-
Jacek Konieczny over 11 years@BatchyX: thanks for the reminder (I think I may need to review my firewall rules… ok, I have this enabled), though wrong or not, it should not break corosync communication, I guess…