add default route for redhat permanently

Question

I have a cluster consists of several RedHat_7 PCs. The master node connects to two networks(Internal and Internet). I used to utilizing NAT to provide Internet service for all the computers.

For some reason, this method doesn't work now. The NAT cmds shows below.

sudo iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -o em4 -j MASQUERADE
sudo sysctl -w net.ipv4.ip_forward=1

and, usually, I have to add a default route,

sudo route add default gw 'my_gateway' em4

Then, I can surf the Internet. However, it works only for a couple of minutes, so I have to execute these cmds periodically.

route -n gets below:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.10.10   0.0.0.0         UG    100    0        0 em1
0.0.0.0         X.X.X.1         0.0.0.0         UG    101    0        0 em4
192.168.10.0    0.0.0.0         255.255.255.0   U     100    0        0 em1
X.X.X.0         0.0.0.0         255.255.224.0   U     0      0        0 em4
X.X.X.0         0.0.0.0         255.255.224.0   U     100    0        0 em4

and ip route show gets below:

default via 192.168.10.10 dev em1  proto static  metric 100 
default via X.X.X.1 dev em4  proto static  metric 101 
192.168.10.0/24 dev em1  proto kernel  scope link  src 192.168.10.10  metric 100 
X.X.X.0/19 dev em4  proto kernel  scope link  src MY_IP 
X.X.X.0/19 dev em4  proto kernel  scope link  src MY_IP  metric 100 

if I execute the route add cmd, changes show below:

0.0.0.0         X.X.X.1      0.0.0.0         UG    0      0        0 em4

default via X.X.X.1 dev em4 

and after several minutes, these changes are gone. I have some questions.

1. why the default route is deleted after some time?
2. the static label in ip route show means what?
3. how to solve the whole problem?

Thanks for any help.

EDIT

The configuration file for these two file are shown below:

/etc/sysconfig/network-scripts/ifcfg-em1

TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
NAME=em1
UUID=MYUUID
ONBOOT=yes
HWADDR=MYMAC
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPADDR=192.168.10.10
PREFIX=24
GATEWAY=192.168.10.10

/etc/sysconfig/network-scripts/ifcfg-em4

TYPE=Ethernet
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_PEERDNS=yes
IPV6_PEERROUTES=yes
IPV6_FAILURE_FATAL=no
NAME=em4
UUID=MYUUID
ONBOOT=yes
HWADDR=MYMAC
PEERDNS=yes
PEERROUTES=yes

Solution

It looks like you might be using DHCP for your internal network, and your DHCP config seems to include a default route.

Your issue is basically down to these two entries:

default via 192.168.10.10 dev em1  proto static  metric 100 
default via X.X.X.1 dev em4  proto static  metric 101

Your em1 default route has a slightly lower cost than your em4 default gateway, meaning the route via 192.168.10.10 is being preferred.

For the one ('master') node providing connectivity with the internet, your best bet is to set a static IP (the master node is presumably a gateway, so a static IP would I think make sense), and not use DHCP.

You could also create a host-specific configuration in DHCP for your master node, but I suspect making the master node static is simpler.

Comments

[stupig]

after adding this line, I cannot restart network service.

[stupig]

Mar 03 17:14:36 master systemd[1]: network.service: control process exited, code=exited status=1 Mar 03 17:14:36 master systemd[1]: Failed to start LSB: Bring up/down networking. Mar 03 17:14:36 master systemd[1]: Unit network.service entered failed state. Mar 03 17:14:36 master systemd[1]: network.service failed.

[mzhaase]

You are not supposed to add it, but change the existing one.

[stupig]

The original file is empty. Now, I cannot start network service, even if I restore the file.

[stupig]

No, my internal network is configured statically through GUI.

[stupig]

Using three cmds: systemctl stop NetworkManager, and systemctl disable NetworkManager, then reboot. The problem is solved. Now I only have one default gateway belongs to em4. This is equivalent to what you said. However, my network service cannot startup.

[iwaseatenbyagrue]

Ah, no, not necessarily: NetworkManager (probably) manages your whole network (so disabling is what is causing no network startup). I think you have a config issue on your master node, which is causing em1 to do things you don't want it. You might want to poke around /etc/sysconfig/network-scripts via the command-line - doing things via the GUI isn't always as straightforward.

[stupig]

Yeh, you are right. I checked from setting panel. It shows that The system network services are not compatible with this version. I added the configuration files of these two NICs. So should I set DEFROUTE=no in ifcfg-em1?

[iwaseatenbyagrue]

I think that should work, yes - good idea. If you do have a totally static config for em1, you could also consider adding NM_CONTROLLED=no.