Configuring multiple ENIs on an EC2 instance

Question

How do you configure the interfaces in the OS without using ec2-net-utils?

I need to configure around 18 public ips to be accessible on a single EC2 instance. Each public IP needs to be bound to a single internal IP on the instance. I have 2 ENIs configured with the necessary Elastic IPs and these are attached to the instance but I am stuck at getting the OS recognise the ENIs.

I have been following this tutorial but can't work out how to extend it to configure multiple IPs per interface. Also there seems to be some debate according to this question about what the correct way to do this is.

If anyone would be able to point me in the right direction I would be really grateful. I have nearly wasted 2 days on this!

Here are the details:

• OS: Centos 7
• ENI1 : Secondary Private IPs = [10.0.1.97 ...10.0.1.106]
• ENI2 : Secondary Private IPs = [10.0.1.218 ... 10.0.1.226]
• VPC gateway 10.0.1.1

Solution 1

The answer by @mootmoot got me most of the way on this one but, just for completeness, here are the steps that eventually got me all the way to a working configuration. I'm sure there are better ways to do this but in case someone else is stuck ...

I made the following modifications to the files in /etc/sysconfig/network-scripts

I left the interface defined in ifcfg-eth0 alone and added aliases as follows:

ifcfg-eth0:0

DEVICE=eth0:0
BOOTPROTO="static"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
PEERDNS="yes"
IPV6INIT="no"
DEFROUTE="no"


IPADDR0="10.0.1.XXX"
PREFIX0="24"

Where 10.0.1.XXX is the first secondary address on the ENI (the primary address remains configured by dhcp as specified in the default ifcfg-eth0 file)

continue adding ifcfg-eth0:X files until all the ips in the ENI are specified.

Then define the second device in the file ifcfg-eth1 using the same template as for the eth0 aliases above and the primary IP address of the ENI.

Then add aliases for eth1 by defining files called ifcfg-eth1:0 etc as above.

define the following routes files:

route-eth0

default via 10.0.1.1 dev eth0  table 1

route-eth1

default via 10.0.1.1 dev eth1  table 2

where 10.0.1.1 is the address of the VPC gateway

I then needed to define rules for all IPs in eth2 and all secondary ips in eth1 as follows:

rule-eth0

from 10.0.1.106 lookup 1
from 10.0.1.105 lookup 1
... etc

rule-eth1

from 10.0.1.41 lookup 1
from 10.0.1.226 lookup 1
... etc   

I also added the line

GATEWAYDEV=eth0

to /etc/sysconfig/networking

Hope that helps someone in a similar situation, thanks for all the advice.

Solution 2

Since the previous answer is clutter, I will put another one with workflow, just by using AWS CLI (you can write fine tune automation script using AWS SDK) (http://docs.aws.amazon.com/cli/latest/userguide/installing.html)

1. Create ENI with private IP using. (aws ec2 create-network-interface) Write down ENI id
2. Allocate EIP for VPC. (aws ec2 allocate-address --domain vpc) write down EIP-id
3. Link EIP to ENI-id, point to correct private IP (aws ec2 associate-address)
4. Create or launch EC2 instance, attach to the ENI. (aws ec2 attach-network-interface)

Once you put everything script in place, recreate the EC2 instance with the proper ENI is just matter of minutes.

(Updated answer below) In Linux, to assign multiple IP address to an interface, the correct assignment to the interface is to add additional ip address to the physical interface. For ubuntu , etc, it is something like eth0:0 , eth0:1, for 1st interface, eth1:0, eth1:1 for subsequent interface.

And this is slightly different for Centos, ie.

enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
link/ether ....
inet 192.168.1.150/24 brd 192.168.1.255 scope global enp0s3
inet 192.168.1.151/24 brd 192.168.1.255 scope global secondary enp0s3
inet 192.168.1.152/24 brd 192.168.1.255 scope global secondary enp0s3

So the correct documentation should be this one. http://www.unixmen.com/linux-basics-assign-multiple-ip-addresses-single-network-interface-card-centos-7/

In short, Centos will automatically create one network-interface file for each interface. Just go /etc/sysconfig/network-scripts/ and check each file name as ifcfg-eth* (don't ask me why the above link show enp0) . The tricky part is whether your Centos enforce to use NetworkManager, and you must configure as required by the centos documentation wiki.centos.org/FAQ/CentOS7

So you should see least 4 interface file for your m3.xlarge , e.g. /etc/sysconfig/network-scripts/ifcfg-eth0 /etc/sysconfig/network-scripts/ifcfg-eth1 /etc/sysconfig/network-scripts/ifcfg-eth2 /etc/sysconfig/network-scripts/ifcfg-eth3

Since the main interface should be running, it will give you the hint of the running instance IP address. So if you open /etc/sysconfig/network-scripts/ifcfg-eth0, if the instance configure as IP 10.0.1.10, you should see this

IPADDR0="10.0.1.10"

To add more IP address, just do as unixmen link say, i.e.

IPADDR1="10.0.1.97"
IPADDR2="10.0.1.98"
IPADDR3="10.0.1.99"

Then open ifcfg-eth1, ifcfg-eth2, ifcfg-eth3 and do repeat the task. After done that, use "systemctl restart network" to restart. (backup all config file so you just override the file in the future)

Next , you play with the routing part. Just print out your VPC route table, Subnet. Now inside /etc/sysconfig/network-scripts/, you deal with route-ethX , i.e. route-eth0, route-eth1, route-eth2,route-eth3. That's mean, you must know your own network to configure this part (which should be shown in your VPC layout). Since you mentioned only one VPC gateway 10.0.1.1, so for all route-* file, it should be something like this

# I just assume your put all your 10.0.1.x in the CIDR /24 segments
#
# file route-eth0
# Assume your first ENI IP address is  10.0.1.10 
# format : default via gateway-ip dev dev-name table route-table-number 
default via 10.0.1.1 dev eth0 table 0

# format : network cidr  dev dev-name src ENI-intrace-IP route-table-number  
10.0.1.0/24 dev eth0 src 10.0.1.10 table 0

# file route-eth1
# Assume your 2nd ENI IP address is  10.0.1.11
default via 10.0.1.1 dev eth1 table 1
10.0.1.0/24 dev eth1 src 10.0.1.15 table 1

# file route-eth2
# Assume your 2nd ENI IP address is  10.0.1.12
default via 10.0.1.1 dev eth1 table 2
10.0.1.0/24 dev eth2 src 10.0.1.12 table 2

Then you follow the document you mentioned instruction, in /etc/sysconfig/network-scripts, create a rule-ethX , i.e. rule-eth0, rule-eth1 1. Increment the table number to match route-ethX 2. Change the IP to the assigned internal network address of the ENI.

# file rule-eth0 ,but as the doc suggest, you should skip this file.
# format : from  ENI_IP/CIDR table <table_number> 
from 10.0.1.10/32 table 0

#file rule-eth1
from 10.0.1.11/32 table 1

You should play the Centos network setup with your Local Vmware/virtualbox virtual network adapter. Then you don't need to worry a sudden

Comments

[Michael - sqlbot]

If this doesn't get you somewhere useful, please tell us where you're stuck, more specifically... what you've tried, and in what way it fails. You might also examine the source code of ec2-net-utils, to give you some idea of what mechanism that code uses.

[Michael - sqlbot]

Are you trying to automate the config, or just make it work? /sbin/ifconfig eth0:1 10.0.0.97/24 up should, for example, be sufficient to bring up the next interface... then eth0:2 eth0:3 and so on. There's (afaik) no DHCP support for the secondary addresses.

[Russell]

Yes I am aware of the limit of the number of addresses per ENI, I am using m3.xlarge so this is not the problem. Splitting to multiple machines is not an option either. This is a staging server set up to replicate one of our physical servers. I know this can be done. It was done before but unfortunately I terminated the previous instance without realizing how complicated it would be to recreate the network set-up.

[mootmoot]

even m3.xlarge only support 4 ENI , 15 Private address per interface. Again, You may request "exceptional request" from AWS; or the "magic" actually is done through ELB or something else. Unless you are the one who create the network interface, otherwise, you need to refer to the person who "make things works before", or any documentation that explain how it is done.

[Russell]

I have 2 ENIs , 10 private interfaces per interface so this is fully supported. In fact if we were using an Amazon Linux AMI you can configure all this with ec2-net-utils (Unfortunately this is not an option). This is fully possible with Centos of course as well, just need to configure manually. Telling me to look at "any documentation" does not really help either. That is why I'm asking on Server Fault. If you don't have an answer then please remove your post.

[mootmoot]

It is an issue of EIP limit towards your Private IP address. If the EIP limit is not raise (5 per VPC) , you cannot make a 1 to 1 assignment from EIP towards the private IP. I just suspect the "working instance" that you terminated, actually achieve the goal using some other method. Looking at the Centos AMI, I have no problem putting Private IP address until it hit the limit. Next , assigning EIP to specific ENI and specific Private IP address, no issue either. But then, I use up my ENI.

[Michael - sqlbot]

@mootmoot EIP is always mapped 1:1 towards a single private IP. Some of the questions raised in the body of this answer might have been more appropriate as comments requesting clarification. We know that OP has a compatible supported VPC/ENI configuration. The question is, simply stated, appears to be "how do you configure the interfaces in the OS without using ec2-net-utils." This answer strays far from there. -1 I'm afraid.

[mootmoot]

@Michael - sqlbot , ec2-net-utils IS NOT a default package for ANY Linux AMI except Amazon Linux. IMHO, if you can do the works using AWS SDK api or aws cli, (create instance+ add private IP+ assign EIP) you should not play with custom tools that you are not sure it will works. (can you confirm ec2-net-utils works under CentOS?)

[mootmoot]

Did you notice the soft limit of EC2 EIP (5 EIP per VPC) ? Imagine you play with VPC and hit the soft limit, what will you do? Will you bet on some tools and hope it will overcome the limit?(And definitely won't work.) Isn't it too soon to make conclusion and say "it cannot be fully done with standard tools". Anyway, I just write another answer with simple workflow with AWS CLI hint.

[Russell]

all this does is attach the ENI to the instance, there is still the problem of configuring the OS to use the secondary IP address(s). See the docs at docs.aws.amazon.com/AWSEC2/latest/UserGuide/MultipleIP.html, specifically the section on "Configuring the Operating System on Your Instance to Recognize the Secondary Private IP Address"

[mootmoot]

Oh, now I understand the issues. Seems I owe an apology to @Michael - sqlbot. So it appear to be bad documentation from those solution . :-D

[mootmoot]

The internetstaff document say "Multiple EC2 Network Interfaces " , it didn't say multiple ENI with multiple private address. I assume you get stuck there.

[Russell]

Thanks for improving the answer @mootmoot. This pretty much gets the job done. The final thing I have left is that, although I can communicate with the machine on all the public addresses fine, the machine cannot resolve external addresses itself, for example ping 8.8.8.8 outputs "Network is Unreachable" this is solved if I type "route add default gw 10.0.1.1 eth0" but I cannot make this change permanent. I have tried adding "GATEWAY=1.0.1.1" in /etc/sysconfig/network and in /etc/sysconfig/network-scripts/ifcfg-eth0 but nothing helps. Any ideas?

[mootmoot]

Since internetstaff suggest use the default route, that's why I suggest to skip route-eth0. Maybe you should add route-eth0 , with the entry "from <First_ENI_IP>/32 table 0" (replace that <> stuff with your first ENI IP), then restart the network. Because it change the default route. As important reminder : always try to this out in your local VM. Unless you are able to ssh the instance via another ENI IP, then you still have a backup connection if route-eth0 screw up.

[mootmoot]

p/s: just a reminder, there is ALWAYS 5 IP address reserved for every subnet by AWS. 1. network address 2. 1st IP address for the subnet for aws router, 3. 2nd IP of subnet for aws DNS, 4. 3rd IP address of subnet for reserved purpose, 5. The broadcast IP address