Example standard VPC Subnetting and Addressing
How many online tutorials and examples instruct Dear Reader to create VPCs with the network address of 192.168.1.0/24? Or 10.0.0.0/24? Legions of AWS users are set up to learn about VPCs the hard way, because this practice not only limits the number of instances available (251 in a /24 on AWS), but it also limits your ability to subnet to cover availability zones, environments and things like private vs. public networks.
If you run out of room, you have to move to a new VPC and that is painful and best avoided.
This post is for folks who are using AWS but might not realise they are now Cloud Network Engineers:) VPCs are simple when you know them, but were all Donald Rumsfeld at some point: we dont know what we dont know (until it bites us in the proverbial).
Learning by doing sounds great, but hold on a minute. When youre using AWS its not just for the sake of it, youve got some end result to deliver. Right? So doing the minimum with VPCs (and unknowingly being productive today by creating technical debt for tomorrow) is fine. Right?
After all, why would you need to know about the best practices for VPCs if all you want to do is launch Wordpress on EC2? Isnt that the Network Engineers job?
No way! Thatt not me! (It is now. Cloud.) Everyone knows that Network Engineers are weird, right? I mean, who starts a fist fight about creating trunk ports to servers? Theyre nearly as bad as developers, talking to rubber ducks! And anyway, if theyre so awesome, why is everything their fault?! Or their networks fault. Isnt it all just bits of string? Who cares?
Fact is, in the cloud, you are the Network Engineer. Being weird is optional, but knowing VPCs is mandatory.
This post is about stepping away from the footgun and doing VPC subnetting in a non-lethal manner.
There are lots of changeable, programmable things in the cloud. Things you can change your mind about later. Change the instance size. Change the database size. Change security groups. Change the IAM role on an instance.
One thing you cant do, though, is modify a VPC network address after it has been created.
Imagine you decide on a basket size before you go shopping: the size of the basket decides how much shopping you can do. Once your shopping is in the basket and you need more basket then you either add a basket and try to carry two, or take all your shopping out of the current basket and put it in a bigger basket. So it is with VPCs. Just swap fruit for EC2 instances.
Non-cloud networks are different to cloud networks. Old school, non-cloud networks (ie. the majority out there in your home and office) worried about things like broadcast domains, spanning trees and all kinds of weird gobbledygook that only Network Engineers knew / cared about / used to protect their job.
In AWS, where you dont have to worry about broadcast domains and all that crap, you do networking differently. For a start, when you create your VPC, you dont carefully size it for what you need right now (your Wordpress on EC2 tutorial, for example): you size it so that you avoid problems in the future. That means: going big! Cloud networks are, in this way, totally different to non-cloud networks.
From the AWS doc on VPCs and Subnets:
VPC and Subnet Sizing for IPv4
You can assign a single CIDR block to a VPC. The allowed block size is between a
/28netmask. In other words, the VPC can contain from 16 to 65,536 IP addresses.
When you create a VPC, we recommend that you specify a CIDR block (of
/16or smaller) from the private IPv4 address ranges as specified in RFC 1918:
Are you familiar with CIDR? Scrumpy jokes aside, this is a tricky bit for even seasoned IT folks. I used to use variable-length subnet masking as an interview question until I saw someone cry, its that fun.
Does this shock you: you know that familiar IP address, 192.168.1.1? Did you know theres actually TWO addresses represented there?
Surely one is having the bantz?
No, Im serious. 192.168.1.1 is two addressesa NETWORK address and a HOST address. Sorry for shouting, but its important, and if we English know anything its that shouting gets results.
The way you find out the network and host address from a single IP address (no bantz) is by applying something called a netmask. It looks like this, the netmask being the /24 at the end of 192.168.1.1/24.
The /24 netmask at the end means that, if you wrote out 192.168.1.1 in binary there would be 32 bits, and the left-most 24 bits represent the network address. 24 is the count of bits from the left. Thats all it is.
The right-most bits (the 8 left of the whole 32) are the host address. In binary, eight bits adds up to 256 (128+64+32+16+8+4+2+1). In AWS this means you can have 251 instances in that network (broadcast is.255, and AWS reserve the first.1,.2,.3 leaving 251 possible instance addresses).
So, in the case of 192.168.1.1/24 the network address is the 192.168.1 octets (fancy talk for eight binary bits) and the host address (of your laptop on your home wifi) is the remaining.1.
The way you tend to describe the network address is to ignore the host bits on the right (cos, hey, were Network Engineers now and who cares about computers?) and just report the network address and the netmask, a la: The VPC network address is 192.168.1.0 / 24.
Dont believe me? Useful site alert.
Pack it in, Steve, whats CIDR got to do with VPCs?
Youve been in the AWS console and created a VPC with the address of 192.168.1.0/24 and you know that lets you have 254 hosts, less a few for AWS reserved use. Lets call is 251 for ca$h.
Quick point here, just checking were all awake: you do know that a VPC isnt a network you can put instances directly into, right? Even though weve given it a network address, its an AWS cloud construct that aligns with a region and spans availability zones. You can attach special things to a VPC like Internet Gateways, but you dont stick instances directly into a VPC: these every day non-networky things go in subnets.
Really, all youve done so far is create your shopping basket (VPC). Now you need compartments inside the basket to put your actual shopping (cloud objects like EC2 instances) inside. These compartments are called subnets.
You can create one or more subnets inside your VPC. You could create one subnet with the same network address as the VPC itself, thats totally possible but not fine at all. One subnet in your VPC means you can only ever have one usable network in your VPC, for your EC2s and everything else.
Why is this a problem? Well, what if you want two kinds of networksa public one connected to the internet for your web server, and a private network that the internet cant get to for your database of customer data? Cant do it with one subnet, its either on or off the internet. To do both in this case youd need two VPCs. Madness.
A public network in AWS is one that is has a route to the VPCs Internet Gateway (the gateway is attached to your VPC, not your subnet). If your house (VPC) didnt have a door (internet gateway), and even if it did have a door your couldnt find it (route), then nobody couldnt get in and out of your house. Same thing. Sometimes you want a door, sometimes you dont.
If you have created just one subnet in your VPC then you have to go all in on public or private, theres no public and private in the same subnet.
Then theres the other problem with one subnet in your VPC: a subnet is aligned to one availability zone, whereas your VPC is across all AZs in the Region. Some AWS features like RDS MultiAZ need more than one AZ. And if the AZ that houses your subnet spontaneously combusts, thats you done.
So, for both reasons (private and public, and multi-AZ) you probably want more than one subnet in your VPC. Without multiple subnets, theres no autoscaling for you. No need for a load balancer. No multi-AZ RDS database. Etc. Not a good direction. One subnet per VPC is bad for these reasons. But totally possible. Footgun.
You probably want six subnets as a minimum in your VPC, but its wise to allow for more (spares, if you like) because you know how the future likes to spring surprises. But lets do six for illustration:
Six subnets across three AZs in oneregion
All those orange boxes are subnets that we actually create in AWS (we dont create one big subnet for the AZ and we dont create one for the VPC).
How do we Make It So if our VPC has the network address of 192.168.1.0/24? Well, with subnets, the clue is in the name: we need to subdivide our network.
We get more networks by increasing the size of the netmask, but it costs us by gobbling up our hosts bits and reducing the number of instances we can shove into our shiny, new, half-size subnets. The netmask, per AWS, can go up to /28, but what should it be?
At this point, you can cheat, if you want to be like a glassy eyed fat seal catching fish in the zoo, and not knowing why youre unhappy.
How it works is this: for every extra bit you increase the netmask by, you are creating two more subnets that are are half the (hosts) size of the original network.
So, in our VPCs case:
we increase the netmask by one bit and it becomes two new networks with half as many hosts:
Note that the second network address is.128, thats because our netmask of /25 is now using the left-most bit of the last octet. That left-most bit is 128remember our binary: 1286432168421
But two subnets is not enough for us! We need six! So lets increase the netmask again by one bit to /26. We are now using the two left-most bits of the last octet, so now our network address boundary has moved from 128 to 64 so, if youre awake, we have four possible subnets and they are called.0,.64,.128 and.192:
Notice something? We started with a possible 251 hosts in our 192.168.1.0/24 network, now we have a possible 240 as each network gobbles up a few special addresses.
What! What? Why 240? Because we have four networks, and each network loses four addresses to the AWS network gods, so with a network of 192.168.1.0 / 26 thats 60 host addresses, and 4 x 60 = 240.
However, we need six subnets, not four increasing the netmask again to /27 creates 8 subnets. There is no netmask boundary for six subnets (but youll see later theres a way to kinda do this).
The /27 is on the boundary of the 3rd left-most bit in the octet, which is 32, so you guessed it again, our network addresses go up in lots of 32.
We have achieved our goal of six subnets across three AZs in our region-wide VPC, but at a cost. Our original single network could host 251 EC2 instances or other AWS gizmos, but now we are down to (8 x 28 =) 224.
Is there a better way? Yes!
Go big is my advice for my fellow VPCing Subnetters!
Going big means creating a VPC with a small netmask. The AWS block size limits go from /16 to /28, so well go for /16.
For our illustration, well pick 172.16.0.0/16. That /16 means the left-most 16 bits, ie the first two octets of 172.16, are our network address and the right-most 16 bits give us room for 65,531 hosts. Before you say Dont be ridiculous, Steve, we dont need 65k hosts!have you forgotten our previous lesson? This is about creating subnets just as much as deploying instances. We need more than one subnet in a VPC and each time we increase the netmask and create two more networks, we are halving the amounts of hosts in those subnets.
Yeah, yeah, yeah. But how?
For our illustration were going to state the following for our VPC design:
With that said, we walk over to the whiteboard and scribble:
Graffiti masquerading as VPCdesign
The thinking that emerges from this whiteboarding is:
But, and this is important, this is design theory at this stage, and here is something that people get tripped up on:
We dont actually create all of these subnets in AWS.
This seemingly obvious fact is not obvious to people new to networking. If you tried to create all these subnets in AWS you would get overlap errors. You only create the leaf node subnets. Its clearer (I think!) when you read this then see the picture, then my list of subnets in AWS.
The full picture looks like this:
And in the VPC page of the AWS Console is looks like this:
The bigger the netmask the smaller the room to maneouvre. Not only do you limit yourself to small number of hosts, when you need to subnet it you have less room. Be careful, though, if you will need to peer that VPC as addresses cant overlap with your peers (routing, innit?) and the larger YOUR network the bigger chance of overlap. Start with a subnet closer to the size of /16 and not /24. You cant change a VPC after it has been created, and if its full of stuff like hosts and gateways then you dont want to go through the pain of migrating them to a new VPC. That way, lies madness.
Face of an AWS Admin who has one VPC with one Subnet with a /28address