The Invisible Highways — VPC Endpoints, Security Groups, NACLs & Private Connectivity on AWS

---

Episode 03: We go under the hood of private connectivity — covering VPC Endpoints, SSM, Security Groups, NACLs, and how traffic actually moves securely inside AWS.

---

Why Private Connectivity Matters

By default, when an EC2 instance needs to talk to an AWS service like S3 or SSM, that traffic goes out to the public internet — even though both are inside AWS. This is inefficient, expensive, and a security risk.

VPC Endpoints solve this by creating a private path between your VPC and AWS services, keeping all traffic on the AWS backbone — no internet required.

---

VPC Endpoints — Types and When to Use Them

There are three types of VPC endpoints:

Type	Works with	Cost	How it works
Gateway Endpoint	S3, DynamoDB only	Free	Entry added to route table — no ENI created
Interface Endpoint	SSM, ECR, SQS, Secrets Manager, CloudWatch, KMS, and 100+ services	Hourly per AZ	Creates an ENI with a private IP in your subnet
Gateway Load Balancer Endpoint	Third-party network appliances (firewalls, IDS/IPS)	Hourly	Routes traffic through security appliances transparently

Gateway Endpoint — Patching via S3

One of the most common uses of a Gateway Endpoint in enterprise environments is EC2 patching via S3. Patches are uploaded to an S3 bucket and EC2 instances pull them down during patching windows. Without an endpoint, that traffic goes through a NAT Gateway — costing money and adding an internet hop.

With a Gateway Endpoint, the flow looks like this:

EC2 Instance (private subnet)
  │
  │  outbound port 443 / 80
  │
Subnet Route Table
  │
  │  destination matches S3 prefix list?
  │
S3 Gateway Endpoint ──────────────────► S3 Bucket (patches/whitelisted URLs)
                                      (traffic stays entirely on AWS)

No NAT Gateway needed. Faster, free, and the traffic never leaves the AWS backbone.

Interface Endpoint — Best Practice

When creating Interface Endpoints, follow this AWS best practice:

Create a dedicated Security Group for the endpoint with inbound allowed on port 443 from your VPC CIDR only. Attach this SG to the endpoint — not the instance SG.

This keeps endpoint access locked down to resources inside the VPC and prevents any unintended external access.

---

SSM — Secure Private Access to EC2

AWS Systems Manager (SSM) allows you to connect to EC2 instances securely without opening port 22 for SSH and without needing a bastion host. It does this through an SSM Agent installed on the instance that communicates privately with the SSM service.

For SSM to work, the instance must be able to reach three specific endpoints:

Endpoint	Purpose
ssm	Core Systems Manager service communication
ec2messages	Relay for run commands sent to the instance
ssmmessages	Handles Session Manager terminal sessions

These can be reached via:

• A NAT Gateway (internet path)
• Interface Endpoints for each of the three services (recommended — fully private)

The instance must also have an IAM role attached with the AmazonSSMManagedInstanceCore policy. Without this, the agent cannot authenticate with SSM regardless of network connectivity.

Interview Answer: EC2 Showing as Unmanaged in AWS Inspector

First I would check Fleet Manager in Systems Manager to see if the instance is registered. If it shows as unmanaged, I would verify the SSM Agent is installed and running, restarting it if needed and reviewing logs at /var/log/amazon/ssm/.

Next I would confirm the instance has an IAM role with the AmazonSSMManagedInstanceCore policy attached.

Then I would check network connectivity — the instance must reach the ssm, ec2messages, and ssmmessages endpoints on port 443, either through a NAT Gateway or Interface Endpoints. The Security Group and NACL must allow outbound 443.

Once resolved, I would confirm the instance appears as managed in Fleet Manager before rechecking Inspector coverage.

---

VPC Endpoint Services — PrivateLink Between VPCs

A VPC Endpoint Service is different from a regular endpoint. It allows you to expose your own private service to other VPCs or AWS accounts — securely, without VPC peering, and without internet exposure.

The architecture looks like this:

VPC A (Consumer)                     VPC B (Provider)
─────────────────                    ──────────────────
EC2 Instance
  │
  ▼
Interface Endpoint ◄──── PrivateLink ────► Endpoint Service
(private IP via ENI)                            │
                                                ▼
                                        Network Load Balancer
                                                │
                                                ▼
                                        Application (EC2 / ECS)

Key things to know:

• VPC A and VPC B never peer — no overlapping CIDR concerns
• The provider must accept each connection request from a consumer
• Works across different AWS accounts — common when platform teams share internal services across business units
• Only works with NLB or Gateway Load Balancer on the provider side
• NLB is preferred because it has static IPs — ALB has dynamic IPs which makes it unsuitable for this use case

---

MongoDB Private Connectivity via Endpoint

Connecting to a managed MongoDB cluster (like MongoDB Atlas) privately requires creating an Interface Endpoint inside your VPC.

The setup needs:

• VPC ID, AWS account ID, and subnet ID shared with the MongoDB team
• MongoDB creates the Endpoint Service on their side
• You accept the connection on your side

Once connected, the Security Group configuration is:

Resource	Direction	Port	Source / Destination
Endpoint SG	Inbound	27017	VPC CIDR
EC2 SG	Outbound	1024–65535	Endpoint SG

Port 27017 is MongoDB's default. The EC2 outbound ephemeral range 1024–65535 is required for TCP return traffic from the database.

---

Security Groups — Stateful Firewall at Instance Level

A Security Group is a virtual firewall that controls inbound and outbound traffic at the EC2 instance level (technically at the ENI).

Key characteristics:

• Stateful — if a request is allowed in, the response is automatically allowed out. You do not need a separate outbound rule for return traffic.
• Allow rules only — you cannot explicitly deny traffic in an SG. Anything not allowed is implicitly denied.
• All rules evaluated — unlike NACLs, there is no rule priority. All rules are checked and the most permissive match wins.

Best Practice: Source Another SG

Instead of specifying an IP range as the source in an SG rule, you can specify another Security Group ID. This means only resources attached to that SG can communicate — not just any IP in a range.

Example: EC2 inbound rule source = ALB Security Group ID. Only the ALB can reach the EC2 — nothing else, even if it has the right IP.

---

ENI — The Elastic Network Interface

Every EC2 instance communicates through an ENI (Elastic Network Interface) — a virtual network card attached to the instance. It holds the private IP, the MAC address, and the Security Group associations.

ENIs are important for troubleshooting because VPC Flow Logs capture traffic at the ENI level. When debugging connectivity issues:

Request (forward flow):   User → ALB → EC2 ENI → Application
Response (reverse flow):  Application → EC2 ENI → ALB → User

If VPC Flow Logs show REJECT on an ENI:

• Inbound REJECT → missing inbound SG rule or NACL deny
• Outbound REJECT → missing outbound SG rule or NACL deny
• Both present as REJECT → NACL is the likely culprit (stateless)

---

NACL — Stateless Firewall at Subnet Level

A NACL (Network Access Control List) acts as a firewall at the subnet level, evaluated before traffic reaches any instance.

The critical difference from Security Groups: NACLs are stateless.

This means return traffic is not automatically allowed. For every connection you permit, you must explicitly allow both directions — including the ephemeral port range 1024–65535 for TCP return traffic.

NACL Rule Structure

Rules are evaluated in order by rule number — lowest number first. The first matching rule wins. The implicit * deny at the bottom catches everything not matched above.

Example — allowing HTTPS inbound to a private subnet:

Rule #	Direction	Protocol	Port	Source	Action
100	Inbound	HTTPS	443	10.0.0.0/16	Allow
200	Inbound	TCP	1024–65535	10.0.0.0/16	Allow
100	Outbound	HTTPS	443	10.0.0.0/16	Allow
200	Outbound	TCP	1024–65535	10.0.0.0/16	Allow
*	Inbound	All	All	0.0.0.0/0	Deny
*	Outbound	All	All	0.0.0.0/0	Deny

The * deny rules are permanent and cannot be removed — they are AWS defaults that exist on every NACL.

SG vs NACL — Quick Reference

	Security Group	NACL
Level	Instance (ENI)	Subnet
State	Stateful	Stateless
Rules	Allow only	Allow + Deny
Return traffic	Automatic	Must be explicitly allowed
Rule evaluation	All rules, most permissive wins	In order, first match wins
Default behaviour	Deny all inbound	Allow all (default NACL)

---

Architecture at a Glance

AWS Account
└── VPC  (10.0.0.0/16)
    │
    ├── Private Subnet
    │   ├── EC2 Instance
    │   │   ├── ENI  ──► VPC Flow Logs (CloudWatch)
    │   │   ├── SG: allow inbound from ALB-SG on 443
    │   │   └── SSM Agent  ──► SSM Interface Endpoint
    │   │
    │   ├── NACL: allow 443 + 1024-65535 both directions
    │   │
    │   └── MongoDB Endpoint  ──► Atlas Endpoint Service (port 27017)
    │
    ├── Route Table
    │   ├── S3 Gateway Endpoint  ──► S3 (free, no NAT needed)
    │   └── local: 10.0.0.0/16
    │
    └── Interface Endpoints (dedicated SG, inbound 443 from VPC CIDR)
        ├── com.amazonaws.ap-south-1.ssm
        ├── com.amazonaws.ap-south-1.ssmmessages
        └── com.amazonaws.ap-south-1.ec2messages

---

Key Takeaways

• VPC Endpoints create private paths to AWS services without touching the internet — Gateway Endpoints are free, Interface Endpoints have a cost.
• The S3 Gateway Endpoint is ideal for patching — EC2 pulls patches from S3 privately via the route table, no NAT Gateway required.
• SSM requires the instance to reach three endpoints (ssm, ec2messages, ssmmessages) and have the AmazonSSMManagedInstanceCore IAM policy attached.
• VPC Endpoint Services (PrivateLink) let you share private services across VPCs and accounts via NLB — without VPC peering.
• Security Groups are stateful and instance-level — return traffic is automatic.
• NACLs are stateless and subnet-level — you must explicitly allow return traffic on ephemeral ports 1024–65535.
• VPC Flow Logs at the ENI level show ACCEPT or REJECT — use this to diagnose whether SG or NACL rules are the issue.

---

This is Episode 03 of the Booting to Cloud — AWS Foundations series.

---

If this helped, drop a reaction and follow the series — more episodes dropping regularly.