Expired certificates are still behind some of the most avoidable outages. A browser warning, failed API calls, broken mobile apps, all from one date someone forgot to track.

Renewal is also about to get more frequent. The CA/Browser Forum approved shorter maximum public TLS certificate lifetimes: 200 days since March 2026, 100 days in 2027, and 47 days by 2029. Manual renewal won’t keep up with that.

Here’s how I manage the full certificate lifecycle in AWS using ACM + Terraform. 👇

🔄 The lifecycle 1️⃣ Request → 2️⃣ Validate → 3️⃣ Deploy → 4️⃣ Monitor → 5️⃣ Renew → 6️⃣ Retire

1️⃣ Request the certificate (ACM)

💡 Use DNS validation, not email. As long as the CNAME record stays in place, ACM renews the certificate automatically.

2️⃣ Automate DNS validation (Route 53)

3️⃣ Deploy to the load balancer

💡 Reference the validation resource’s ARN, not the certificate’s. That way Terraform waits until the certificate is actually issued before attaching it.

💡 For CloudFront, the certificate must be in us-east-1. Use a provider alias:

4️⃣ Monitor expiry. Trust auto-renewal, but verify it.

💡 If one of these alerts fires on an ACM-managed DNS-validated certificate, auto-renewal has failed. The usual cause is a deleted validation CNAME.

5️⃣ Renew & 6️⃣ Retire ✅ ACM renews DNS-validated certificates on its own, with no redeploy needed for integrated services (ALB, CloudFront, API Gateway). ✅ create_before_destroy makes replacements (new SANs, key algorithm change) zero-downtime. ✅ Imported third-party certificates do not auto-renew, so automate re-import in your pipeline or move them to ACM. ✅ Run terraform destroy on retired environments so orphaned certificates and validation records don’t pile up.

#AWS #Terraform #DevOps #CloudEngineering #InfrastructureAsCode #ACM #CloudSecurity #SRE #PlatformEngineering

While pushing my Terraform infrastructure code to GitHub, I encountered a frustrating error:

“File terraform‑provider‑aws_v6.67.0_x5.exe exceeds GitHub’s 100 MB file‑size limit.”

Terraform automatically downloads provider binaries into the .terraform directory, and these files can easily exceed GitHub’s upload limit. Even after adding a .gitignore, Git had already committed those large files, causing repeated push failures.

⚙️ My Project Setup

Inside my infrastructure folder (roboshop‑dev‑infra), I had multiple Terraform modules:

Code

00-vpc/
10-sg/
20-sg-rules/
30-bastion/
40-databases/
.gitignore
.gitattributes 

Each module contained its own main.tf, variables.tf, and terraform.lock.hcl.

🧾 The Initial Attempt I tried re‑initializing the repository:

bash

cd roboshop-dev-infra/
rm -rf .git
git init
git branch -M main
git remote add origin https://github.com/mokridevops/roboshop-dev-infra.git
git add .
git commit -m "updated"
git push origin main 

But GitHub rejected the push because the large provider binaries were already tracked.

🧠 The Root Cause Even though I added the following patterns to .gitignore, the oversized files had already been committed:

bash

Git doesn’t retroactively untrack files that were committed before .gitignore was updated.

🛠️ The Fix

To resolve this, I created a new folder and initialized a fresh Git repository:

bash

Before committing, I ensured .gitignore contained all Terraform‑related exclusions. Then I committed and pushed again:

bash

git add .
git commit -m "created infra v1"
git push origin main 

✅ This time it worked perfectly!

🧩 Why This Works

• The new repo starts clean, without previously tracked large files.

• .gitignore prevents Terraform binaries and state files from being added.

• Only essential configuration files (.tf, .tfvars, .hcl, .sh) are committed.

🌟 Final Outcome

After re‑initializing and cleaning up, I successfully pushed my Terraform infrastructure code to GitHub without hitting the file‑size limit. This experience reinforced the importance of proper .gitignore management and repository hygiene in DevOps workflows.

It started with a basic requirement:
Server A needed to connect to MySQL running on Server B over a private VPC network.

Everything looked perfect.

• Same VPC

• Same subnet

• Same Availability Zone

• Security Groups open

• NACLs open

• MySQL running

And yet… every connection attempt timed out.

This is the kind of problem that forces you to slow down, think systematically, and peel back each layer until the truth reveals itself.

The Problem: MySQL Was Completely Unreachable

Server A simply couldn’t reach Server B’s private IP.
ping failed.
nc -zv timed out.
MySQL refused to respond.

When both ping and nc fail, the issue is no longer the application — it’s the network path itself.

So I began troubleshooting layer by layer.

Step 1: Application Layer — MySQL Was Listening Only on IPv6

From the host:

“tcp6 :::3306 LISTEN”

That was the first red flag.
VPC communication uses IPv4 private IPs, so an IPv6‑only listener is effectively invisible.

Fix:
I updated MySQL’s configuration:

bind-address = 0.0.0.0

After restart:

“tcp 0.0.0.0:3306 LISTEN”

Application layer cleared.

Step 2: OS Layer — Host Firewall Ruled Out

I checked both firewalld and iptables:

• firewalld → inactive

• iptables → ACCEPT on INPUT, FORWARD, OUTPUT

No host‑level firewall was interfering.

This matters because OS firewalls can silently drop packets even when AWS networking is perfectly configured.

Step 3: AWS Security Groups — All Good

Inbound:

• Port 3306 allowed

• Source: 0.0.0.0/0 + Server A’s private IP

Outbound:

• All traffic allowed

Security Groups are stateful, but both directions still need to allow the traffic you expect. Everything checked out.

Step 4: AWS NACLs — Also Good

Inbound: ALLOW ALL
Outbound: ALLOW ALL

NACLs are stateless — return traffic is not automatic — but in this case, both directions were fully open.

Step 5: Subnet & Availability Zone — Verified

Both servers were in:

• us‑east‑1a

• subnet‑0c4c44c9d55e260e4

Subnets are AZ‑specific, so comparing IDs (not names) is critical. No mismatch here.

Step 6: Connectivity Tests — Both Failed

• ping 172.31.11.193 → no replies

• nc -zv 172.31.11.193 3306 → timeout

At this point, the application, OS, SGs, NACLs, and subnet were all correct.

Only one layer remained: routing.

Step 7: The Real Fix — A Missing VPC‑Local Route

This was the “aha” moment.

Using EC2 instance metadata, I inspected the route table associated with Server A’s subnet.

The route that should always be present was missing:

“172.31.0.0/16 → local”
MISSING

This single missing route explained everything:

• No private IP communication

• All internal traffic dropped

• Only internet‑bound traffic worked

Even though both servers lived inside the same VPC, Server A had no internal path to reach Server B.

Final Fix: Add the VPC‑Local Route

I added the missing route:

• Destination: 172.31.0.0/16

• Target: local

Instantly, everything came back to life:

nc -zv 172.31.11.193 3306
Connected!

MySQL login succeeded.
Private EC2‑to‑EC2 communication fully restored.

Before & After: What Actually Happened

Before Fix — Broken Private Network

Server A’s route table only had:

• 0.0.0.0/0 → igw

It was missing:

• 172.31.0.0/16 → local

Result:
Packets had no internal route inside the VPC.
Everything timed out.

After Fix — Private Path Restored

Once the local route was added:

• Internal routing worked end‑to‑end

• MySQL became reachable

• Ping and nc succeeded

A single missing route caused a full outage.

Key Takeaway: Troubleshoot Layer by Layer

This experience reinforced a powerful lesson:

Cloud troubleshooting is never guesswork — it’s a systematic, multi‑layer process.

• Application

• OS

• Security Groups

• NACLs

• Subnet

• Connectivity

• Routing

And above all:

EC2 metadata is the source of truth.
Never assume the AWS console tells the full story.

This was a proud DevOps learning moment — and a checklist I’ll reuse for every future outage.

Recently, I ran into one of those moments.

I was working inside an EC2 instance and needed to launch another EC2 instance using AWS CLI. Simple task, right? Except every AWS CLI command I ran — from sts get-caller-identity to ec2 describe-security-groups — returned absolutely nothing.

No JSON.
No error.
No text.
Just a silent return to the shell prompt.

This wasn’t a failure. It was worse: a non‑response.

And that kicked off one of the most interesting debugging sessions I’ve had in a while.

🔍 The Mystery: AWS CLI Commands Returning No Output

When AWS CLI fails, it usually tells you:

• “Unable to locate credentials”

• “You must specify a region”

• “AccessDenied”

• “Endpoint connection timed out”

But this time?
Nothing. Not even an error message.

That meant the problem wasn’t IAM, wasn’t region, wasn’t CLI installation — something deeper was happening.

So I started peeling the layers.

🧪 Step 1 — Verify AWS CLI Installation

I checked:

which aws
aws --version

Everything looked perfect. AWS CLI was installed and executable.

So the binary wasn’t corrupted.

🧪 Step 2 — Test Internet Connectivity

I tried:

wget https://aws.amazon.com

It worked beautifully. DNS resolution was fine. Outbound internet was fine.

So the instance wasn’t isolated.

🧪 Step 3 — Test AWS API Endpoint Connectivity

This is where things got interesting.

I ran:

curl https://sts.amazonaws.com
curl https://ec2.amazonaws.com

Both commands returned nothing.

Not an error.
Not a timeout.
Not even HTML.

Just silence.

This told me something very specific:

👉 My EC2 instance could reach public websites,
but could NOT reach AWS API endpoints.

That’s a huge clue.

🧩 The Root Cause: Outbound HTTPS Traffic Was Blocked

AWS CLI communicates with AWS services using HTTPS (port 443).
If outbound HTTPS is blocked, AWS CLI simply cannot talk to AWS.

And that’s exactly what was happening.

The EC2 instance’s security group outbound rules were too restrictive.

Even though inbound rules were fine, outbound rules were silently blocking AWS API calls.

Once I updated the outbound rule to:

Type: HTTPS
Port: 443
Destination: 0.0.0.0/0

everything changed instantly.

🎉 The Breakthrough

I ran:

aws sts get-caller-identity

And finally saw:

{
    "UserId": "AIDAZRHYHRONASSESAPVL",
    "Account": "655503100826",
    "Arn": "arn:aws:iam::655503100826:user/shell_user"
}

AWS CLI was alive again.

Curl started returning XML.
EC2 commands started working.
The instance could finally talk to AWS APIs.

🧠 What This Taught Me

This issue was a great reminder of how AWS networking layers interact:

✔ Security groups affect outbound traffic

Even though they’re stateful, outbound rules still matter.

✔ Public internet ≠ AWS API access

Reaching aws.amazon.com doesn’t mean you can reach sts.amazonaws.com.

✔ AWS CLI silence is a networking symptom

When AWS CLI returns nothing, it’s often because the API endpoint never responded.

✔ DevOps troubleshooting is layered

You need to check:

• CLI installation

• IAM credentials

• DNS

• VPC routing

• Security groups

• API endpoint reachability

One missing piece can break the entire chain.

🛠 Final Fix

To restore AWS CLI functionality:

• I tightened outbound rules to HTTPS 443 → 0.0.0.0/0

• Verified AWS API connectivity with curl

• Re-tested AWS CLI commands

• Confirmed IAM identity and EC2 access

Everything worked perfectly.

📌 Conclusion

This wasn’t just a networking issue — it was a great real‑world example of how DevOps requires thinking across multiple AWS layers simultaneously.

A silent AWS CLI isn’t a broken CLI.
It’s a sign that your EC2 instance can’t reach AWS itself.

And sometimes, the fix is as simple as allowing outbound HTTPS.