03 · AWS by handLab 18 / 59
Guided labaws
EC2 Operations: SSM, CloudWatch Logs & Metrics
Operate an instance without SSH: run commands, ship logs, and alarm on something that matters.
- Time
- 50 min
- Level
- Intermediate
- Objectives
- 4 objectives
- Cost
- Low cost
Where this fits in the platform
Already built
This lab adds
- An instance you can reach without opening SSH to the world
Which lets you
—
Before you start
You will need
- AWS CLI v2, configured
- An AWS account
You do not need these already — the lab environment below provides them.
You will be able to
- Administer an instance with no inbound ports open
- Ship application logs to CloudWatch and query them
- Alarm on a symptom rather than on CPU
Cost — Low cost
— one `t3.micro`, and CloudWatch's free tier covers 5 GB of logs and 10 custom metrics. Leave the instance running and expect ~$8/month after the first year.
Nothing to pay in the browser. Open the terminal runs this against a simulated cloud — the same API calls and the same commands, with no account and no bill. The figure above applies only if you build it in your own.
The scenario#
An instance has port 22 open to the world, a key everyone shares, and logs that exist only on its disk — so when it is replaced, the evidence goes with it.
All three are avoidable, and the alternatives are free.
Hands-on environment
Run this lab in a real terminal, free and in your browser. The environment is temporary and yours alone — break it as much as you like.
Open the terminalOpens in Killercoda, in a new tab — keep this page open for the steps.
Run it on AWS
This lab builds real cloud infrastructure, so it needs your own AWS account. Follow the cost and cleanup notes above — the resources are yours, and so is the bill.
Anything you tick here is your own record. EgyKode cannot see inside that terminal, so the success criteria stay self-assessed even when the environment checks your work for you.
An instance with no inbound ports
Step 1 of 4
What you are proving: You can run a command on an instance with no SSH key and no inbound rule
Marking this settles success criterion 1.
Terminal
aws iam create-role --role-name lab-ec2-ssm \
--assume-role-policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"ec2.amazonaws.com"},"Action":"sts:AssumeRole"}]}'
aws iam attach-role-policy --role-name lab-ec2-ssm \
--policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
aws iam attach-role-policy --role-name lab-ec2-ssm \
--policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
aws iam create-instance-profile --instance-profile-name lab-ec2-ssm
aws iam add-role-to-instance-profile --instance-profile-name lab-ec2-ssm --role-name lab-ec2-ssmLaunch with that profile and no SSH ingress rule at all, then:
Terminal
aws ssm start-session --target <instance-id>
aws ssm send-command --instance-ids <instance-id> \
--document-name "AWS-RunShellScript" \
--parameters 'commands=["uptime","df -h"]' \
--query 'Command.CommandId' --output textsend-command runs across many instances at once and records who ran what.
That audit trail is something a shared SSH key can never provide.
What you are proving: You can ship logs and memory metrics off an instance, and explain why the default EC2 metrics include neither
Marking this settles success criterion 4.
Terminal
sudo dnf install -y amazon-cloudwatch-agentjson
{
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/app/app.log",
"log_group_name": "/egykode/lab/app",
"log_stream_name": "{instance_id}",
"retention_in_days": 7
}
]
}
}
},
"metrics": {
"metrics_collected": {
"mem": { "measurement": ["mem_used_percent"] },
"disk": { "measurement": ["used_percent"], "resources": ["/"] }
}
}
}retention_in_days is not optional. A log group defaults to never expire,
and CloudWatch Logs charges for storage — an unbounded log group is one of the
quietest ways to accumulate a bill.
Why memory and disk are in there: EC2's default metrics come from the hypervisor, which can see CPU, network and disk I/O but has no visibility inside the guest. Memory and filesystem usage require an agent. A great deal of "CloudWatch does not show memory" confusion is this, and only this.
What you are proving: You can query those logs for the errors you care about, over a time window
Marking this settles success criterion 2.
Terminal
aws logs start-query \
--log-group-name /egykode/lab/app \
--start-time $(date -d '1 hour ago' +%s) --end-time $(date +%s) \
--query-string 'fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 20'What you are proving: You can alarm on something that reflects user impact, and justify the threshold you chose
Marking this settles success criterion 3.
Terminal
aws logs put-metric-filter \
--log-group-name /egykode/lab/app \
--filter-name errors \
--filter-pattern 'ERROR' \
--metric-transformations metricName=AppErrors,metricNamespace=EgyKode,metricValue=1
aws cloudwatch put-metric-alarm \
--alarm-name egykode-lab-errors \
--metric-name AppErrors --namespace EgyKode \
--statistic Sum --period 300 --evaluation-periods 2 \
--threshold 10 --comparison-operator GreaterThanThreshold \
--treat-missing-data notBreachingTwo deliberate choices:
evaluation-periods 2— the condition must hold for two consecutive windows. One bad minute during a deploy should not page anyone.treat-missing-data notBreaching— no errors produces no data points, and the default (missing) would leave the alarm inINSUFFICIENT_DATAforever.
Alarm on errors, not CPU. CPU at 90% with happy users is not an incident; errors at 10 per five minutes is, whatever the CPU is doing.
The instance does not appear in Session Manager
Missing instance profile, or no path to the SSM endpoints. aws ssm describe-instance-information lists what SSM can actually see.
No logs arrive
The agent is not running, the file path is wrong, or the role lacks CloudWatchAgentServerPolicy. Check /opt/aws/amazon-cloudwatch-agent/logs/.
The alarm sits in INSUFFICIENT_DATA
A metric filter emits data points only when the pattern matches. Set --treat-missing-data notBreaching.
No memory metric
Expected. The hypervisor cannot see inside the guest — the agent provides it.
Clean up#
Run this even if you did not finish.
Destructive — This removes real resources. Check which environment you are in first.
Terminal
aws ec2 terminate-instances --instance-ids <id>
aws logs delete-log-group --log-group-name /egykode/lab/app
aws cloudwatch delete-alarms --alarm-names egykode-lab-errors
aws iam remove-role-from-instance-profile --instance-profile-name <p> --role-name <r>
aws iam delete-instance-profile --instance-profile-name <p>Cost of this lab: Free tier — one t3.micro, and CloudWatch's free tier covers 5 GB of logs and 10 custom metrics. Leave the instance running and expect ~$8/month after the first year.
Success criteria
0 of 4
The concept behind it
Next up
Lab 18 of 59 on the project path