Challenges
Focused, hands-on DevOps problems based on real production incidents. Each one drops you into a broken server with a simulated terminal, and comes with hints, an automated solution check and a walkthrough explaining the fix.
10 challenges
Linux
Disks, processes, services and permissions: your first on-call week.
Challenge on Linux
Disk full on prod
by @loopedin.daily
Deploys fail with 'No space left on device'. Find what's eating / and free it without stopping the API.
#df#du#logs#truncate
Easy~5 minSolveChallenge on Linux
"I deleted it, still 100%"
by @loopedin.daily
A teammate rm'd the giant log but df hasn't moved. Find where the space went and get it back.
#lsof#df#deleted-files#systemctl
Medium~8 minSolveChallenge on Linux
Backup cron silently failing
by @loopedin.daily
The nightly DB backup hasn't produced a file in 9 days and nobody noticed. Make the cron job work again.
#cron#chmod#syslog#bash
Easy~6 minSolve
Networking
Ports, reverse proxies, DNS and TLS between the user and your process.
Challenge on Networking
Nginx returns 502
by @loopedin.daily
Every request gets 502 Bad Gateway. Nginx is up, so who isn't? Find the dead upstream and bring it back.
#nginx#curl#journalctl#systemctl
Easy~5 minSolveChallenge on Networking
DNS resolves to the wrong IP
by @loopedin.daily
DNS was migrated yesterday and dig shows the new IP, yet the app still talks to the old server. Why?
#dig#getent#hosts#dns
Medium~7 minSolveChallenge on Networking
Address already in use :8080
by @loopedin.daily
After the deploy the service won't start: EADDRINUSE on :8080. Something is squatting on the port.
#ss#lsof#kill#systemctl
Easy~5 minSolve
Containers
Docker containers that won't start, can't connect or eat the disk.
Kubernetes
Pods that crash, can't pull, or never get traffic.
Challenge on Kubernetes
ImagePullBackOff
by @loopedin.daily
The new release never starts: pods sit in ImagePullBackOff. Find out why and ship the right image.
#kubectl#describe#set-image#registry
Easy~5 minSolveChallenge on Kubernetes
Pod in CrashLoopBackOff
by @loopedin.daily
The API pod restarts every 30 seconds. The current logs are empty. Find the crash reason and fix the config.
#kubectl#logs#configmap#rollout
Medium~8 minSolve
Observability
Turn raw logs into answers with a few pipes.
AI infra challenges are cooking 🤖
GPUs, model serving and LLM inference break in brand-new ways. We're building incidents for exactly that, and they won't be behind Pro.
GPU node says CUDA out of memory
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 GiB
vLLM pod stuck pulling a 40 GB model
Back-off pulling image "llm-server:7b" ImagePullBackOff
Inference p99 jumped from 300 ms to 9 s
p99=9.2s queue_depth=148 gpu_util=31%
Driver mismatch after a node upgrade
Failed to initialize NVML: Driver/library version mismatch
Advanced challenges
Interview-grade incidents: OOMKilled pods, stuck rollouts, leaked secrets, Terraform locks, expired TLS.
Challenge on Networking
Certificate expired at midnight
by @loopedin.daily
Browsers scream NET::ERR_CERT_DATE_INVALID at 00:07. Confirm it, find why auto-renew didn't, and fix it.
#tls#openssl#certbot#nginx
Challenge on Kubernetes
Exit code 137 at peak traffic
by @loopedin.daily
Every evening at peak the API pods restart. Exit 137, OOMKilled. Stabilise prod and name the real cause.
#kubectl#oomkilled#limits#memory
Challenge on Kubernetes
Rollout stuck at 1/3
by @loopedin.daily
A release hangs forever: new pods start but never become Ready. Restore service, then find the bad change.
#kubectl#rollout#readiness#undo
Challenge on CI/CD
AWS key pushed to GitHub
by @loopedin.daily
Security pinged you: an AWS access key is in the repo. Find the commit and run the response in the right order.
#git#secrets#aws#incident-response
Challenge on Cloud
Terraform: Error acquiring state lock
by @loopedin.daily
A CI job died mid-apply and now every plan fails on the state lock. Unlock it safely, without stomping anyone.
#terraform#state#lock#dynamodb
Challenge on Containers
App can't reach the DB container
by @loopedin.daily
ECONNREFUSED 127.0.0.1:5432 inside the container, but Postgres is clearly running. Make them talk.
#docker#compose#networking#postgres
Challenge on Containers
The 2.1 GB Docker image
by @loopedin.daily
Deploys take 6 minutes because the image is 2.1 GB. Find the fat layers and pick the fix that matters.
#docker#history#multi-stage#images
Challenge on Kubernetes
Pod stuck in Pending
by @loopedin.daily
A new deployment never schedules. No logs, no crash, just Pending forever. Get it running.
#kubectl#scheduling#requests#describe