-o wide, describe pod, cluster info and other approved read-only commands.kubectl get pods -o wide, kubectl describe pod NAME, kubectl cluster-info or kubectl api-resources. No real shell is exposed.terraform/, helm/, .github/workflows/, app/, migration/, docs/, ansible/ and the Jenkinsfile — rather than a generic reference architecture. Click a section to expand it.This project is an Azure Kubernetes Service (AKS) migration and operations lab: a workload was containerized, moved onto AKS, and is now operated through this console instead of raw kubectl. The flow below is the real pipeline in this repository.
Underneath all of this sits Terraform (terraform/), which provisions the Azure infrastructure the pipeline above depends on: the Resource Group, Virtual Network/Subnet, the AKS cluster, the Container Registry and the Key Vault. Terraform is applied manually against a remote state file — it is not triggered by a push to main. It is the layer that has to exist before the diagram above can run at all.
What can go wrong: a red step in the GitHub Actions run means the build/push/deploy stage itself failed — read that job's log. A green pipeline run with a broken app means the problem moved into Kubernetes (see Troubleshooting). A Terraform-layer problem, such as a resource created outside Terraform, shows up as drift on the next terraform plan, not as a pipeline failure.
A management/billing boundary — everything else in this project lives inside it (or, for AKS, inside its auto-managed node resource group). Defined once in terraform/main.tf as azurerm_resource_group.main, location westeurope. Deleting it deletes everything inside it — the single biggest blast-radius resource in the project.
AKS nodes get private IPs from snet-aks (10.10.1.0/24) inside vnet-migration-lab (10.10.0.0/16), both in main.tf. This lets Pods, the internal load-balancer frontend and the API server talk to each other privately instead of over the public internet.
Defined in main.tf as azurerm_kubernetes_cluster.main: Free SKU tier, one system node pool (Standard_B2s_v2), SystemAssigned managed identity. AKS also auto-creates a second, hidden resource group (MC_rg-migration-lab_aks-migration-lab_swedencentral) holding the actual VM scale set, disks, the Standard Load Balancer and the public IP the Ingress controller uses — none of that is declared in main.tf, AKS manages it internally.
Basic SKU, admin_enabled = false. Both application images (migration-app, pong-app) live here. AKS pulls from it using the kubelet's managed identity, granted the AcrPull role via azurerm_role_assignment.aks_acr_pull — no registry password stored anywhere. If that role assignment were removed, every Pod would fail with ImagePullBackOff/401 even though the image exists.
Holds the PostgreSQL password. Defined in terraform/keyvault.tf with rbac_authorization_enabled = true (Azure RBAC controls access, not vault access policies). Pods never call Key Vault directly — a CSI driver mounts the secret in (see Security).
Not in Terraform at all — the NGINX Ingress Controller's LoadBalancer Service made Azure auto-provision a Standard SKU, Static public IP and Standard Load Balancer inside the AKS node resource group. Real, working, currently stable — but nothing yet protects it from reallocation if that Service were ever deleted and recreated (see Terraform section).
Four files. providers.tf pins the azurerm provider to ~>4.0. backend.tf configures a remote state backend — an Azure Storage Account (stmigrationlabtfstate, container tfstate, key migration-lab.tfstate) — instead of a local terraform.tfstate. main.tf declares the Resource Group, VNet, Subnet, ACR, AKS cluster and the AcrPull role assignment. keyvault.tf declares the Key Vault.
Local state is one JSON file describing what Terraform believes exists. Kept on one machine, two people (or one person from two machines) could run apply against stale state and corrupt or double-create resources. A Storage Account backend gives one shared source of truth and, via blob leases, locks concurrent apply runs out of each other.
terraform state list against this backend shows exactly 8 objects: resource group, VNet, subnet, ACR, AKS cluster, Key Vault, the AcrPull role assignment, and the azurerm_client_config data source. Nothing about ingress-nginx, cert-manager, the Ingress Public IP, or any Kubernetes/Helm object is in Terraform — that layer was set up imperatively with az/helm/kubectl. A real, current gap: terraform plan never shows the true end-to-end platform state, only the base Azure layer.
The most common failure is drift: something Terraform manages changed or was deleted outside Terraform, and the next plan either wants to revert it or errors because reality no longer matches state. Always run plan before apply and read the diff — a resource that already exists but isn't tracked yet (like the Ingress Public IP) needs terraform import, never a fresh apply, or Terraform will try to create a duplicate and Azure will reject it on a name collision.
FROM python:3.12-slim, installs requirements.txt, then copies only config.py db.py k8s_ops.py main.py and static/ — not the whole build context. Creates and switches to a non-root user (useradd -u 10001, USER 10001) before any application code runs, then starts uvicorn main:app on port 8000. app/.dockerignore excludes __pycache__/ and *.pyc from the build context.
A much simpler, unrelated image: FROM nginx:1.27-alpine, copies a static index.html and an nginx.conf that adds a plain-text /health endpoint on port 8080. Deliberately "dumb" — its only job is to be something that can be rolling-restarted and scaled repeatedly with no real backend dependency.
The console and the demo target are deployed, scaled and restarted independently (two Deployments, two Services, two Ingress paths). One shared image would couple two unrelated failure domains for no reason.
ImagePullBackOff usually means the tag CI pushed doesn't match the tag Helm was told to deploy, or the AcrPull role assignment is missing. A build failure shows directly in the GitHub Actions "Build Docker image" step log.
The smallest deployable unit. Every running piece of this project — console, Pong, PostgreSQL — is ultimately one or more Pods. Nothing here creates a Pod directly; a Deployment or StatefulSet does.
Manages a set of identical, stateless Pods (migration-app-prod, pong-app): desired replica count, Pod template, rollout strategy. Scale and restart in this console both work by patching the Deployment, never a Pod directly.
The object a Deployment creates to hold one Pod template's replicas. Every rolling restart creates a new ReplicaSet and scales the old one to zero — that is what "rolling update" means at the API level.
A stable virtual IP/DNS name in front of a changing set of Pods. migration-app-prod, pong-app and migration-postgresql are all ClusterIP. The app reaches Postgres by the Service's DNS name, never a Pod IP.
Routes external HTTP traffic by host/path via the NGINX Ingress Controller. The console's Ingress serves /; Pong's serves /pong(/|$)(.*) with a rewrite annotation stripping the prefix before it reaches nginx.
A logical partition inside one cluster. This project uses prod (real deployment) plus dev and test for lighter Helm releases, all on the same AKS cluster. RBAC Roles are namespace-scoped, so a Role bound in prod has zero effect elsewhere.
The identity a Pod uses against the Kubernetes API. Console Pods run as migration-app-sa, which the chart does not create in prod (serviceAccount.create: false) because it is pre-wired for Workload Identity. Every RBAC binding in this project targets this one ServiceAccount.
Two matter here: migration-app-keyvault-secret (populated from Key Vault via the CSI driver) and migration-ops-secret (LOGIN_PASSWORD/JWT_SECRET). Neither is created by a Helm template with a literal value — both must already exist in the cluster.
Like a Deployment, but for workloads needing stable identity and storage — only migration-postgresql. Its Pod name is always migration-postgresql-0, with its own PersistentVolumeClaim that survives Pod restarts.
The StatefulSet's volumeClaimTemplates requests a PVC (size from postgresql.storage, e.g. 5Gi in prod, ReadWriteOnce). Azure's CSI storage driver provisions a matching PersistentVolume (an Azure Disk) automatically, so Postgres data survives a Pod reschedule onto a different node.
Tells Kubernetes when a Pod may receive traffic. The console's probe hits /readyz, which returns 503 until db.is_up() is true — a Pod that can't reach PostgreSQL is pulled out of the Service's Endpoints instead of serving broken requests.
Tells Kubernetes when to kill and restart a Pod. The console's /healthz is deliberately DB-independent, so a temporary Postgres blip marks the Pod not-ready instead of restarting it in a loop.
Pong's Deployment explicitly sets maxUnavailable: 0, maxSurge: 1: a new Pod must pass readiness before an old one terminates, so replica count never drops during a redeploy. Demo Mode and the ROLLING RESTART button both exercise exactly this.
The +/− REPLICA buttons call PATCH on the Deployment's /scale subresource (k8s_ops.scale()), bounded by MAX_REPLICAS (5 in prod). The ReplicaSet controller then creates or deletes Pods to match.
A restart without a new image patches the Pod template's annotations (kubectl.kubernetes.io/restartedAt) so the template hash changes and a new ReplicaSet is created even with an identical image tag — k8s_ops.restart(), the same mechanism as kubectl rollout restart.
Every console action — read, scale, restart, kill, the terminal — goes through the official kubernetes Python client, authenticated as migration-app-sa. No kubectl binary and no shell exist inside the app container.
helm/migration-app and helm/pong-app are two independent charts. Each Chart.yaml declares apiVersion: v2, a chart version (currently 0.1.0 for both) and an appVersion — metadata only, it does not drive the deployed image tag.
values.yaml holds chart defaults, kept in sync with the current FastAPI app: containerPort: 8000, liveness path /healthz, readiness path /readyz — so a bare helm install with no environment file already produces a Pod that can become Ready. values-dev.yaml/values-test.yaml stay minimal on top of that (replica count, environment tag) because those are the only things they genuinely need to differ on. values-prod.yaml goes further and overrides Key Vault, ingress and the operations-console settings, because those are real per-environment differences. This is exactly why keeping the base values.yaml accurate matters: dev/test inherit everything they don't explicitly override from it, so a stale default would silently break them even while prod, which overrides everything, looked completely fine.
Deployment, Service, Ingress, ServiceAccount, HPA, the Postgres StatefulSet/Service/Secret, the SecretProviderClass, and the RBAC Role/ClusterRole. _helpers.tpl centralizes name/label generation for consistent app.kubernetes.io/* labels across every object.
A release is one named, tracked deployment of a chart into a namespace. This project currently has four: migration-app-prod (the real one), migration-app-dev, migration-app-test, and a bare migration-app release in the default namespace left over from earlier iterations before namespaces were introduced. helm upgrade --install creates the release if missing or upgrades it if present — idempotent, which is why CI can call it on every push. --set overrides individual values on the command line (used for image.repository/image.tag, only known at CI runtime). Every upgrade creates a numbered revision (prod is at revision 12); helm rollback <release> <revision> reverts to a prior revision's rendered manifests, documented in docs/rollback.md.
This project's own helm history shows a real failed revision (revision 8: a field-ownership conflict on .spec.replicas, caused by a raw kubectl scale changing replicas at the same moment Helm applied). The fix is always the same: re-run helm upgrade once the conflicting change has settled, or helm rollback to the last good revision.
Every image tag is the exact Git commit SHA that triggered the build — no latest tag anywhere in this flow, so the running image always maps back to one exact commit.
A separate, deliberately manual-only workflow (workflow_dispatch, no schedule) that builds and deploys the Pong image, tagged by the GitHub run number. Demo Mode does not use this workflow — it triggers a rolling restart of the already-deployed Pong image directly through the Kubernetes API, so Demo Mode does not cost a Docker build every 3 minutes.
An older, parallel pipeline that builds and deploys the same console image, targeting the exact same production release as GitHub Actions: helm upgrade --install migration-app-prod ./helm/migration-app --namespace prod -f helm/migration-app/values-prod.yaml. What legitimately still differs is the platform plumbing around that, not the target: Jenkins authenticates with stored service-principal credentials (azure-client-id/secret/tenant-id) via az login instead of the azure/login action, and tags the image with Jenkins' own $BUILD_NUMBER instead of github.sha. Neither pipeline injects the PostgreSQL password directly any more — with keyVault.enabled: true in values-prod.yaml, the chart never renders the plain-Secret template that would need one, so the CSI driver / Workload Identity chain supplies it in both cases.
A red step in the Actions log names the exact failed stage (login, build, push, Helm). A green workflow with a broken app means the problem moved into Kubernetes — the job's job is done the moment helm upgrade exits 0, it does not wait for the rollout to actually finish.
migration-postgresql is a single-replica StatefulSet running postgres:16, data directory (PGDATA) on a PersistentVolumeClaim so data survives Pod restarts and rescheduling.
The app connects to the Service's cluster-internal DNS name, never a Pod IP — a Postgres Pod restart is transparent to the app once the Service's endpoint updates.
migration/backup-postgres.sh runs pg_dump -Fc (custom format) against a source database; migration/restore-postgres.sh runs pg_restore --clean --if-exists against the target. migration/README.md documents the cutover: dump → restore → validate → cut writes over → final dump/restore → switch the app's connection → validate again → keep the source around for rollback.
docs/rollback.md checks /healthz (liveness) and /readyz (fails with 503 until db.is_up() is true, 200 once PostgreSQL is reachable) after cutover — the same two endpoints the Kubernetes probes themselves use, plus the dashboard's own POSTGRESQL status card for a human-readable view. Keeping a runbook's endpoint names in sync with whatever the application actually exposes is what makes it safe to follow literally during a real rollback, instead of chasing a route that no longer exists.
See "PostgreSQL connection failure" in Troubleshooting — the console degrades gracefully rather than crashing, because every DB call in db.py checks if not _pool first.
No client secret, connection string or password is stored in the cluster or the repo to make this work — trust is established once between the AKS OIDC issuer and the Managed Identity, and after that a Pod authenticates purely by presenting its own ServiceAccount token.
A Role (migration-app-operations) grants permissions only inside the namespace it is created in (prod) — get/list/watch/delete on pods, get/list/watch on services/events, get/list/watch/patch on deployments, get/patch/update on deployments/scale, read on ingresses. A ClusterRole (migration-app-node-reader) grants permissions cluster-wide, needed because Nodes aren't namespaced — narrowed to get/list/watch on nodes only. Neither uses a wildcard verb or resource. Both bind to the single migration-app-sa ServiceAccount.
Because the Role is bound only in prod, the console's ServiceAccount has zero permissions in dev, test or kube-system — the API server itself would reject a request there with 403, regardless of what the terminal's whitelist allows. It is also why the terminal only accepts the pinned TARGET_NAMESPACE for -n.
k8s_ops.parse_readonly_command() uses shlex.split() only to tokenize typed text for comparison against a fixed set of allowed token sequences — every match dispatches to a specific Python function calling the Kubernetes client library directly. There is no subprocess, no os.system, no shell=True, no eval/exec anywhere in the codebase. An unrecognized command is rejected with an error, never passed to anything that could execute it.
Enforced server-side with a FastAPI dependency (require(*roles)) reading the role claim from the signed JWT — never trusted from the request body. guest reaches only read-only endpoints; DELETE /api/pods/{name} requires admin; scale/restart/Demo Mode require user or admin. Registration always assigns the hardcoded role "user" — there is no path for a self-registered account to become admin.
LOGIN_PASSWORD and JWT_SECRET are read from a Kubernetes Secret (migration-ops-secret) that Helm references but never creates with a literal value. The PostgreSQL password never touches the repo or CI logs; it flows from Key Vault through the CSI driver straight into a Secret object. .gitignore also blocks *.zip, __pycache__ and Terraform state/plan files from ever being committed by accident.
The same order used in docs/troubleshooting.md — start at the Pod, work outward only once the Pod itself looks healthy.
kubectl logs <pod> --previous to see why the last attempt crashed. In this project the most likely cause is k8s_ops.init() failing on startup with no in-cluster or kubeconfig credentials reachable — confirmed directly by running the console container outside AKS. Inside AKS this should not happen since the ServiceAccount token is auto-mounted.
kubectl describe pod Events show either "not found" (Helm was given a tag never pushed to ACR) or "unauthorized" (the AcrPull role assignment on the kubelet identity is missing).
Pod is Running but 0/1 Ready and gets no traffic. For the console this means /readyz is 503 — check whether migration-postgresql is reachable. For Pong it means nginx isn't answering on 8080 yet.
Pod restarts repeatedly after briefly being Ready. Since the console's liveness check (/healthz) is deliberately DB-independent, a liveness failure here points at something more serious than a database blip.
kubectl get endpoints <service> empty means the Service's selector matches no Ready Pod — check the Pod's labels against the selector, and that the Pod is passing readinessProbe (a not-Ready Pod is excluded from Endpoints even if it matches).
A 502 almost always means the Ingress points at a Service with no healthy Endpoints — check the backend Service first. For pong-app, a wrong rewrite/regex sends nginx a path it doesn't recognize, which shows as a 404 from Pong's own nginx rather than a 502.
migration-app-keyvault-secret never appears, or the Pod is stuck ContainerCreating on the CSI volume. Check the SecretProviderClass's clientId/tenantId/keyvaultName, and that the federated identity credential trusts this exact ServiceAccount + namespace pair — a namespace typo here is a classic silent failure.
The console does not crash — db.connect() catches the exception, logs a connection-failed message, and every DB call short-circuits on if not _pool. Visible symptom: a degraded but running console (no history, no registered-user login, Demo Mode/registration return 503). Check kubectl logs migration-postgresql-0 and the Secret the app reads its password from.
helm history <release> -n <namespace> shows past attempts. This project's own revision 8 failed with a field-ownership conflict on .spec.replicas because a raw kubectl scale changed it outside Helm at the same moment. Re-running helm upgrade once things settle resolves it.
Most likely here: managing a resource that already exists but isn't in state yet (e.g. the Ingress Public IP) — Terraform tries to create a duplicate and Azure rejects it on a name collision. Fix with terraform import, then plan to confirm no unwanted changes, never a forced apply.
A 403 from the Kubernetes API means the request went outside what migration-app-operations or migration-app-node-reader actually grant — most commonly touching a namespace other than prod, or a resource type not listed in either rule set.
app/**, helm/migration-app/** or the workflow file itself and lands on main, deploy.yml runs: authenticate to Azure, build the console image tagged with the commit SHA, push to ACR, fetch AKS credentials, then helm upgrade --install migration-app-prod with -f values-prod.yaml and --namespace prod. A push outside those paths does not trigger it at all.helm rollback <release> <revision> re-applies the exact rendered manifests from that earlier revision. It does not rebuild anything — a purely Kubernetes-object-level revert, so the image referenced by that old revision must still exist in ACR for the rollback to work.migration-app-prod, pong-app) manages interchangeable, stateless Pods with random name suffixes. A StatefulSet (migration-postgresql) gives each Pod a stable, predictable name and its own dedicated PVC that follows that specific Pod — required because a database can't be "any interchangeable replica."prod, and a ClusterRole only for the one thing that must be cluster-wide: reading Nodes.migration-app-prod, namespace prod, values-prod.yaml. What legitimately differs is the platform plumbing: GitHub Actions authenticates via the azure/login action and tags images with github.sha; Jenkins authenticates with stored service-principal credentials and tags images with $BUILD_NUMBER. Neither one injects the PostgreSQL password directly — both rely on values-prod.yaml's Key Vault/CSI-driver configuration for that..tf files — how terraform plan tells "doesn't exist yet" apart from "exists and unchanged." Without accurate state Terraform can't tell the difference and may try to recreate something already there.stmigrationlabtfstate) instead of a local file, so it's identical for everyone applying changes and supports locking — two concurrent apply runs against the same local state would otherwise corrupt it or double-create resources.kubectl logs <pod> --previous to read the last crash's output, then kubectl describe pod for the Events section, which usually names the exact reason: probe failure, OOMKilled, non-zero exit code, or missing config.migration-app-dev, migration-app-test, migration-app-prod) in three separate namespaces, driven by three separate values files. Only values-prod.yaml carries the full production configuration — Key Vault, ingress, operations-console settings, the correct port and probes.maxUnavailable: 0 and maxSurge: 1, Kubernetes starts one extra new Pod, waits for it to pass readinessProbe, then terminates one old Pod, repeating until every Pod is on the new template. Total replica count never drops below desired during the process./ routes to the console's Service and /pong(/|$)(.*) (rewritten to strip the prefix) routes to Pong's Service — both on the same controller and public IP, distinguished purely by path since neither Ingress sets a host.