devlab — a development context for devmid

Read patches/ first. start-local.sh in your devmid checkout is broken: it sets YAML_COMMIT_REPO_OWNER / YAML_COMMIT_REPO / GITHUB_TOKEN, but the code reads FORGEJO_YAML_COMMIT_REPO_OWNER, FORGEJO_YAML_COMMIT_REPO, FORGEJO_TOKEN and FORGEJO_BASE_URL — and GitHubRepo.__init__ raises when any is missing. Since REPO_HANDLER = GitHubRepo() runs at module import, the app cannot start at all. tests/conftest.py sets the new names, which is why the suite passes and the dev server does not.

A self-contained stand-in for the parts of production devmid touches, so the mandatory-scanning work can be exercised end to end without a dev cluster.

The premise: devmid never talks to the Kubernetes API. Its production side effects are commits to the infra repo, writes to its own Postgres, outbound HTTP to four services, and a kubectl | kubeseal subprocess. None of that needs a cluster or a GPU. Point those somewhere harmless and you have a real environment.

Because devmid takes its configuration from the cluster, the lab starts from the actual Deployment environment and overrides only the variables that point at production. Anything not overridden in docker-compose.yml is running with its real value — so a passing test means something.

        registers a model
  you ─────────────────────► devmid ──── commits ────► forgejo (sandbox repo)
                               │                            │
                    GET /security/to_scan                   │ polls scan/
                    POST /security/claim                    ▼
                               │                       fake-argo
                               │                    ┌──────────────────────┐
                               ▼                    │ Argo CD              │
                            watcher ◄─── commits ───┤ sealed-secrets ctrl  │
                               ▲                    │ model scanner        │
                               │  POST /completed   └──────────────────────┘
                               │  (RSA-PSS signed)
                    POST /security/response
                               │
                               ▼
                            devmid  ──► model becomes loadable

fake-argo collapses three production components into one small process. It polls the sandbox repo for scan Jobs, decrypts the SealedSecrets with the lab's sealing key (the real sealed-secrets wire format — verified against kubeseal v0.38.4, all three scopes), checks the unsealed private key matches the public key committed beside it, and posts a correctly signed verdict.

Running against the OLD (pre-v3) code

The lab supports this deliberately: prove the harness against code you trust, then swap in v3. Two things must change.

echo 'SCAN_PROTOCOL=v1' >> .env      # signature + callback contract
echo 'DEVMID_PATH=devmid' >> .env    # v2 commits inference manifests here

SCAN_PROTOCOL is the one that bites. The signed message went from {salt}:{model_name}:{verdict} in v1 to {salt}:{scan_id}:{verdict}:{content_digest} in v3. Leave it on v3 against a v2 watcher and every callback is rejected as a bad signature — which looks exactly like a broken crypto implementation and is not one.

bootstrap.sh also writes sealed/scan-certificate.pem alongside kubeai-certificate.pem, because v2 devmid seals the HF token twice and needs both. That file was never in the repo — it was supplied at deploy time — so check what path variables your .example.env uses and adjust KUBEAI_CERT_PATH / SCAN_CERT_PATH in .env to match. v3 needs only the kubeai one.

Then start with the version-agnostic check:

export DEVMID_TOKEN='<a real Keycloak access token>'
./scripts/smoke.sh

smoke.sh makes no assumptions about scanning at all. It proves devmid boots against the lab database, accepts a real token, has its outbound calls intercepted, registers a model, and commits a manifest to the sandbox repo — reporting which folder it landed in. That last part is the answer to "does devmid write correctly", and it is worth knowing for certain before you upgrade, because devmid/ vs kubeai/custom is exactly what moved in v3 and the two services have historically disagreed about it.

e2e.sh is v3-only. Against v2 it will fail on its first assertion because /security/to_scan does not exist yet. That is the test being right, not the lab being broken.

Troubleshooting

password authentication failed for user "forgejo" / host db

Nothing in this compose file defines a host called db or a user called forgejo — those are the Forgejo image's own defaults, which means the FORGEJO__* overrides did not reach it. Usually a stale app.ini in the volume: the image writes it from the template only on the first run, so a bad first attempt sticks forever.

make reset-forgejo          # or: FORGEJO_RESET=1 ./scripts/bootstrap.sh

The compose file now sets the database three ways (bare DB_* for first-run templating, plus both the FORGEJO__ and legacy GITEA__ prefixes), so whichever your build honours, it lands on sqlite.

podman

Supported. scripts/compose.sh picks docker compose, podman compose or podman-compose automatically; override with COMPOSE="podman-compose" make up. If your podman-compose is old and ignores condition: service_healthy, start in two steps: make bootstrap (brings up postgres/forgejo/stubs and waits), then make up.

Setup

Requires Docker, jq, curl, openssl, and kubectl for the env dump.

cp .env.example .env
$EDITOR .env                 # set DEVMID_REPO and WATCHER_REPO

./scripts/bootstrap.sh       # sealing keys, forgejo, token, sandbox repo

# Start from the real cluster configuration:
./scripts/pull-cluster-env.sh <ns> devmid            > .env.devmid
./scripts/pull-cluster-env.sh <ns> security-watcher  > .env.watcher

docker compose up -d --build
docker compose logs -f devmid watcher fake-argo

Then drive it:

export DEVMID_TOKEN='<a real Keycloak access token>'
./scripts/e2e.sh

make help lists everything.

Facts taken from the code, not guessed

	Value
Port	8095 (Dockerfile CMD), not 8000
Database	single DSN in DATABASE
Git	FORGEJO_BASE_URL, FORGEJO_TOKEN, FORGEJO_YAML_COMMIT_REPO_OWNER, FORGEJO_YAML_COMMIT_REPO
v2 commit folder	hardcoded devmid/ — no env var
Certificates	hardcoded RESOURCES_DIR/kubeai-certificate.pem and .../scan-certificate.pem, mounted over, not configured
Build stage	service (named explicitly, since a test_runner stage also exists)
GITHUB_ENCODE_KEY	must be valid base64 — it is urlsafe_b64decoded at startup

Seeweb cannot be intercepted over HTTP

seeweb_api.fetch_seeweb_data goes through the ecsapi SDK, so there is no URL for the stub server to catch. And an empty catalog is not a soft failure: POST /inference/load looks the requested GPU up in shared_catalog.seeweb_catalog and refuses the load when it is absent — so nothing could ever be loaded.

stubs/sitecustomize.py therefore replaces the Seeweb entry points at import time, the same way conftest.py does, feeding fixtures/seeweb_catalog.json (32 instance types generated from the repo's own instance_to_gpu.csv, in the exact shape csv_to_gpu_json produces). SEEWEB_API_KEY is deliberately left empty so shared_catalog never tries the real call.

Two things to get right first

PYTHONPATH. Outbound HTTP is intercepted by stubs/sitecustomize.py, which Python imports automatically at startup — no code change in devmid. It only works if /devlab is early on PYTHONPATH, so the compose file sets PYTHONPATH=/devlab:/app/src:/app:/src. If your image sets its own PYTHONPATH, merge them rather than replacing. Confirm with:

docker compose logs devmid | grep 'devlab: redirecting'

No such line means nothing is being intercepted and devmid is calling the real billing API.

Keycloak is not stubbed, on purpose. Point the lab at the real one and use a real token. Validating a token mutates nothing in production, and a fake issuer is the single most likely way to make this lab lie to you. If local_dev_hooks.py already bypasses auth locally, check it covers the /security/* endpoints — those are gated by SECURITY_WATCHER_API_TOKEN, not by SSO.

Discovering what to stub

The stub server answers for the billing API, HuggingFace and Seeweb. The Seeweb catalog shape is a guess. Anything unrecognised returns 501 and is recorded:

make unmatched

Capture a real response, drop it at fixtures/<host>/<path_with_underscores>.json, and it takes priority over the built-in handler. That loop — run, read unmatched, add fixture — is the intended way to complete the lab, and the list doubles as an inventory of devmid's external dependencies.

What e2e.sh proves

1. Registration leaves a model pending and not runnable
2. The watcher dispatches it and the signed verdict comes back
3. A cleared model loads, and the manifest lands in KUBEAI_COMMIT_PATH, revision-pinned (org/model@sha)
4. unsafe blocks inference with a 409 that explains itself — including with force=true
5. inconclusive blocks it too
6. A second model on the same artefact is safe immediately (scan cache hit)
7. An unrecognised verdict re-queues instead of being cached as safe
8. No plaintext HF token or private key is ever committed

Items 4 (force bypass) and 7 (poisoned cache) are the ones worth watching. Both are security properties, and neither has run as a chain before.

Steering individual scans:

curl -X POST localhost:8090/verdict -H 'Content-Type: application/json' \
     -d '{"scan_id":"...","verdict":"FAIL"}'
curl -X POST localhost:8090/pause     # let a claim time out and get re-queued
make scans                            # everything fake-argo has processed

Verdicts otherwise follow the model name: unsafe → FAIL, inconclusive → INCONCLUSIVE, bogus → an unrecognised verdict string, anything else → PASS.

The migration

You have no Alembic, so the v3 columns go in as hand-run SQL — and writing that by hand is where the mistakes live. Generate it from the models instead:

make schema-diff

Then rehearse it properly: restore a production dump into the lab's Postgres, run the generated SQL, boot v3 against it, and check what scan_status pre-existing models get. Models that were serving happily yesterday will start returning 409 the moment v3 ships unless you backfill them. That is invisible in a fresh-database test and is the most likely thing to page you at 3am.

What this does not cover

• Real Seeweb and billing responses. Stubbed until you add fixtures.
• Whether a Model CRD is actually valid. Nothing here admits the YAML. Add kubeconform against the KubeAI CRD schema in CI — cheap, and catches schema drift that git will otherwise accept silently.
• Real sealed-secrets, Argo sync, vLLM, GPUs. For manifest admission and a genuine unseal, add a kind cluster with the KubeAI CRDs and the sealed-secrets controller; no GPU needed, since you are testing that the object is accepted, not that it serves tokens.
• Race conditions at production scale. SCAN_CLAIM_TIMEOUT_SECONDS is 300 here instead of 10800 so timeout behaviour is observable, which means the timing you observe is not the timing you will get.

Safety

.env.devmid and .env.watcher contain real production secrets pulled from the cluster. They are gitignored — keep them that way, and delete them when you are done. The sandbox repo is a throwaway inside the lab's own Forgejo, so no credential in it can reach your real infra repo.