{ "slug": "code/operator-runbook", "title": "Code Apps and Registry operator runbook", "description": "Operate Cuitty Code Apps, Registry, SpiceDB, and Airflow in self-hosted deployments.", "url": "https://cuitty.com/docs/code/operator-runbook", "markdown_url": "https://cuitty.com/docs/code/operator-runbook.md", "json_url": "https://cuitty.com/docs/code/operator-runbook.json", "frontmatter": { "title": "Code Apps and Registry operator runbook", "description": "Operate Cuitty Code Apps, Registry, SpiceDB, and Airflow in self-hosted deployments.", "order": 41, "section": "Code", "updatedAt": "2026-05-24" }, "headings": [ { "depth": 2, "slug": "required-services", "text": "Required services" }, { "depth": 2, "slug": "startup-order", "text": "Startup order" }, { "depth": 2, "slug": "environment-variables", "text": "Environment variables" }, { "depth": 2, "slug": "migrations", "text": "Migrations" }, { "depth": 2, "slug": "spicedb-checks", "text": "SpiceDB checks" }, { "depth": 2, "slug": "registry-client-smoke-checks", "text": "Registry client smoke checks" }, { "depth": 2, "slug": "airflow-checks", "text": "Airflow checks" }, { "depth": 2, "slug": "playwright-e2e", "text": "Playwright E2E" }, { "depth": 2, "slug": "failure-modes-and-recovery", "text": "Failure modes and recovery" }, { "depth": 2, "slug": "related-pages", "text": "Related pages" } ], "body_markdown": "This runbook covers local and self-hosted operation for Code Apps, App Market, Code Registry, SpiceDB authorization, and Airflow execution.\n\n## Required services\n\n| Service | Purpose | Local default |\n| --- | --- | --- |\n| Cuitty Git API | REST API, git HTTP, app, registry, and callback endpoints | `http://localhost:4351` |\n| Cuitty Git frontend | Browser UI | `http://localhost:4350` |\n| Cuitty Auth | OAuth/OIDC sign-in | `http://localhost:7705` |\n| Database | Persistent metadata and run state | Deployment-specific |\n| SpiceDB | Relationship authorization | `http://localhost:50051` |\n| Airflow | App run orchestration | `http://localhost:8080` |\n| Registry artifact storage | Package tarballs, crates, wheels, and image layers | Deployment-specific |\n| Object storage | Optional backing storage for large artifacts and logs | Deployment-specific |\n\n## Startup order\n\n1. Start the database.\n2. Start SpiceDB.\n3. Load the SpiceDB schema.\n4. Start Cuitty Auth.\n5. Start Airflow.\n6. Run Cuitty Git API migrations, then start the API.\n7. Start the Cuitty Git frontend.\n\n## Environment variables\n\n```bash\nPUBLIC_API_URL=http://localhost:4351\nPUBLIC_CUITTY_AUTH_URL=http://localhost:7705\nAUTH_ISSUER=http://localhost:7705\nAUTH_CLIENT_ID=cuitty-git\nCUITTY_PUBLIC_URL=http://localhost:4350\nCUITTY_GIT_SECRET_KEY=dev-secret-change-me\nSPICEDB_ENDPOINT=http://localhost:50051\nSPICEDB_PRESHARED_KEY=dev-secret\nAIRFLOW_URL=http://localhost:8080\nAIRFLOW_USERNAME=airflow\nAIRFLOW_PASSWORD=airflow\nCUITTY_APP_EXECUTION_MODE=airflow\nCUITTY_APP_RUN_CALLBACK_SECRET=dev-callback-secret\nREGISTRY_STORAGE_URL=file:///var/lib/cuitty/registry\n```\n\n## Migrations\n\nRun migrations before traffic reaches a new API build:\n\n```bash\ncargo run -p git-server -- migrate\n```\n\nVerify the API starts with the expected database URL and no pending migration errors.\n\n## SpiceDB checks\n\n```bash\nzed schema write spicedb/schema.zed\nzed permission check user:alice read repository:acme/demo\nzed permission check user:alice view package:acme/npm/acme-widget\nzed permission check user:alice install app_listing:acme/review-bot\n```\n\nInspect failed authorization relationship writes:\n\n```sql\nSELECT id, operation_kind, resource_kind, resource_id, status, attempts, last_error\nFROM authz_outbox\nWHERE status IN ('pending', 'processing', 'failed_retryable', 'failed_terminal')\nORDER BY created_at ASC\nLIMIT 50;\n```\n\n## Registry client smoke checks\n\n```bash\nnpm config set @acme:registry http://localhost:4351/api/v1/npm/\nnpm publish --registry http://localhost:4351/api/v1/npm/\nnpm install @acme/button --registry http://localhost:4351/api/v1/npm/\n```\n\n```toml\n[registries.cuitty]\nindex = \"sparse+http://localhost:4351/api/v1/cargo/index/\"\n```\n\n```bash\ncargo publish --registry cuitty\ntwine upload --repository-url http://localhost:4351/api/v1/pypi/ dist/*\ndocker push localhost:4351/acme/image:tag\n```\n\n## Airflow checks\n\nSubmit or trigger an app run, then confirm the run has an Airflow DAG ID and DAG run ID. Reconcile submitted runs when callbacks are delayed:\n\n```bash\ncurl -X POST http://localhost:4351/api/v1/apps/runs/reconcile -H \"Authorization: Bearer $CUITTY_TOKEN\"\n```\n\nCallback failures usually mean the timestamp is stale, the `x-cuitty-signature` HMAC is wrong, or Airflow is signing with a different secret than `CUITTY_APP_RUN_CALLBACK_SECRET`.\n\n## Playwright E2E\n\nFrom the Cuitty Git `tests` directory, run the mocked market, registry, and social specs:\n\n```bash\nbun run test:e2e:apps-registry -- --project=chromium\nbun run test:e2e:apps-registry:headed\n```\n\nThe specs set browser auth state and mock API responses by path, so they cover both direct API calls on `http://localhost:4351/api/v1/...` and frontend-proxied calls on `http://localhost:4350/api/v1/...`.\n\n## Failure modes and recovery\n\n- SpiceDB unavailable: pause security-expanding app, registry, and transfer writes; restore SpiceDB; drain the authz outbox.\n- Authz outbox backlog: inspect `last_error`, reset stale `processing` rows to `pending` only after confirming no dispatcher is active, and restart the dispatcher.\n- Airflow DAG submission failure: check `AIRFLOW_URL`, credentials, DAG ID derivation, API logs, and Airflow scheduler/webserver logs.\n- App run stuck in `submitted` or `running`: poll Airflow for the DAG run ID, check task logs, verify callback delivery, and run reconciliation.\n- npm publish succeeds but artifact is missing: check package version records, artifact storage, API logs, and upload temp directories.\n- Private package returns `404` versus `403`: `404` can be correct when the caller lacks view permission; `403` means the caller can see the resource but lacks the requested operation permission.\n\n## Related pages\n\n- [Self-Hosted Registry](/docs/code/self-hosted-registry)\n- [Cuitty Code Apps](/docs/code/apps)\n- [Code App Market](/docs/code/app-market)\n- [Code Registry](/docs/code/registry)\n- [Authorization with SpiceDB](/docs/code/authz-spicedb)\n- [App execution with Airflow](/docs/code/app-execution-airflow)", "body_html": "

This runbook covers local and self-hosted operation for Code Apps, App Market, Code Registry, SpiceDB authorization, and Airflow execution.

\n

Required services

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ServicePurposeLocal default
Cuitty Git APIREST API, git HTTP, app, registry, and callback endpointshttp://localhost:4351
Cuitty Git frontendBrowser UIhttp://localhost:4350
Cuitty AuthOAuth/OIDC sign-inhttp://localhost:7705
DatabasePersistent metadata and run stateDeployment-specific
SpiceDBRelationship authorizationhttp://localhost:50051
AirflowApp run orchestrationhttp://localhost:8080
Registry artifact storagePackage tarballs, crates, wheels, and image layersDeployment-specific
Object storageOptional backing storage for large artifacts and logsDeployment-specific
\n

Startup order

\n
    \n
  1. Start the database.
    2. \n
  2. Start SpiceDB.
    3. \n
  3. Load the SpiceDB schema.
    4. \n
  4. Start Cuitty Auth.
    5. \n
  5. Start Airflow.
    6. \n
  6. Run Cuitty Git API migrations, then start the API.
    7. \n
  7. Start the Cuitty Git frontend.
    8. \n
\n

Environment variables

\n
PUBLIC_API_URL=http://localhost:4351\nPUBLIC_CUITTY_AUTH_URL=http://localhost:7705\nAUTH_ISSUER=http://localhost:7705\nAUTH_CLIENT_ID=cuitty-git\nCUITTY_PUBLIC_URL=http://localhost:4350\nCUITTY_GIT_SECRET_KEY=dev-secret-change-me\nSPICEDB_ENDPOINT=http://localhost:50051\nSPICEDB_PRESHARED_KEY=dev-secret\nAIRFLOW_URL=http://localhost:8080\nAIRFLOW_USERNAME=airflow\nAIRFLOW_PASSWORD=airflow\nCUITTY_APP_EXECUTION_MODE=airflow\nCUITTY_APP_RUN_CALLBACK_SECRET=dev-callback-secret\nREGISTRY_STORAGE_URL=file:///var/lib/cuitty/registry
\n

Migrations

\n

Run migrations before traffic reaches a new API build:

\n
cargo run -p git-server -- migrate
\n

Verify the API starts with the expected database URL and no pending migration errors.

\n

SpiceDB checks

\n
zed schema write spicedb/schema.zed\nzed permission check user:alice read repository:acme/demo\nzed permission check user:alice view package:acme/npm/acme-widget\nzed permission check user:alice install app_listing:acme/review-bot
\n

Inspect failed authorization relationship writes:

\n
SELECT id, operation_kind, resource_kind, resource_id, status, attempts, last_error\nFROM authz_outbox\nWHERE status IN ('pending', 'processing', 'failed_retryable', 'failed_terminal')\nORDER BY created_at ASC\nLIMIT 50;
\n

Registry client smoke checks

\n
npm config set @acme:registry http://localhost:4351/api/v1/npm/\nnpm publish --registry http://localhost:4351/api/v1/npm/\nnpm install @acme/button --registry http://localhost:4351/api/v1/npm/
\n
[registries.cuitty]\nindex = \"sparse+http://localhost:4351/api/v1/cargo/index/\"
\n
cargo publish --registry cuitty\ntwine upload --repository-url http://localhost:4351/api/v1/pypi/ dist/*\ndocker push localhost:4351/acme/image:tag
\n

Airflow checks

\n

Submit or trigger an app run, then confirm the run has an Airflow DAG ID and DAG run ID. Reconcile submitted runs when callbacks are delayed:

\n
curl -X POST http://localhost:4351/api/v1/apps/runs/reconcile   -H \"Authorization: Bearer $CUITTY_TOKEN\"
\n

Callback failures usually mean the timestamp is stale, the x-cuitty-signature HMAC is wrong, or Airflow is signing with a different secret than CUITTY_APP_RUN_CALLBACK_SECRET.

\n

Playwright E2E

\n

From the Cuitty Git tests directory, run the mocked market, registry, and social specs:

\n
bun run test:e2e:apps-registry -- --project=chromium\nbun run test:e2e:apps-registry:headed
\n

The specs set browser auth state and mock API responses by path, so they cover both direct API calls on http://localhost:4351/api/v1/... and frontend-proxied calls on http://localhost:4350/api/v1/....

\n

Failure modes and recovery

\n\n

Related pages

\n", "links_out": [ "/docs/code/self-hosted-registry", "/docs/code/apps", "/docs/code/app-market", "/docs/code/registry", "/docs/code/authz-spicedb", "/docs/code/app-execution-airflow" ] }