CILocal.git · README.md
CILocal.git / README.md
revision 1afe81ed7def116ffc969fd75cf9db405fe6298b · complete file
# CI Control Plane Welcome to the **CI Control Plane**, a Swift‑based command‑line tool for orchestrating your local CI stack. The goal of this project is to provide a **super‑powered local to remote CI tool** that brings together Gitea, Jenkins, a local Docker registry and Watchtower and mirrors your work into the Fountain Coach GitHub organisation. It uses the [FountainStore](https://github.com/Fountain-Coach/Fountain-Store) persistence engine and [Swift‑SecretStore](https://github.com/Fountain-Coach/swift-secretstore) for secrets management, both developed in‑house and tailored for this purpose. ## Motivation Many teams rely on hosted CI services like GitHub Actions. While convenient, those platforms can be opaque, slow to iterate and require handing over your build pipeline. This control plane lets you **take control of your build and deployment lifecycle locally** while still benefiting from GitHub as a collaboration hub. By wrapping Gitea, Jenkins and Docker in a single CLI you can spin up a complete CI environment, run tests, publish images and push a mirrored history to GitHub. Key features include: - **Stateful orchestration:** [FountainStore](https://github.com/Fountain-Coach/Fountain-Store) serves as an embedded, ACID storage engine that persists pipeline definitions, build results and mirroring metadata. It uses an LSM‑style architecture, exposes a small HTTP/JSON surface and treats its OpenAPI spec as the source of truth. - **Secure secret handling:** [Swift‑SecretStore](https://github.com/Fountain-Coach/swift-secretstore) provides a unified `SecretStore` protocol with backends for the macOS Keychain, Linux Secret Service and a file‑based keystore. API tokens, passwords and SSH keys are never written to disk in plain text. - **Extensible CLI:** Built on Swift’s `ArgumentParser`, the tool exposes subcommands for setup, status checking and mirroring. The canonical contract for every flag, request and response lives in [`cicontrol.openapi.yaml`](cicontrol.openapi.yaml); the summaries below simply restate that specification. You can add your own commands for running pipelines or deploying images by updating the OpenAPI first. - **Integration with existing tools:** The control plane shells out to `gitea`, `docker`, `jenkins-cli.jar` and the official `gh` CLI. You can replace these with direct HTTP calls in the future by generating clients from their OpenAPI specs. ## Prerequisites To use this project you will need: 1. **Swift 6.0+ toolchain.** Install via [swift.org](https://www.swift.org/download/). 2. **Gitea CLI** (comes with the server) available on your `PATH`. You can start the server with `gitea web` or run it in Docker. The CLI supports user creation and repository management; see the [Gitea command reference](https://docs.gitea.com/administration/command-line). 3. **Jenkins with CLI access.** Download `jenkins-cli.jar` from your Jenkins instance and ensure it can connect over SSH or HTTP. The Jenkins CLI allows you to script routine tasks and trigger builds; consult the [Jenkins CLI guide](https://www.jenkins.io/doc/book/managing/cli/). Set `JENKINS_CLI_JAR` to its absolute path. 4. **Docker Engine** with the `docker` command available. This is used to manage your registry, Watchtower and service containers. 5. **GitHub CLI (`gh`).** Authenticate with your GitHub account and ensure you have rights to create repositories in the Fountain Coach organisation. 6. **Secret store credentials.** On macOS the Keychain backend requires no additional setup. On Linux set the `KEYSTORE_PASSWORD` environment variable to protect the file keystore. Secrets used by setup and mirroring (SecretStore keys with environment fallbacks): - `gitea-admin-token` (`GITEA_ADMIN_TOKEN`) – created or imported during setup. - `jenkins-admin-token` – created or imported during setup. - `github-mirror-username` (`GITHUB_MIRROR_USERNAME`) – GitHub account for push mirrors. - `github-mirror-token` (`GITHUB_MIRROR_TOKEN`) – PAT for the above account with repo scope. 7. **Docker Desktop.** To spin up the local CI services you will need Docker installed and running. The repository includes a shell script at `scripts/setup_local_ci.sh` that will start Gitea, Jenkins, a Docker registry and Watchtower for you. ## Getting Started Clone this repository and build the executable: ```bash git clone https://github.com/your-org/ci-control-plane.git cd ci-control-plane swift build -c release ``` The compiled binary will be located at `.build/release/cicontrol`. Add it to your `PATH` or run it directly via `swift run` during development. ### Bootstrap your CI stack Run the setup command to provision your local services (see the `POST /cli/setup` operation in [`cicontrol.openapi.yaml`](cicontrol.openapi.yaml) for the authoritative list of inputs and outputs): ```bash ./cicontrol setup \ --gitea-url http://localhost:3000 \ --admin-user admin \ --jenkins-url http://localhost:8080 \ --jenkins-user admin ``` This will: 1. Create a Gitea admin user (if not present) and generate an API token. The token is stored securely using Swift‑SecretStore. 2. Provision Jenkins admin access: verify connectivity via Jenkins CLI, optionally reset the admin password when `--jenkins-password` is provided, then generate an API token and store it securely. 3. Initialise FountainStore for persisting pipeline definitions and results. 4. (Optional) Configure push mirrors so changes are automatically forced to the Fountain Coach GitHub organisation. After running `setup` you can begin defining repositories and pipelines in FountainStore and run builds via Jenkins. #### Jenkins connection inputs `cicontrol setup` now accepts explicit Jenkins parameters that mirror the OpenAPI contract: - `--jenkins-url <uri>` – Base URL for the Jenkins controller (defaults to `http://localhost:8080`). - `--jenkins-user <username>` – Jenkins administrator account that should exist or be created (defaults to `admin`). - `--jenkins-password <password>` – Optional password for the Jenkins admin user when not using token authentication. - `--jenkins-api-token <token>` – Optional pre-generated Jenkins API token to import into Swift‑SecretStore. Set the `JENKINS_CLI_JAR` environment variable to the absolute path of `jenkins-cli.jar` so setup can reach the Jenkins CLI. The setup flow uses a temporary `@auth` file to avoid leaking credentials and never prints tokens or passwords. #### Setup with optional mirroring You can bootstrap GitHub push mirrors during setup: ```bash ./cicontrol setup \ --gitea-url http://localhost:3000 \ --jenkins-url http://localhost:8080 \ --jenkins-user admin \ --configure-mirror \ --mirror-org Fountain-Coach \ --mirror-include 'ci-*' \ --mirror-exclude 'legacy-*' ``` Requirements: - Store `gitea-admin-token`, `github-mirror-username`, and `github-mirror-token` in SecretStore (or export their env var fallbacks) before enabling mirrors. - Use `--mirror-dry-run` to preview changes without creating repos or mirrors. ### Local CI stack setup Before running any of the CLI commands, you need a local CI environment. This repository includes a helper script that uses Docker to start all services: ```bash ./scripts/setup_local_ci.sh ``` The script pulls trusted images (Gitea, Jenkins, Docker Registry and Watchtower) and exposes them on ports 3000, 8080 and 5000 respectively. It is safe to rerun the script; your data persists in Docker volumes. Once the containers are up you can visit Gitea and Jenkins in your browser and complete their first‑run wizards. See the top of `scripts/setup_local_ci.sh` for details. To keep the stack hardened, run optional vulnerability scans against the pinned images once you have either [Trivy](https://aquasec.com/products/trivy/) or the [Docker Scout](https://docs.docker.com/scout/) plugin installed: ```bash ./scripts/scan_images.sh ./scripts/scan_images.sh --scanner scout # pick a specific scanner ``` The helper script uses the same image list defined in `scripts/docker-compose.local-ci.yml` so scans stay aligned with what you deploy. ### Checking status To see the current state of your CI stack, run (contract defined by `POST /cli/status` in [`cicontrol.openapi.yaml`](cicontrol.openapi.yaml)): ```bash ./cicontrol status --org Fountain-Coach ``` The status command fetches Prometheus‑style metrics from FountainStore’s optional HTTP server, lists running Docker containers and queries your mirrored GitHub repositories using the `gh` CLI. Use this to monitor memory usage, compaction activity and whether the mirror is up to date. Pass `--org <github-org>` to scope the GitHub queries to the organisation you mirror (for example `--org Fountain-Coach`). ### Mirroring to GitHub Keep your private Gitea repositories in sync with the official Fountain Coach GitHub organisation (per the `POST /cli/mirror` operation in [`cicontrol.openapi.yaml`](cicontrol.openapi.yaml)): ```bash ./cicontrol mirror --org Fountain-Coach ``` For each repository recorded in FountainStore, the control plane will create it on GitHub (if necessary) and configure a push mirror in Gitea. Push mirrors perform force pushes; ensure that local Gitea is the source of truth before enabling. The `--org` flag is required so the tool knows which GitHub organisation should receive the mirror configuration. ## Extending the Control Plane This project is just a starting point. You can customise it by: - Adding new subcommands in `Sources/CIControlPlane/main.swift`. - Generating typed clients for Jenkins or Gitea using Apple’s Swift OpenAPI Generator and switching from shell calls to pure Swift. Both services expose OpenAPI specs. - Implementing a `Runner` actor that orchestrates pipelines, stores results in FountainStore and publishes events. - Surfacing more metrics and logs in `status` and wiring them to a local dashboard. - Writing integration tests for your commands using XCTest. ## Documentation High‑level design and agent requirements are described in [`docs/OVERVIEW.md`](docs/OVERVIEW.md) and [`AGENTS.md`](AGENTS.md). Security practices, including secret storage and audit logging, are summarised in [`docs/SECURITY.md`](docs/SECURITY.md). For the complete, source‑of‑truth contract of every CLI command, consult [`cicontrol.openapi.yaml`](cicontrol.openapi.yaml). ## License This project is released under the MIT License. See `LICENSE` for details.