Connecting your build tools

Every client talks to a repository of its own type, in its own protocol, and presents the same kind of credential: a key issued under Access → Credentials. This chapter lists the type, the URL and the credential form for each client. The examples use repo.example.com, the releases tenant, a repository named <repo> created with the type in the table, and a key in $KEY.

A repository's URL is /repository/<tenant>/<repo>/, and a repository of one format leaves that format's name out of the paths after it. Maven and the Jenesis module layout are the two that keep theirs - maven/, and module/ beside artifact/ - which is what lets the java type hold both in one repository. Container images are addressed under /v2/ instead, with the tenant and the repository as the first two parts of the image name. A repository is created before a client uses it; Repositories shows how.

How a client presents its key

A key travels in whichever form a client can send, and the server accepts all three:

A request without a key is answered 401 with a challenge, which is what Maven and Docker wait for before they send the credentials they hold. A key that lacks the right is answered 403.

The clients

Each client needs a repository of its type and is pointed at an address inside it. The addresses start with the repository's own URL, written $REPO below:

REPO=https://repo.example.com/repository/releases/<repo>

Java and the JVM

Language package managers

Native code, data and models

Operating-system packages

Containers and infrastructure

Anything else

Where a URL carries <name>, the format keeps separate spaces inside the one repository - a Cargo registry, a Helm chart repository, a Conda channel, a Swift registry, a Hugging Face hub - and the name is yours to choose. Publishing under a new name creates that space; the repository itself has to exist first. The Terraform discovery document at /.well-known/terraform.json names one registry path for the whole host, terraform.prefix, which is /repository/releases/terraform/registry - a terraform repository named terraform - unless you set it.

Maven, Gradle and Jenesis

A repository holds one format, so each tool needs a repository of the type it speaks. Maven, Gradle and a Jenesis build all read the Maven layout from a maven or java repository at /repository/releases/<repo>/maven/; a Gradle build that publishes Ivy descriptors needs an ivy repository, and a Jenesis build that resolves modules by name a jenesis or java repository. What a client uploads - POMs, checksums and maven-metadata.xml included - is stored and served back verbatim. A key goes wherever the tool keeps credentials outside the project, so it is never committed with the build.

Maven

The key is the password of a server entry in ~/.m2/settings.xml; the user name is not checked:

<settings>
  <servers>
    <server>
      <id>jenesis</id>
      <username>jenesis</username>
      <password>jenk_releases.…</password>
    </server>
  </servers>
</settings>

The project names the repository by that id, to publish with mvn deploy and to resolve from:

<project>
  <distributionManagement>
    <repository>
      <id>jenesis</id>
      <url>https://repo.example.com/repository/releases/<repo>/maven/</url>
    </repository>
  </distributionManagement>

  <repositories>
    <repository>
      <id>jenesis</id>
      <url>https://repo.example.com/repository/releases/<repo>/maven/</url>
    </repository>
  </repositories>
</project>

To send every download through the repository instead - when it proxies Maven Central or groups a proxy with your own releases - name it as a mirror in settings.xml rather than in each project:

<mirrors>
  <mirror>
    <id>jenesis</id>
    <mirrorOf>*</mirrorOf>
    <url>https://repo.example.com/repository/releases/<repo>/maven/</url>
  </mirror>
</mirrors>

A folder URL - one ending in /, such as .../maven/com/acme/lib/ - answers 404 unless the repository's Folder listings setting is on. Maven and Gradle never need it: they read maven-metadata.xml. Coursier and sbt list a folder where that file is missing - an imported repository, say - to find the versions a range like 1.+ can pick, and a person may want to browse. With the setting on, a folder answers a plain HTML index of what a download would serve, a thousand names at a time with a link to the next page, or JSON when asked for with Accept: application/json. A withheld version is left out, so a listing never offers what then fails to download. It is off by default because every page reads the store once per name it shows; set it for the repositories whose clients need it, or for a tenant or the whole deployment:

jenrepo repos settings <repo> set folder-listing true

maven-metadata.xml

By default a maven-metadata.xml is served byte for byte as the client uploaded it, with the checksums the client uploaded beside it. The same holds for every checksum: a .sha1, .md5, .sha256 or .sha512 is stored and served exactly as uploaded, never checked against the file it describes, and never made up where none was uploaded. A checksum is the publisher's to provide and the client's to check. One beside a release is as fixed as the release: a different one uploaded over it is refused with 409, as a release's own bytes are, unless allow-redeploy is on.

With maven-metadata-compute switched on - a deployment setting, applied on the next restart - the server computes each artifact's document instead. Its <versions> list is reconciled against the versions stored here, every other field is kept as published, and a document is derived for an artifact that never had one uploaded, as after an import. The computed document comes with all four checksums.

The two differ most in a repository that both hosts and proxies:

• Computed, the document lists the upstream's versions and the ones published here as one list, kept live: a release published upstream later appears in it, and a version yanked here is left out whichever side lists it.
• By default, mvn deploy uploads the document it read before deploying - the upstream's, relayed through this repository - with its own version added. That stored copy then answers ahead of the upstream's, so releases the upstream publishes afterwards are not listed for that artifact.

Gradle

A repository named jenesis with PasswordCredentials reads its user name and key from jenesisUsername and jenesisPassword, which belong in ~/.gradle/gradle.properties:

jenesisUsername=jenesis
jenesisPassword=jenk_releases.…

The build resolves from the repository and publishes to it with ./gradlew publish:

// build.gradle.kts
plugins {
    `java-library`
    `maven-publish`
}

repositories {
    maven {
        name = "jenesis"
        url = uri("https://repo.example.com/repository/releases/<repo>/maven/")
        credentials(PasswordCredentials::class)
    }
}

publishing {
    publications {
        create<MavenPublication>("library") {
            from(components["java"])
        }
    }
    repositories {
        maven {
            name = "jenesis"
            url = uri("https://repo.example.com/repository/releases/<repo>/maven/")
            credentials(PasswordCredentials::class)
        }
    }
}

A build that publishes Ivy descriptors uses an ivy repository instead, at the repository's own URL. Gradle's default Ivy layout is the one the repository accepts, so nothing more needs saying:

plugins {
    `java-library`
    `ivy-publish`
}

repositories {
    ivy {
        name = "jenesis"
        url = uri("https://repo.example.com/repository/releases/<ivy-repo>/")
        credentials(PasswordCredentials::class)
    }
}

publishing {
    publications {
        create<IvyPublication>("library") {
            from(components["java"])
        }
    }
    repositories {
        ivy {
            name = "jenesis"
            url = uri("https://repo.example.com/repository/releases/<ivy-repo>/")
            credentials(PasswordCredentials::class)
        }
    }
}

Jenesis

A Jenesis build resolves Maven coordinates from jenesis.maven.uri and modules by name from jenesis.module.uri, and both can be the one java repository: its Maven layout under maven/, and every modular jar published into it served by module name as well. A token is never read from a project's own jenesis.properties - it would be committed with the build - so the address and the key both go in your user-global ~/.jenesis/jenesis.properties:

jenesis.maven.uri=https://repo.example.com/repository/releases/<repo>/maven/
jenesis.maven.token=jenk_releases.…
jenesis.module.uri=https://repo.example.com/repository/releases/<repo>/
jenesis.module.token=jenk_releases.…

or in the environment, which suits a CI job:

export MAVEN_REPOSITORY_URI=https://repo.example.com/repository/releases/<repo>/maven/
export MAVEN_REPOSITORY_TOKEN="$KEY"
export JENESIS_REPOSITORY_URI=https://repo.example.com/repository/releases/<repo>/
export JENESIS_REPOSITORY_TOKEN="$KEY"
java build/jenesis/Make.java

Either way a build that names only the repository resolves from it alone. Append ,@ to an address to fall back to the public defaults for whatever the repository does not hold - …/<repo>/maven/,@ - though a repository that proxies them already answers for them.

In a java repository, every modular jar is a published module too. When a jar published through Maven carries a module-info or an Automatic-Module-Name, a java repository also serves it by module name under module/, and under artifact/ beside the POM published with it. So a Jenesis build that requires that module resolves it from the same repository with no second upload: artifact/ gives its POM, and the Maven coordinate that POM names gives its jar. A jar published under a classifier is served beside the module's own as <module>-<classifier>.jar. A maven repository serves the Maven layout alone; creating it again with the type java makes it a java repository, with every URL it answered still answering.

A Jenesis build puts its own modules into a jenesis repository with release, once jenesis.release.uri names the repository's address and jenesis.release.token a key that may publish to it: one put of each module's jar under its version, below module/. The repository moves the module's version-less <module>/<module>.jar to the version released last, so that path is not an upload target, and neither is anything else. A java repository refuses that put - Maven drives its publication - so modules a build releases go to a jenesis repository, and modules published through Maven reach module consumers from a java one. Both settings have environment variables of their own, JENESIS_RELEASE_URI and JENESIS_RELEASE_TOKEN, so a CI job keeps the key it releases with apart from the key it resolves with. Publishing describes the release.

Containers

The registry answers at the host root, because the container protocol fixes it at /v2/. An image's name begins with the tenant and the oci repository it lives in, so an image my-app in a repository named images is releases/images/my-app:

docker login repo.example.com -u jenesis -p "$KEY"
docker tag my-app repo.example.com/releases/images/my-app:1.0
docker push repo.example.com/releases/images/my-app:1.0
docker pull repo.example.com/releases/images/my-app:1.0

The registry's catalog, GET /v2/_catalog, lists every image in the tenant's oci repositories by the name a client pulls it by - releases/images/my-app - and pages with a Link header.

Image layers are stored by their digest, so a layer shared by many images - or identical to a file stored by another format - is kept once.

Homebrew

A homebrew repository is a bottle domain for the bottles you build yourself. With HOMEBREW_BOTTLE_DOMAIN=$REPO/<name>, brew install asks for each bottle as one file under that address, named the way the formula names it, and a bottle the domain does not hold is fetched from Homebrew's default domain instead.

homebrew-core's own bottles are container-image blobs on ghcr.io, and they pull through an oci repository rather than a homebrew one. HOMEBREW_ARTIFACT_DOMAIN=https://repo.example.com makes brew ask for them at /v2/homebrew/core/<formula>/…, which names the tenant homebrew and the repository core. So the mirror is an oci repository named core, defined as fallback https://ghcr.io/homebrew/core, in a deployment whose default tenant is homebrew (JENREPO_DEFAULT_TENANT=homebrew): a request that carries no key is answered for the default tenant alone. An anonymous install also needs anonymous reads allowed (JENREPO_ANONYMOUS_RIGHTS=repository:read); HOMEBREW_DOCKER_REGISTRY_TOKEN=$KEY presents a key instead. Each bottle is kept under its digest after the first install, and served from the store from then on.

Raw files

For artifacts that belong to no ecosystem - installers, archives, datasets - a raw repository is a plain file store. One named files holds paths directly under its URL:

curl -H "Jenesis-Repository-Key: $KEY" -T installer.msi \
  https://repo.example.com/repository/releases/files/tools/installer-1.2.msi
curl -H "Jenesis-Repository-Key: $KEY" https://repo.example.com/repository/releases/files/tools/installer-1.2.msi -o installer.msi
curl -H "Jenesis-Repository-Key: $KEY" https://repo.example.com/repository/releases/files/tools/     # lists the folder

Deprecating and withdrawing a version

A published version can be marked deprecated - still served, with a warning - or yanked - withdrawn from what a resolver picks, while a build that already pins it keeps working. Each client sees the mark in its own terms: npm prints the deprecation, Cargo's index says yanked, a yanked gem leaves the RubyGems index, NuGet lists the version as unlisted, and PyPI marks it yanked.

Where a client has its own command for it, that command sets the same mark:

npm deprecate acme-demo@1.2.0 "use 1.3 or later"     # and an empty message to undo it
cargo yank --registry jenesis --version 1.2.0 acme-demo  # cargo yank --undo to take it back
gem yank acme-demo -v 1.2.0 --host https://repo.example.com/repository/releases/gems
dotnet nuget delete Acme.Demo 1.2.0 -s jenesis          # unlists the version, as nuget.org does

Each needs the key's write right on the repository, and each is recorded on the audit trail. For every other format, and to see or clear what is marked, the repository's Deprecations & yanks page in the console and jenrepo lifecycle do the same:

jenrepo lifecycle mark libraries com.acme:widget 2.0.1 deprecated --message "use 2.0.2"
jenrepo lifecycle libraries
jenrepo lifecycle clear libraries com.acme:widget 2.0.1

A format with nowhere to show a mark to its client refuses one rather than storing a mark nobody would see.

Switching a format off

Every format is on until you switch it off. JENREPO_<FORMAT>=false keeps one from starting, exactly as if it were not installed: no repository can be created with its type, a repository that holds it answers 404, and nothing is imported for it. The names are the types in the table above - maven, npm, pypi, oci, go, cargo, nuget, rubygems, helm, and so on - and Settings → Modules switches them from the console, taking effect on the next restart. A combined type such as java is offered only while every format it holds is on.