Migrating in and out

Most teams adopting Jenesis Repository already have artifacts somewhere else. This chapter shows how to move them in, how to follow and resume the job, how to load an archive of files in one go, and how to take everything back out when you leave - as a listing, or by exporting it straight into another repository.

Starting an import

An import is a background job that walks another repository manager and publishes every artifact it finds into this repository, through the same gate a client's upload passes. The repository is created first, with the type of what you are bringing in (see Repositories): an import into a repository that holds no type is refused before it starts, and a repository holds one type, so an import lays out only the formats it holds - a Nexus instance with Maven and Docker repositories is brought into a maven repository and an oci one, one import each. Open the repository you are importing into and choose Import. Under Start a migration:

Start migration answers at once, and Migrations below lists every job with how many artifacts it has imported, skipped, held for review and rejected, refreshing while one runs. A job that stopped part-way offers Resume from cursor, which continues from its last checkpoint rather than starting over; Dismiss removes a finished job from the list, and finished jobs are dismissed on their own after seven days.

The same import can be started from a script with a POST to /api/repository/import, naming the repository in ?repo=, and a key that may write to the repository:

curl -X POST 'https://repo.example.com/api/repository/import?repo=libraries' \
  -H "Jenesis-Repository-Key: $KEY" -H 'Content-Type: application/json' \
  -d '{
        "source": "nexus",
        "url": "https://nexus.example.com",
        "repository": "maven-releases",
        "username": "migrator",
        "password": "…"
      }'
# 202  {"job":"a1b2c3…","state":"running"}

The request fields, which match the form:

The other connectors report a format per asset, so they take none. Only POST starts a job; any other method on /api/repository/import answers 405, and an import into a repository that holds no type answers 400. A deployment in read-only mode refuses imports with 403.

Following and resuming a job

The job writes its state into the store, so it survives a restart and any node can answer for it. Read it with the id the POST returned:

curl -H "Jenesis-Repository-Key: $KEY" 'https://repo.example.com/api/repository/import/a1b2c3…?repo=libraries'

{"state":"completed","imported":128,"skipped":0,"held":0,"rejected":0,
 "skippedFormats":[],"cursor":null,"asset":"maven/org/example/app/1.0/app-1.0.jar","error":null}

A job that stopped - a network fault, a restart - is continued by submitting the same request again with "resume": "<job id>". The walk resumes under that same job id, from its recorded cursor, carrying imported and skipped forward; held and rejected restart at zero and count the resumed run alone.

The content-addressed store makes any overlap cost nothing: re-importing bytes that are already stored needs no space and changes nothing, so a re-run after a partial migration is always safe.

The connectors

Each connector reads one kind of source. All of them stream every artifact straight into the store, and all but one checkpoint as they go: the Artifactory Pro deep listing arrives as a single response, so a job walking it has no mid-walk resume point.

Nexus

nexus pages the Sonatype Nexus 3 components API and downloads every asset a component lists. Nexus names a format per asset (maven2, docker, raw, and others), so one job can migrate a Nexus instance that holds repositories of several formats. The credentials you pass travel only to the base origin; a download URL on another host is fetched without them.

Artifactory

artifactory lists a repository with the storage API and downloads each file. An Artifactory repository has a single package type, so the request must name its format. Against Artifactory Pro it uses the deep file listing, which is one response and carries no resume point; against an OSS instance, which refuses that API, it falls back to the per-folder listing and checkpoints after every top-level entry, folder or file, so an interrupted OSS migration resumes without re-walking.

Each file is published through its format's own publish path, so it is read, screened and indexed as if a client had published it. Indexes and other documents the format derives are skipped, since the import rebuilds them. Three package types are read at the paths Artifactory documents for them:

Alpine branches have no level of their own here. Packages from two branches of one repository and architecture end up in one apk repository, and a version both branches hold with different bytes is refused as a republish. The incumbent's APKINDEX.tar.gz and a provider's SHA256SUMS are signed with the incumbent's key, so they are not carried over: this repository signs its own.

Maven

maven walks any repository that serves the Maven layout over plain HTTP - a Nexus or Artifactory repository root, an nginx or httpd directory listing, a static bucket, or another Jenesis Repository. Where the server exposes a listing, the tree is walked; where it does not, the connector falls back to the .index/nexus-maven-repository-index.gz index and refreshes each coordinate through its maven-metadata.xml. Every asset is reported as maven.

Index

index walks a format's own published index rather than a vendor API, which is why it needs the format up front: a PyPI Simple index, a NuGet V3 feed, a Debian archive's Packages indexes, an RPM repository's repodata, a conda channel's repodata.json, a Composer repository's packages.json or a registry's catalog. The walk starts at the base URL with repository appended - . appends nothing - which is where the format's own client is pointed. For an OCI registry that is the registry's root; to walk one repository of another Jenesis Repository, point it at that repository's own registry, https://repo.example.com/v2/<tenant>/<repository>/.

An RPM repository, a conda channel and a Composer repository are each named by the last segment of the address their index is read from, and the import lays the packages out in a repository, channel or registry of the same name here, so a client keeps the address it had.

Jenesis

jenesis walks another Jenesis Repository through its /api/assets listing (below), so one instance migrates into another. repository names the source repository, and the credential is the source's API key, passed as the password. Each asset is downloaded from the path the listing says the source serves it at, so nothing about the source's layout needs configuring, and it is laid out at that same path here: an RPM repository, a conda channel, or a Composer, Cargo, Helm, CocoaPods, Hugging Face, Conan, winget, Swift or Terraform registry keeps its name. A version comes across whole: a container image as its config and layers, then the manifests of its platforms, then its tag; a Go module as its .info, .mod and .zip; a winget version as its manifest, then its installers.

What the importers write

A connector hands each asset to the importer for its format. Every format carries one, and each accepts the names Nexus and Artifactory give that format as well as its own - yum for RPM, apt for Debian, alpine for apk, golang, gems, crates, huggingfaceml - and publishes each asset through the format's own publish path. Three are worth knowing in detail:

Two Maven details matter before you cut clients over. The source's maven-metadata.xml files are always left behind - their checksums describe bytes the source served, not the copy you now hold. Jenesis Repository serves a stored maven-metadata.xml verbatim, so an imported coordinate has no version listing until one is published, and a client asking "which versions exist?" gets a 404. Switch on jenrepo.maven-metadata-compute=true and the server derives the listing from the version folders it holds instead. An artifact's own checksums (.sha1, .md5, .sha256, .sha512) do come across: the server serves an artifact's checksums as they were uploaded and derives none, so the maven connector imports the ones a directory listing shows beside each artifact, and the Nexus and Artifactory connectors import the sidecars they list.

When a job finishes, the server builds the listings the imported artifacts imply - an OCI repository's tag list and catalogue, a raw folder's listing - before it reports completed, and regenerates ones that already existed, so a client pointed at the new repository sees a correct listing on its first read. A listing that cannot be built is a slow first read, never a failed migration.

Loading an archive in one request

For a one-off load without a source to walk - a backup, a hand-built tree - upload a zip and let the server publish each entry as if it had been deployed on its own. The feature is off by default; switch it on with jenrepo.batch-upload=true, then PUT or POST the archive to the repository path the entries are relative to, with the Jenesis-Explode: zip header:

curl -X PUT -H "Jenesis-Repository-Key: $KEY" https://repo.example.com/repository/releases/libraries/maven/ \
  -H 'Jenesis-Explode: zip' \
  --data-binary @artifacts.zip

Every entry is published through the same path a single deploy takes, so a Maven jar is laid out and cross-published, and an entry no format claims is reported rather than stored. The response is a per-entry manifest (path and status - stored, quarantined, rejected or unclaimed), with "capped": true when the walk stopped at jenrepo.batch-upload-max-entries (default 10 000). An entry whose name tries to escape its folder is rejected before it reaches the store, and a malformed archive answers 400. Only zip is understood; another encoding answers 415.

Listing everything back out

Leaving is as easy as arriving. GET /api/assets?repo=<repository> lists every published artifact in a repository as a flat, stably ordered, paged JSON list - the same listing the jenesis connector reads when another instance imports from this one:

curl -H "Jenesis-Repository-Key: $KEY" 'https://repo.example.com/api/assets?repo=libraries&limit=500'

{"repository":"libraries",
 "assets":[{"path":"/maven/org/example/app/1.0/app-1.0.jar",
            "served":"/repository/releases/libraries/maven/org/example/app/1.0/app-1.0.jar",
            "size":48213,"sha256":"9f3b…",
            "format":"maven","ecosystem":"Maven","coordinate":"org.example:app","version":"1.0",
            "prerelease":false}],
 "cursor":"bWF2ZW4v…"}

Each entry carries the path, size and SHA-256 straight from the publication record - no artifact is opened - plus served, the URL path a client downloads it from, and the format's reading of it: format, ecosystem, coordinate, version and prerelease. Pass the returned cursor back as ?cursor= for the next page; it is null once the listing is exhausted. repo is required and names a repository of the tenant the server serves; limit defaults to 500 and is capped at 1 000. The read needs repository:read on the repository it lists.

A format that keeps its packages in an index of its own - npm, PyPI, NuGet, Debian, RPM and most others - is listed after the files published at a path of their own, version by version, newest first. A page of those holds the files of at most limit versions, so it can carry more entries than limit, and a page can be empty while its cursor is not: follow the cursor until it is null.

The bytes themselves are addressed by the served paths the listing returns, so any HTTP client can copy a repository out - and another Jenesis Repository imports one directly with the jenesis connector above.

Moving to another repository

An export publishes every version a repository holds into another repository - another Jenesis Repository, or any repository manager - through the protocol the format's own client publishes with, so the other side needs no importer and sees nothing it would not see from a client. It is a background job, like an import.

Open the repository to export and choose its Export page. Enter the Target URL - the URL the format's own client would be pointed at to publish into the other repository - and a credential for it: a Username with a Password, or a Token. Start export answers at once, and the list below shows each job's state, how many versions it published, how many the target already held and how many it withheld, refreshing while one runs.

The same job starts from a script with a POST to /api/repository/export, naming the repository in ?repo=:

curl -X POST 'https://repo.example.com/api/repository/export?repo=libraries' \
  -H "Jenesis-Repository-Key: $KEY" -H 'Content-Type: application/json' \
  -d '{"url": "https://other.example.com/repository/maven-releases/", "token": "…"}'
# 202  {"job":"d4e5f6…","state":"running"}

curl -H "Jenesis-Repository-Key: $KEY" \
  'https://repo.example.com/api/repository/export/d4e5f6…?repo=libraries'
# {"state":"completed","published":412,"present":0,"withheld":3,…}

and from the command line, where --refresh follows the job until it finishes:

jenrepo export libraries --url https://other.example.com/repository/maven-releases/ --token "$TOKEN"
jenrepo export status libraries d4e5f6… --refresh

What an export sends, per format, relative to the target URL:

Before sending a file the export asks the target whether it already holds it, and a file the target serves with the same bytes is not sent again - so a job can be run twice, or resumed, and a repository already partly moved is completed rather than duplicated. A target that refuses a file it already holds with the same bytes - a release repository refusing a second deploy - is not a failure. A version the gate is holding is never sent. Finished export jobs are removed after seven days.

Settings

Every key is also an environment variable (JENREPO_BATCH_UPLOAD=true) or a -D system property.