From ab128d72bb89f0d43f74f953cb1336be73bba960 Mon Sep 17 00:00:00 2001
From: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date: Sat, 23 Jan 2021 15:13:23 +0100
Subject: [PATCH] Squash and rebase multiarch feature branch

Moves manifest-approach to a parallel role besides the 1.3.0 approach.

Also adds pipeline mechanics to automatically detect and handle
architecture builds.

Furthermore squashed commit of the following:

commit 1b3bd0fdce08778f3df4b13d7ed33a78dcc79f0b
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Sat Jan 16 16:48:26 2021 +0100

    CI: fix syntax error in CI script

commit 9ce8c0b7d50b7181578693ea5fec3db652f83ceb
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Sat Jan 16 14:23:00 2021 +0100

    shell-tools: add test-cases for manifests

commit 02a128dc08dad494536ebb01647e01e9d3a6dfd3
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Sat Jan 16 14:22:38 2021 +0100

    refactor remaining code to manifest

    Now it's fully manifest approach with the naming, thinking and
    everything.

    Makes a lot of things clearer and should help getting multiarch done.

    In theory this is sufficient going forward, since manifests can also be
    used for single-arch. In order to be backwards compatible and still
    support the simple tagging approach we can re-introduce the tagging
    part and split it by using different CI templates, while the
    build-ah-engine uses the multiarch templates.

    For compatibility reasons I'd use a new template name for multiarch in
    that case.

commit 94f91a01ba0e05d7c4d8efc9ed10ee9c55ae7181
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Sat Jan 16 14:18:04 2021 +0100

    shell-tools: remove latest test

    The latest test is incompatible with the full-metadata approach.

    Could be restored, when legacy tagging is restored as well.

commit 79f9f29a7afd3987a27980a5720bb1ae0383a897
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Fri Jan 15 22:29:49 2021 +0100

    shell-tools: go all in with manifest

    This removes all the images code and relies solely on manifests.

    The templates always create images with manifest suffix.

    What's left todo is to get rid of the pull-code and just try to add all
    built archs to the manifest directly from the remote, since it should
    be pushed to the remote anyway.

commit 8d4e70bac998d82062e4c175b494901357b8f4e2
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Fri Jan 15 22:20:45 2021 +0100

    tests: remove obsolete statements from multiarch

commit 756beaf2953849a9eb85ada4dbf4f6064aae3aa0
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Fri Jan 15 21:33:39 2021 +0100

    shell-tools: fix inaccessible pushImageOrManifest

    The function has to be exported and used with bash -c here in order to
    work with xargs.

    See for more info:
    https://stackoverflow.com/questions/11003418/calling-shell-functions-with-xargs

commit 4897440367aad609a2be1041cf6987a2ab708a87
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Thu Jan 14 14:37:11 2021 +0100

    disable shellcheck rule SC2015

    The rule notes that this could be mis-used as if-then-else. This is
    intended here, since the logic is not meant to be if-then-else but
    rather if-then-anyway

commit 5a6633e8bfaf71927d88d722ad65d32fe6107189
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Thu Jan 14 14:30:46 2021 +0100

    add aarch64 build job

    Since we support the alternative name aarch64 for arm64 we should also
    provide an optional build job for it in the template.

    The build-ah-engine will per default only build amd64 and arm64

commit 6609af6abc6a2b38a5c51fe721995d290f11b968
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Thu Jan 14 14:29:44 2021 +0100

    extend push tooling by manifest type

    The push tool will now perform a simple check if the image reference is
    of type manifest and will invoke the correct command for that.

    The --all parameter is used to push all images on the local machine
    referenced in the manifest which has shown to reduce errors at
    registries.

    The --format v2s2 tag is crucial to push the manifest in docker format
    which is the only supported format of quay.io.

commit 6b330f9e0da7cc70db50d87eed8f452c4d5e9405
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Thu Jan 14 14:27:28 2021 +0100

    add si-pull tool

    The image pull is now done in it's own script file to handle the logic
    of trying different expected images and fail if none was found.

commit 5a683bc18d9c5743004f598f9d97bb58c842e527
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Thu Jan 14 13:30:35 2021 +0100

    Revert removal of var exports

    We don't want to break backwards compatiblity.

commit caf9812d3f18dc57f9372e60b58fa6d5bf7d3d4f
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Wed Jan 13 00:57:00 2021 +0100

    fix test for manifest based build

    Don't test the tagging tools, since we already do this in a separate
    job.

commit 9271931792f176763379edcd0cf379600a640ede
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Wed Jan 13 00:56:15 2021 +0100

    pin build base image to build-ah-engine 2.0.0

commit 265caf5bdff5a4f343d67edbcae3aded2ad766b8
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Wed Jan 13 00:06:55 2021 +0100

    Exclude runs on merge requests

    Reference: https://docs.gitlab.com/ee/ci/yaml/README.html#prevent-duplicate-pipelines

commit d4580afbbed9f05d678d55f6f2bebcf063b8602c
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Wed Jan 13 00:00:47 2021 +0100

    add missing storage.conf

commit bc6fe549e4f77390dcdec9d26f59378e701008a4
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Tue Jan 12 23:53:35 2021 +0100

    remove syntax error in before_script

commit 189c14fcd9b4387b0fe73adc9ec580e42d0ac606
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Tue Jan 12 23:49:04 2021 +0100

    fix typo in path to shell-tools

commit abdad0e1d58246f2c4275e27e88a1594666a0aa5
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Tue Jan 12 23:44:18 2021 +0100

    fix before_script execution order

    First install podman, then use it.

commit 47f90cf1fe4005dad127164ab30d69af70b3d86d
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Tue Jan 12 23:41:25 2021 +0100

    remove arch variable

commit bd5d66d785450c7b74a96c7f2f0e0791b2f6d20f
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Tue Jan 12 23:37:15 2021 +0100

    setup initial clean build on plain fedora

commit 83c2cc9566c329ed829528326c853a4f1cc37352
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Tue Jan 12 23:20:18 2021 +0100

    add default build archs ci variable

    Setting the default for the build-ah-engine repo to all archs and the
    templates included in other projects to amd64.

commit f1464918d39d08c049fa483b4aed4aa056055a38
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Tue Jan 12 23:03:54 2021 +0100

    fix syntax of variables hash

commit 8017c09ab5c0653db5faa9e935a7b101021fd3f7
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Tue Jan 12 23:02:11 2021 +0100

    fix tilde typo

commit 9b82e942f533b61690b1cfe630d96a75dd695648
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Tue Jan 12 23:00:03 2021 +0100

    shell-tools: add multiarch tagging and tests

    Behavior is to check if any multiarch tags / images are present in local
    storage and if so create a manifest instead of a normal image tag.

commit b8263166ed4bbc7734dc69705f95eaf20b17cfed
Author: Alexander Wellbrock <a.wellbrock@mailbox.org>
Date:   Tue Jan 12 22:42:19 2021 +0100

    rewrite using gitlab templates
---
 .gitlab-ci.yml                                |  32 +++--
 gitlab-ci-template-manifest.yml               |  53 ++++++++
 resources/shell-tools/.gitlab-ci.yml          |   4 +
 resources/shell-tools/bin/manifest-push.sh    |  54 ++++++++
 resources/shell-tools/bin/manifest.sh         | 124 ++++++++++++++++++
 resources/shell-tools/install.sh              |   6 +
 resources/shell-tools/test/manifest-ci.sh     |  36 +++++
 resources/shell-tools/test/manifest-latest.sh |  34 +++++
 resources/shell-tools/test/manifest-print.sh  |  25 ++++
 resources/shell-tools/test/manifest-push.sh   |  31 +++++
 resources/shell-tools/test/manifest.sh        |  37 ++++++
 11 files changed, 419 insertions(+), 17 deletions(-)
 create mode 100644 gitlab-ci-template-manifest.yml
 create mode 100755 resources/shell-tools/bin/manifest-push.sh
 create mode 100755 resources/shell-tools/bin/manifest.sh
 create mode 100755 resources/shell-tools/test/manifest-ci.sh
 create mode 100755 resources/shell-tools/test/manifest-latest.sh
 create mode 100755 resources/shell-tools/test/manifest-print.sh
 create mode 100755 resources/shell-tools/test/manifest-push.sh
 create mode 100755 resources/shell-tools/test/manifest.sh

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 90535ac..cc3e9a0 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -1,5 +1,5 @@
 include:
-  - local: /gitlab-ci-template.yml
+  - local: /gitlab-ci-template-manifest.yml
   - local: /resources/shell-tools/.gitlab-ci.yml
 
 stages:
@@ -10,30 +10,28 @@ stages:
   - tag
 
 variables:
-  CI_REGISTRY_IMAGE_VERSION: "1.3.0"
-  CI_REGISTRY_BUILD_ARG: "--build-arg SI_TOOLS_VERSION=0.3.1"
+  CI_REGISTRY_IMAGE_VERSION: "2.0.0"
 
+container-build-x86_64:
+  extends: .container-build
+  tags:
+    - x86_64
+
+container-build-aarch64:
+  extends: .container-build
+  tags:
+    - aarch64
 
 container-test:
   stage: test
   inherit:
     default: false
-    variables:
-      - CI_REGISTRY
-      - CI_REGISTRY_USER
-      - CI_REGISTRY_PASSWORD
-      - CI_REGISTRY_IMAGE
-      - CI_REGISTRY_BUILD_ARGS
-  image: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA
+    variables: true
+  image: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA-x86_64
   before_script:
-    - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
+    - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
   script:
     - podman build --pull $CI_REGISTRY_BUILD_ARGS -t "build-image:test" .
-    - si-tagging -l "build-image:test" "example.com/tagging-image" "0.1.2.3"
-    - podman images --format "{{.Repository}}:{{.Tag}}" | grep -Pe "^example.com/tagging-image:0.1.2$"
-    - podman images --format "{{.Repository}}:{{.Tag}}" | grep "latest"
-    - podman images --format "{{.Repository}}:{{.Tag}}" | grep -Pe "^example.com/tagging-image:0$"
-
 
-container-tagging:
+container-manifest:
   stage: tag
diff --git a/gitlab-ci-template-manifest.yml b/gitlab-ci-template-manifest.yml
new file mode 100644
index 0000000..a111018
--- /dev/null
+++ b/gitlab-ci-template-manifest.yml
@@ -0,0 +1,53 @@
+stages:
+  - build
+  - tag
+
+variables:
+  CI_REGISTRY_BUILD_ARGS: ""
+
+.container-build:
+  stage: build
+  inherit:
+    default: false
+    variables: true
+  image: quay.io/sheogorath/build-ah-engine:2.0.0
+  artifacts:
+    paths:
+      - logs/
+    expire_in: 1 week
+  before_script:
+    - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
+    - export VCS_REF="$CI_COMMIT_SHA"
+    - export VCS_URL="$CI_PROJECT_URL"
+    - export BUILD_DATE="$(date --rfc-3339 ns)"
+  script:
+    - si-fix "${CI_REGISTRY_BUILD_DOCKERFILE:-./Dockerfile}"
+    - podman build --pull 
+      --label "org.opencontainers.image.source=$CI_PROJECT_URL"
+      --label "org.opencontainers.image.revision=$CI_COMMIT_SHA"
+      --label "org.opencontainers.image.created=$(date --rfc-3339 ns)"
+      --label "org.opencontainers.image.title=$CI_PROJECT_TITLE"
+      $CI_REGISTRY_BUILD_ARGS
+      -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA-$(uname -i)"
+      --format docker
+      .
+    - podman push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA-$(uname -i)"
+    - mkdir logs/
+    - echo "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG-$CI_COMMIT_SHORT_SHA-$(uname -i)" > "logs/build-$(uname -i).log"
+
+container-manifest:
+  stage: tag
+  inherit:
+    default: false
+    variables: true
+  image: quay.io/sheogorath/build-ah-engine:2.0.0
+  before_script:
+    - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
+    - export BUILD_DATE="$(date --rfc-3339 ns)"
+  script:
+    - IMAGE_REFERENCES=()
+    - for log in logs/*.log; do IMAGE_REFERENCES+=("$(cat "$log")"); done
+    - si-manifest --push -l "$CI_REGISTRY_IMAGE" "${CI_REGISTRY_IMAGE_VERSION}" "${IMAGE_REFERENCES[@]}"
+  resource_group: latest
+  rules:
+    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
diff --git a/resources/shell-tools/.gitlab-ci.yml b/resources/shell-tools/.gitlab-ci.yml
index e06e7ff..1212afc 100644
--- a/resources/shell-tools/.gitlab-ci.yml
+++ b/resources/shell-tools/.gitlab-ci.yml
@@ -17,3 +17,7 @@ shell-tools-test:
   script:
     - ./resources/shell-tools/test/latest.sh
     - ./resources/shell-tools/test/fix-dockerfile-pinning.sh
+    - ./resources/shell-tools/test/manifest.sh
+    - ./resources/shell-tools/test/manifest-latest.sh
+    - ./resources/shell-tools/test/manifest-print.sh
+    - ./resources/shell-tools/test/manifest-push.sh
diff --git a/resources/shell-tools/bin/manifest-push.sh b/resources/shell-tools/bin/manifest-push.sh
new file mode 100755
index 0000000..def9257
--- /dev/null
+++ b/resources/shell-tools/bin/manifest-push.sh
@@ -0,0 +1,54 @@
+#!/bin/bash
+
+set -o pipefail
+set -u
+set -e
+
+printUsage() {
+    echo "
+    Shivering-Isles push tool
+
+    This tool will push all tagged versions of a container image upstream
+
+    Usage of $0:
+        $0 <IMAGE REFERENCE>
+
+    Example:
+        $0 registry.example.com/example/app:1 registry.example.com/example/app:1.2
+    "
+    exit 1
+}
+
+if [ "$1" = "--help" ]; then
+    printUsage
+fi
+
+DRY=0
+
+for i in "$@"
+do
+case $i in
+    --dry)
+    DRY=1
+    shift
+    ;;
+    -h|--help)
+    printUsage
+    shift
+    ;;
+    *)
+          # further/unknown options
+    ;;
+esac
+done
+
+TAGS_TO_PUSH=("$@")
+
+if [ "$DRY" = "1" ]; then
+    echo "${TAGS_TO_PUSH[@]}"
+else
+    for IMAGE_REFERENCE in "${TAGS_TO_PUSH[@]}"
+    do
+        podman manifest push --all --format v2s2 "$IMAGE_REFERENCE" "docker://$IMAGE_REFERENCE"
+    done
+fi
diff --git a/resources/shell-tools/bin/manifest.sh b/resources/shell-tools/bin/manifest.sh
new file mode 100755
index 0000000..2c9f3e5
--- /dev/null
+++ b/resources/shell-tools/bin/manifest.sh
@@ -0,0 +1,124 @@
+#!/bin/bash
+
+set -o pipefail
+set -u
+set -e
+
+printUsage() {
+    echo "
+    Shivering-Isles manifest tool
+
+    Usage of $0:
+        $0 [PARAMS...] <NAME> <VERSION> <IMAGE REFERENCE> [<IMAGE REFERENCE>...]
+
+    Example:
+        $0 myApp 2.0.24 myimage:test
+        $0 --suffix=alpine myApp 2.0.24 myimage:test
+    "
+    exit 1
+}
+
+createManifest() {
+    TARGET_IMAGE_REFERENCE=$1
+    podman manifest create "$TARGET_IMAGE_REFERENCE" >/dev/null
+    MANIFESTS_TO_PUSH+=("$TARGET_IMAGE_REFERENCE")
+    for IMAGE_REFERENCE in "${IMAGE_REFERENCES[@]}"
+    do
+        MANIFEST_OPTIONS=()
+        if [[ "$IMAGE_REFERENCE" =~ "aarch64"|"arm64" ]]; then
+            MANIFEST_OPTIONS+=(--variant v8)
+        fi
+        if [ "$LOCAL" = "1" ]; then
+            podman manifest add "${MANIFEST_OPTIONS[@]}" "$TARGET_IMAGE_REFERENCE" containers-storage:localhost/"$IMAGE_REFERENCE" >/dev/null
+        else
+            podman manifest add "${MANIFEST_OPTIONS[@]}" "$TARGET_IMAGE_REFERENCE" "docker://$IMAGE_REFERENCE" >/dev/null
+        fi
+    done
+}
+
+PREFIX=""
+SUFFIX=""
+LATEST=0
+LOCAL=0
+PRINT=0
+PUSH=0
+
+if [ "$1" = "--help" ]; then
+    printUsage
+fi
+
+for i in "$@"
+do
+case $i in
+    -p=*|--prefix=*)
+    PREFIX="${i#*=}"
+    shift
+    ;;
+    -s=*|--suffix=*)
+    SUFFIX="${i#*=}"
+    shift # past argument=value
+    ;;
+    -l|--latest)
+    LATEST=1
+    shift # past argument=value
+    ;;
+    --local)
+    LOCAL=1
+    shift # past argument=value
+    ;;
+    --print)
+    PRINT=1
+    shift # past argument=value
+    ;;
+    --push)
+    PUSH=1
+    shift # past argument=value
+    ;;
+    -h|--help)
+    printUsage
+    shift
+    ;;
+    *)
+          # further/unknown options
+    ;;
+esac
+done
+
+TARGET_IMAGE_NAME=${1:-invalid}
+TARGET_IMAGE_VERSION=${2:-invalid}
+FIRST_IMAGE_REFERENCE=${3:-invalid}
+
+if [ "$TARGET_IMAGE_NAME" = "invalid" ] || [ "$TARGET_IMAGE_VERSION" = "invalid" ] || [ "$FIRST_IMAGE_REFERENCE" = "invalid" ]; then
+    echo "Error: Invalid image name or version" >&2
+    printUsage
+fi
+
+shift 2
+
+IMAGE_REFERENCES=($@)
+MANIFESTS_TO_PUSH=()
+counter=1
+new_version="$(echo "$TARGET_IMAGE_VERSION" | cut -d. -f$counter)"
+last_version=""
+while [ "$last_version" != "$new_version" ]; do
+    createManifest "${TARGET_IMAGE_NAME}:${PREFIX}${new_version}${SUFFIX}"
+    last_version="$new_version"
+    ((counter++))
+    new_version="$(echo "$TARGET_IMAGE_VERSION" | cut -d. -f-"$counter")"
+done
+
+if [ "$LATEST" = "1" ]; then
+    createManifest "${TARGET_IMAGE_NAME}:latest"
+fi
+
+if [ "$PRINT" = "1" ]; then
+    echo "${MANIFESTS_TO_PUSH[@]}"
+fi
+
+if [ "$PUSH" = "1" ]; then
+    if ! command -v "si-manifest-push" >/dev/null 2>&1; then
+        ./manifest-push.sh "${MANIFESTS_TO_PUSH[@]}"
+    else
+        si-manifest-push "${MANIFESTS_TO_PUSH[@]}"
+    fi
+fi
diff --git a/resources/shell-tools/install.sh b/resources/shell-tools/install.sh
index 6480b76..9a88726 100755
--- a/resources/shell-tools/install.sh
+++ b/resources/shell-tools/install.sh
@@ -4,10 +4,16 @@ BASENAME="$(dirname "$0")"
 
 cp "$BASENAME"/./bin/tagging.sh /usr/local/bin/si-tagging
 cp "$BASENAME"/./bin/push.sh /usr/local/bin/si-push
+cp "$BASENAME"/./bin/manifest.sh /usr/local/bin/si-manifest
+cp "$BASENAME"/./bin/manifest-push.sh /usr/local/bin/si-manifest-push
 cp "$BASENAME"/./bin/fix-dockerfile.sh /usr/local/bin/si-fix
 chown root:root /usr/local/bin/si-tagging
 chown root:root /usr/local/bin/si-push
+chown root:root /usr/local/bin/si-manifest
+chown root:root /usr/local/bin/si-manifest-push
 chown root:root /usr/local/bin/si-fix
 chmod 0755 /usr/local/bin/si-tagging
 chmod 0755 /usr/local/bin/si-push
+chmod 0755 /usr/local/bin/si-manifest
+chmod 0755 /usr/local/bin/si-manifest-push
 chmod 0755 /usr/local/bin/si-fix
diff --git a/resources/shell-tools/test/manifest-ci.sh b/resources/shell-tools/test/manifest-ci.sh
new file mode 100755
index 0000000..4282e8f
--- /dev/null
+++ b/resources/shell-tools/test/manifest-ci.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+
+set -o pipefail
+set -u
+
+BASEDIR=$(dirname "$0")
+
+podman build --format docker -t mytest-latest:test-amd64 -f- <<EOF
+FROM scratch
+EOF
+podman build --format docker -t mytest-latest:test-arm64 -f- <<EOF
+FROM scratch
+EOF
+
+mkdir logs
+echo "mytest-latest:test-arm64" > logs/base-aarch64.log
+echo "mytest-latest:test-amd64" > logs/base-x86_64.log
+IMAGE_REFERENCES=()
+for log in logs/*.log; do IMAGE_REFERENCES+=("$(cat "$log")"); done
+"$BASEDIR"/../bin/manifest.sh --local registry.example.com/latest-test 1.2.3 "${IMAGE_REFERENCES[@]}"
+
+EXIT_CODE=1
+
+
+if podman manifest inspect registry.example.com/latest-test:1.2.3; then
+  EXIT_CODE=0
+fi
+
+podman rmi -f registry.example.com/latest-test:1 \
+  registry.example.com/latest-test:1.2 \
+  registry.example.com/latest-test:1.2.3
+podman rmi -f "$(podman images -q mytest-latest:test-amd64)"
+podman rmi -f "$(podman images -q mytest-latest:test-arm64)"
+rm -r logs/
+
+exit $EXIT_CODE
diff --git a/resources/shell-tools/test/manifest-latest.sh b/resources/shell-tools/test/manifest-latest.sh
new file mode 100755
index 0000000..b87f67c
--- /dev/null
+++ b/resources/shell-tools/test/manifest-latest.sh
@@ -0,0 +1,34 @@
+#!/bin/bash
+
+set -o pipefail
+set -u
+
+BASEDIR=$(dirname "$0")
+
+podman build --format docker -t mytest-latest:test-amd64 -f- <<EOF
+FROM scratch
+EOF
+podman build --format docker -t mytest-latest:test-arm64 -f- <<EOF
+FROM scratch
+EOF
+
+"$BASEDIR"/../bin/manifest.sh --local --latest registry.example.com/latest-test 1.2.3 mytest-latest:test-arm64 mytest-latest:test-amd64
+
+ERRORS=0
+podman manifest inspect registry.example.com/latest-test:latest
+ERRORS=$((ERRORS + "$?"))
+
+if [ "$ERRORS" = "0" ]; then
+  EXIT_CODE=0
+else
+  EXIT_CODE=1
+fi
+
+podman rmi -f registry.example.com/latest-test:1 \
+  registry.example.com/latest-test:1.2 \
+  registry.example.com/latest-test:1.2.3 \
+  registry.example.com/latest-test:latest
+podman rmi -f "$(podman images -q mytest-latest:test-amd64)"
+podman rmi -f "$(podman images -q mytest-latest:test-arm64)"
+
+exit $EXIT_CODE
diff --git a/resources/shell-tools/test/manifest-print.sh b/resources/shell-tools/test/manifest-print.sh
new file mode 100755
index 0000000..dfd1cad
--- /dev/null
+++ b/resources/shell-tools/test/manifest-print.sh
@@ -0,0 +1,25 @@
+#!/bin/bash
+
+set -o pipefail
+set -u
+
+BASEDIR=$(dirname "$0")
+
+podman build --format docker -t mytest-latest:test -f- <<EOF
+FROM scratch
+EOF
+
+MANIFESTS_TO_PUSH=$("$BASEDIR"/../bin/manifest.sh --local --print registry.example.com/latest-test 1.2.3 mytest-latest:test)
+
+EXIT_CODE=1
+
+if [ "$MANIFESTS_TO_PUSH" == "registry.example.com/latest-test:1 registry.example.com/latest-test:1.2 registry.example.com/latest-test:1.2.3" ]; then
+  EXIT_CODE=0
+fi
+
+podman rmi -f registry.example.com/latest-test:1 \
+  registry.example.com/latest-test:1.2 \
+  registry.example.com/latest-test:1.2.3
+podman rmi -f "$(podman images -q mytest-latest:test)"
+
+exit $EXIT_CODE
diff --git a/resources/shell-tools/test/manifest-push.sh b/resources/shell-tools/test/manifest-push.sh
new file mode 100755
index 0000000..b084397
--- /dev/null
+++ b/resources/shell-tools/test/manifest-push.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+
+set -o pipefail
+set -u
+
+BASEDIR=$(dirname "$0")
+
+podman build --format docker -t mytest-latest:test-amd64 -f- <<EOF
+FROM scratch
+EOF
+podman build --format docker -t mytest-latest:test-arm64 -f- <<EOF
+FROM scratch
+EOF
+
+MANIFESTS_TO_PUSH=$("$BASEDIR"/../bin/manifest.sh --local --print registry.example.com/latest-test 1.2.3 mytest-latest:test-arm64 mytest-latest:test-amd64)
+
+EXIT_CODE=1
+
+PUSHED=$("$BASEDIR"/../bin/manifest-push.sh --dry "$MANIFESTS_TO_PUSH")
+
+if [ "$PUSHED" == "registry.example.com/latest-test:1 registry.example.com/latest-test:1.2 registry.example.com/latest-test:1.2.3" ]; then
+  EXIT_CODE=0
+fi
+
+podman rmi -f registry.example.com/latest-test:1 \
+  registry.example.com/latest-test:1.2 \
+  registry.example.com/latest-test:1.2.3
+podman rmi -f "$(podman images -q mytest-latest:test-amd64)"
+podman rmi -f "$(podman images -q mytest-latest:test-arm64)"
+
+exit $EXIT_CODE
diff --git a/resources/shell-tools/test/manifest.sh b/resources/shell-tools/test/manifest.sh
new file mode 100755
index 0000000..ce5404b
--- /dev/null
+++ b/resources/shell-tools/test/manifest.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+
+set -o pipefail
+set -u
+
+BASEDIR=$(dirname "$0")
+
+podman build --format docker -t mytest-latest:test-amd64 -f- <<EOF
+FROM scratch
+EOF
+podman build --format docker -t mytest-latest:test-arm64 -f- <<EOF
+FROM scratch
+EOF
+
+"$BASEDIR"/../bin/manifest.sh --local registry.example.com/latest-test 1.2.3 mytest-latest:test-arm64 mytest-latest:test-amd64
+
+ERRORS=0
+podman manifest inspect registry.example.com/latest-test:1
+ERRORS=$((ERRORS + "$?"))
+podman manifest inspect registry.example.com/latest-test:1.2
+ERRORS=$((ERRORS + "$?"))
+podman manifest inspect registry.example.com/latest-test:1.2.3
+ERRORS=$((ERRORS + "$?"))
+
+if [ "$ERRORS" = "0" ]; then
+  EXIT_CODE=0
+else
+  EXIT_CODE=1
+fi
+
+podman rmi -f registry.example.com/latest-test:1 \
+  registry.example.com/latest-test:1.2 \
+  registry.example.com/latest-test:1.2.3
+podman rmi -f "$(podman images -q mytest-latest:test-amd64)"
+podman rmi -f "$(podman images -q mytest-latest:test-arm64)"
+
+exit $EXIT_CODE
-- 
GitLab