How to store releases/binaries in GitLab?

Question

I am building a workflow with Gitlab, Jenkins and - probably - Nexus (I need an artifact storage). I would like to have GitLab to store releases/binaries - is it possible in a convenient way?

I would not like to have another service from which a release (and documentation) could be downloaded but to have it somehow integrated with repository manager, just like releases are handled in e.g. GitHub. Any clues?

Solution 1

Update Oct. 2020:

GitLab 13.5 now offers:

Attach binary assets to Releases

If you aren’t currently using GitLab for your releases because you can’t attach binaries to releases, your workflow just got a lot simpler.

You now have the ability to attach binaries to a release tag from the gitlab.ci-yml. This extends support of Release Assets to include binaries, rather than just asset links or source code. This makes it even easier for your development teams to adopt GitLab and use it to automate your release process.

See Documentation and Issue.

---

Update Nov. 2015: GitLab 8.2 now supports releases.

With its API, you now can create and update a relase associated to a tag.
For now, it is only the ability to add release notes (markdown text and attachments) to git tags (aka Releases).

• first upload the release binary
• create a new release and place a link to the uploaded binary in the description

Update May 2019: GitLab 1.11 introduces an interesting "Guest access to Releases":

It is now possible for guest users of your projects to view releases that you have published on the Releases page.
They will be able to download your published artifacts, but are not allowed to download the source code nor see repository information such as tags and commits.

---

Original answer March 2015

This is in progress, and suggested in suggestions 4156755:

We’re accepting merge requests for the minimal proposal by Ciro:

1. For each repository tag under https://github.com/cirosantilli/test/releases/tag/3.0, allow to upload and download a list of files. 2. The upload and download can be done directly from the tag list view. 3. If a tag gets removed, the uploads are destroyed. (we’re not accepting the tag message edit mentioned in recent comments)

---

The comments to that suggestion include:

What makes it more interesting is the next step.
I would really like a way to let public download artifacts from "releases" without being able to access source code (i.e. make sources private for project team only except anything else like wiki, "releases", issue tracker).

However, such additional feature looks more generic and I submitted a separate feature request for that.

Nevertheless, I repeat my point here:
While the simplistic version of "releases" is still nice to have, many people can easily set up external file server and point URLs in release/tag description to this server outside GitLab.
In other words, "releases" may not look attractive now without some future picture of integration.

Solution 2

This answer is going to be basically the same, as the one from VonC, just described in a step-by-step manner for less experienced CI users.

So, let's say, that you have a really cool commit 30728cab and you'd like to make this version of your code a new release...

1) Create a tag for your commit

git tag -a MY_TAG_NAME 30728cab

After this command you'll be asked to fill in a description, just like when you're committing new changes to your code.

2) Push the tag to your remote repository

Tag are NOT going to be pushed there automatically with your commits! You need to push them manually to your remote.

git push REMOTE_REPO_NAME REMOTE_BRANCH_NAME MY_TAG_NAME

3) Upload a file

Now you can either a) upload a file to the GitLab repository, b) upload it anywhere else and save the link in both cases.

WARNING: Files uploaded to the GitLab repository can't be easily deleted then and you can't see their link later!

While I'm NOT recommending uploading binaries to the repository because of the reason above, you asked for it, so here is the way:

curl --request POST --header "Private-Token: YOUR_PRIVATE_TOKEN" --form "file=@/PATH/TO/THE/FILE/file.txt" "https://MY_GITLAB_HOSTING.COM/api/v4/projects/MY_PROJECT_ID/uploads"

The private token can be created in User settings -> Access tokens.

Also, if you really needed to delete the file, you can export the project, remove manually the folder updates from your downloaded archive, delete the former remote repository and create a new one by importing your downloaded and modified archive.

4) Create a release

Now we can finally tie it all together using Release.

curl --request POST --header 'Content-Type: application/json' --header "Private-Token: YOUR_PRIVATE_TOKEN" --data '{"name": "MY_RELEASE_NAME", "tag_name": "MY_TAG_NAME", "description": "Release with the binary LINK_TO_YOUR_BINARY"}' "https://MY_GITLAB_HOSTING.COM/api/v4/projects/MY_PROJECT_ID/releases"

You can see, that Release is inherently tied with a specific tag, which is subsequently tied to a specific commit. Connection with binaries is then performed simply by providing links to those files.

The funny point is, that your description supports Markdown, but it's really hard to write some larger *.md document in such cumbersome one-liner. So I've written a short Bash script, which allows us to write the Markdown file aside and then it reads it and sends it automatically:

#!/bin/bash

RELEASE_NAME="$1"
TAG_NAME="$2"
PROJECT_ID="$3"
DESCRIPTION_FILE_PATH="$4"
PRIVATE_TOKEN="$5"

if [ "$5" == "" ]; then
    echo "Missing parameter! Parameters are RELEASE_NAME, TAG_NAME, PROJECT_ID, DESCRIPTION_FILE_PATH and PRIVATE_TOKEN.";
    exit 1;
fi

DESCRIPTION=''

# Load data from file
while read -r line; do
    DESCRIPTION="${DESCRIPTION}${line}\n";
done < "${DESCRIPTION_FILE_PATH}"

curl --request POST\
     --header 'Content-Type: application/json'\
     --header "Private-Token: ${PRIVATE_TOKEN}"\
     --data-binary "{\"name\": \"${RELEASE_NAME}\", \"tag_name\": \"${TAG_NAME}\", \"description\": \"${DESCRIPTION}\"}"\
     "https://MY_GITLAB_HOSTING.com/api/v4/projects/${PROJECT_ID}/releases" 

echo

so you can use it just like

./upload_release.sh MY_RELEASE_NAME MY_TAG_NAME MY_PROJECT_ID MY_MARKDOWN_FILE_PATH MY_PRIVATE_TOKEN

And now that's it! You have your first complete release!

Until you realizes, that you've made a terrible typo in the header of your release description...

5) Delete a release (if needed)

Here you're lucky! Unlike uploaded binaries you can delete your releases using the REST API, too!

curl --request DELETE --header "Private-Token: MY_PRIVATE_TOKEN" "https://MY_GITLAB_HOSTING.com/api/v4/projects/MY_PROJECT_ID/releases/MY_TAG_NAME"

And as it is still pretty annoying to type this several times in a row, I've made another Bash script:

#!/bin/bash

PROJECT_ID=$1
TAG_NAME=$2
PRIVATE_TOKEN=$3

if [ "$3" == "" ]; then
    echo "Missing parameter! Parameters are PROJECT_ID, TAG_NAME and PRIVATE_TOKEN.";
    exit 1;
fi

curl --request DELETE --header "Private-Token: ${PRIVATE_TOKEN}" "https://MY_GITLAB_HOSTING.com/api/v4/projects/${PROJECT_ID}/releases/${TAG_NAME}"

echo

which can be used like ./delete_release.sh MY_PROJECT_ID MY_TAG_NAME MY_PRIVATE_TOKEN.

Solution 3

I am using a command line tool I've written in python as a shortcut. Does some steps of the API methods described in previous answers (Upload, Release, Delete Release).

You can check the source code or just use it yourself.

For even simpler version, if anyone is interested in doing these with python and requests.py; it is pretty trivial.

For Upload

import requests
secret_token = "<your secret token>"
project_id = "<your project id>"
file_path = "your_file.txt"
api_root = "https://gitlab.com/api/v4"

headers = {
    'PRIVATE-TOKEN': secret_token,
}

uri = '{}/projects/{}/uploads'.format(api_root, project_id)

files = {
    'file': open(file_path, 'rb')
}


r_upload = requests.post(uri, headers=headers, files=files)
if(r_upload.status_code != 201 and r_upload.status_code != 200):
    raise ValueError(f"Upload API responded with unvalid status {r_upload.status_code}")  # noqa: E501

upload = r_upload.json()

Upload returns a json with the link and the markdown formatted link. I found markdown formatted link pretty useful for the release description. Since you have to add the link to the description yourself.

For Creating a Release

import requests
secret_token = "<your secret token>"
project_id = "<your project id>"
api_root = "https://gitlab.com/api/v4"

desc = """
Your release description. **You can use markdown !**
Don't forget to add download link here.
"""

headers = {
    'PRIVATE-TOKEN': secret_token,
}

data = {
    "description": desc
}

r_new_release = requests.post(uri, data=data, headers=headers)
if(r_new_release.status_code != 201 and r_new_release.status_code != 200):
    raise ValueError(f"Releases API responded with invalid status {r_new_release.status_code}")

new_release = r_new_release.json()

JSON Response is the release.

For the other steps basically follow API guidelines and make the adjustments. All of them are pretty similar.

Solution 4

We're using scp to copy files, such as binaries or reports, generated in GitlabCI.

# capture test exit code
set +e
bash build/ci/test.sh; TESTS_EXIT_CODE=$?
set -e

# copy reports
sshpass -p "$SFTP_PASS" ssh -o StrictHostKeyChecking=no [email protected] "mkdir -p ${CI_REPORTS_PATH}"
sshpass -p "$SFTP_PASS" scp -r ${CI_APP_VOLUME}/tests/_output/* [email protected]:${CI_REPORTS_PATH}

# return test exit-code
exit ${TESTS_EXIT_CODE}

View more solutions

Comments

[Kasia Gauza]

Well, yes and no, git annex allows to deal with binaries in git. GitLab CI is out of question, I will have to choose only between Jenkins and TeamCity.

[volker]

git annex is an exception of git with a modified back end. While you can always hack your CI and CD environment I don't think it is feasible to push release builds to git-annex (even if it can delete old releases, which is the benefit of Nexus & co). The checkout will be just more complicated as a simple get request from Nexus/FTP/HTTP(S)/S3/etc.

[Ciro Santilli OurBigBook.com]

Can you attach files larger than 10Mb to releases like you can on GitHub? Is it configurable locally?

[VonC]

@CiroSantilli包子露宪六四事件法轮功 Not sure: gitlab.com/gitlab-com/support-forum/issues/2654, redirecting to gitlab.com/gitlab-org/gitlab-ce/issues/39938

[Kasia Gauza]

Guest access to Releases is a really interesting feature!

[James Mak]

I was able to use this script only once to create a release and now i'm trying to do the same exact thing but it the script doesn't fully run. It exits when appending DESCRIPTION variable to the curl command. At least that what it looks like when use `bash -x release.sh'

[Eenoku]

@JamesMak And how do you launch the script normally?

[James Mak]

@Eonoku I was using WSL. Turned out that the loop that stores the file to a variable just wasn't doing it properly. The special characters like # and * kept messing up the curl command. I still have no idea why that was happening (I'm pretty new to bash). But instead of reading the file using a loop I used this command sed -zE 's/\r\n|\n/\\n/g' < ${DESCRIPTION_FILE_PATH} and that solved my issues.