Open a file directly from a GitLab private repository

Question

I have a private repository on a GitLab server and using the SSH I can pull a project using git clone.

But I want to run a script on linux command line directly from the server (more specific, a Drupal / Drush .make file)

I tried to run it using the raw file:

drush make http://server.com/user/project/raw/master/file.make

(for the convenience of non Drupal users let’s say)

curl http://server.com/user/project/raw/master/file.make

Without success. Of course, it returns me the login page.

Is it possible?

Solution 1

With Chris's valuable help, here is how you can run a script (drupal .make file in my case) from a GitLab server. (Probably it works for GitHub but I didn't test it. Maybe the syntax will be a bit different). (Of course this works for any type of script)

It can be done using the authentication tokens. Here is the documentation of the GitLab's API and here is the GitHub's API

For convenient I will use the https://gitlab.com as the example server.

• Go to https://gitlab.com/profile/account and find your "Private token"

• Then print the list of the projects and find the id of your project you are looking for

  curl https://gitlab.com/api/v3/projects?private_token=<your_private_token>

  or go there with your browser (a json viewer will help a lot)

• Then print the list of the files that are on this project and find the id of your file you are looking for

  curl https://gitlab.com/api/v3/projects/<project_id>/repository/tree?private_token=<your_private_token>

• Finally get / run the file!

  curl https://gitlab.com/api/v3/projects/<project_id>/repository/raw_blobs/<file_id>?private_token=<your_private_token>

In case you want to run the script (drupal .make)

drush make https://gitlab.com/api/v3/projects/<project_id>/repository/raw_blobs/<file_id>?private_token=<your_private_token> <drupal_folder>

(If you are here looking for a workflow to integrate GitLab with Aegir .make platforms without using tokens (maybe SSH?) please make a thread and paste here the link.)

EDIT

You can get the file without the project_id by using the encoded project name. For example the my-user-name/my-project will become: my-user-name%2Fmy-project

Solution 2

Update 2018-12-25:

as long as you're not downloading huge files, this should work:

curl -s -H "Private-Token: <token>" "https://gitlab.com/api/v4/projects/<urlencode("gitlab_username/project_name")>/repository/files/<path/to/file>/raw?ref=<branch_name>"

, a real example, downloading the file /README.md from the private repository https://gitlab.com/divinity76/Yur17, where the web download url is https://gitlab.com/divinity76/Yur17/raw/master/README.md?inline=false, is:

curl -s -H "Private-Token: afF2s1xgk6xcwXHy3J4C" "https://gitlab.com/api/v4/projects/divinity76%2Fyur17/repository/files/README%2Emd/raw?ref=master"

take special note of how the gitlab_username/repo_name was url-encoded, eg / became %2F (you can check how your username & repo name is as url-encoded by opening your browser javascript terminal and write encodeURIComponent("your_username/repo_name"); in the terminal and press enter.)

thanks to Jonathan Hall @ gitlab at mg.gitlab.com, and https://docs.gitlab.com/ee/api/repository_files.html , and tvl for helping reach a solution.

---

---

Update 2018-12-11: this method no longer works, now it just serves the login page, with a message saying need to log in to continue, even using HTTP 200 OK (shame on them), will update if i find out why ( @XavierStuvw claims it's security concerns related)

---

---

i found a much easier way to do it than @tvl 's answer,

first create an access token with API access here: https://gitlab.com/profile/personal_access_tokens , then do:

wget --header 'PRIVATE-TOKEN: <token>' 'https://gitlab.com/<username>/<repo>/raw/master/path/to/file.ext'

i found the solution here.

Solution 3

If you would like to access a file from private GitLab, you could use the below approach which worked for me:)

Construct the URL:

https://url/api/v4/projects/projectId/repository/files/fileName/raw?ref=master&private_token=Generated_private_token

---

• url is your Gitlab url ex: git.lab.com.
• /api/v4/projects is a constant.
• projectId is the projectId of your project, which you can find below the name of your project in gitlab.
• /repository/files is again a constant.
• fileName is the name of the file ex: sagar.txt
• /raw?ref= is a constant and the value of ref can be master or any branch which you would like to take the file from. I am retrieving the file from Master.
• Generated_private_token should be generated from gitlab, please follow the steps in mentioned in the link : Generate Private Token

Comments

[Chris]

Does http://user:[email protected]/user/project/raw/master/file‌​.make, replacing user and password with your credentials, work?

[tvl]

Thank for the help but it doesn't work. Also I would like a solution without exposing my password. (but even if that works it will be really good!)

[Chris]

You're going to have a tough time accessing files over HTTP without exposing your password. In theory you could use a client-side SSL certificate to authenticate, but that's likely a lot of work. I seriously doubt that it's supported by GitLab. You may want to create a dedicated read-only account and use that in your command to limit your exposure.

[Chris]

systemseed.com/blog/drush-make-private-git-repository-github

[tvl]

To be honest my final goal is to grab the .make files from a GitLab server and feed them on a Aegir server so it can create "platforms". But, for the moment, I try the middle linux-steps (as Aegir uses drush commands). So if I add on the Aegir server an SSL certificate I can verify it on the GitLab and will have a password-less client-to-server communication?

[Chris]

Only if you manage to get GitLab to rely on client SSL certificates as an authentication method. If GitLab offers something similar to GitHub's API, the link I added above may get you started.

[tvl]

Man you are awesome! I knew about the tokens, but I didn't try them. Thanks for the guid-inspiration. GitLab have tokens and an excellent API. here. So if you want to grab a raw file you must do somethink like: gitlabserver.com/api/v3/projects/:project-id/repository/….

[Chris]

Glad to see that this helped. Once you've got it working, please consider answering your own question with details about what you needed to do. I don't have a GitLab box handy and don't feel comfortable writing an answer without one.

[tvl]

Let us continue this discussion in chat.

[Bruce Yong Li]

How to access a file in a folder? Is it possible to use a fixed file id? The id changes every time I update the file.

[tvl]

Sorry @Yong I can't help you as I don't use this method for long time now because it wasn't practical.

[Bruce Yong Li]

Thanks for your reply. I figured out another way to make it work. This is how I did: https://gitlab.com/<group_name>/<project_name>/raw/master/<f‌​older>/<file_name>?p‌​rivate_token=<your_k‌​ey>

[hanshenrik]

@Yong doesn't work for me (or rather, only works as long as i also use a cookie session which is already logged in, and the private_token is ignored

[XavierStuvw]

Currently the private token is probably called 'access token'. I got one from gitlab.com/profile/personal_access_tokens and have used it successfully for downloading from a private repository with wget (stackoverflow.com/a/31442131/5459638 for the interested ones). By asking for an access token you will need to specify expiry date and purpose (in my case, read_repository)

[XavierStuvw]

This solution has also worked out in stackoverflow.com/a/31442131/5459638, and for me

[BLuEGoD]

This no longer works for security concerns - but the API is available and you can just use the read_repository scope (no need for the API scope).

[hanshenrik]

@BLuEGoD yeah doesn't work now :< - werid, TLS certificates are verified before headers are sent so it's probably not a MITM concern, do you know what security concerns they are talking about?

[BLuEGoD]

@hanshenrik see "Improper Enforcement of Token Scope" in about.gitlab.com/2018/11/28/…

[hanshenrik]

@BLuEGoD thanks for the heads-up, found a new (albeit terrible from a bandwidth point-of-view) way to do it, that is still easier than tvl's answer ^^

[oliolioli]

@hanshenrik Regarding your update "Update 2018-12-25" with gitlab-API v4: please add a "/raw" before ?ref=<branch_name>" then you can remove the | jd ... > filename.ext

[oliolioli]

For gitlab-API v4 I had to insert <project_id> instead of <urlencode("gitlab_username/project_name")>

[oliolioli]

<path/to/file> must also be urlencoded, e.g. . with %2E and / with %2F.

[Oo'-]

I downvoted the answer and @BruceYongLi's because the path to access a file in a folder does not work. I found GitLab very hostile than GitHub, regards cURL, SVN and wget. I'll switch to GitHub and use the old and good SVN.

[lights]

Note that file path needs to be url encoded ie. app/models/key.rb goes in the url as app%2Fmodels%2Fkey%2Erb

[C. Feng]

This is the answer I was looking for. GitLab docs suck if you use private GitLab. This one works great. Thanks!