What does git mean by, "unable to migrate objects to permanent storage"?

Question

What does git mean by, "unable to migrate objects to permanent storage"?

 Counting objects: 4, done.
 Delta compression using up to 8 threads.
 Compressing objects: 100% (4/4), done.
 Writing objects: 100% (4/4), 956 bytes | 0 bytes/s, done.
 Total 4 (delta 2), reused 0 (delta 0)
 error: failed to push some refs to 'https://git.patrikx3.tk/server-scripts.git'
 To https://git.patrikx3.tk/server-scripts.git
 !  refs/heads/master:refs/heads/master [remote rejected] (unable to migrate objects to permanent storage)
 Done

Solution 1

You can see the error message "unable to migrate objects to permanent storage" introduced in commit 722ff7f for Git 2.11 in Oct. 2016.

The explanation is:

receive-pack: quarantine objects until pre-receive accepts

When a client pushes objects to us, index-pack checks the objects themselves and then installs them into place.
If we then reject the push due to a pre-receive hook, we cannot just delete the packfile; other processes may be depending on it. We have to do a normal reachability check at this point via git gc.

But such objects may hang around for weeks due to the gc.pruneExpire grace period. And worse, during that time they may be exploded from the pack into inefficient loose objects.

Instead, this patch teaches receive-pack to put the new objects into a "quarantine" temporary directory.
We make these objects available to the connectivity check and to the pre-receive hook, and then install them into place only if it is successful (and otherwise remove them as tempfiles).

The code is:

    /*
     * Now we'll start writing out refs, which means the objects need
     * to be in their final positions so that other processes can see them.
     */
    if (tmp_objdir_migrate(tmp_objdir) < 0) {
        for (cmd = commands; cmd; cmd = cmd->next) {
            if (!cmd->error_string)
                cmd->error_string = "unable to migrate objects to permanent storage";
        }
        return;
    }
tmp_objdir = NULL;

The tmp_objdir_migrate() function comes from commit 2564d99 (still for Git 2.11)

it helps callers create a temporary directory inside the object directory, and a temporary environment which can be passed to sub-programs to ask them to write there (the original object directory remains accessible as an alternate of the temporary one).

As mentioned, this could result from a permission issue (or disk space issue)

Also, using (on the server side) a git 2.10 would likely make that error disappear.

---

Git 2.13 (Q2 2017) will expand on that quarantine notion:
See commit d8f4481, commit eaeed07, commit 360244a (10 Apr 2017) by Jeff King (peff).
^{(Merged by Junio C Hamano -- gitster -- in commit 9f1384f, 24 Apr 2017)}

git receive-pack man page now includes:

Quarantine Environment

When receive-pack takes in objects, they are placed into a temporary "quarantine" directory within the $GIT_DIR/objects directory and migrated into the main object store only after the pre-receive hook has completed. If the push fails before then, the temporary directory is removed entirely.

This has a few user-visible effects and caveats:

1. Pushes which fail due to problems with the incoming pack, missing objects, or due to the pre-receive hook will not leave any on-disk data. This is usually helpful to prevent repeated failed pushes from filling up your disk, but can make debugging more challenging.

2. Any objects created by the pre-receive hook will be created in the quarantine directory (and migrated only if it succeeds).

3. The pre-receive hook MUST NOT update any refs to point to quarantined objects. Other programs accessing the repository will not be able to see the objects (and if the pre-receive hook fails, those refs would become corrupted).

---

razor7 proposes in the comments:

In my case, using GitLab from Docker, ran those commands from the host server (not the GitLAb container)

cd /media/data/gitlab/data/git-data/repositories/@hashed 
grep "gitlab-user/repo-name" . -R 
cd /media/data/gitlab/data/git-data/repositories/@hashed/found-folder 
chown -R 998:998 ./hash-repo-folder-name.git/. 

Have in mind that 998 may vary for you.

Solution 2

Presumably when it compresses objects it puts them somewhere temporarily, then in a separate action tries to move them to their permanent location. Moving them means copying them from the temporary location to the permanent location, then either deleting them from the temporary location or leaving them to be removed by other processing, possibly at another time. For example, many of the files and directories that get written to the /tmp directory in Linux are customarily left there, until the operating system removes them upon the next reboot (if there ever is a reboot).

The failure looks like a lack of permission to write to the remote location. The failure occurred when git tried to move the objects that it compressed to the remote location https://git.patrikx3.tk/server-scripts.git that you would have specified in the project's configuration. Less likely, you could have specified it in the command or (I think?) in your global configuration.

Comments

[Martlark]

Permissions! The previous guy doing our repositories had a habit of always using root. chmod -R g+w *, fixed the error.

[Java Dude]

chmod -R g+w * fix my issue

[MensSana]

chown -R git:giton the objects folders (gitlab server: repositories) fixed the issue. some objects folders were owned by root mysteriously (2 in fact).

[VonC]

@razor7 Thank you for this feedback. I have included your comment in the answer for more visibility.