How can I use git-archive to include submodules from a bare repository

Question

I'm in the process of setting up a deployment script. The basic process is:

1. Push changes to a bare repository on the server
2. Then based on new tags will create a new folder for the release.
3. Use git archive to move the files into the release directory
4. Runs some migrations scripts and puts it live (if all is successful).

The issue is my repository contains a submodule, which doesn't get put in the archive, and therefore doesn't get put in the release directory.

I've seen git-archive-all, but that doesn't work on a bare repository.

If its not possible, I'm considering,

1. making the repository not bare, and updating the working copy, which would allow me to use git-archive-all. Or
2. having a second bare repository of the submodule on the server, which I could get an archive from (would have to look into this to make sure I'm getting the right revision).

Solution 1

I use this python package https://github.com/Kentzo/git-archive-all. You can install it by using

pip install git-archive-all

On OSX, you can install it also using brew install git-archive-all

Solution 2

Based on answers and comments here. You can create non-bare repo and run:

git ls-files --recurse-submodules | tar caf ../prog.tar.gz -T-

• To handle files starting with hypen, use --verbatim-files-from
• To put files in folder inside the archive, use --xform s:^:prog/:

Full version:

git ls-files --recurse-submodules | tar caf ../prog.tar.gz --xform s:^:prog/: --verbatim-files-from -T-

Solution 3

If your submodule is in a repo accessible from the server, I would rather have a post-receive hook which would

• update a full non-bare repo (so a second repo beside your original bare one), including the submodule (git submodule update --init)
• git archive from that second repo (you would be sure to get the right version since the non-bare repo would reference the right version of the submodule)
  Since the non-bare repo would contain the parent repo and its submodules, git archive-all would be able to detect the .git subdirectories and would archive everything.

If the submodule isn't accessible from the server, that means:

• it needs to be pushed in its own repo on the server
• the submodule in the parent repo needs to be reference with a relative path, in order for the parent repo to still be able to retrieve said submodule once pushed on the server.

Solution 4

This should do the trick: https://github.com/meitar/git-archive-all.sh/wiki

Solution 5

Here it is as a few-liner:

prefix=$(basename "$(pwd -P)")
{
  git ls-files
  git submodule foreach --recursive --quiet \
                'git ls-files --with-tree="$sha1" | sed "s#^#$path/#"'
} | sed "s#^#$prefix/#" | xargs tar -c -C.. -f "$prefix.tar.xz" --

View more solutions

Comments

[Jacob]

Even with submodules initialized and updated. They don't appear in an archive produced with git-archive. Hence git-archive-all. Would there be any reason to keep both a bare and non-bare repository?

[VonC]

@Jacob: the submodules won't appear if their repo referenced in the .gitsubmodules file isn't accessible from the server. On the other point, git archive-all will work better on a non-bare repo (which will contain the extra .git subdirectories) with all the right submodules in place than in a bare repo (without any .git to grep and parse).

[VonC]

@Jacob: that is why I recommended a non-bare repo with submodules checked out in it: git archive-all will detect and archive everything. I have edited my answer to add that point.

[Brandon]

Actually, looks like it has a few small issues so it might not be ready for prime time yet.

[Jarl]

If you update a full non-bare repo (with modules updated using git submodule update --init), what does git-archive-full.sh provide that tar does not already do for you? Like tar --exclude-vcs --exclude=.gitmodules -c .

[Jonatan Littke]

git-submodule cannot be used without a working tree. This question is regarding a bare repo.

[hilcharge]

this worked for me after changing it to use the -o output_file instead of > output_file format for git archive

[Robie Basak]

It's worth noting that this is a different git-archive-all from the one mentioned in @Brandon's answer.

[Robie Basak]

It's worth noting that this is a different git-archive-all from the one mentioned in @ismailsunni's answer.

[Qix - MONICA WAS MISTREATED]

Downside to this is that you can't specify a ref for the root repository :P Always runs for whatever is currently checked out.

[Michal Sojka]

Another problem is that if the repo (including submodules) have more files than fits on tar's command line, the archive contains only a subset of files. Solution is to use --files-from rather than xargs: tar -c -C.. -f "$prefix.tar.xz" --verbatim-files-from --files-from=-

[noelbk]

git ls-files --recurse-submodules | cpio -o --format=ustar > HEAD.tar

[Akom]

Thanks! Version to go directly to a directory of files (bypassing tar.gz): git ls-files --recurse-submodules | tar c -T- | tar x -C TARGET_DIR (Effectively converts a git non-bare repo to a git-free directory)

[Andrea Reina]

It's worth noting that this will give the files as they exist in the (possibly dirty) worktree, rather than what's committed to a particular ref.

[Grammargeek]

Note: This might behave unexpectedly with symlinks. Ex: foo/bar -> ../baz becomes prog/foo/bar -> prog/../baz