How can I have linked dependencies in a git repo?

Question

In my scripts, I often use libraries (mine or others') that have their own repos. I don't want to duplicate those in my repo and get stuck with updating them every time a new version comes out. However, when somebody clones the repo, it should still work locally and not have broken links.

Any ideas about what I could do?

Solution

You can do this with submodules in git. In your repository, do:

git submodule add path_to_repo path_where_you_want_it

So, if the library's repository had a URL of git://github.com/example/some_lib.git and you wanted it at lib/some_lib in your project, you'd enter:

git submodule add git://github.com/example/some_lib.git lib/some_lib

Note that this needs to be done from the top-level directory in your repository. So don't cd into the directory where you're putting it first.

After you add a submodule, or whenever someone does a fresh checkout of your repository, you'll need to do:

git submodule init
git submodule update

And then all submodules you've added will be checked out at the same revision you have.

When you want to update to a newer version of one of the libraries, cd into the submodule and pull:

cd lib/some_lib
git pull

Then, when you do a git status you should see lib/somelib listed in the modified section. Add that file, commit, and you're up to date. When a collaborator pulls that commit into their repository, they'll see lib/somelib as modified until they run git submodule update again.

Comments

[Lea Verou]

Thanks for your answer, upvoted! (and will probably accept it tomorrow) Is there a way to add only one file from the other repo as a dependency? Or does it have to be a whole folder?

[Emily]

If you want to track it as a submodule (and thus, be able to easily pull in updates) you'll have to pull in the whole repository. Unlike a lot of other VCSs, git really only wants to deal with the top-level repository.

[minghua]

read this to know what a submodule is good for and the ways to workaround potential issues. and a discussion about it.

[andrevenancio]

lets say you import as a submodule another repo you own. Can you make changes to the original repo through your local changes in the submodule? Lets say you found a bug and need to update the original?

[Gabriel Hautclocq]

If you want to add a GitHub submodule into another GitHub repository, you should use the https:// URL, not the git:// URL, or GitHub will not recognize the submodule repository correctly.

[Chris]

I know this is an old answer, but git submodules can be updated using git submodule update --remote I recommend using more flags though for more complex projects. I use git submodule update --remote --recursive --init

[django-unchained]

@Emily This looks related to what I want to achieve. How about maintaining dependencies on the scripts within same repo? Let's say I've two scripts Script 1 and Script 2. How should I track if Script 2 is dependent on Script 1?

[topher217]

Is there a reason this has to be done in the top-level dir like you mention? I'm looking to add a submodule, but would like it in a child dir in order to better fit the project architecture.

[jena]

@django-unchained I think the content of your scripts is where the dependencies happen - your files nor git don't need to know anything about how your scripts depend on each other. When you call the scripts, they call their own dependencies (I only script in R, but that's how it's done).

[jena]

@topher217 As I understand the answer, you can pick whatever subdirectory you want to put your submodule into, you just call git command from the top level dir (I assume because that's where git configs are).

[jena]

Just to clarify something for myself - is this the process that will allow me to use git clone --recursive <url> to clone the repo together with it's dependencies? I mean as seen for example in this repo: github.com/vcflib/vcflib#build-from-source (notice the dependency dirs at the top of the page).