Git-based CI/CD Pipeline for Monorepo

Question

Are there established CI/CD strategies & best practices for working with a distributed team on a project that is utilizing monorepos?

I can provide two specific use-cases:

1. Working on content distribution, e.g. a daily/weekly HTML email newsletter, and

2. A multi-project directory for building LaTeX documents

Both of these scenarios will utilize the same CI/CD pipeline (build/send an email, compile and upload a LaTeX document) but with a growing & variable amount of content that makes independent repos per document undesirable.

Most of the projects using monorepos currently seem to be javascript projects with a lot of cross-dependencies on code. This is a step a way from that, but I'd still like to embrace the audit trail of version control with an automated development process for producing content.

Do these projects compile every piece of code on change? Is there a way to embrace a git diff or use some sort of 'flag' file in the repo root to use CI/CD tools with a monorepo?

Solution 1

The fact that all the code is in a single repository doesn't mean that all the code is changing every time a commit is pushed. I would first make the "pathways" in the code explicit. E.g. perhaps you have a few subdirectories:

- App1/
  - Docs/
  - Code/
  - Tests/
  - Makefile
- App2/
  - Docs/
  - Code/
  - Tests/
  - Makefile

Assuming that App1 is independent from App2, you could trigger it's build by checking if anything in that directory changed (see https://stackoverflow.com/questions/424071/how-to-list-all-the-files-in-a-commit).

Most continuous integration tools have a git plugin or native feature that allows you to filter on a path. If you were using Jenkins, you could set up a few jobs, each with the included region to be be the pathway that should be triggered -- so, e.g. changes to App1/* would trigger jobs/app1/ which executes make -f App1/Makefile or similar.

The trick then to maintaining this project is really in the Makefiles, or other build tool of your choice. You would write the dependencies in there.

You could also have a single global Makefile and use the git diff-tree trick in the question mentioned above:

if grep -q App1 `git diff-tree --no-commit-id --name-only -r <commit>` ; then 
  # trigger pipeline App1
  cd App1
  make build
  make test
  make publish
fi

Solution 2

I created some showcase for monorepo with Gradle as build tool and CircleCI or Bitbucket pipelines as CI tool.

It's based on the same ideas as in accepted answer and I used similar setup in two projects till now.

See: https://github.com/zladovan/monorepo

Gradle specific stuff should be replaceable quite easily by something else. Just change tools/ci/core/list-dependencies.sh.

Comments

[Kaymaz]

What if you change what's in the root directly like .gitlab-ci file? No build would be triggered because no file has been changed in App*

[Bruce Becker]

of course the logic of the build trigger would have to accomodate your use cases.

[Dean Hiller]

This is really missing the fact of libraries and if you build library A, you must also build services C and D that depend on A. (I am trying to figure this out myself having worked at twitter which had it but they used pants build system to do it and pants kinda sucked...it was pretty slow)

[Dean Hiller]

Totally awesome!!!! Added one issue to help so master branch never breaks to the project.