CodeBuild - multiple input artifacts in AWS Pipeline

Question

I am building an AWS CodePipeline which depends on two source repositories. If the 'CodeBuild' step is specified with multiple Input artifacts, then you get the error: .. declares 2 input artifacts which is more than the maximum count. Documentation has a strict requirement for CodeBuild to have only a single input artifact.

Lambdas, on another hand, do not have this limitation. I've been looking for a way how a Lambda could "merge" two artifacts into a single one which I can pass to the CodeBuild then. Unfortunately, information on this subject is quite limited.

Does anyone have a working CodePipeline Lambda which would merge two Artifacts into one?

Note: Documentation about How to access artifacts.

Solution 1

As of Sep 4 2018 Amazon added support for multiple input sources:

https://aws.amazon.com/about-aws/whats-new/2018/08/aws-codebuild-adds-ability-to-create-build-projects-with-multiple-input-sources-and-output-artifacts/

From documentation:

{
"name": "sample-project",
"source": {
  "type": "S3",
  "location": "bucket/sample.zip"
},
"secondarySources": [
  {
    "type": "CODECOMMIT",
    "location": "https://git-codecommit.us-west-2.amazonaws.com/v1/repos/repo"
    "sourceIdentifier": "source1"
  },
  {
    "type": "GITHUB",
    "location": "https://github.com/awslabs/aws-codebuild-jenkins-plugin"
    "sourceIdentifier": "source2"
  }
],

---

version: 0.2

phases:
  build:
    commands:
      - cd $CODEBUILD_SRC_DIR_source1
      - touch file1
      - cd $CODEBUILD_SRC_DIR_source2
      - touch file2

artifacts:
  secondary-artifacts:
    artifact1:
      base-directory: $CODEBUILD_SRC_DIR_source1
      files:
        - file1
    artifact2:
      base-directory: $CODEBUILD_SRC_DIR_source2
      files:
        - file2

Solution 2

We've had exactly the same problem - our build stage needs to use information from two separate locations.

To solve this in a generic manner, I've created two general purpose Lambdas - one that will merge artifacts, and one that will extract specific files from an artifact. In both cases a new artifact is created that can be passed on to a subsequent CodePipeline stage.

See https://github.com/tooltwist/codepipeline-artifact-munge

If it does not meet your exact requirement, it should be easy enough to tweak to do what you need.

Solution 3

I have actually solved the problem by cloning repository directly in codebuild:

Pass GitHub token:

      - { Name: GITHUB_TOKEN, Value: {Ref: GitHubToken } }
      - { Name: GITHUB_BRANCH, Value: {Ref: GitHubBranch } }

# Execute git clone in install step:

      - git clone --single-branch --depth=1 -b $GITHUB_BRANCH https://gitorgname:[email protected]/gitorgname/reponame.git  src/reponame

I found that this solution is easier to implement, does not rely on Lambdas and even though performs two pulls, works quite reliably.

Comments

[romaninsh]

Awesome work, Philip, thanks for sharing this Lambda.

[Alex R]

Where exactly do you put the - { Name ... } ?

[Tensibai]

I fail to see how this answer the question.

[romaninsh]

I create CodeBuild through cloudformation, but that is an environment variable which you can specify through console also.