Work around circular dependency in AWS CloudFormation

Question

The following AWS CloudFormation gives a circular dependency error. My understanding is that the dependencies flow like this: rawUploads -> generatePreview -> previewPipeline -> rawUploads. Although it doesn't seem like rawUploads depends on generatePreview, I guess CF needs to know what lambda to trigger when creating the bucket, even though the trigger is defined in the lambda part of the CloudFormation template.

I've found some resources online that talk about a similar issue, but it doesn't seem to apply here. https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-circular-dependency-cloudformation/

What are my options for breaking this circular dependency chain? Scriptable solutions are viable, but multiple deployments with manual changes are not for my use case.

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Resources:
  rawUploads:
    Type: 'AWS::S3::Bucket'
  previewAudioFiles:
    Type: 'AWS::S3::Bucket'

  generatePreview:
    Type: AWS::Serverless::Function
    Properties:
      Handler: generatePreview.handler
      Runtime: nodejs6.10
      CodeUri: .
      Environment:
        Variables:
          PipelineId: !Ref previewPipeline
      Events:
        BucketrawUploads:
          Type: S3
          Properties:
            Bucket: !Ref rawUploads
            Events: 's3:ObjectCreated:*'

  previewPipeline:
    Type: Custom::ElasticTranscoderPipeline
    Version: '1.0'
    Properties:
      ServiceToken:
        Fn::Join:
        - ":"
        - - arn:aws:lambda
          - Ref: AWS::Region
          - Ref: AWS::AccountId
          - function
          - aws-cloudformation-elastic-transcoder-pipeline-1-0-0
      Name: transcoderPipeline
      InputBucket:
        Ref: rawUploads
      OutputBucket:
        Ref: previewAudioFiles

Solution 1

This post helped me out in the end: https://aws.amazon.com/premiumsupport/knowledge-center/unable-validate-destination-s3/

I ended up configuring an SNS topic in CloudFormation. The bucket would push events on this topic, and the Lambda function listens to this topic. This way the dependency graph is as follows:

S3 bucket -> SNS topic -> SNS topic policy
Lambda function -> SNS topic
Lambda function -> transcoder pipeline

Something along the lines of this (some policies omitted)

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Resources:
  SNSTopic:
    Type: AWS::SNS::Topic
  SNSTopicPolicy:
    Type: AWS::SNS::TopicPolicy
    Properties:
      PolicyDocument:
        Id: MyTopicPolicy
        Version: '2012-10-17'
        Statement:
        - Sid: Statement-id
          Effect: Allow
          Principal:
            AWS: "*"
          Action: sns:Publish
          Resource:
            Ref: SNSTopic
          Condition:
            ArnLike:
              aws:SourceArn:
                !Join ["-", ['arn:aws:s3:::rawuploads', Ref: 'AWS::StackName']]
      Topics:
      - Ref: SNSTopic

  rawUploads:
    Type: 'AWS::S3::Bucket'
    DependsOn: SNSTopicPolicy
    Properties:
      BucketName: !Join ["-", ['rawuploads', Ref: 'AWS::StackName']]
      NotificationConfiguration:
        TopicConfigurations:
          - Topic:
              Ref: "SNSTopic"
            Event: 's3:ObjectCreated:*'

  previewAudioFiles:
    Type: 'AWS::S3::Bucket'


  generatePreview:
    Type: AWS::Serverless::Function
    Properties:
      FunctionName: !Join ["-", ['generatepreview', Ref: 'AWS::StackName']]
      Handler: generatePreview.handler
      Runtime: nodejs6.10
      CodeUri: .
      Environment:
        Variables:
          PipelineId: !Ref previewPipeline
      Events:
        BucketrawUploads:
          Type: SNS
          Properties:
            Topic: !Ref "SNSTopic"

  previewPipeline:
    Type: Custom::ElasticTranscoderPipeline
    DependsOn: 'rawUploads'
    Version: '1.0'
    Properties:
      ServiceToken:
        Fn::Join:
        - ":"
        - - arn:aws:lambda
          - Ref: AWS::Region
          - Ref: AWS::AccountId
          - function
          - aws-cloudformation-elastic-transcoder-pipeline-1-0-0
      Name: transcoderPipeline
      InputBucket:
        !Join ["-", ['arn:aws:s3:::rawuploads', Ref: 'AWS::StackName']]
      OutputBucket:
        Ref: previewAudioFiles

Solution 2

One way is to give the S3 buckets explicit names so that later, instead of relying on Ref: bucketname, you can simply use the bucket name. That's obviously problematic if you want auto-generated bucket names and in those cases it's prudent to generate the bucket name from some prefix plus the (unique) stack name, for example:

InputBucket: !Join ["-", ['rawuploads', Ref: 'AWS::StackName']]

Another option is to use a single CloudFormation template but in 2 stages - the 1st stage creates the base resources (and whatever refs are not circular) and then you add the remaining refs to the template and do a stack update. Not ideal, obviously, so I would prefer the first approach.

You can also use the first technique in cases when you need a reference to an ARN, for example:

!Join ['/', ['arn:aws:s3:::logsbucket', 'AWSLogs', Ref: 'AWS:AccountId', '*']]

When using this technique, you may want to also consider using DependsOn because you have removed an implicit dependency which can sometimes cause problems.