persistent storage solutions for aws fargate

Question

I'm running an apache webapp on a container in ecs fargate, but I need to be able to persist certain files so that any changes to them are available to all users through all versions of the webapp. Unfortunately, fargate's support for EFS is in-development, so I need a persistent storage solution at least until that's useable (or until a release date for that feature is announced so I can just say I'll deal with this then).

I tried to switch over from fargate to ec2, as EFS is supposedly compatible with ec2, but I couldn't figure out a way to get past the "unable to place a task because no container instance met all of its requirements. Reason: No Container Instances were found in your cluster." error. I tried all solutions in AWS ECS Error when running task: No Container Instances were found in your cluster , as well as everything I could find in a few google searches, but to no avail.

The way the files are accessed is essentially:

1. A php file calls the database
2. database points to the location on the webapp that the application data is stored
3. data is read / written to the file in the webapp

does anyone know of a persistent storage solution for ecs fargate? And how would I go about setting it up?

Solution 1

EDIT : Fargate now supports EFS as persistent storage. So you don't need to use something like S3FS or any other hacky solution :

https://aws.amazon.com/about-aws/whats-new/2020/04/amazon-ecs-aws-fargate-support-amazon-efs-filesystems-generally-available/

---

Original answer (not relevent any more):

Fargate does not support persistent storage at the moment, because you don't have access to the underlying infrastructure, and as such, you cannot mount an EFS volume. And it might not happen any time soon.

But one solution would be to use S3FS, which mounts an S3 bucket as a file system, directly on the container, so you don't need to have access to the infrastructure.

You can find it here : https://github.com/s3fs-fuse/s3fs-fuse

The project is popular and well maintained. It should work, but keep in mind that it's a bit of a hacky solution. It's not a fully POSIX compliant solution, but it's OK for storing simple files. Performance will also be degraded compared to a local file system. If you're serving the stored files, it's better to add a CDN on top, with optimized caching.

Solution 2

Different horse for different courses, Fargate is designed to zero management of infrastructure which means just specify the Docker image and then leave everything on AWS.

Bind mount host volumes are supported when using either the EC2 or Fargate launch types. Fargate tasks only support nonpersistent storage volumes, so the host and sourcePath fields are not supported.

The work arround is volume mounts.

Fargate Task Storage

When provisioned, each Fargate task receives the following storage. Task storage is ephemeral. After a Fargate task stops, the storage is deleted.

10 GB of Docker layer storage

An additional 4 GB for volume mounts. This can be mounted and shared among containers using the volumes, mountPoints and volumesFrom parameters in the task definition.

To provide nonpersistent empty storage for containers in a Fargate task

In this example, you may have two database containers that need to access the same scratch file storage location during a task.

In the task definition volumes section, define a volume with the name database_scratch.

  "volumes": [
    {
      "name": "database_scratch",
      "host": {}
    }
  ]

In the containerDefinitions section, create the database container definitions so they mount the nonpersistent storage.

  "containerDefinitions": [
    {
      "name": "database1",
      "image": "my-repo/database",
      "cpu": 100,
      "memory": 100,
      "essential": true,
      "mountPoints": [
        {
          "sourceVolume": "database_scratch",
          "containerPath": "/var/scratch"
        }
      ]
    },
    {
      "name": "database2",
      "image": "my-repo/database",
      "cpu": 100,
      "memory": 100,
      "essential": true,
      "mountPoints": [
        {
          "sourceVolume": "database_scratch",
          "containerPath": "/var/scratch"
        }
      ]
    }
  ]

If you are looking for a way to bind host volume then you are expecting wrong from fargate as there is no host in case of fargate.

You need the Ec2 type ECS Task.

To provide persistent storage for containers using a bind mount

When using bind mounts, if a sourcePath value is specified the data persists even after all containers that referenced it have stopped. Any files that exist at the sourcePath are presented to the containers at the containerPath value, and any files that are written to the containerPath value are written to the sourcePath value on the container instance.

In the task definition volumes section, define a bind mount with name and sourcePath values.

  "volumes": [
    {
      "name": "webdata",
      "host": {
        "sourcePath": "/ecs/webdata"
      }
    }
  ]

In the containerDefinitions section, define a container with mountPoints values that reference the name of the defined bind mount and the containerPath value to mount the bind mount at on the container.

  "containerDefinitions": [
    {
      "name": "web",
      "image": "nginx",
      "cpu": 99,
      "memory": 100,
      "portMappings": [
        {
          "containerPort": 80,
          "hostPort": 80
        }
      ],
      "essential": true,
      "mountPoints": [
        {
          "sourceVolume": "webdata",
          "containerPath": "/usr/share/nginx/html"
        }
      ]
    }
  ]

bind-mounts

again

The host and sourcePath parameters are not supported for Fargate tasks.

Solution 3

Fargate nows supports EFS. I've personally set up an app that uses it and I can say that it works, in general, but the performance is terrible. So really, it depends on what kind of I/O performance you need.

Specifically, I found that EFS has a hard time listing files in a directory with thousands of files, as well as may be slow when reading the files' contents.

There is a lot of content regarding EFS performance out there, so make sure you look into it. However, you may now finally have your solution for file persistence in Fargate!