How to append write to google cloud storage file from app engine?

Question

This deprecated API supports appending content to google cloud storage file:

FileWriteChannel FileService.openWriteChannel(AppEngineFile file,
                                boolean lock)

But,

GcsOutputChannel GcsService.createOrReplace(GcsFilename filename,
                               GcsFileOptions options)

doesn't seem so.

What is the solution here?

Solution 1

Sorry to tell you but it cannot be done without replacing the file each time.

As stated in the Cloud Services docs Under Object immutability

Objects are immutable, which means that an uploaded object cannot change throughout its storage lifetime. An object's storage lifetime is the time between successful object creation (upload) and successful object deletion. In practice, this means that you cannot make incremental changes to objects, such as append operations or truncate operations. However, it is possible to overwrite objects that are stored in Google Cloud Storage because an overwrite operation is in effect a delete object operation followed immediately by an upload object operation. So a single overwrite operation simply marks the end of one immutable object's lifetime and the beginning of a new immutable object's lifetime.

Solution 2

While Google Cloud Storage objects are immutable, there is something that could help you if you need to do limited appends: object compose

This operation basically concatenates the contents of a number of objects in the same bucket under a new name, like cat file1 file2 > newfile but without re-writing data.

So you could create a new object, upload the contents to append to it, close and subsequently compose this new piece at the end of your main file.

There is a caveat: a maximum of 1024 'pieces' per object are allowed, and composing objects that are themselves composed counts as composing the original pieces (so you can't cheat the limit by iterated composing).

Comments

[manesioz]

There is no longer any limit to the amount of objects each object can be composed of

[Nathan F.]

There is a limit of 32 objects per call to compose, however a good solution is to simply upload(b), compose(a, b) > a, delete(b)

[stkvtflw]

There is limit though on how many objects you can compose in a single operation: 32. cloud.google.com/storage/docs/composite-objects