How do I upload a base64 encoded image (string) directly to a Google Cloud Storage bucket using Node.js?

Question

Currently, I am using the @google-cloud/storage NPM package to upload a file directly to a Google Cloud Storage bucket. This requires some trickery as I only have the image's base64 encoded string. I have to:

• Decode the string
• Save it as a file
• Send the file path to the below script to upload to Google Cloud Storage
• Delete the local file

I'd like to avoid storing the file in the filesystem altogether since I am using Google App Engine and I don't want to overload the filesystem / leave junk files there if the delete operation doesn't work for whatever reason. This is what my upload script looks like right now:

// Convert the base64 string back to an image to upload into the Google Cloud Storage bucket
var base64Img = require('base64-img');
var filePath = base64Img.imgSync(req.body.base64Image, 'user-uploads', 'image-name');

// Instantiate the GCP Storage instance
var gcs = require('@google-cloud/storage')(),
    bucket = gcs.bucket('google-cloud-storage-bucket-name');

// Upload the image to the bucket
bucket.upload(__dirname.slice(0, -15) + filePath, {
    destination: 'profile-images/576dba00c1346abe12fb502a-original.jpg',
    public: true,
    validation: 'md5'
}, function(error, file) {

    if (error) {
        sails.log.error(error);
    }

    return res.ok('Image uploaded');
});

Is there anyway to directly upload the base64 encoded string of the image instead of having to convert it to a file and then upload using the path?

Solution 1

The solution, I believe, is to use the file.createWriteStream functionality that the bucket.upload function wraps in the Google Cloud Node SDK.

I've got very little experience with streams, so try to bear with me if this doesn't work right off.

First of all, we need take the base64 data and drop it into a stream. For that, we're going to include the stream library, create a buffer from the base64 data, and add the buffer to the end of the stream.

var stream = require('stream');
var bufferStream = new stream.PassThrough();
bufferStream.end(Buffer.from(req.body.base64Image, 'base64'));

More on decoding base64 and creating the stream.

We're then going to pipe the stream into a write stream created by the file.createWriteStream function.

var gcs = require('@google-cloud/storage')({
  projectId: 'grape-spaceship-123',
  keyFilename: '/path/to/keyfile.json'
});

//Define bucket.
var myBucket = gcs.bucket('my-bucket');
//Define file & file name.
var file = myBucket.file('my-file.jpg');
//Pipe the 'bufferStream' into a 'file.createWriteStream' method.
bufferStream.pipe(file.createWriteStream({
    metadata: {
      contentType: 'image/jpeg',
      metadata: {
        custom: 'metadata'
      }
    },
    public: true,
    validation: "md5"
  }))
  .on('error', function(err) {})
  .on('finish', function() {
    // The file upload is complete.
  });

Info on file.createWriteStream, File docs, bucket.upload, and the bucket.upload method code in the Node SDK.

So the way the above code works is to define the bucket you want to put the file in, then define the file and the file name. We don't set upload options here. We then pipe the bufferStream variable we just created into the file.createWriteStream method we discussed before. In these options we define the metadata and other options you want to implement. It was very helpful to look directly at the Node code on Github to figure out how they break down the bucket.upload function, and recommend you do so as well. Finally, we attach a couple events for when the upload finishes and when it errors out.

Solution 2

Posting my version of the answer in response to @krlozadan 's request above:

// Convert the base64 string back to an image to upload into the Google Cloud Storage bucket
var mimeTypes = require('mimetypes');

var image = req.body.profile.image,
    mimeType = image.match(/data:([a-zA-Z0-9]+\/[a-zA-Z0-9-.+]+).*,.*/)[1],
    fileName = req.profile.id + '-original.' + mimeTypes.detectExtension(mimeType),
    base64EncodedImageString = image.replace(/^data:image\/\w+;base64,/, ''),
    imageBuffer = new Buffer(base64EncodedImageString, 'base64');

// Instantiate the GCP Storage instance
var gcs = require('@google-cloud/storage')(),
    bucket = gcs.bucket('my-bucket');

// Upload the image to the bucket
var file = bucket.file('profile-images/' + fileName);

file.save(imageBuffer, {
    metadata: { contentType: mimeType },
    public: true,
    validation: 'md5'
}, function(error) {

    if (error) {
        return res.serverError('Unable to upload the image.');
    }

    return res.ok('Uploaded');
});

This worked just fine for me. Ignore some of the additional logic in the first few lines as they are only relevant to the application I am building.

Solution 3

If you want to save a string as a file in Google Cloud Storage, you can do it easily using the file.save method:

const {Storage} = require('@google-cloud/storage');
const storage = new Storage();
const myBucket = storage.bucket('my-bucket');

const file = myBucket.file('my-file.txt');
const contents = 'This is the contents of the file.';

file.save(contents).then(() => console.log('done'));

Solution 4

:) what an issue !! Have tried it and got the issue Image has uploaded on firebase Storage but not download and just loader is moving around and around... After spending time... Got the success to upload the image on firebase storage with downloading... There was an issue in an access token...

check the screenshot

If you check in the file location section on the right side bottom there is an option "create access token" and not showing any "access token" on there if you create manually access token on there then refresh the page image will showing... So now the question is how to create it by code...

just use below code to create the access token

const uuidv4 = require('uuid/v4');
const uuid = uuidv4();
metadata: { firebaseStorageDownloadTokens: uuid }

Full code is given below for uploading an image to storage image on firebase storage

const functions = require('firebase-functions')
var firebase = require('firebase');
var express = require('express');
var bodyParser = require("body-parser");

const uuidv4 = require('uuid/v4');
const uuid = uuidv4();

    const os = require('os')
    const path = require('path')
    const cors = require('cors')({ origin: true })
    const Busboy = require('busboy')
    const fs = require('fs')
    var admin = require("firebase-admin");


    var serviceAccount = {
        "type": "service_account",
        "project_id": "xxxxxx",
        "private_key_id": "xxxxxx",
        "private_key": "-----BEGIN PRIVATE KEY-----\jr5x+4AvctKLonBafg\nElTg3Cj7pAEbUfIO9I44zZ8=\n-----END PRIVATE KEY-----\n",
        "client_email": "[email protected]",
        "client_id": "xxxxxxxx",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
        "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/firebase-adminsdk-5rmdm%40xxxxx.iam.gserviceaccount.com"
      }

    admin.initializeApp({
        credential: admin.credential.cert(serviceAccount),
        storageBucket: "xxxxx-xxxx" // use your storage bucket name
    });


    const app = express();
    app.use(bodyParser.urlencoded({ extended: false }));
    app.use(bodyParser.json());
app.post('/uploadFile', (req, response) => {
    response.set('Access-Control-Allow-Origin', '*');
    const busboy = new Busboy({ headers: req.headers })
    let uploadData = null
    busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
        const filepath = path.join(os.tmpdir(), filename)
        uploadData = { file: filepath, type: mimetype }
        console.log("-------------->>",filepath)
        file.pipe(fs.createWriteStream(filepath))
      })

      busboy.on('finish', () => {
        const bucket = admin.storage().bucket();
        bucket.upload(uploadData.file, {
            uploadType: 'media',
            metadata: {
              metadata: { firebaseStorageDownloadTokens: uuid,
                contentType: uploadData.type,
              },
            },
          })

          .catch(err => {
            res.status(500).json({
              error: err,
            })
          })
      })
      busboy.end(req.rawBody)
   });




exports.widgets = functions.https.onRequest(app);

Solution 5

You have to convert base64 to image buffer then upload as below, you need to provide image_data_from_html variable as the data you extract from HTML event.

const base64Text = image_data_from_html.split(';base64,').pop();
const imageBuffer = Buffer.from(base64Text, 'base64');
const contentType = data.image_data.split(';base64,')[0].split(':')[1];
const fileName = 'myimage.png';
const imageUrl = 'https://storage.googleapis.com/bucket-url/some_path/' + fileName;

await admin.storage().bucket().file('some_path/' + fileName).save(imageBuffer, {
    public: true,
    gzip: true,
    metadata: {
        contentType,
        cacheControl: 'public, max-age=31536000',
    }
});

console.log(imageUrl);

View more solutions

Comments

[JackH]

Thank you for posting this! I actually did something similar except, I used the file.save() API which is a wraparound of createWriteStream.

[forrestmid]

@Nag That definitely works! I read through that API but didn't notice it's operation matched that which you were looking for. Glad you were able to figure it out.

[krlozadan]

@Nag how exactly did you manage to do that? do you have the source code we can take a look at?? I'm struggling a lot with this. I'm trying to upload a base64 encoded image string to Firebase Storage from the Firebase Cloud Functions

[JackH]

@krlozadan Please see my answer below. I'm not sure what the differences are between Cloud Storage and Firebase Storage so I cannot comment on Firebase. Hope this helps.

[Fabian Vilers]

Note that the Buffer constructor is deprecated due to security issues. We should use Buffer.from(req.body.base64Image, 'base64') instead.

[mikey]

I am trying to do something similar, but I have binary data coming in.. upload works but when I open the image on the bucket it is 'empty'. @forrestmid do you have any idea why that might happening? I have played around with encoding, but no luck so far

[Mason]

The question is about uploading "base64 encoded image". This does not work.

[wrymug]

This works for base64 strings if you set contents to this, where data is a base64 encoded file: Buffer.from(data.replace(/^data:image\/(png|gif|jpeg);base64‌​,/, ''), 'base64');

[xaphod]

Not sure where you get isBuffer: true, raw: true from -- I don't see those in the SDK.

[xaphod]

It's worth pointing out that if your base64-encoded string is actually a dataURL (see: developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/…) like one you generated from a canvas or fabric, then you need this solution's first line to get rid of the extra URL parts. Works for me.

[scientiffic]

Thanks for sharing your code! Could you also share how your request was formatted (what was the body of the POST request?)