How to use JNI bitmap operations for helping to avoid OOM when using large images?

Question

Background

most of the times, getting OOM on android is due to using too many bitmaps and/or creating large bitmaps.

recently i've decided to try out JNI in order to allow avoiding OOM by storing the data itself on the JNI side.

after messing around with JNI for a while, i've created some posts on SO asking for help and sharing my knowledge, and i've now decided to share some more code with you. here are the posts in case anyone is interested in reading the findings or contributing :

• How to cache bitmaps into native memory

• image decoding and manipulation using JNI on android

• JNI - how to use multiple Jni wrapper instances with different fields?

• Rotating a bitmap using JNI & NDK

this time, i've added the ability to store,restore, crop and rotate bitmaps. it should be easy to add more options and I would be happy if other people here would add their own code to more useful functions .

so the code i'm about to show is actually merging of all the things i've created.

Sample code of usage:

Bitmap bitmap=BitmapFactory.decodeResource(getResources(),R.drawable.ic_launcher);
final int width=bitmap.getWidth(),height=bitmap.getHeight();
// store the bitmap in the JNI "world"
final JniBitmapHolder bitmapHolder=new JniBitmapHolder(bitmap);
// no need for the bitmap on the java "world", since the operations are done on the JNI "world"
bitmap.recycle();
// crop a center square from the bitmap, from (0.25,0.25) to (0.75,0.75) of the bitmap.
bitmapHolder.cropBitmap(width/4,height/4,width*3/4,height*3/4);
//rotate the bitmap:
bitmapHolder.rotateBitmapCcw90();
//get the output java bitmap , and free the one on the JNI "world"
bitmap=bitmapHolder.getBitmapAndFree();

The project is available on github

• project page is available on github here .

• feel free to give advises and contribute.

Important notes

same notes as shown here, plus:

• current features that are written here (more updated on the project page) :

  • store

  • restore

  • rotate 90 degrees CCW

  • crop.

• the approach i've taken for this code is both memory efficiency (use only memory that i need, and free it when not needed), and CPU efficiency (i tried to use pointers and CPU memory cache optimizations whenever possible).

• for best performance, i've done really few validations, especially on the JNI part. it might be best to manage the validations on the java "world".

• there are still many missing features that i think should be added, and i hope that i will have the time to add them . if anyone wishes to contribute, i will be glad to add they code too. here are the functions that i think could be useful:

  • get current bitmap info

  • scale bitmaps, including choice of which algorithm to use (nearest neighbour and bilinear interpolation should be enough).

  • use different bitmap formats

  • do the decoding within JNI, to avoid creation of the java bitmap (and not use the heap on the java world) from the beginning, only at the end, when you finished with all of the operations.

  • face detection

  • rotation in any angle, or at least the obvious ones . currently i only added rotation of 90 degrees counter clock wise .

Solution

NOTE: this is a bit old code. for the most updated one, check out the project page on github.

jni/Android.mk

LOCAL_PATH := $(call my-dir)

#bitmap operations module
include $(CLEAR_VARS)

LOCAL_MODULE    := JniBitmapOperations
LOCAL_SRC_FILES := JniBitmapOperations.cpp
LOCAL_LDLIBS := -llog
LOCAL_LDFLAGS += -ljnigraphics

include $(BUILD_SHARED_LIBRARY)
APP_OPTIM := debug
LOCAL_CFLAGS := -g

#if you need to add more module, do the same as the one we started with (the one with the CLEAR_VARS)

jni/JniBitmapOperations.cpp

#include <jni.h>
#include <jni.h>
#include <android/log.h>
#include <stdio.h>
#include <android/bitmap.h>
#include <cstring>
#include <unistd.h>

#define  LOG_TAG    "DEBUG"
#define  LOGD(...)  __android_log_print(ANDROID_LOG_DEBUG,LOG_TAG,__VA_ARGS__)
#define  LOGE(...)  __android_log_print(ANDROID_LOG_ERROR,LOG_TAG,__VA_ARGS__)

extern "C"
  {
  JNIEXPORT jobject JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniStoreBitmapData(JNIEnv * env, jobject obj, jobject bitmap);
  JNIEXPORT jobject JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniGetBitmapFromStoredBitmapData(JNIEnv * env, jobject obj, jobject handle);
  JNIEXPORT void JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniFreeBitmapData(JNIEnv * env, jobject obj, jobject handle);
  JNIEXPORT void JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniRotateBitmapCcw90(JNIEnv * env, jobject obj, jobject handle);
  JNIEXPORT void JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniCropBitmap(JNIEnv * env, jobject obj, jobject handle, uint32_t left, uint32_t top, uint32_t right, uint32_t bottom);
  }

class JniBitmap
  {
  public:
    uint32_t* _storedBitmapPixels;
    AndroidBitmapInfo _bitmapInfo;
    JniBitmap()
      {
      _storedBitmapPixels = NULL;
      }
  };

/**crops the bitmap within to be smaller. note that no validations are done*/ //
JNIEXPORT void JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniCropBitmap(JNIEnv * env, jobject obj, jobject handle, uint32_t left, uint32_t top, uint32_t right, uint32_t bottom)
  {
  JniBitmap* jniBitmap = (JniBitmap*) env->GetDirectBufferAddress(handle);
  if (jniBitmap->_storedBitmapPixels == NULL)
    return;
  uint32_t* previousData = jniBitmap->_storedBitmapPixels;
  uint32_t oldWidth = jniBitmap->_bitmapInfo.width;
  uint32_t newWidth = right - left, newHeight = bottom - top;
  uint32_t* newBitmapPixels = new uint32_t[newWidth * newHeight];
  uint32_t* whereToGet = previousData + left + top * oldWidth;
  uint32_t* whereToPut = newBitmapPixels;
  for (int y = top; y < bottom; ++y)
    {
    memcpy(whereToPut, whereToGet, sizeof(uint32_t) * newWidth);
    whereToGet += oldWidth;
    whereToPut += newWidth;
    }
  //done copying , so replace old data with new one
  delete[] previousData;
  jniBitmap->_storedBitmapPixels = newBitmapPixels;
  jniBitmap->_bitmapInfo.width = newWidth;
  jniBitmap->_bitmapInfo.height = newHeight;
  }

/**rotates the inner bitmap data by 90 degress counter clock wise*/ //
JNIEXPORT void JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniRotateBitmapCcw90(JNIEnv * env, jobject obj, jobject handle)
  {
  JniBitmap* jniBitmap = (JniBitmap*) env->GetDirectBufferAddress(handle);
  if (jniBitmap->_storedBitmapPixels == NULL)
    return;
  uint32_t* previousData = jniBitmap->_storedBitmapPixels;
  AndroidBitmapInfo bitmapInfo = jniBitmap->_bitmapInfo;
  uint32_t* newBitmapPixels = new uint32_t[bitmapInfo.height * bitmapInfo.width];
  int whereToPut = 0;
  // A.D D.C
  // ...>...
  // B.C A.B
  for (int x = bitmapInfo.width - 1; x >= 0; --x)
    for (int y = 0; y < bitmapInfo.height; ++y)
      {
      uint32_t pixel = previousData[bitmapInfo.width * y + x];
      newBitmapPixels[whereToPut++] = pixel;
      }
  delete[] previousData;
  jniBitmap->_storedBitmapPixels = newBitmapPixels;
  uint32_t temp = bitmapInfo.width;
  bitmapInfo.width = bitmapInfo.height;
  bitmapInfo.height = temp;
  }

/**free bitmap*/  //
JNIEXPORT void JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniFreeBitmapData(JNIEnv * env, jobject obj, jobject handle)
  {
  JniBitmap* jniBitmap = (JniBitmap*) env->GetDirectBufferAddress(handle);
  if (jniBitmap->_storedBitmapPixels == NULL)
    return;
  delete[] jniBitmap->_storedBitmapPixels;
  jniBitmap->_storedBitmapPixels = NULL;
  delete jniBitmap;
  }

/**restore java bitmap (from JNI data)*/  //
JNIEXPORT jobject JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniGetBitmapFromStoredBitmapData(JNIEnv * env, jobject obj, jobject handle)
  {
  JniBitmap* jniBitmap = (JniBitmap*) env->GetDirectBufferAddress(handle);
  if (jniBitmap->_storedBitmapPixels == NULL)
    {
    LOGD("no bitmap data was stored. returning null...");
    return NULL;
    }
  //
  //creating a new bitmap to put the pixels into it - using Bitmap Bitmap.createBitmap (int width, int height, Bitmap.Config config) :
  //
  //LOGD("creating new bitmap...");
  jclass bitmapCls = env->FindClass("android/graphics/Bitmap");
  jmethodID createBitmapFunction = env->GetStaticMethodID(bitmapCls, "createBitmap", "(IILandroid/graphics/Bitmap$Config;)Landroid/graphics/Bitmap;");
  jstring configName = env->NewStringUTF("ARGB_8888");
  jclass bitmapConfigClass = env->FindClass("android/graphics/Bitmap$Config");
  jmethodID valueOfBitmapConfigFunction = env->GetStaticMethodID(bitmapConfigClass, "valueOf", "(Ljava/lang/String;)Landroid/graphics/Bitmap$Config;");
  jobject bitmapConfig = env->CallStaticObjectMethod(bitmapConfigClass, valueOfBitmapConfigFunction, configName);
  jobject newBitmap = env->CallStaticObjectMethod(bitmapCls, createBitmapFunction, jniBitmap->_bitmapInfo.width, jniBitmap->_bitmapInfo.height, bitmapConfig);
  //
  // putting the pixels into the new bitmap:
  //
  int ret;
  void* bitmapPixels;
  if ((ret = AndroidBitmap_lockPixels(env, newBitmap, &bitmapPixels)) < 0)
    {
    LOGE("AndroidBitmap_lockPixels() failed ! error=%d", ret);
    return NULL;
    }
  uint32_t* newBitmapPixels = (uint32_t*) bitmapPixels;
  int pixelsCount = jniBitmap->_bitmapInfo.height * jniBitmap->_bitmapInfo.width;
  memcpy(newBitmapPixels, jniBitmap->_storedBitmapPixels, sizeof(uint32_t) * pixelsCount);
  AndroidBitmap_unlockPixels(env, newBitmap);
  //LOGD("returning the new bitmap");
  return newBitmap;
  }

/**store java bitmap as JNI data*/  //
JNIEXPORT jobject JNICALL Java_com_jni_bitmap_1operations_JniBitmapHolder_jniStoreBitmapData(JNIEnv * env, jobject obj, jobject bitmap)
  {
  AndroidBitmapInfo bitmapInfo;
  uint32_t* storedBitmapPixels = NULL;
  //LOGD("reading bitmap info...");
  int ret;
  if ((ret = AndroidBitmap_getInfo(env, bitmap, &bitmapInfo)) < 0)
    {
    LOGE("AndroidBitmap_getInfo() failed ! error=%d", ret);
    return NULL;
    }
  LOGD("width:%d height:%d stride:%d", bitmapInfo.width, bitmapInfo.height, bitmapInfo.stride);
  if (bitmapInfo.format != ANDROID_BITMAP_FORMAT_RGBA_8888)
    {
    LOGE("Bitmap format is not RGBA_8888!");
    return NULL;
    }
  //
  //read pixels of bitmap into native memory :
  //
  //LOGD("reading bitmap pixels...");
  void* bitmapPixels;
  if ((ret = AndroidBitmap_lockPixels(env, bitmap, &bitmapPixels)) < 0)
    {
    LOGE("AndroidBitmap_lockPixels() failed ! error=%d", ret);
    return NULL;
    }
  uint32_t* src = (uint32_t*) bitmapPixels;
  storedBitmapPixels = new uint32_t[bitmapInfo.height * bitmapInfo.width];
  int pixelsCount = bitmapInfo.height * bitmapInfo.width;
  memcpy(storedBitmapPixels, src, sizeof(uint32_t) * pixelsCount);
  AndroidBitmap_unlockPixels(env, bitmap);
  JniBitmap *jniBitmap = new JniBitmap();
  jniBitmap->_bitmapInfo = bitmapInfo;
  jniBitmap->_storedBitmapPixels = storedBitmapPixels;
  return env->NewDirectByteBuffer(jniBitmap, 0);
  }

src/com/jni/bitmap_operations/JniBitmapHolder.java

package com.jni.bitmap_operations;
import java.nio.ByteBuffer;
import android.graphics.Bitmap;
import android.util.Log;

public class JniBitmapHolder
  {
  ByteBuffer _handler =null;
  static
    {
    System.loadLibrary("JniBitmapOperations");
    }

  private native ByteBuffer jniStoreBitmapData(Bitmap bitmap);

  private native Bitmap jniGetBitmapFromStoredBitmapData(ByteBuffer handler);

  private native void jniFreeBitmapData(ByteBuffer handler);

  private native void jniRotateBitmapCcw90(ByteBuffer handler);

  private native void jniCropBitmap(ByteBuffer handler,final int left,final int top,final int right,final int bottom);

  public JniBitmapHolder()
    {}

  public JniBitmapHolder(final Bitmap bitmap)
    {
    storeBitmap(bitmap);
    }

  public void storeBitmap(final Bitmap bitmap)
    {
    if(_handler!=null)
      freeBitmap();
    _handler=jniStoreBitmapData(bitmap);
    }

  public void rotateBitmapCcw90()
    {
    if(_handler==null)
      return;
    jniRotateBitmapCcw90(_handler);
    }

  public void cropBitmap(final int left,final int top,final int right,final int bottom)
    {
    if(_handler==null)
      return;
    jniCropBitmap(_handler,left,top,right,bottom);
    }

  public Bitmap getBitmap()
    {
    if(_handler==null)
      return null;
    return jniGetBitmapFromStoredBitmapData(_handler);
    }

  public Bitmap getBitmapAndFree()
    {
    final Bitmap bitmap=getBitmap();
    freeBitmap();
    return bitmap;
    }

  public void freeBitmap()
    {
    if(_handler==null)
      return;
    jniFreeBitmapData(_handler);
    _handler=null;
    }

  @Override
  protected void finalize() throws Throwable
    {
    super.finalize();
    if(_handler==null)
      return;
    Log.w("DEBUG","JNI bitmap wasn't freed nicely.please rememeber to free the bitmap as soon as you can");
    freeBitmap();
    }
  }

Comments

[SysHex]

I just wish you had released this as a lib somewhere.

[android developer]

@SysHex i think i will. i think that if it was somewhere else, people would be more willing to contribute. do you have any suggestions?

[SysHex]

yeah, plenty. I've implemented the cw rotation bit cause I needed it. I've integrated this with the "simple-crop-image-lib" but don't quite like the way it is working (the interface doesn't work very well) so I might be redoing that bit at some point in the next months.

[SysHex]

started a lib and used your jni code here . Let me know once you start you lib somewhere so I can link my stuff with your original work and so I can commit some improvements. I hope it is OK by you to have used your code.

[android developer]

@SysHex OK i've put the entire project on github: github.com/AndroidDeveloperLB/AndroidJniBitmapOperations . what can be done about licenses?

[SysHex]

I would go with as permissive as possible. Maybe simplified BSD License?

[android developer]

i'm very newb in licenses and consider the text written in licenses to be very hard to understand, as if i have to be in law school in order to understand them. i wish there could be a nice table of all known licenses and a super short and easy to understand description of each of them. the only one that i know is apache-2, which allows everyone to do whatever they want with it, yet for some reason leave the upper comments area in the code.

[SysHex]

yeah , I understand you! I'm good with both apache-2 and BSD-2-clause. Check these links wikipedia and Link 2 . As long as there is no liability to original authors then I'm good, what other people do with my code I don't care, don't need no know, and I really don't care if they remove or change license. This is in general my view on this ;)

[android developer]

then what is the difference between BSD-2-clause , apache-2 and MIT ? are there other licenses that are so similar to those?

[SysHex]

well, I'm not real fluent in licenses. I like the BSD-2-clause because it is real simple, like 2-clauses and a liability disclaimer. Basically, redistribution of source with disclaimer and copyright, redistribution of binary with disclaimer, the 2 clauses and copyright. Check here 2clause Simplified BSD. I'm not particularly biased about BSD, as long has there is a liability disclaimer for me, then I couldn't care less how my shared source is used, by whom, etc

[Dan Levin]

hey can you please help me out? i tried importing the library and the sample, which seems fine in eclipse, but when i try to run in i get an error message in the console saying "Could not find JniBitmapOperationsLibrary.apk!" the logcat gives me a NoClassDefFoundError exception when initializing JniBitmapHolder

[android developer]

@DanLevin does the app itself work? if so, it's ok, as it seems like a very common and known bug when running android apps that use android library projects.

[Dan Levin]

i was just trying to run the sample that came with the zip, it crashed on fatal exception with NoClassDefFoundError, i guess it has something to do with the project setup, i'm still too noob to know whats happening there :\ i will try to create a new project and just copy the classes. any ideas what else i should do?

[android developer]

@DanLevin since there are 2 projects, one should use the other (use the library project). this isn't done using the "standard java-way". instead you should do it in the android-category-settings of the project that should use the library project. other than that, there is nothing special that you should do. but my projects are supposed to be working out of the box. maybe you didn't update ADT or the SDK ?

[Dan Levin]

i tired :/ still doesn't work, i get a message saying the library couldn't be found from System.loadLibrary("JniBitmapOperations"); it returns null, so frustrating :/

[android developer]

I've just tested it now. I just do "Import..."=>"Android-Existing Android Code Into Workspace" , and then I chose the folder. after a short time of building the projects, all worked fine. I just installed&run... Maybe you should update your Eclipse,Java,Sdk,ADT ? make sure you use Java 1.6 .

[Dan Levin]

hey! i just tried to do it all over again and it worked :) i dunno what the problem was... but thanks for your help! :)

[android developer]

@DanLevin i've recently read about an alternative, similar library for handling images. Maybe it could be useful for you: github.com/nirvanfallacy/AndroidGraphicsUtility

[Dan Levin]

unfortunately i have another problem now :( when rotating the bitmap it gets screwed up :/ looks interlaced, any idea why this could happen? if i don't use rotation the image looks fine

[android developer]

sure you've use the project link? did the sample work well ?

[Dan Levin]

im still getting images corrupted when rotating, so i couldn't figure out what i was doing wrong, so i posted a brand question here stackoverflow.com/questions/21913972/… if you can please check it out, maybe you will have an idea - Cheers

[android developer]

you should have put this in the issues of github, but ok, i will take a look. can you please make a minimal project that demonstrate the problem instead of what you did?

[AndroidLearner]

@androiddeveloper i have implemented successfully your lib,i am able to rotate the image and then grayscale to image,but after grayscale if i am trying t save that image in sd card,it shows black image.can you please help me? stackoverflow.com/questions/22953988/…

[android developer]

@AndroidLearner You have already written 2 threads and didn't accept the answer of any of them even though you got answers.. anyway, i've written a comment to your 3rd question

[AndroidLearner]

@androiddeveloper i have already accepted the answer of my previous threads.

[android developer]

@AndroidLearner what about this one: stackoverflow.com/questions/22908781/…

[Vikalp Patel]

@androiddeveloper : Can we rebuild image from pixel by stretching a image with X and Y(Touch Points) Just like Virtual Make Over Application where one can stretch hair wig and also squeeze it?

[android developer]

@VikalpPatel I'm not sure what you are talking about. Can you show a video or something?

[Vikalp Patel]

@androiddeveloper : I'm try to develop application like play.google.com/store/apps/… which adjust hair with TouchEvents. Tried to achieve it using drawBitmapMesh(), but which is not as smooth as I want. SO : stackoverflow.com/questions/26404564/…

[android developer]

Are you talking only about the movement and resizing, or also the stretching of the image?

[Vikalp Patel]

@androiddeveloper : What I want is stretching image from one point which will give warp effect to bitmap on stretching. Morever Image can also be moved and resize. How one can achieve something like that?

[android developer]

@VikalpPatel About the special warp effect, you should learn how to do it and probably use either C/C++ or Renderscript. Or use what you've suggested and see if they have a similar effect. about just moving and resizing, I've put some links that I think might be useful, in the thread you've created: stackoverflow.com/questions/26404564/…

[Vikalp Patel]

Let us continue this discussion in chat.

[android developer]

@VikalpPatel Just answer in the post you've made.

[rule]

@androiddeveloper This is pretty nice project. Have you already implemented bitmap decoding directly in JNI memory or you have that in plans? If not can you give me some directions how to do it, so I will give a try and push to your github.

[android developer]

@rule I tried to read about JPG and PNG (in Wikipedia), how they work etc... but it got so complex that I ditched it. Sorry. However, there are other interesting libraries that you could look at: github.com/facebook/fresco github.com/suckgamony/RapidDecoder android-arsenal.com/tag/63

[rule]

@androiddeveloper Thanks for pointing on these libraries, but did you found any way to load bitmap into JNI memory using your library. I like it because it is simple as it should be.

[android developer]

@rule You need to parse a file's content, like on C/C++, and put the pixels' data into byte array. In order to know how to decode files, you will need to research about it.

[rule]

@androiddeveloper ok, I'll give a try

[android developer]

@rule Thanks. And please, if you succeed, do a pull request. It can really be nice to have this feature on my tiny library...