android mmap fails with out of memory

Question

I've searched everywhere for an answer but I think I'm hitting the limits of what I can find. My question seems somewhat related to this one : Android NDK mmap call broken on 32-bit devices after upgrading to Lollipop but no answer has been provided.

My problem is that I try to memory map 457232384 bytes from a file through a mmap call. On two different devices (Samsung Galaxy Note 3 & OnePlus One, 3GB RAM each) with Android 5.1.1, that call fails with errno 12 "Out of memory". Actually, the call fails when I try to allocate more than 300MB of memory. 313524224 bytes (299MB) works, 314572800 (300MB) won't.

Thing is, the very same call works on a third device which stayed on Android 4.4.2. Even stranger, this call works on the Android ARM emulator with SDK 21 (Android 5.0). Needless to say, the same amount of data (not mmap'ed) can be loaded without any issue.

dmesg reports this to me:

<3>[ 1137.488411] [0:Thread-298: 4267] arch_get_unmapped_area (TASK_SIZE - len < addr) len=457232384 task size=3204448256 pid=4267 do_align=0 addr=3034054656 mmap_base=3069939712

The function (from openfst) which tries to map the file is the following:

MappedFile* MappedFile::Map(istream* s, const FstReadOptions &opts,
                        size_t size) {
  size_t pos = s->tellg();

  if (opts.mode == FstReadOptions::MAP && pos >= 0 &&
      pos % kArchAlignment == 0) {
    int fd = open(opts.source.c_str(), O_RDONLY);
    if (fd != -1) {
      int pagesize = getpagesize();
      off_t offset = pos % pagesize;
      off_t upsize = size + offset;
      void *map = mmap(NULL, upsize, PROT_READ, MAP_SHARED, fd, pos - offset);
      char *data = reinterpret_cast<char*>(map);
      if (close(fd) == 0 && map != MAP_FAILED) {
        MemoryRegion region;
        region.mmap = map;
        region.size = upsize;
        region.data = reinterpret_cast<void*>(data + offset);
        MappedFile *mmf = new MappedFile(region);
        s->seekg(pos + size, ios::beg);
        if (s) {
          VLOG(1) << "mmap'ed region of " << size << " at offset " << pos
                  << " from " << opts.source.c_str() << " to addr " << map;
          return mmf;
        }
        delete mmf;
      } else {
        LOG(INFO) << "Mapping of file failed: " << strerror(errno);
      }
    }
  }
  // If all else fails resort to reading from file into allocated buffer.
  if (opts.mode != FstReadOptions::READ) {
    LOG(WARNING) << "File mapping at offset " << pos << " of file "
                 << opts.source << " could not be honored, reading instead.";
  }
  MappedFile* mf = Allocate(size);
  if (!s->read(reinterpret_cast<char*>(mf->mutable_data()), size)) {
    delete mf;
    return NULL;
  }
  return mf;
}

Return from mmap is MAP_FAILED everytime.

Does someone has suggestions on where can I look to solve my issue? Thanks!

EDIT :

here is the content of /proc/self/maps right after the infamous mmap call : http://pastebin.com/1864jZC2

A little gap analysis:

Gap between 00000000 and 12c00000 (diff = 314572800 bytes, 300 MB)
Gap between 42c00000 and 55281000 (diff = 308809728 bytes, 294.50390625 MB)
Gap between 67e80000 and 67ea4000 (diff = 147456 bytes, 0.140625 MB)
Gap between 7778b000 and 77800000 (diff = 479232 bytes, 0.45703125 MB)
Gap between 77a80000 and 77a82000 (diff = 8192 bytes, 0.0078125 MB)
Gap between 77c00000 and 77c04000 (diff = 16384 bytes, 0.015625 MB)
Gap between 78080000 and 780b7000 (diff = 225280 bytes, 0.21484375 MB)
Gap between 79ac1000 and 79ac2000 (diff = 4096 bytes, 0.00390625 MB)
Gap between 7db70000 and 7db71000 (diff = 4096 bytes, 0.00390625 MB)
Gap between 7e000000 and 7e001000 (diff = 4096 bytes, 0.00390625 MB)
Gap between 7e0fe000 and 7e0ff000 (diff = 4096 bytes, 0.00390625 MB)
Gap between 7e145000 and 7e146000 (diff = 4096 bytes, 0.00390625 MB)
Gap between b6fb9000 and be6ff000 (diff = 125067264 bytes, 119.2734375 MB)
Gap between beeff000 and ffff0000 (diff = 1091506176 bytes, 1040.94140625 MB)

EDIT:

the solution that worked for me in the comments of @fadden's answer.

TL;DR: set dalvik.vm.heapsize to 512m.

Solution

Dump a copy of /proc/self/maps after mmap() fails (just open the file from your code and copy the contents to a temp file). You may be having a problem getting a large contiguous virtual address range due to ASLR.

Some of the Zip file handling code in Android was using mmap() to map the entire file, and just operate on it in memory. One day somebody created a 1GB Zip file and was unable to open it. While the process' virtual address space had enough free pages, there weren't enough contiguous pages to create a single linear mapping. (IIRC, the solution was to mmap() just the central directory.)

The maps output will show you what your process' address space looks like. 300MB seems a bit low for ASLR / fragmentation to be an issue, but it's a good place to start looking, and might explain the inconsistent behavior.