Prevent OutOfMemory when using java.nio.MappedByteBuffer

Consider application, which create 5-6 threads, each thread in cycle allocate MappedByteBuffer for 5mb page size.

MappedByteBuffer b = ch.map(FileChannel.MapMode.READ_ONLY, r, 1024*1024*5);

Sooner or later, when application works with big files, oom is thrown

java.io.IOException: Map failed  at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
Caused by: java.lang.OutOfMemoryError: Map failed
        at sun.nio.ch.FileChannelImpl.map0(Native Method)
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)

According to specification, MappedBuffer should dispose direct memory as soon as it is GC itself. Looks like the problem is, that MappedBuffer-s are GC-ed too late, later then direct memory finished.

How to avoid this situation ? Probably say MappedBuffer to dispose implicitly or use some kind of pool of MappedBuffer

Answers


You can avoid having to trigger a GC by cleaning up the mapped byte buffers directly.

public static void clean(ByteBuffer bb) {
    if(bb == null) return;
    Cleaner cleaner = ((DirectBuffer) bb).cleaner();
    if(cleaner != null) cleaner.clean();
}

Provided you call this before discarding, you won't run out of virtual memory.

Perhaps you can look at creating larger ByteBuffers less often (unless you have a large number of files) Creating a MappedByteBuffer is not free (takes about 50 micro-seconds on some machines)


The error message says "map failed", not "heap space" or "permgen space". This means the JVM doesn't have enough address space available.

See this bug in Sun's database, and also this question.

The first link provides a workaround (ewww) which is close the what the second link says:

    try {
        buffer = channel.map(READ_ONLY, ofs, n);
    } catch (java.io.IOException e) {
        System.gc();
        System.runFinalization();
        buffer = channel.map(READ_ONLY, ofs, n);
    }

Maybe a WeakHashMap to pool those MappedBuffers would work.

But before you guess about the root cause, I'd recommend hooking your app up to Visual VM 1.3.3, with all the plugins installed, so you can see exactly what's causing the OOM error. You're presuming that these MappedBuffers are doing it, but they're only 5MB each for 5-6 threads - 25-30MB total.

Better to have data than guess. Visual VM will get it for you.


MappedBuffer should dispose direct memory as soon as it is GC itself

It doesn't actually say that anywhere that I can see. There is a long-standing Bug Parade item that says it is never released.

It does say this:

It is therefore recommended that direct buffers be allocated primarily for large, long-lived buffers


Need Your Help

RDF namespaces (Semantic data)

semantic-web

I am trying to come up with an ontology (internal) for my application. However, I want to borrow as much as I can. When I look around different existing ontology, I observe an overlap. For eg:

HQL Inner join on same table

hibernate grails hql gorm

I asked an earlier question today which was about GORM: How to fetch records in grails by max date and group at same time However, someone suggested it can be easily achieved using HQL. But using ...

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.