Background
As I mentioned in previous blog, there is a memory fragmentation problem within our application, and tunning the JVM parameters can resolve the problem, but it doesn't answer the question who is creating so many direct buffers and use them for short time? Because it's against the recommendation of
ByteBuffer.allocateDirect():
It is therefore recommended that direct buffers be allocated primarily for large, long-lived buffers that are subject to the underlying system's native I/O operations.
Analysis
To answer this question, I created a simple program to monitor the caller of
ByteBuffer.allocateDirect(int) through
JDI, and got following output (some stack are omitted):
java.nio.ByteBuffer.allocateDirect(int): count=124, size=13463718
sun.nio.ch.Util.getTemporaryDirectBuffer(int): count=124, size=13463718
sun.nio.ch.IOUtil.write(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher): count=1, size=447850
sun.nio.ch.SocketChannelImpl.write(java.nio.ByteBuffer): count=1, size=447850
org.eclipse.jetty.io.ChannelEndPoint.flush(java.nio.ByteBuffer[]): count=1, size=447850
sun.nio.ch.IOUtil.write(java.io.FileDescriptor, java.nio.ByteBuffer[], int, int, sun.nio.ch.NativeDispatcher): count=102, size=12819260
sun.nio.ch.SocketChannelImpl.write(java.nio.ByteBuffer[], int, int): count=102, size=12819260
org.eclipse.jetty.io.ChannelEndPoint.flush(java.nio.ByteBuffer[]): count=102, size=12819260
sun.nio.ch.IOUtil.read(java.io.FileDescriptor, java.nio.ByteBuffer, long, sun.nio.ch.NativeDispatcher): count=21, size=196608
sun.nio.ch.SocketChannelImpl.read(java.nio.ByteBuffer): count=21, size=196608
org.eclipse.jetty.io.ChannelEndPoint.fill(java.nio.ByteBuffer): count=21, size=196608
The
count is the number of direct buffers, and the
size is the total size of buffers.
According the monitoring result, those direct buffers are all allocated by JDK default implementation of
SocketChannel, when the user pass a non-direct buffer to perform I/O. After read the source code of
sun.nio.ch.Util, I found it uses a thread local cache to keep the direct buffer. While the
sun.nio.ch.Util is designed carefully to limit the total number of cached buffers (8 per thread), and clean direct buffer while it's removed from the cache. The only possible reason is there are too many new threads, and the buffers in cache are not used at all.
Root Cause
Our application uses Jetty to handle HTTP requests, and there is a custom class wraps Jetty Server, in this wrapper class, a
ThreadPoolExecutor is used like following:
ExecutorService executor = new ThreadPoolExecutor(
minThreads,
maxThreads,
maxIdleTimeMs,
TimeUnit.MILLISECONDS,
new SynchronousQueue());
Server server = new Server(new ExecutorThreadPool(executor));
Unfortunately, the
minThreads is
2 and
maxIdleTimeMs is 5000ms, and the Jetty Server will use 12 threads from the pool for accepting and selector. Which means the total number of threads is always bigger than the
minThreads. When the service is not very busy, a worker thread will be discarded, and a new worker thread will be created when a new request comes. In this situation, the cached buffers in
sun.nio.ch.Util will no longer be used and will only been collected when GC triggered.
