I need some help with a spark memory issue. I have my Spark application running inside a single JVM as a Kubernetes pod. The nature of the job is that it is a batched job, it runs for 4-5 hours and then the pod is idle. Each batch is processed after the 12-hour gap.

Issue: The pod's memory keeps increasing during this 4-5 hours job cycle and once all the jobs are done in the batch, and pod is idle, the memory does not get lowered.

Let's say I have allocated 100GB to a pod, it increases to 80GB during the batch cycle, and once all the jobs are completed, and the pod is idle, the memory does not come down from 80 GB. In the next batch cycle (the gap is 12 hours), memory starts increasing from 80GB until it gets OOM killed at 100GB

I tried taking a heap dump as well. The heap dump points to an unsafe Memory allocator. I suspect that physical memory is getting allocated outside JVM and not cleaned up, but I'm unsure what Spark operation/configuration can cause this issue.

Also, in case it is allocated outside the heap, why it is not getting cleaned up?

One instance of .apache.spark.unsafe.memory.HeapMemoryAllocator loaded by jdk.internal.loader.ClassLoaders$AppClassLoader 1,61,06,14,312 (89.24%) bytes. The memory is accumulated in one instance of java.util.LinkedList, loaded by <system class loader>, which occupies 1,61,06,14,112 (89.24%) bytes.

Also, I checked in the "Storage" tab of the Spark UI as well, it doesn't show any RDD cached.

I need some help with a spark memory issue. I have my Spark application running inside a single JVM as a Kubernetes pod. The nature of the job is that it is a batched job, it runs for 4-5 hours and then the pod is idle. Each batch is processed after the 12-hour gap.

Issue: The pod's memory keeps increasing during this 4-5 hours job cycle and once all the jobs are done in the batch, and pod is idle, the memory does not get lowered.

Let's say I have allocated 100GB to a pod, it increases to 80GB during the batch cycle, and once all the jobs are completed, and the pod is idle, the memory does not come down from 80 GB. In the next batch cycle (the gap is 12 hours), memory starts increasing from 80GB until it gets OOM killed at 100GB

I tried taking a heap dump as well. The heap dump points to an unsafe Memory allocator. I suspect that physical memory is getting allocated outside JVM and not cleaned up, but I'm unsure what Spark operation/configuration can cause this issue.

Also, in case it is allocated outside the heap, why it is not getting cleaned up?

One instance of .apache.spark.unsafe.memory.HeapMemoryAllocator loaded by jdk.internal.loader.ClassLoaders$AppClassLoader 1,61,06,14,312 (89.24%) bytes. The memory is accumulated in one instance of java.util.LinkedList, loaded by <system class loader>, which occupies 1,61,06,14,112 (89.24%) bytes.

Also, I checked in the "Storage" tab of the Spark UI as well, it doesn't show any RDD cached.

• java
• apache-spark
• memory-management
• apache-spark-sql
• memory-leaks

• Memory that appears in a heap dump is allocated inside the heap. There may be other memory reserved outside the heap too, but that will not appear in a heap dump. – aled Commented Mar 14 at 17:31

1 Answer 1

What I understood is from your log that there is an instance of java.util.LinkedList which has consumed 90% of the memory.

From Spark's Memory Tuning Blog, the memory issues can happen because of the following reasons given below:

1. Each distinct Java object has an “object header”, which is about 16 bytes and contains information such as a pointer to its class. For an object with very little data in it (say one Int field), this can be bigger than the data.

2. Java Strings have about 40 bytes of overhead over the raw string data (since they store it in an array of Chars and keep extra data such as the length), and store each character as two bytes due to String’s internal usage of UTF-16 encoding. Thus a 10-character string can easily consume 60 bytes.

3. Common collection classes, such as HashMap and LinkedList, use linked data structures, where there is a “wrapper” object for each entry (e.g. Map.Entry). This object not only has a header, but also pointers (typically 8 bytes each) to the next object in the list.

4. Collections of primitive types often store them as “boxed” objects such as java.lang.Integer.

Now, what I got to know from Apache Spark Docs is that for maintaining efficiency and preventing OutOfMemory errors, Spark can use the memory outside of the heap when there is a high consumption on the heap and the garbage collection by JVM is not guaranteed on those memory allocations. So, that's why it's not getting cleaned.

Spark calls it as off-heap memory (memory allocation outside of the heap). The off-heap memory allocation is disabled by default. But it can be enabled irrespective of what was set in spark.memory.offHeap.enabled configuration by Spark if it sees a high heap memory usage.

Now, the solutions that are suggested by the Spark's Data Structure Tuning Blog :

1. Design your data structures to prefer arrays of objects, and primitive types, instead of the standard Java or Scala collection classes (e.g. HashMap). The fastutil library provides convenient collection classes for primitive types that are compatible with the Java standard library.

2. Avoid nested structures with a lot of small objects and pointers when possible.

3. Consider using numeric IDs or enumeration objects instead of strings for keys.

4. If you have less than 32 GiB of RAM, set the JVM flag -XX:+UseCompressedOops to make pointers be four bytes instead of eight. You can add these options in spark-env.sh.

See if you can use these tuning options to optimise the memory utilization.

Otherwise, adjust the configuration available in Spark's Memory Management Configuration Options which is around configuring the optimal usage of off-heap when the memory heap usage is too high.

• spark.driver.memoryOverhead

• spark.driver.memory

• spark.driver.memoryOverheadFactor

• spark.shuffle.io.preferDirectBufs (only works with Netty Server)

• spark.memory.offHeap.enabled

• spark.memory.offHeap.size

Read about these configurations here to understand their usage.