We are installing DSE Cassandra as a tarball install by creating our own Docker Image. As per their recommended best practices we have to set few values like "memlock" to unlimited for better performance. Refer: .9/managing/configure/recommended-settings.html#set-user-resource-limits

We have tried the same by adding the following lines into "/etc/security/limits.conf" file.

<non-root-dbuser> - memlock unlimited
<non-root-dbuser> - nofile 1048576
<non-root-dbuser> - nproc 32768
<non-root-dbuser> - as unlimited

The non-root-dbuser is our user with which we start our Cassadra Docker container

#Docker file reference
ADD limits.conf /etc/security/limits.conf
ADD dse-version-bin.tar.gz /opt    
USER <non-root-dbuser>
ENTRYPOINT <start-cassandra>

But when we start the DSE container, as a POD in Kubernetes. We are not able to see the memlock value being persisted.

$ ulimit -l
64

Also getting the following WARN in logs.

WARN  [DSE main thread] 2024-11-18 09:08:04,077 NativeLibrary.java:195 - \
  Unable to lock JVM memory (ENOMEM). This can result in part of the JVM \
  being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK \
  or run Cassandra as root.

Please let us know if anyone has already faced this issue, and how did you overcome. Also does anybody know the impact if the user-limits (memlock unlimited) is not set for cassandra.

We are installing DSE Cassandra as a tarball install by creating our own Docker Image. As per their recommended best practices we have to set few values like "memlock" to unlimited for better performance. Refer: https://docs.datastax/en/dse/6.9/managing/configure/recommended-settings.html#set-user-resource-limits

We have tried the same by adding the following lines into "/etc/security/limits.conf" file.

<non-root-dbuser> - memlock unlimited
<non-root-dbuser> - nofile 1048576
<non-root-dbuser> - nproc 32768
<non-root-dbuser> - as unlimited

The non-root-dbuser is our user with which we start our Cassadra Docker container

#Docker file reference
ADD limits.conf /etc/security/limits.conf
ADD dse-version-bin.tar.gz /opt    
USER <non-root-dbuser>
ENTRYPOINT <start-cassandra>

But when we start the DSE container, as a POD in Kubernetes. We are not able to see the memlock value being persisted.

$ ulimit -l
64

Also getting the following WARN in logs.

WARN  [DSE main thread] 2024-11-18 09:08:04,077 NativeLibrary.java:195 - \
  Unable to lock JVM memory (ENOMEM). This can result in part of the JVM \
  being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK \
  or run Cassandra as root.

Please let us know if anyone has already faced this issue, and how did you overcome. Also does anybody know the impact if the user-limits (memlock unlimited) is not set for cassandra.

• cassandra
• datastax
• datastax-enterprise

• Without memlock unlimited, you couild get the error describe here: cassandra.apache./doc/stable/cassandra/faq/#oom-map-failed – Aaron Commented Dec 10, 2024 at 14:44
• Can you run cat /etc/security/limits.conf with Docker exec and verify that the file is set properly? – Aaron Commented Dec 10, 2024 at 14:45
• Yes the file "/etc/security/limits.conf" contains the contents set in Docker Image <non-root-dbuser> - memlock unlimited <non-root-dbuser> - nofile 1048576 <non-root-dbuser> - nproc 32768 <non-root-dbuser> - as unlimited – Arun Veeramani Commented Dec 11, 2024 at 10:24

1 Answer 1

DataStax Enterprise (DSE) locks memory with mlock to prevent the JVM from being swapped out, particularly with files mapped with mmap. The recommendation is to allow unlimited locks so files can be mapped to available memory in the JVM.

Configuring resource limits in DSE instances deployed on OSIs (VMs or bare-metal servers) is different compared to Docker environments. By default, Docker containers have limited privileges so memory locking is not allowed (see Docker Runtime privilege and Linux capabilities).

Enable the capability to lock memory with the Docker runtime flag --cap-add:

$ docker run --cap-add=IPC_LOCK

This will allow you to set memlock to unlimited on containers:

--ulimit memlock=-1:-1

For details, see DSE recommended settings for Docker. Cheers!