We are installing DSE Cassandra as a tarball install by creating our own Docker Image. As per their recommended best practices we have to set few values like "memlock" to unlimited for better performance. Refer: .9/managing/configure/recommended-settings.html#set-user-resource-limits
We have tried the same by adding the following lines into "/etc/security/limits.conf" file.
<non-root-dbuser> - memlock unlimited
<non-root-dbuser> - nofile 1048576
<non-root-dbuser> - nproc 32768
<non-root-dbuser> - as unlimited
The non-root-dbuser is our user with which we start our Cassadra Docker container
#Docker file reference
ADD limits.conf /etc/security/limits.conf
ADD dse-version-bin.tar.gz /opt
USER <non-root-dbuser>
ENTRYPOINT <start-cassandra>
But when we start the DSE container, as a POD in Kubernetes. We are not able to see the memlock value being persisted.
$ ulimit -l
64
Also getting the following WARN in logs.
WARN [DSE main thread] 2024-11-18 09:08:04,077 NativeLibrary.java:195 - \
Unable to lock JVM memory (ENOMEM). This can result in part of the JVM \
being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK \
or run Cassandra as root.
Please let us know if anyone has already faced this issue, and how did you overcome. Also does anybody know the impact if the user-limits (memlock unlimited) is not set for cassandra.
We are installing DSE Cassandra as a tarball install by creating our own Docker Image. As per their recommended best practices we have to set few values like "memlock" to unlimited for better performance. Refer: https://docs.datastax/en/dse/6.9/managing/configure/recommended-settings.html#set-user-resource-limits
We have tried the same by adding the following lines into "/etc/security/limits.conf" file.
<non-root-dbuser> - memlock unlimited
<non-root-dbuser> - nofile 1048576
<non-root-dbuser> - nproc 32768
<non-root-dbuser> - as unlimited
The non-root-dbuser is our user with which we start our Cassadra Docker container
#Docker file reference
ADD limits.conf /etc/security/limits.conf
ADD dse-version-bin.tar.gz /opt
USER <non-root-dbuser>
ENTRYPOINT <start-cassandra>
But when we start the DSE container, as a POD in Kubernetes. We are not able to see the memlock value being persisted.
$ ulimit -l
64
Also getting the following WARN in logs.
WARN [DSE main thread] 2024-11-18 09:08:04,077 NativeLibrary.java:195 - \
Unable to lock JVM memory (ENOMEM). This can result in part of the JVM \
being swapped out, especially with mmapped I/O enabled. Increase RLIMIT_MEMLOCK \
or run Cassandra as root.
Please let us know if anyone has already faced this issue, and how did you overcome. Also does anybody know the impact if the user-limits (memlock unlimited) is not set for cassandra.
Share Improve this question edited Jan 14 at 4:28 Erick Ramirez 16.4k2 gold badges21 silver badges31 bronze badges asked Dec 3, 2024 at 5:32 Arun VeeramaniArun Veeramani 791 silver badge9 bronze badges 3 |1 Answer
Reset to default 1DataStax Enterprise (DSE) locks memory with mlock to prevent the JVM from being swapped out, particularly with files mapped with mmap. The recommendation is to allow unlimited locks so files can be mapped to available memory in the JVM.
Configuring resource limits in DSE instances deployed on OSIs (VMs or bare-metal servers) is different compared to Docker environments. By default, Docker containers have limited privileges so memory locking is not allowed (see Docker Runtime privilege and Linux capabilities).
Enable the capability to lock memory with the Docker runtime flag --cap-add:
$ docker run --cap-add=IPC_LOCK
This will allow you to set memlock to unlimited on containers:
--ulimit memlock=-1:-1
For details, see DSE recommended settings for Docker. Cheers!
memlock unlimited, you couild get the error describe here: cassandra.apache./doc/stable/cassandra/faq/#oom-map-failed – Aaron Commented Dec 10, 2024 at 14:44cat /etc/security/limits.confwith Docker exec and verify that the file is set properly? – Aaron Commented Dec 10, 2024 at 14:45