document nofile limit

solutions/observability/apm/apm-server/binary.md

@@ -61,6 +61,8 @@
 tar xzvf apm-server-{{version.stack}}-linux-x86_64.tar.gz
 ```
 
+Refer to [modifying the `nofile` ulimit](#modify-nofile-ulimit).
+
 $$$apm-mac$$$
 **Mac:**
 
@@ -926,4 +928,69 @@
 ```dockerfile
 FROM docker.elastic.co/apm/apm-server:9.0.0
 COPY --chmod=0644 --chown=1000:1000 apm-server.yml /usr/share/apm-server/apm-server.yml
-```
+```
+
+#### Modifying `nofile` ulimit [ulimit-on-docker]
+
+Limits can be set from the command line using `--ulimit=soft:hard`, refer to [Set ulimits in container (--ulimit)](https://docs.docker.com/reference/cli/docker/container/run/#ulimit) in the Docker documentation.
+
+```sh
+docker run -d \
+  -p 8200:8200 \
+  --name=apm-server \
+  --user=apm-server \
+  --volume="$(pwd)/apm-server.docker.yml:/usr/share/apm-server/apm-server.yml:ro" \
+  docker.elastic.co/apm/apm-server:9.0.0 \
+  --strict.perms=false -e \
+  --ulimit=524287:524287 \
+  -E output.elasticsearch.hosts=["elasticsearch:9200"] <1> <2>
+```
+
+1. Substitute your {{es}} hosts and ports.
+2. If you are using {{ech}}, replace the `-E output.elasticsearch.hosts` line with the Cloud ID and elastic password using the syntax shown earlier.
+
+
+## Modify the `nofile` ulimit [modify-nofile-ulimit]
+
+When run as a standalone binary APM Server inherits the `nofile` limit from the user running the process. On most system this is configured to `1024`. This limit is too low for higher throughput scenarios or when using Tail Based Sampling.
+
+To chose the new limit, consider these guidelines:
+- there is no system performance impact of using a limit of ``;
+- a limit of `1024` would suffice for low throughput use cases;
+- the major contributor to open files is the number of incoming connections;
+- Tail Based Sampling is file based, when enabling it the number of open files is higher in proportion to the throughput and sampling policies.
+
+To configure the limit for your user, you need to know the username you run APM Server process with.
+
+```sh
+whoami
+```
+
+Edit `/etc/security/limits.conf` with root privileges:
+
+```sh
+sudo nano /etc/security/limits.conf
+```
+
+Add the following lines to set soft and hard limits for your user:
+
+```text
+apm-server soft nofile 524287 <1>
+apm-server hard nofile 524287 <1>
+```
+
+1. Replace `apm-server` with the username you run APM Server process with.
+
+To update the `nofile` ulimit of a running process you need to know the process ID (PID):
+
+```sh
+pgrep -f apm-server
+```
+
+Then apply the new limits:
+
+```sh
+prlimit --pid PID --nofile=524287:524287 <1>
+```
+1. Replace `PID` with your APM Server process ID.
+

solutions/observability/apm/apm-server/systemd.md

@@ -88,6 +88,16 @@ systemctl restart apm-server
 It is recommended that you use a configuration management tool to include drop-in unit files. If you need to add a drop-in manually, use `systemctl edit apm-server.service`.
 ::::
 
+#### Configuring the NOFILE limit [configuring-nofile-limit]
+
+::::{note}
+There should be no need to manually configure this limit when running APM Server.
+::::
+
+In systemd the `LimitNOFILE` defaults are set to `1024` (soft) and `524288` (hard) and most Linux systems with systemd do not change these values or reduce them drastically. Golang, starting from version 1.19 (refer to [golang/go#46279](https://github.com/golang/go/issues/46279)), automatically bump the process limit up to the available hard limit. This means that by default APM Server runs with the limit set to the hard limit value by the Operating System is being run on, generally `524287` on a recent system. There should be no reason to change this limit, as back-pressure from too many open files happens through memory usage.
+
+For guidelines on the value to set this value to refer to [modifying the `nofile` ulimit](/solutions/observability/apm/apm-server/binary.md#modify-nofile-ulimit).
+
 #### Configuration file ownership [apm-config-file-ownership]
 
 On systems with POSIX file permissions, the APM Server configuration file is subject to ownership and file permission checks. These checks prevent unauthorized users from providing or modifying configurations that are run by APM Server.

solutions/observability/apm/apm-server/tail-based-sampling.md

@@ -20,6 +20,14 @@ Most options on this page are supported by all APM Server deployment methods whe
 Enhanced privileges are required to use tail-based sampling. For more information, refer to [Create a tail-based sampling role](/solutions/observability/apm/create-assign-feature-roles-to-apm-server-users.md#apm-privileges-tail-based-sampling).
 ::::
 
+::::{note}
+If you are manually configuring systemd `LimitNOFILE` or `LimitNOFILESoft` when using Tail Based Sampling and it affects the APM Server process, this can result in a `too many open files` error. Refert to [configuring the NOFILE limit](/solutions/observability/apm/apm-server/systemd.md#configuring-nofile-limit) for further instructions.
+::::
+
+::::{note}
+If you are running the binary standalone (not using the provided dev or rpm packages or the docker image) you need to adjust the `nofile` limit based on your throughput requirements. Refer to [modifying the `nofile` ulimit](/solutions/observability/apm/apm-server/binary.md#modify-nofile-ulimit).
+::::
+
 Tail-based sampling configuration options.
 
 :::::::{tab-set}