fix: Respect container memory and CPU limits from cgroups

[Mantisus]

Description

get_memory_info and get_cpu_info reported the host machine even when the crawler ran under a container limit, because /proc/meminfo is not namespaced. The autoscaler sized its budget from host RAM and kept scaling until the container got killed, and a container pinned to two cores read the load of the whole machine as idle. Both now come from the cgroup of the process, with nothing to configure.

Two decisions worth a look:

• The control files are resolved through /proc/self/mountinfo and /proc/self/cgroup instead of assuming /sys/fs/cgroup, which is what makes this work under --cgroupns=host.
• Several cgroups can limit the same process, as they do under Kubernetes. The tightest limit is the ceiling a budget has to fit under, and the utilization is taken from the level closest to running out.

Without a limit, and outside Linux, the host values are used as before. A crawler in a container whose limit is below the host RAM now gets a smaller max_memory_size, so its concurrency drops.

Issues

• Closes: Memory metrics ignore container memory limits (no cgroup awareness) #2095
• Closes: Crawlee increasing concurrency until it dies #1535

Testing

• Add new tests with a fake cgroup filesystem

[Copilot]

Pull request overview

This PR makes Crawlee’s CPU and memory metrics container-aware on Linux by reading cgroup v1/v2 limits/usage (instead of relying solely on host-wide /proc/meminfo / psutil), so the autoscaler sizes budgets and CPU utilization against the resources actually available to the running process.

Changes:

• Add a new cgroup discovery + metrics reader (src/crawlee/_utils/cgroup.py) that resolves controllers via /proc/self/mountinfo + /proc/self/cgroup and reads the tightest applicable limits.
• Update get_memory_info() / get_cpu_info() to prefer cgroup-scoped totals/usage when limits apply, falling back to host metrics otherwise.
• Add comprehensive unit tests with a fake cgroup filesystem, and document autoscaling behavior under resource limits.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
tests/unit/_utils/test_cgroup.py	Adds unit tests covering cgroup v1/v2 discovery and limit/usage semantics, plus system.py integration behavior.
src/crawlee/_utils/system.py	Switches CPU/memory “system-wide” metrics to cgroup-aware readings when applicable; keeps host fallbacks.
src/crawlee/_utils/cgroup.py	Implements cgroup controller discovery and reads memory/cpu limits + usage from the most relevant hierarchy levels.
docs/guides/scaling_crawlers.mdx	Documents how autoscaling respects cgroup CPU/memory limits when running under container/systemd constraints.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[vdusek]

If I am correct, all the tests run against a fake cgroup filesystem. Is there a way to cover this end-to-end? e.g., running a crawler in a container with --memory and/or --cpus set, and asserting get_memory_info() and get_cpu_info()? Or have you tested it manually? Also, we should make sure we cover both --cgroupns=private and --cgroupns=host.

[Mantisus]

If I am correct, all the tests run against a fake cgroup filesystem. Is there a way to cover this end-to-end?

That's right. To add the test, we'll probably need to add a new job to CI. With my permissions in the repository, I won't be able to test it properly.

Or have you tested it manually?

Yes, I tested it locally with systemd and docker

(crawlee) mantisus@Kleimor:~/repos/apify/crawlee-python$ docker run --rm $MOUNTS $IMG $RUN
configuration
  available_memory_ratio : 0.25
  max_used_memory_ratio  : 0.9
  max_used_cpu_ratio     : 0.95
  memory_mbytes          : None
  system_info_interval   : 0:00:01
limits read from the cgroup
  memory      : None
  cpu quota   : None
  cpu set     : 20
latest memory snapshot
  current_size            : 60.27 MB
  max_memory_size         : 4.88 GB
  system_wide_used_size   : 2.63 GB
  system_wide_memory_size : 19.53 GB
  is_overloaded           : False
latest cpu snapshot
  used_ratio              : 0.216
  max_used_ratio          : 0.95
  is_overloaded           : False
system status over the last 6 cpu samples
  cpu        : 0.00 of 0.40 allowed   ok
  memory     : 0.00 of 0.20 allowed   ok
  event_loop : 0.00 of 0.60 allowed   ok
  client     : 0.00 of 0.30 allowed   ok
  is_system_idle: True

(crawlee) mantisus@Kleimor:~/repos/apify/crawlee-python$ docker run --rm -m 512m --cpus 1 $MOUNTS $IMG $RUN
configuration
  available_memory_ratio : 0.25
  max_used_memory_ratio  : 0.9
  max_used_cpu_ratio     : 0.95
  memory_mbytes          : None
  system_info_interval   : 0:00:01
limits read from the cgroup
  memory      : MemoryLimit(limit=536870912, working_set=84348928)
  cpu quota   : 1.0
  cpu set     : 20
latest memory snapshot
  current_size            : 60.19 MB
  max_memory_size         : 128.00 MB
  system_wide_used_size   : 80.98 MB
  system_wide_memory_size : 512.00 MB
  is_overloaded           : False
latest cpu snapshot
  used_ratio              : 0.996
  max_used_ratio          : 0.95
  is_overloaded           : True
system status over the last 6 cpu samples
  cpu        : 0.74 of 0.40 allowed   OVERLOADED
  memory     : 0.00 of 0.20 allowed   ok
  event_loop : 0.00 of 0.60 allowed   ok
  client     : 0.00 of 0.30 allowed   ok
  is_system_idle: False

(crawlee) mantisus@Kleimor:~/repos/apify/crawlee-python$ docker run --rm -m 512m --cpus 6 $MOUNTS $IMG $RUN
configuration
  available_memory_ratio : 0.25
  max_used_memory_ratio  : 0.9
  max_used_cpu_ratio     : 0.95
  memory_mbytes          : None
  system_info_interval   : 0:00:01
limits read from the cgroup
  memory      : MemoryLimit(limit=536870912, working_set=85512192)
  cpu quota   : 6.0
  cpu set     : 20
latest memory snapshot
  current_size            : 60.32 MB
  max_memory_size         : 128.00 MB
  system_wide_used_size   : 82.48 MB
  system_wide_memory_size : 512.00 MB
  is_overloaded           : False
latest cpu snapshot
  used_ratio              : 0.723
  max_used_ratio          : 0.95
  is_overloaded           : False
system status over the last 6 cpu samples
  cpu        : 0.00 of 0.40 allowed   ok
  memory     : 0.00 of 0.20 allowed   ok
  event_loop : 0.00 of 0.60 allowed   ok
  client     : 0.00 of 0.30 allowed   ok
  is_system_idle: True

(crawlee) mantisus@Kleimor:~/repos/apify/crawlee-python$ docker run --rm -m 512m --cgroupns=host $MOUNTS $IMG $RUN 
configuration
  available_memory_ratio : 0.25
  max_used_memory_ratio  : 0.9
  max_used_cpu_ratio     : 0.95
  memory_mbytes          : None
  system_info_interval   : 0:00:01
limits read from the cgroup
  memory      : MemoryLimit(limit=536870912, working_set=89137152)
  cpu quota   : None
  cpu set     : 20
latest memory snapshot
  current_size            : 60.34 MB
  max_memory_size         : 128.00 MB
  system_wide_used_size   : 85.98 MB
  system_wide_memory_size : 512.00 MB
  is_overloaded           : False
latest cpu snapshot
  used_ratio              : 0.200
  max_used_ratio          : 0.95
  is_overloaded           : False
system status over the last 6 cpu samples
  cpu        : 0.00 of 0.40 allowed   ok
  memory     : 0.00 of 0.20 allowed   ok
  event_loop : 0.00 of 0.60 allowed   ok
  client     : 0.00 of 0.30 allowed   ok
  is_system_idle: True

[Pijukatel]

Great idea. But let's focus on testing this in different environment setups. This could be a standalone repo, as it can be useful outside of Crawlee and it has quite different testing requirements.

So I would suggest having it in this placeholder repo as an experiment, and once we are sure it works in different environments and setups, we can publish it and integrate it into Crawlee.