Feature: Add external database volume and external database connection by nshandra · Pull Request #155 · EyeSeeTea/d2-docker

nshandra · 2025-12-05T08:55:45Z

📌 References

Issue: Closes [d2-docker] External database storage or connection

📝 Implementation

Add parameters to d2-docker start to allow either:

Store the DB in an external persistent volume
Use an external DB instead of a container

Testing

External Persistent Volume Case

This case will store the DB volume at the specified absolute path.

./d2-docker-dev.sh --log-level=DEBUG start <data_image> --external-db-volume="/absolute/path/to/volume_dir"

External DB Case

This case needs a psql DB runnig in the machine hosting the dockers or in a accessible network.

This DB has to have the correct settings:

/etc/postgresql/{ver}/main/postgresql.conf

Set listen_addresses to:
- '*' (easiest for testing)
- 'localhost,172.17.0.1' (docker and DB in same system)
- 'localhost,<reachable_ip>'

/etc/postgresql/{ver}/main/pg_hba.conf

Add an entry to allow the connection. Example:

# Docker bridge subnet (check with: docker network inspect bridge)
host    dhis2           dhis            172.17.0.0/16           scram-sha-256
# LAN
host    dhis2           dhis            192.168.0.0/24          scram-sha-256

Running d2-cloner:

./d2-docker-dev.sh --log-level=DEBUG start <data_image> --external-db-url="postgresql://user:pass@host:port/db"

#1ck80y8

tokland · 2026-01-26T08:20:53Z

@nshandra Sorry, I had this pending. Can you merge with development? (re-add me when done)

nshandra · 2026-01-26T08:53:45Z

@tokland done.

tokland

Features worked for me (only I had to add a trailing comma, probably a merge artifact)

In-line comments about the code:

tokland · 2026-04-14T08:20:20Z

src/d2_docker/utils.py

@@ -300,6 +318,9 @@ def run_docker_compose(
        ("ROOT_PATH", ROOT_PATH),
        ("PSQL_ENABLE_QUERY_LOGS", "") if not enable_postgres_queries_logging else None,
        ("GLOWROOT_PORT", get_port_glowroot(glowroot_port))


Missing trailing comma, the code does not run as-is.

tokland · 2026-04-14T08:47:14Z

src/d2_docker/commands/start.py

+        "--external-db-volume",
+        metavar="DIRECTORY",
+        help="Directory for external database volume",
+    )


This option worked for me. Only problem is that the final permissions of the folder are off:

$ ls -al | grep volume drwx------ 19 70 70 4.0K Apr 14 10:21 volume

However, that's something d2-docker has been ignoring all along - we probably should have user: "${UID}:${GID}" or similar in our docker-compose to avoid this kind of problems. Not sure it's the moment to try that (it may have unintended consequences), but I'd be a step in the good direction.

tokland · 2026-04-14T08:47:27Z

src/d2_docker/commands/start.py

+        type=str,
+        metavar="postgresql://user:pass@host:port/dbname",
+        help="Use external PostgreSQL database"
+    )


It worked for me

tokland · 2026-04-14T08:48:18Z

src/d2_docker/commands/start.py

+        "--load-dump-from-data",
+        action="store_true",
+        help="Load database dump from data container (only with --external-db-url)",
+    )


It worked for me, only that I had to explicitly create the database with psql for it to work. Is that the intended workflow?

tokland · 2026-04-14T08:53:25Z

src/d2_docker/config/dhis2-core-start.sh

    else
        echo "Normal mode: $path"
-        $psql_cmd < "$path"
+        $psql_cmd <"$path"


Formatting is being applied to code outside of the PR. It would be fine if the formatting were incorrect, but we don't have a defined prettifier, so there is no "incorrect" for now. Would the best approach be a .vscode/settings.json file where we set up the formatter and its options?

tokland · 2026-04-14T08:57:29Z

src/d2_docker/utils.py

        ("GLOWROOT_PORT", get_port_glowroot(glowroot_port))
+        ("EXTERNAL_DB_VOLUME", external_db_volume) if external_db_volume else None,
+        ("EXTERNAL_DB_URL", external_db_url) if external_db_url else None,
+        ("LOAD_DUMP_FROM_DATA", "yes" if load_dump_from_data else "no") if external_db_url else "no",


The grouping/parentheses look off, right?

tokland · 2026-04-14T09:02:19Z

src/d2_docker/utils.py

+    logger.info("Validating external database connection...")
+    try:
+        psql_cmd = ["psql", "-d", db_url, "-c", "SELECT now();", "&> /dev/null"]
+        run(psql_cmd, capture_output=False)


it seems it worked, but I am curious about "&> /dev/null", since the default for run is shell=False.

tokland · 2026-04-14T09:07:01Z

src/d2_docker/commands/start.py

+
+    if args.external_db_url:
+        utils.validate_external_db_connection(args.external_db_url)
+


Should we add a check to raise error if both options are passed, as they are mutually exclusive?

Also, a standalone --load-dump-from-data is not correct workflow.

tokland · 2026-04-14T09:17:00Z

src/d2_docker/config/dhis2-core-start.sh

 psql_cmd="$psql_base_cmd -v ON_ERROR_STOP=0"
 psql_strict_cmd="$psql_base_cmd -v ON_ERROR_STOP=1"
-pgrestore_cmd="pg_restore -h db -U dhis -d dhis2"
+pgrestore_cmd="pg_restore ${db_url:-"-h db -U dhis dhis2"}"


Before we had -d dhis2 but now it's directly dhis2. Is that correct?

tokland · 2026-04-14T09:18:29Z

src/d2_docker/utils.py

+        if parsed.hostname == "localhost":
+            hostname = "host.docker.internal"
+        else:
+            hostname = parsed.hostname


[minor] we can use hostname = ..

nshandra added 3 commits November 7, 2025 09:54

feat: add external persisted volume for DB container

7573e69

feat: add external DB connection option to start

2cb916a

Merge branch 'development' into feature/external-database

74a84de

nshandra requested a review from tokland December 5, 2025 08:55

nshandra self-assigned this Dec 5, 2025

nshandra marked this pull request as draft December 5, 2025 13:06

nshandra marked this pull request as ready for review January 21, 2026 21:28

Merge branch 'development' into feature/external-database

4a880ad

tokland requested review from tokland and removed request for tokland January 26, 2026 09:13

tokland reviewed Apr 14, 2026

View reviewed changes


		if args.external_db_url:
		utils.validate_external_db_connection(args.external_db_url)

Conversation

nshandra commented Dec 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📌 References

📝 Implementation

Testing

External Persistent Volume Case

External DB Case

Uh oh!

tokland commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nshandra commented Jan 26, 2026

Uh oh!

tokland left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nshandra commented Dec 5, 2025 •

edited

Loading

tokland commented Jan 26, 2026 •

edited

Loading

tokland left a comment •

edited

Loading