Autodiscover paths in the no-engine case

[vineetbansal]

Summary of Changes

Adds support for inspecting the current running context using:

• get_directory_context() <- The top-level directory specified by the @flow.
• get_context_path() <- The position of the running code relative to the top-level @flow as /-separated string elements.

Together, these allow the code to create a folder using - settings.RESULTS_DIR / Path(get_directory_context()) / get_context_path().

calc_setup uses this to create timestamped folders for output.

Still a WIP. I need to add support for a few things we discussed, and add tests for these:

• @jobs might be running without an encompassing @flow and need to be able to set their own top-level directory.
• A @flow might have several @subflows with the same name. Thus the subflow level folders need to have a timestamp appended to them, just like the folders at the @job level.
• Add tests in test_autodiscover_paths to cover the case of QUACC_AUTODISCOVER_DIR to True/False to demonstrate the directory structure in both modes. Add tree to their docstrings so it's clear what to expect in each case.

[Andrew-S-Rosen]

Thanks, @vineetbansal! One other ToDo here: can you also update the docs with info about your new feature? https://quantum-accelerators.github.io/quacc/user/settings/file_management.html

[codecov]

Codecov Report

❌ Patch coverage is 97.32143% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.59%. Comparing base (221643b) to head (1ba97ae).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/quacc/wflow_tools/context.py	96.05%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3139      +/-   ##
==========================================
- Coverage   97.68%   97.59%   -0.10%     
==========================================
  Files          97       98       +1     
  Lines        4190     4282      +92     
==========================================
+ Hits         4093     4179      +86     
- Misses         97      103       +6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[vineetbansal]

@Andrew-S-Rosen - I think that should do it to fix the tests. I'm not sure why Jenkins is stuck - perhaps a restart of server infrastructure that happened last Tuesday?

[vineetbansal]

The structure during runtime is now such that we still create soft links from RESULTS_DIR to SCRATCH_DIR when running, and these links are created at the top-level (just like before) for running jobs, but point to the corresponding folder for the job within the SCRATCH_DIR tree.

These symlinks are in addition to any other structure that the running flow creates, and removed once the job completes.

In case of failures, these symlinks are called symlink-failed- (like before) and they point to the corresponding folder for the job within SCRATCH_DIR tree, but now the target folder is failed-<job-name>-.. instead of just <job-name>...

This naming scheme is the same as before, but happens at the job-level (since it is the job that has failed, not the flow).

I've tried to detail this in the updated docs. Let me know if it is too much detail and/or should be broken up to avoid overload.

[Andrew-S-Rosen]

@vineetbansal thanks!! Is this ready for review?

After review, I will probably ask someone else in the group to try it out and make sure the hierarchy makes sense to them.

[vineetbansal]

@Andrew-S-Rosen - yes this is ready to be reviewed now.

Two things that we can either do as part of this PR or later:

• One of the issues raised during the meeting was the ability to assign the "top-level" folder name within RESULTS_DIR instead of it being auto-generated from the flow name. This should be possible with a few lines of code.
• It is not too much more code (nowhere compared to PR WIP: Auto-discovery of Paths #3137) to support all this for jobflow as well. Let me know if this is a good time to put that in (along with tests).

[Andrew-S-Rosen]

Thanks! I will review it this weekend. 👍

1. That'd be great! Perhaps you should open that in a new PR? I leave that to you.
2. I think it could be interesting to support this for jobflow --- the only concern is that I am not sure that the runtime folder would be properly stored in the MongoDB collection for the job metadata. If we start moving directories around within quacc, I don't know how jobflow would know where they are anymore.

[Andrew-S-Rosen]

Hi @vineetbansal! I have some comments worth addressing below. As for your "Two things that we can either do as part of this PR or later", please see my comment above.

Directory in Output

When I run the following example, the output schema (JSON) has a dir_name field that includes the directory. When I run with AUTODISCOVER_DIR set to False, the full path is shown (this is the desired behavior). When I run with AUTODISCOVER_DIR set to True, the relative path is shown. Can you ensure that when it gets stored in the output, it is resolved so that the user can find the files afterwards?

Note that this is also a relevant concern with the logging, where it does things like

INFO:quacc.runners.prep:Moving C:\Users\asros\Desktop\tmp-bulk_to_slabs_flow-2026-02-22-01-32-35-651667-90541\bulk_to_slabs_subflow-2026-02-22-01-32-35-652280-77098\relax_job-2026-02-22-01-32-35-825258-48557 contents to bulk_to_slabs_flow-2026-02-22-01-32-35-651667-90541\bulk_to_slabs_subflow-2026-02-22-01-32-35-652280-77098\relax_job-2026-02-22-01-32-35-825258-48557

from quacc.recipes.emt.core import relax_job
from ase.build import bulk

atoms = bulk("Cu")
output = relax_job(atoms)
print(output["dir_name"])

Tmp Path Naming on Failed Jobs

There is now a consideration that we did not need to think about before. Namely, if someone runs a flow that launches 10 independent and concurrent jobs, they might (or might not) be fine with 9/10 finishing depending on whether the data for the 9/10 might still be useful.

As it stands right now, the parent flow will stay in the scratch directory with the `tmp- name if 1/10 jobs fails. Admittedly, I am not sure really what to do about this and am open to suggestions. I will need to think on this some more too.

from quacc.recipes.emt.slabs import bulk_to_slabs_flow
from ase.build import bulk

atoms = bulk("Xe")
bulk_to_slabs_flow(atoms)

[Andrew-S-Rosen]

Can we make this False instead?

[Andrew-S-Rosen]

Can we change to False?

[Andrew-S-Rosen]

Looking at the diff, I'm a bit confused where the if settings.CREATE_UNIQUE_DIR business went. Can you explain?

[vineetbansal]

On main, CREATE_UNIQUE_DIR is checked in both functions:

def calc_setup:                                                                                                                            
  job_results_dir = settings.RESULTS_DIR.resolve()
  if settings.CREATE_UNIQUE_DIR:                                                                                                                
      job_results_dir /= f"{tmpdir.name.split('tmp-')[-1]}"

def calc_cleanup (main):
  if settings.CREATE_UNIQUE_DIR:
      move(tmpdir, job_results_dir)
  else:
      for file_name in os.listdir(tmpdir):
          move(tmpdir / file_name, job_results_dir / file_name)
      rmtree(tmpdir)

When CREATE_UNIQUE_DIR=True, job_results_dir doesn't exist yet, so move(tmpdir, job_results_dir) works as a rename.

On this branch, CREATE_UNIQUE_DIR is only checked in calc_setup. In calc_cleanup, we do:

def calc_cleanup:
  job_results_dir.mkdir(parents=True, exist_ok=True)
  for file_name in os.listdir(tmpdir):
      move(tmpdir / file_name, job_results_dir / file_name)

Once mkdir pre-creates job_results_dir (the mkdir call is needed for the AUTODISCOVER_DIR case where job_results_dir is a deep nested path), move(tmpdir, job_results_dir) would move tmpdir inside job_results_dir rather than as it. So we don't do that, and move its contents to job_results_dir instead.

So this branch collapses both cases into a single strategy as far as calc_cleanup is concerned, and CREATE_UNIQUE_DIR only needs to live in calc_setup. Effectively we're still moving everything inside tmpdir to job_results_dir.

Some newly introduced tests in tests/core/wflow/test_autodiscover_paths.py::test_results_dir_safe illustrate this.

[Andrew-S-Rosen]

I think this comment might need updating.

[vineetbansal]

@Andrew-S-Rosen. For the two comments you made:

Directory in Output

Fair point - this is fixed by returned the .resolve()ed path to the caller in calc_setup, which I'm doing now. The old code was already returning resolved paths.

Tmp Path Naming on Failed Jobs

With NESTED_RESULTS=False and SCRATCH_DIR set, a failure results in:

scratch/failed-quacc-<uuid>/

With NESTED_RESULTS=True, a failure leaves:

scratch/tmp-<flow>-ts/.../failed-<job>-ts/

In either case, stuff stays in scratch when a job fails. The two minor differences are:

• Extra layers of parent directories around the failed job directory. This can only be a good thing, since it provides additional information about what failed exactly.
• the tmp- prefix on the parent dir name, which might look odd. But it could be argued that it is indeed temporary (because its in the SCRATCH_DIR but we chose not to delete it in case it's worth a review). It can be nuked if the user sees it out of context and is not going to be doing anything with it.