Skip to content

Make queue-executed ASE (UMA/fairchem) jobs runnable on a cluster - #985

Open
alongd wants to merge 3 commits into
mainfrom
queue-executed-ase-adapter
Open

Make queue-executed ASE (UMA/fairchem) jobs runnable on a cluster#985
alongd wants to merge 3 commits into
mainfrom
queue-executed-ase-adapter

Conversation

@alongd

@alongd alongd commented Aug 15, 2026

Copy link
Copy Markdown
Member

What

Make queue-executed ASEAdapter jobs actually run on a remote cluster. The ASE adapter had only ever been exercised incore; its queue path was latent and broke on the first real submission. This lets ASE-backed calculators (e.g. UMA / fairchem MLIPs) run hindered-rotor and other scans as ordinary PBS/Slurm jobs.

Why

Running a real ARC job that delegates its hindered-rotor scans to an ASE calculator on a cluster queue crashed the moment the first scan job was dispatched:

arc.exceptions.InputError: Cannot upload a non-existing file.
Check why file in path .../directed_scan_a2781/submit.sh is missing.

Two independent defects in the queue path, both masked because ASE had only ever run incore:

  1. Files were never written. JobAdapter.execute() calls upload_files() before execute_queue(), and _initialize_adapter() calls set_files() while the job is constructed. The Gaussian/Orca/xTB adapters write submit.sh / input inside set_files(); ASEAdapter.set_files() only appended their names to files_to_upload and never wrote them, so the calcs dir was empty when ssh.upload_file() ran its os.path.isfile pre-flight.
  2. Submission was dead code. ASEAdapter.execute_queue() guarded on self.server_adapter, an attribute nothing in ARC sets (hasattrFalse), so even with the files present the job would never submit — the write fix alone just moves the crash one line down to an AttributeError.

Changes (confined to the ASE adapter)

  • set_files() now writes submit.sh and input.yml for non-incore jobs, mirroring the other adapters. Incore is unchanged — it still writes its input in execute_incore().
  • execute_queue() now delegates to the shared legacy_queue_execution() that every other adapter uses (which also records job_status / job_id), replacing the bespoke never-reached block.
  • write_submit_script()'s non-queue branch falls back to local_path when remote_path is None (a server-less job), fixing a latent crash surfaced by writing at construction time.
  • Adds test_set_files_writes_the_files_of_a_queue_job (red against the pre-fix adapter).

Verification

  • New/updated unit tests: pytest arc/job/adapters/ase_test.py -q -n010 passed (the file requires serial -n0, a pre-existing xdist setUpClass race unrelated to this change).
  • Incore path unregressed: full execute()execute_incore()ase_script.pyparse_results() reproduces the prior geometry/energy to 5 decimals.
  • Live cluster run: an ARC job (DFT opt/freq/sp on the queue, hindered-rotor scans via a UMA/ASE calculator) now runs end-to-end — the DFT stage completes and the rotor jobs are written with their submit.sh + input.yml and submitted to the queue, where the pre-fix run crashed.

The diff is limited to arc/job/adapters/ase_adapter.py, arc/job/adapters/ase_test.py, and a 1-indexed→0-indexed constraint fix in arc/job/adapters/scripts/ase_script.py (ASE FixInternals is 0-indexed; ARC constraints are 1-indexed). No other adapter's submit path is touched.

Comment thread arc/job/adapters/ase_adapter.py Dismissed
@codecov

codecov Bot commented Aug 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.17%. Comparing base (9f41176) to head (366c5d0).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #985      +/-   ##
==========================================
- Coverage   64.17%   64.17%   -0.01%     
==========================================
  Files         119      119              
  Lines       39554    39584      +30     
  Branches    10264    10269       +5     
==========================================
+ Hits        25385    25403      +18     
- Misses      11199    11204       +5     
- Partials     2970     2977       +7     
Flag Coverage Δ
functionaltests 64.17% <ø> (-0.01%) ⬇️
unittests 64.17% <ø> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes the previously-latent “queue execution” path for ASEAdapter so ASE-backed calculators (e.g., UMA/fairchem MLIPs) can run as real cluster jobs, including rotor/directed-scan workloads that ARC dispatches via the scheduler.

Changes:

  • Write submit + input.yml during adapter construction for non-incore ASE jobs so upload_files() succeeds.
  • Replace ASE’s bespoke queue submission logic with the shared legacy_queue_execution() flow.
  • Correct ASE internal constraint indexing by translating ARC’s 1-indexed constraints to ASE FixInternals’ 0-indexed expectations.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
arc/job/adapters/ase_adapter.py Writes queue job files in set_files(), adds directed-scan constraints translation, and delegates queue submission to legacy_queue_execution().
arc/job/adapters/ase_test.py Adds unit coverage asserting queue-job construction writes/uploadables exist on disk (and that incore does not).
arc/job/adapters/scripts/ase_script.py Converts constraint atom indices from ARC’s 1-indexing to ASE’s 0-indexing before applying FixInternals.
Suppressed comments (1)

arc/job/adapters/ase_adapter.py:415

  • This method always writes the submit script to submit.sh. On Slurm, ARC submits submit.sl (see settings['submit_filenames']), so Slurm jobs will still error unless the script is written under the Slurm filename.
        with open(os.path.join(self.local_path, 'submit.sh'), 'w') as f:
            f.write(content)

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread arc/job/adapters/ase_adapter.py
@alongd

alongd commented Aug 15, 2026

Copy link
Copy Markdown
Member Author

Added a third commit (dc23057): fix a stale directed-scan parser call site — check_directed_scan_job called parser.parse_e_elect(path=...) but the refactored API is log_file_path= (as every other call site uses). Without it, ARC crashes assembling V(φ) the moment a directed-scan job's energy is parsed. Verified by parsing a real UMA rotor output (−500058.16 kJ/mol). This is needed for ASE/UMA rotor scans to run end-to-end.

alongd added 3 commits August 15, 2026 21:47
ASEAdapter.write_submit_script() emitted a bare two-line bash script: no
scheduler directives, no queue, no environment activation, and the ARC
host's conda python path, which does not exist on the server. It never
reached ARC's submit_scripts templates, so there was no {env_setup} hook
to populate either. It now composes a PBS or Slurm script from
args['block'] (queue, env_setup, gpu_resource, python), pins the thread
pools to the granted core count so torch cannot oversubscribe a shared
node, and stamps initial_time/final_time so ARC can report a run time.
Incore jobs keep the bare script.

Directed scans were also not constrained: Scheduler.run_job() always
passes constraints=None and hands the adapter torsions + dihedrals
instead, so every point of a brute_force_opt scan optimized freely and
relaxed to the same minimum. ASEAdapter.determine_constraints() derives
the constraint, and apply_constraints() converts ARC's 1-indexed atom
indices to ASE's 0-indexed FixInternals.

The shared Scheduler/Gaussian side of that second defect is left alone
here; a Gaussian directed_scan job has the same missing constraint and
needs its own fix.
An ASE job submitted to a queue never wrote its submit.sh or input.yml.
set_files() only listed them for upload, but JobAdapter.execute() uploads
before it calls execute_queue(), so the upload died with
"InputError: Cannot upload a non-existing file". Write them in set_files(),
where Gaussian, Orca and xTB write theirs; the incore path is untouched and
still writes its input in execute_incore().

execute_queue() then never submitted anything either: it guarded on
self.server_adapter, an attribute nothing sets. Use legacy_queue_execution(),
as every other adapter does, which also records the job id and status.

Also fall back to the local path when writing a non-queue submit script for a
job with no server, since such a job has no remote path.
check_directed_scan_job called the old parser signature; ARC's refactor made
parse_e_elect = make_parser(...) take log_file_path (as every other call site
uses). Crashed I-009 Arm A after rotor jobs succeeded on Zeus. One-token fix;
verified by parsing a real UMA rotor output (-500058.16 kJ/mol).
@alongd
alongd force-pushed the queue-executed-ase-adapter branch from dc23057 to 366c5d0 Compare August 15, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants