Make queue-executed ASE (UMA/fairchem) jobs runnable on a cluster - #985
Make queue-executed ASE (UMA/fairchem) jobs runnable on a cluster#985alongd wants to merge 3 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #985 +/- ##
==========================================
- Coverage 64.17% 64.17% -0.01%
==========================================
Files 119 119
Lines 39554 39584 +30
Branches 10264 10269 +5
==========================================
+ Hits 25385 25403 +18
- Misses 11199 11204 +5
- Partials 2970 2977 +7
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
This PR fixes the previously-latent “queue execution” path for ASEAdapter so ASE-backed calculators (e.g., UMA/fairchem MLIPs) can run as real cluster jobs, including rotor/directed-scan workloads that ARC dispatches via the scheduler.
Changes:
- Write
submit+input.ymlduring adapter construction for non-incore ASE jobs soupload_files()succeeds. - Replace ASE’s bespoke queue submission logic with the shared
legacy_queue_execution()flow. - Correct ASE internal constraint indexing by translating ARC’s 1-indexed constraints to ASE
FixInternals’ 0-indexed expectations.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| arc/job/adapters/ase_adapter.py | Writes queue job files in set_files(), adds directed-scan constraints translation, and delegates queue submission to legacy_queue_execution(). |
| arc/job/adapters/ase_test.py | Adds unit coverage asserting queue-job construction writes/uploadables exist on disk (and that incore does not). |
| arc/job/adapters/scripts/ase_script.py | Converts constraint atom indices from ARC’s 1-indexing to ASE’s 0-indexing before applying FixInternals. |
Suppressed comments (1)
arc/job/adapters/ase_adapter.py:415
- This method always writes the submit script to submit.sh. On Slurm, ARC submits submit.sl (see settings['submit_filenames']), so Slurm jobs will still error unless the script is written under the Slurm filename.
with open(os.path.join(self.local_path, 'submit.sh'), 'w') as f:
f.write(content)
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Added a third commit ( |
ASEAdapter.write_submit_script() emitted a bare two-line bash script: no
scheduler directives, no queue, no environment activation, and the ARC
host's conda python path, which does not exist on the server. It never
reached ARC's submit_scripts templates, so there was no {env_setup} hook
to populate either. It now composes a PBS or Slurm script from
args['block'] (queue, env_setup, gpu_resource, python), pins the thread
pools to the granted core count so torch cannot oversubscribe a shared
node, and stamps initial_time/final_time so ARC can report a run time.
Incore jobs keep the bare script.
Directed scans were also not constrained: Scheduler.run_job() always
passes constraints=None and hands the adapter torsions + dihedrals
instead, so every point of a brute_force_opt scan optimized freely and
relaxed to the same minimum. ASEAdapter.determine_constraints() derives
the constraint, and apply_constraints() converts ARC's 1-indexed atom
indices to ASE's 0-indexed FixInternals.
The shared Scheduler/Gaussian side of that second defect is left alone
here; a Gaussian directed_scan job has the same missing constraint and
needs its own fix.
An ASE job submitted to a queue never wrote its submit.sh or input.yml. set_files() only listed them for upload, but JobAdapter.execute() uploads before it calls execute_queue(), so the upload died with "InputError: Cannot upload a non-existing file". Write them in set_files(), where Gaussian, Orca and xTB write theirs; the incore path is untouched and still writes its input in execute_incore(). execute_queue() then never submitted anything either: it guarded on self.server_adapter, an attribute nothing sets. Use legacy_queue_execution(), as every other adapter does, which also records the job id and status. Also fall back to the local path when writing a non-queue submit script for a job with no server, since such a job has no remote path.
check_directed_scan_job called the old parser signature; ARC's refactor made parse_e_elect = make_parser(...) take log_file_path (as every other call site uses). Crashed I-009 Arm A after rotor jobs succeeded on Zeus. One-token fix; verified by parsing a real UMA rotor output (-500058.16 kJ/mol).
dc23057 to
366c5d0
Compare
What
Make queue-executed
ASEAdapterjobs actually run on a remote cluster. The ASE adapter had only ever been exercised incore; its queue path was latent and broke on the first real submission. This lets ASE-backed calculators (e.g. UMA / fairchem MLIPs) run hindered-rotor and other scans as ordinary PBS/Slurm jobs.Why
Running a real ARC job that delegates its hindered-rotor scans to an ASE calculator on a cluster queue crashed the moment the first scan job was dispatched:
Two independent defects in the queue path, both masked because ASE had only ever run incore:
JobAdapter.execute()callsupload_files()beforeexecute_queue(), and_initialize_adapter()callsset_files()while the job is constructed. The Gaussian/Orca/xTB adapters writesubmit.sh/ input insideset_files();ASEAdapter.set_files()only appended their names tofiles_to_uploadand never wrote them, so the calcs dir was empty whenssh.upload_file()ran itsos.path.isfilepre-flight.ASEAdapter.execute_queue()guarded onself.server_adapter, an attribute nothing in ARC sets (hasattr→False), so even with the files present the job would never submit — the write fix alone just moves the crash one line down to anAttributeError.Changes (confined to the ASE adapter)
set_files()now writessubmit.shandinput.ymlfor non-incore jobs, mirroring the other adapters. Incore is unchanged — it still writes its input inexecute_incore().execute_queue()now delegates to the sharedlegacy_queue_execution()that every other adapter uses (which also recordsjob_status/job_id), replacing the bespoke never-reached block.write_submit_script()'s non-queue branch falls back tolocal_pathwhenremote_pathisNone(a server-less job), fixing a latent crash surfaced by writing at construction time.test_set_files_writes_the_files_of_a_queue_job(red against the pre-fix adapter).Verification
pytest arc/job/adapters/ase_test.py -q -n0→ 10 passed (the file requires serial-n0, a pre-existing xdistsetUpClassrace unrelated to this change).execute()→execute_incore()→ase_script.py→parse_results()reproduces the prior geometry/energy to 5 decimals.submit.sh+input.ymland submitted to the queue, where the pre-fix run crashed.The diff is limited to
arc/job/adapters/ase_adapter.py,arc/job/adapters/ase_test.py, and a 1-indexed→0-indexed constraint fix inarc/job/adapters/scripts/ase_script.py(ASEFixInternalsis 0-indexed; ARC constraints are 1-indexed). No other adapter's submit path is touched.