Fix gaussian galloc underallocation - #960
Draft
calvinp0 wants to merge 2 commits into
Draft
Conversation
A `galloc: could not allocate memory.` line only appears in a Gaussian output
file, and a Gaussian output file only exists if the job started, which means the
scheduler granted the reservation it asked for. galloc therefore always means
Gaussian was given too little memory, never too much: it asked the OS for what
%mem promised and did not get it. ARC's observable over-allocation signal is a
different one - the scheduler job log ("using less than N percent of requested"),
classified by determine_job_log_memory_issues as "Memory requested is too high"
and already excluded from this branch by the `'too high' not in error` guard.
determine_ess_status called it 'Memory allocation failed (did you ask for too
much?)'. That question has the sign backwards, and it is what led a later change
to invert the remedy and reduce the request. The remedy itself - double the total
memory, bounded by servers[server]['memory'] * job_max_server_node_memory_
allocation - is right and is unchanged.
What was wrong at the cap was only the advice. "Use a higher-memory node" blames
a node that was never the constraint: servers[server]['memory'] is a deliberate
policy cap on how large one job may grow, so reaching it is the policy working,
and moving the job to a physically larger node changes nothing because ARC would
still request the same amount. Report that the job needs more than the configured
cap and name the three levers that actually move it: servers[<server>]['memory'],
job_max_server_node_memory_allocation, or a cheaper method or level of theory.
Nothing here trades cpu cores for memory per core. Gaussian's %mem is one total
pool for the job: GaussianAdapter.set_input_file_memory derives it from
submit_script_memory_mib alone, with no cpu_cores term, and cpu_cores reaches
only %NProcShared. Halving the cores leaves %mem identical and resubmits the
same failing job. Orca is the opposite case - it writes %maxcore, which is
per-core, which is why its own trsh branch may legitimately trade cores.
Escalation from 42 GB on 12 cores of a 240 GB server (cap 228.00 GB):
84 GB, 168 GB, 228 GB, then give up naming the cap and the three levers, with
the cpu core count untouched throughout.
The galloc.out fixture is a real Gaussian log of the failure, taken from
arcbench's 3945043; cherry-picking this commit onto arcbench will conflict on
that file.
set_cpu_and_mem clamps an over-large request to max_mem * job_max_server_node_memory_allocation and then, because that value is exactly the threshold it was just compared against with a strict `>`, falls through to the uncapped branch on every subsequent call. The clamped job is therefore given DEFAULT_JOB_MEMORY_OVERHEAD (1.10) instead of CAPPED_JOB_MEMORY_OVERHEAD (1.05), so the submit script asks the scheduler for 0.95 * 1.10 = 104.5% of the memory the server is configured to have, and the 'max_total_job_memory' keyword is never appended - making that keyword unreachable for any job that arrived at the cap through the troubleshooting loop, which is the only way a job gets there. trsh_ess_job clamps Gaussian memory to exactly this value, so the boundary is hit by ordinary troubleshooting rather than by an unusual input. On the 256 GB server2 fixture the request drops from 273941 MiB (267.5 GiB, more than the node has) to 261489 MiB (255.4 GiB). Only the Orca branch of trsh_ess_job reads 'max_total_job_memory'; the Gaussian branch carries it without acting on it, so newly emitting it changes no troubleshooting decision.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #960 +/- ##
==========================================
+ Coverage 63.54% 63.59% +0.05%
==========================================
Files 115 115
Lines 38696 38696
Branches 10106 10106
==========================================
+ Hits 24588 24610 +22
+ Misses 11154 11138 -16
+ Partials 2954 2948 -6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request improves the handling and testing of Gaussian memory allocation failures in job execution and troubleshooting. It ensures that jobs hitting the configured memory cap are handled more clearly, updates error messages for clarity, and adds comprehensive tests to verify the new behavior.
Memory allocation handling improvements:
set_cpu_and_memto cap job memory when it reaches (not just exceeds) the configured per-node memory allocation limit, and updated the warning message for clarity.determine_ess_statusandtrsh_ess_jobto clearly indicate when Gaussian cannot allocate the requested memory and that the job has hit the configured memory cap, including guidance on how to resolve the issue. [1] [2]Testing enhancements:
test_trsh_ess_job_gaussian_galloc_escalates_memory_to_the_cap) to verify that jobs escalate memory requests up to the cap and then fail gracefully with the correct error when the cap is reached.gaussian/galloc.out) to support the new tests for memory allocation failures.Test setup improvements:
adapter_test.pyto importdefault_job_settingsand use it in memory cap calculations for better accuracy and maintainability. [1] [2]