feat(lockfile): completely remove the use of any 'lock' files - #323
feat(lockfile): completely remove the use of any 'lock' files#323Dan Streetman (ddstreet) wants to merge 13 commits into
Conversation
Remove the manual-bump field from component locks and eliminate the component update --bump workflow that incremented it. Fingerprints, freshness validation, source preparation, history output, and synthetic-history tests now depend only on concrete component inputs and resolved source identity. Rebuilds can no longer be requested by changing lock-only metadata; callers must make an explicit input change that explains the resulting release.
Delete the version field and current-version gate from component lock serialization, parsing, constructors, store behavior, and fixtures. Lock compatibility is now determined by whether the current TOML structure can be decoded. Existing lock workflows no longer reject structurally compatible files because of an independent format integer, and generated lock files contain only state used by component behavior.
Remove the write-once import-commit field and all update, resolver, history, and source-preparation logic that propagated it. Synthetic dist-git reconstruction now walks first-parent history from the configured upstream commit back to the repository root instead of stopping at a lock-recorded fork point. History output no longer reports import state, and lock files no longer preserve import-only metadata.
Delete resolution-input-hash, freshness classifications derived from it, and the --force-recalculate escape hatch used to bypass those classifications. Component update now resolves every selected source identity and compares the provider result directly with stored state. Snapshot, distro, and pin changes no longer require a second hash in the lock, while HEAD-tracking and externally advanced upstream repositories are naturally re-evaluated on every update.
Remove input-fingerprint and the fingerprint package that hashed component configuration, overlays, build settings, distro release data, and local source content into lock state. Locks now carry only upstream source identity. Local components no longer produce locks, overlay and build metadata no longer make a lock stale, component changed compares upstream commits plus rendered sources, and synthetic history is driven by committed upstream-commit transitions rather than arbitrary input hashes.
Delete the lockfile package, validation, drift warnings, manual bump state, and skip-lock-validation command surface. Continue to parse the legacy project lock-dir field while ignoring its value. Write snapshot-selected upstream commits as normal layered TOML under base/upstream-commits by default. Pin files contain only spec.upstream-commit, load before component definitions, and replace stale content when refreshed. They use the same loading, merging, provenance, and validation as all TOML. Merge partial component definitions before semantic validation. Refresh pins only for effective upstream sources, remove generated pins for selected non-upstream components, and support check-only and orphan-pruning behavior. Make render, build, changed detection, component history, and synthetic dist-git history consume configured upstream commits. Track commit provenance independently so later partial layers cannot redirect synthetic history.
Replace the component update entry point with the explicit component refresh-upstream-commit command across Cobra registration, generated CLI documentation, agent skills, tests, scenarios, and fix suggestions. Retain component update as a hidden compatibility command. Ignore all former arguments and flags, perform no project operation, and print a message that the command no longer does anything and should no longer be used. Emit the refresh command in generated TOML markers while continuing to recognize older update marker formats as generated override configuration.
Load and resolve the complete project configuration independently at the --from and --to Git refs instead of reading stored component fingerprints. Resolve recursive includes, merged definitions, inherited defaults, overlay files, and generated commit pins before comparing TOML object models. Compare the remaining build inputs represented by the former fingerprint: normalized component configuration, upstream or local source identity, overlay source content hashes, and effective distro release version. The removed lock-file manual-bump input has no equivalent. Ignore documentation, publishing, test-selection, scheduling hints, snapshot timestamps, checkout paths, and other non-build metadata. Compare rendered sources manifests separately using the directory configured at each ref. Select components from the union of both historical projects so additions and deletions do not depend on the current checkout. Apply component name, group, and spec-path selections across both refs, including explicit group members. Remove the changed command's --upstream-commits-dir flag because normal project includes determine which configuration files participate in the comparison. Expand unit and scenario coverage for the TOML comparison inputs, historical selection, valid generated commit pins, additions, deletions, and MCP metadata.
Remove the component history command registration and its implementation. The command's customization counting, source-TOML git metrics, generated upstream commit metrics, shared-file filtering, and specialized result rendering are no longer exposed or maintained. Delete the command-specific unit and scenario coverage. Regenerate the parent component command documentation and MCP tool snapshot so help, generated CLI reference pages, and tool discovery no longer advertise component history. Retain the reusable synthetic source-history implementation because component build, render, and prepare-sources still use it when producing dist-git history.
Remove the component query command registration and implementation. Component spec parsing remains available through the reusable component and RPM APIs, but is no longer exposed as a dedicated CLI operation. Delete the command-specific batch spec-query processor, embedded Python worker, query-only mock packages, unit tests, and scenario coverage. Keep the shared mock batch runner used by rendering while removing its query-only integration. Regenerate the parent component documentation so command help and the generated CLI reference no longer advertise query or link to its removed reference page.
Add an explicit annotation for commands that must remain registered for compatibility but should not appear in generated Markdown documentation. Honor the annotation even when supported hidden commands are included, and restore each command's original hidden state after generation. Exclude the no-op component update, history, and query compatibility commands. Regenerate the CLI reference to remove their pages and parent-command links, and cover both excluded and supported hidden commands in generator tests.
Document the move from component lock files and fingerprints to generated upstream commit configuration. Describe the refresh command, configuration- based changed detection, synthetic history behavior, removed legacy commands, and updated workflow guidance.
Allow commands to request permissive project configuration loading before Cobra dispatch, and enable it for component refresh-upstream-commit. This lets the command remove stale generated pins that would otherwise invalidate the resolved model after a component is removed or converted to a local source. Add application-level regression coverage for removed and converted components. Update the generated CLI reference, agent workflow guidance, and changelog to document the cleanup behavior and validation warnings.
There was a problem hiding this comment.
Pull request overview
This PR removes the lock-file system from azldev and replaces it with generated, layered per-component TOML that pins resolved upstream commits (written under base/upstream-commits/ by default). It also updates config loading/validation and CLI/docs to reflect the new “refresh upstream commit + render” workflow.
Changes:
- Replace per-component lock files with generated upstream-commit TOML files and add
azldev component refresh-upstream-commit. - Rework config merging/validation to support partial component definitions across included TOML files (including tracking provenance of upstream-commit pins).
- Remove lock/fingerprint/query/history plumbing and update tests, snapshots, schema, and documentation accordingly.
Reviewed changes
Copilot reviewed 123 out of 124 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| schemas/azldev.schema.json | Updates schema around legacy lock-dir handling. |
| scenario/component_query_test.go | Removes scenario coverage for component query (command removed / replaced with legacy shim). |
| scenario/component_history_test.go | Removes scenario coverage for component history (command removed / replaced with legacy shim). |
| scenario/component_e2e_test.go | Updates E2E expectations to use refresh-upstream-commit instead of update. |
| scenario/component_build_test.go | Updates build scenario pre-step to refresh-upstream-commit. |
| internal/utils/git/git.go | Refines comments around first-parent history behavior. |
| internal/upstreamcommit/store.go | Adds store for generated upstream-commit TOMLs (read/write/prune). |
| internal/upstreamcommit/store_test.go | Adds unit tests for upstream-commit TOML store behavior. |
| internal/providers/sourceproviders/* | Removes lock-based precedence and updates identity/checkout tests and docs. |
| internal/projectconfig/* | Removes lock-dir defaults, adds legacy lock-dir compatibility field, adjusts merging/validation, and updates component provenance tracking. |
| internal/lockfile/* | Deletes lockfile store/show code. |
| internal/fingerprint/* | Deletes fingerprint package and related docs/tests. |
| internal/app/azldev/* | Removes lock store from Env; adds command-level permissive-config annotation plumbing; updates docs generation filtering. |
| internal/app/azldev/cmds/component/* | Adds refresh-upstream-commit command, removes query/history/update implementations, and adds hidden legacy no-op shims. |
| internal/app/azldev/agentskill/* | Replaces “update-component/lock” skill content with refresh-upstream-commit guidance and updates bindings. |
| go.mod / go.sum | Removes hashstructure dependency. |
| docs/user/** | Updates user docs to describe generated upstream-commit TOMLs and include ordering. |
| CHANGELOG.md | Documents the lockfile removal and new upstream-commit workflow. |
| .github/instructions/* | Updates testing/go instructions to remove lockfile guidance. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| // DeprecatedLockDir accepts the legacy [project.lock-dir] field for backward | ||
| // compatibility. Lock files are no longer used, so this value is ignored. | ||
| DeprecatedLockDir string `toml:"lock-dir,omitempty" json:"-" jsonschema:"deprecated=true,description=Deprecated and ignored. Lock files are no longer used."` |
|
To add more context: The 'lock' files contain these fields:
The The The 'import-commit' field is used only in changelog generation, and as the PR description says, this doesn't attempt to fix changelog generation which is already partly/mostly broken; that will need a follow-up PR, and won't need the info this field provides. The 'upstream-commit' field is moved into standard TOML files, and is managed by the azldev The 'input-fingerprint' is used by azldev comp changed to detect if a component has changed between commits; this is replaced by standard toml parsing of the complete object model at each commit, and a simple object model component comparison between the commits, using only the specific fields that were previously used in the generation of this hash value. The 'resolution-input-hash' is used to determine if the inputs into an upstream commit have changed. This is no longer needed because the upstream commit hash is (and always has been, actually) the actual 'source of truth' about the upstream commit. With all those replaced or dropped, we have to generate all the upstream commit hash values in I'll add detail about the release values in the next comment. |
|
This doesn't do anything to modify the logic that updates/sets the release value in rendered dist-git dirs; that logic does need a separate PR to ensure it does everything completely correctly. Additionally, the separate PR needs to handle correct changelog generation, to fill our specific changelog needs/goals as well as matching the changelog entries with our actual component EVR values for released packages. I have not started that work yet, as I want to get feedback on this first, since it's already up to: I will now start working on it (using copilot), but I don't want to proceed too far into its implementation without feedback on this PR. If this is not something we think can/should get merged, then follow-on work will either be unneeded and/or implemented completely differently. |
|
Finally, I believe we need a (textual, e.g. RFCish-format) specification for the details of the process we use to (get and) convert upstream dist-git repos into local dist-git dirs (i.e. our I think we must have a specification that covers (at a level expected by a RFC-like specification) exactly how to fully parse all our TOML files into an object model; exactly what all parts of that object model represent (e.g. valid field names, type of value, object/field usage, etc); and exactly what the process is for using our object model to convert a dist-git repo (or a local dist-git dir, as some of our components are) into a fully-processed dist-git dir. This part is, IMHO, absolutely critical and should initially be crafted based on the current (or post-merge of this and following PRs) implementation of the |
|
Thanks for sharing this, Dan Streetman (@ddstreet). As you call out, there are also changes needed to rendering process, My recommendation would be to introduce a feature flag (or more if needed for decoupled features) that guard sthe behavioral changes. For example, perhaps a |
Warning: this PR was entirely generated using AI. I have not looked at any of the code that's in this PR, I have only interacted with copilot to generate and test all the contained commits.
The intention of this is to completely remove the use of any 'lock' files, and instead place each component's default upstream-commit hash (i.e. the hash for the current upstream repo's branch at the specified 'snapshot' date).
Note that I made no attempt to fix any of the changelog generation issues; my reasoning is that changelog generation currently doesn't work (at least not for many/most components), so fixing it should be done in a separate PR, not here.
Personally, if I had to review this, I would not know where to begin, as the quantity of changes is (IMHO) enormous. However, much of the changes are documentation or comments, and this codebase already is (IMHO) enormous, so this PR might not be unusual for this code; I honestly do not know.
Please feel free to provide completely unfiltered feedback to this PR.