actions: expand to the most specific semver tag

[thepwagner]

Please describe the enhancement

Given a reference like actions/checkout@v3.

I'd prefer the pinned version to be: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.6.0 instead of actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3

This difference makes the result (including the comment) compatible with Dependabot. Dependabot will update the commit of actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3, but leave the comment at v3.

This should only be done when multiple tags reference the same commit.

Solution Proposal

When pinning, list all tags in the repository.
When a commit to be pinned can be resolved to multiple tags, choose the "most specific" tag.

Describe alternatives you've considered

We could wrap frizbee, or use a linter to discourage using major version tags.

This could be WONTFIX, treated as a bug in Dependabot: dependabot/dependabot-core#8011 . (I have not confirmed how RenovateBot handles this case).

Additional context

No response

Acceptance Criteria

1. Have a repository using actions/checkout@v3.
2. Run frizbee to pin the actions in the repository.
3. Enable Dependabot for GitHub Actions: https://docs.github.com/en/code-security/dependabot/working-with-dependabot/keeping-your-actions-up-to-date-with-dependabot
4. Receive a clean pull request upgrading to the latest pinned version (at the time of writing): actions/checkout@692973e3d937129bcbf40652eb9f2f61becf3332 # v4.1.7

[jhrozek]

Sorry that this request went unanswered for such a long time, it simply fell through cracks during the summer holidays!

I think this makes sense, but I feel there might be an option needed to select between the behaviours.

Thanks for filing the issue!

[blkt]

Hey @thepwagner I just wanted to check if this is still relevant.
My guess is that listing all tags in the repository might be expensive, but I might be wrong.

Would you mind opening a PR to address this issue?

[xopham]

I second @thepwagner in their request! Pinning by digest is super critical but only if you do not loose information and updating is simple which in this case means full semver tag versions and dependabot.

[woodruffw]

Apologies for the bump on this issue, but I've also recently run into this -- pinning on a non-specific tag like this appears to render Dependabot ineffective, which leaves repos that use frizbee in a harder-to-maintain default position (since actions are silently ignored for updates and slowly go stale over time).

[p-linnane]

+1 on this. I've been using frizbee to help hash pin workflows, and have found myself having to manually verify commits/tags, which dramatically reduces the efficacy of the tool.

[JAORMX]

Thanks for reaching out folks! You are right and we should solve this. The team will get to this and submit a solution soon

[rdimitrov]

Hey, all 👋 So I've spent a bit of time researching/playing around this issue and these are my findings so far.

The TLDR is I think this is indeed a bug with Dependabot and it might be hard to work around it properly from frizbee.

Details:

What frizbee does so far:

• In the example of actions/checkout@v3, the version is v3, frizbee looks if there's a tag for that version and gets its hash. If there's no such tag, it looks if there's branch named like that and returns its commit hash. If it doesn't find anything, it moves forward and doesn't update the reference. Because tags can be anything, we really look for what hash stands behind the tag and use that for pinning the referenced action.

What Dependabot does so far:

• I haven't checked the code for Dependabot, but from what I found this is indeed a bug on their side (already raised in 2023 but also confirmed by the tests I ran).

Can we work around this?
The suggested approach of listing all tags that point to the same commit, try to compare them and choose the "most specific" one for the comment might not always work.

• The main issue is we cannot ensure version comparison will work deterministically as these tags are not necessarily semver compatible.
• Even if Frizbee did that, for some strange reason Dependabot would still sometimes fail to update the reference/comment correctly (see Test 1 below).

Example test cases

I've ran a few manual tests by following the flow in @thepwagner's acceptance criteria - setting a given version in the workflow file and letting Dependabot update it. It failed for almost all cases which leads me to believe that even if frizbee can come up with the most specific semver tag, Dependabot still hits some bug with a particular versioning since it worked fine for sigstore/cosign-installer but failed for actions/checkout.

• Test 1 - actions/checkout - 🔴 (Expected to get v4-ish, but instead it defaulted to latest main, not the latest available tag/release)

- uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.6.0
+ uses: actions/checkout@85e6279cec87321a52edac9c87bce653a07cf6c2 # v3.6.0

• Test 2 - actions/checkout - 🔴 (defaulted to latest main)

- uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.0.0
+ uses: actions/checkout@85e6279cec87321a52edac9c87bce653a07cf6c2 # v3.0.0

• Test 3 - actions/checkout - 🔴 (defaulted to latest main)

- uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3
+ uses: actions/checkout@85e6279cec87321a52edac9c87bce653a07cf6c2 # v3

• Test 4 - sigstore/cosign-installer - 🟢

- uses: sigstore/cosign-installer@6e04d228eb30da1757ee4e1dd75a0ec73a653e06 # v3.1.1
+ uses: sigstore/cosign-installer@3454372f43399081ed03b604cb2d021dabca52bb # v3.8.2

• Test 5 - sigstore/cosign-installer - 🔴 (defaulted to latest main)

- uses: sigstore/cosign-installer@d7d6bc7722e3daa8354c50bcb52f4837da5e9b6a # v3
+ uses: sigstore/cosign-installer@3454372f43399081ed03b604cb2d021dabca52bb # v3

• Test 6 - sigstore/cosign-installer - 🔴 (defaulted to latest main, definitely not v2)

- uses: sigstore/cosign-installer@c85d0e205a72a294fe064f618a87dbac13084086 # v2
+ uses: sigstore/cosign-installer@3454372f43399081ed03b604cb2d021dabca52bb # v2

That said I'm not sure we can come up with a fix for solving this deterministically. Of course, we all agree this is rather annoying so please share if you have more suggestions/ideas about a work around at least until Dependabot fixes it on their side.

[woodruffw]

Thanks for the response @rdimitrov!

I agree that there's no sound solution here. However, I think frizbee could be more precise here in a way that'll both be more compatible with Dependabot and handle the happy path here.

The suggested approach of listing all tags that point to the same commit, try to compare them and choose the "most specific" one for the comment might not always work.

• The main issue is we cannot ensure version comparison will work deterministically as these tags are not necessarily semver compatible.

In practice, there's a heuristic here that's going to be reliable 99% of the time: get the list of tags and pick the longest. That'll handle the most common mutable tag trick with GitHub Actions, i.e. where both v1 and v1.2.3 point to the same SHA ref.

For context, that's what zizmor does:

https://github.com/woodruffw/zizmor/blob/e7d8899eec0320dd39e0e603c64383d5e022f8da/src/github_api.rs#L246-L268

(This could be made even more precise by additionally filtering on tag names that match v?\d+(\.\d+)*, i.e. tags that roughly resemble semver.)

TL;DR: I agree completely that Dependabot's behavior is incorrect here. I'm curious what you think about the workaround above!

[rdimitrov]

@woodruffw - thanks for the prompt reply! I think your suggestion makes sense and it should be fairly straightforward to add semver validation/parsing on top so we don't just take the longest tag (v1.0 < v1.0.0) but also cover use cases (albeit weird ones) like v1.0.0 and v1.0.1 pointing to the same hash 😃

The 2 concerns that I have with this are:

• From what I saw above unfortunately Dependabot would still fail to bump the comment correctly even in this case where both v3 and v3.6.0 were pointing to the same commit and frizbee did put v3.6.0 in the comment suffix.
• The other one is if maintainers tagged the same commit with v0.1 and v0.1-prerelease and someone put v0.1 in their yaml file this means frizbee will put v0.1-prerelease in the comment part which will be unexpected from the user's point of view and may be considered a bug.

Of course we can always default to the current behavior if we cannot semver-compare all matching tags.

I'll be happy to draft a PR with this, so let me know what you think? 😃

ps. The more I think about it it seems that dependabot fails to work properly when it faces some weird mixture of tags (v3, v3.0.0 and so forth) and decides to play it safe by defaulting to the latest commit instead of trying to guess the next version bump.

[woodruffw]

• From what I saw above unfortunately Dependabot would still fail to bump the comment correctly even in this case where both v3 and v3.6.0 were pointing to the same commit and frizbee did put v3.6.0 in the comment suffix.

Yeah, I'm at a loss to explain that behavior on Dependabot's side 😅 -- your theory about the fallback makes sense to me, however.

• The other one is if maintainers tagged the same commit with v0.1 and v0.1-prerelease and someone put v0.1 in their yaml file this means frizbee will put v0.1-prerelease in the comment part which will be unexpected from the user's point of view and may be considered a bug.

I think it'd be fair to consider this an "unhappy path" case, insofar as there (hopefully...) aren't too many repos doing wonky things like that. In terms of fixing it more generally, however, I think the general solution would be:

• Resolve all tags corresponding to a SHA ref
• Deduplicate in favor of longer tags (e.g. v1.0.0 instead of v1)
• Select the semver-highest tag (v0.1.0 > v0.1-prerelease)

That's a bit of a pain though, so I do think it would be fair to keep the current behavior in cases where the set of tags isn't trivially unambiguous (and maybe spit out a warning in that case?)

I'll be happy to draft a PR with this, so let me know what you think? 😃

Thanks, I'd be happy to!

[rdimitrov]

That sounds great! If you're volunteering, we definitely welcome external contributions 😃 Otherwise, I'm happy to take a stab at it next week (I'm on PTO until Wednesday) 😃