Skip to content

Remove Basilisk from conformance results - #2330

Merged
AlexWaygood merged 1 commit into
python:mainfrom
Nimblesite:remove-basilisk-conformance-results
Aug 5, 2026
Merged

Remove Basilisk from conformance results#2330
AlexWaygood merged 1 commit into
python:mainfrom
Nimblesite:remove-basilisk-conformance-results

Conversation

@MelbourneDeveloper

Copy link
Copy Markdown
Contributor

Reverts #2316. I'm the author of Basilisk, and I'm asking for it to be taken out of the conformance results. Our published score doesn't measure what the table says it measures, so it needs to be removed.

Why

Many of Basilisk's rules match against raw source text and hard-coded typing symbol names instead of resolved symbols on the AST. The result is that our score reflected the exact spelling of the test files rather than conformance to the specification.

A mutation harness makes this unambiguous. It renames imported typing symbols and adjusts whitespace. It should not move any checker's score. Ours moves. I reproduced this myself and 113 of 141 tests are affected. We will investigate exactly how much work is necessary to fix this.

How it happened

Despite our best efforts to enforce accuracy, we relied too heavily on AI to validate the work we did, and we focused too heavily on conformance results instead of thoroughly testing real world scenarios. It looks as though we didn't pick up code that clearly should have only been just a placeholder and that code proliferated. This should have been predictable and I take full responsibility for letting this happen.

I am sorry for breaking trust, especially when the whole aim of Basilisk as a tool is to build trust. We have an absolute commitment to building the trust of the Python community so requesting this removal is a necessary step until we can work through the issues.

Resubmission

I'm asking that you please leave the door open to make another attempt in future. We still have full confidence that Basilisk is a worthy type checker.

Before we submit again, we'll hold ourselves to the test that proved the issues. If a rename or a whitespace change moves our score, the rule isn't implemented and the number doesn't count. We'll publish that measurement alongside any resubmission, and if the honest score is lower, we'll publish the lower score.

We will do a full audit to remove any code that pattern-matches source text. We will track it publicly in our repo. We will take as long as it takes to get this right and not submit again until we've got proof that we're not making the same mistakes again.

Sincere apologies to the maintainers here. You accepted our submission in good faith and our carelessness led to misleading results being posted here.

Remove Basilisk from conformance results

Reverts python#2316, at the request of Basilisk's author.

Some of Basilisk's rules match raw source text and hard-coded typing symbol
names rather than resolved symbols on the AST, so its published score
reflects the spelling of the test files rather than conformance to the
specification. Semantics-preserving changes to the suite — renaming
imported typing symbols, adjusting whitespace — collapse the result.

Removes the basilisk result files, the BasiliskTypeChecker adapter, the
dependency entries, and the README reference, and regenerates
results.html from the remaining checkers. No other type checker's
results are affected.

@AlexWaygood AlexWaygood left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! I really appreciate the honesty and self-reflection here.

I'm asking that you please leave the door open to make another attempt in future. We still have full confidence that Basilisk is a worthy type checker.

I can't speak for the other maintainers of this repo, but I certainly would have no problem with this, assuming the underlying issues here are addressed

@AlexWaygood
AlexWaygood merged commit c43d32e into python:main Aug 5, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants