Skip to content

hash/maphash: add a TinyGo version of the package - #5707

Open
davecheney wants to merge 1 commit into
tinygo-org:devfrom
davecheney:maphash-go127
Open

davecheney wants to merge 1 commit into
tinygo-org:devfrom
davecheney:maphash-go127

Conversation

@davecheney

@davecheney davecheney commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

On Go 1.27, TinyGo cannot compile any program that imports hash/maphash, directly or through a dependency:

$ tinygo run maphash.go
# internal/runtime/maps
/usr/lib/go/src/internal/runtime/maps/group.go:298:39: undefined: abi.MapType
/usr/lib/go/src/internal/runtime/maps/map.go:290:17: undefined: abi.MapGroupSlots
/usr/lib/go/src/internal/runtime/maps/table.go:1032:32: undefined: abi.MapGroupSlotsBits
...

What was wrong

TinyGo always sets the purego build tag (compileopts/config.go:114). Up to Go 1.26 that silently selected maphash_purego.go, which avoids internal/runtime/maps. Go 1.27 deleted that file, so maphash.go now imports internal/runtime/maps for every build, and that package needs abi.MapType, abi.MapGroupSlots and abi.MapGroupSlotsBits. TinyGo's internal/abi cannot provide them, because abi.Type is deliberately an empty struct: TinyGo represents types in its own way.

How it was fixed

A TinyGo src/hash/maphash override, as is already done for unique and reflect. It keeps the Go 1.27 API and documentation, and reroutes the three places that reach into runtime internals:

upstream Go 1.27 this override
byte hash runtime.memhash, with maps.Use64BitHash runtime.memhash, with a local use64BitHash
seed runtime.rand unchanged
comparableHash abi.TypeOf and abi.MapType.Hasher a new runtime.comparablehash, which wraps hashmapInterfaceHash

hashmapInterfaceHash is the hasher that TinyGo maps already use for interface keys, so this adds no new hash code to the binary. The change also adds the pathsToOverride entry, a trimmed copy of the upstream tests, and hash/maphash in TEST_PACKAGES_FAST.

Two things worth knowing

  1. This also corrects a second defect. internal/abi.EscapeNonString is a stub that calls panic("intrinsic") and has no compiler implementation, and Go 1.26+ maphash.Comparable calls it. Comparable thus panicked on TinyGo also on Go 1.26, where the package did compile. The override does not call it, because TinyGo does not move stacks.
  2. A test failure showed a seed defect. The TinyGo FNV hash seeds with result *= uint32(seed), so a seed of 0 makes the hash 0, and int(0) and struct{}{} then collide. The two seed halves in comparableHash are made odd. The upstream test file thus needs no change to its logic.

Known deviation

TinyGo gives NaN the same hash every time. This is deliberate, see hashmapFloat64Hash in src/runtime/hashmap.go. The upstream cases that require two NaN values to hash differently are thus removed, and the test header says so. To change this would change how TinyGo maps behave.

Verified

check result
tinygo run on the reproducer works
tinygo test hash/maphash, amd64 pass
tinygo test hash/maphash, GOARCH=386 pass, this is the 32-bit use64BitHash == false branch
-target=wasm, -target=wasip1 compile
-target=microbit links, 49 kB flash
hash, hash/adler32, hash/crc64, hash/fnv pass, through the new merge directory
go test ./compileopts ./goenv pass
gofmt clean

The wasm tests did not run, because node, wasmtime and wasmer are not installed here. 386 thus takes the place of the 32-bit run time path.

This is item 1 of the eight gaps listed in #5684. The other seven are not touched here.

## What was wrong

TinyGo always sets the `purego` build tag (`compileopts/config.go:114`). Up to Go 1.26 that silently selected `maphash_purego.go`, which avoids `internal/runtime/maps`. Go 1.27 deleted that file, so `maphash.go` now imports `internal/runtime/maps` for every build, and that package needs `abi.MapType`, `abi.MapGroupSlots` and `abi.MapGroupSlotsBits`, which TinyGo's `internal/abi` cannot provide.

## How it was fixed

A TinyGo `src/hash/maphash` override, as is already done for `unique`. It keeps the Go 1.27 API and documentation, and reroutes three things:

- `rthash` to the existing `runtime.memhash`
- the seed to `runtime.rand`
- `comparableHash` to a new five line `runtime.comparablehash`, which wraps `hashmapInterfaceHash`, the hasher that TinyGo maps already use for interface keys

This change also adds the `pathsToOverride` entry, a trimmed copy of the upstream tests, and `hash/maphash` in `TEST_PACKAGES_FAST`.

## Two things worth knowing

1. This also corrects a second defect. `internal/abi.EscapeNonString` is a stub that calls `panic("intrinsic")` and has no compiler implementation, and Go 1.26+ `maphash.Comparable` calls it. `Comparable` thus panicked on TinyGo also on Go 1.26, where the package did compile. The override does not call it, because TinyGo does not move stacks.
2. A test failure showed that the TinyGo FNV hash seeds with `result *= uint32(seed)`, so a seed of `0` makes the hash `0`. `int(0)` and `struct{}{}` then collide. The two seed halves in `comparableHash` are made odd, instead of a change to the test, so the upstream test file needs no change to its logic.

## Known deviation

TinyGo gives `NaN` the same hash every time. This is deliberate, see `hashmapFloat64Hash` in `src/runtime/hashmap.go`. The upstream cases that require two `NaN` values to hash differently are thus removed, and the test header says so. To change this would change how TinyGo maps behave.

## Verified

- `tinygo run` works on the reproducer
- `tinygo test hash/maphash` passes on `amd64` and on `GOARCH=386`, which is the 32-bit `use64BitHash == false` branch
- `-target=wasm` and `-target=wasip1` compile, and `-target=microbit` links at 49 kB flash
- `hash`, `hash/adler32`, `hash/crc64` and `hash/fnv` still pass through the new merge directory
- `go test ./compileopts ./goenv` pass, and `gofmt` is clean
- the `wasm` tests did not run because `node`, `wasmtime` and `wasmer` are not installed, so `386` takes the place of the 32-bit run time path

Updates tinygo-org#5684
@deadprogram

Copy link
Copy Markdown
Member

Thank you @davecheney for working on this. The following is edited from an automated review.

  1. src/hash/maphash/maphash.go:360comparableHash calls runtime.comparablehash, which is hashmapInterfaceHash (src/runtime/hashmap.go:782). That function returns a plain 0 and ignores the seed for a nil interface (line 785), a struct with no fields, and a zero-length array. The | 1 on the seed halves does not help, because those paths never read the seed. So Comparable(seed, struct{}{}) gives 0 for every seed. writeComparable then assigns that result straight to the state, so WriteComparable(&h, struct{}{}) sets h.state.s to 0 and every later write on that Hash uses seed 0:
var h1, h2 maphash.Hash
h1.SetSeed(s1) // two different seeds
h2.SetSeed(s2)
maphash.WriteComparable(&h1, struct{}{})
maphash.WriteComparable(&h2, struct{}{})
h1.WriteString("abc")
h2.WriteString("abc")
// h1.Sum64() == h2.Sum64(), although the seeds differ

TestComparable and TestWriteComparable cannot catch this, because they only compare a value against itself, so both sides change in the same way. One possible repair is to mix the seed on the zero-field paths in runtime.comparablehash. This changes map iteration order for those key types only, not map behavior.

  1. src/hash/maphash/maphash.go, rthash has the same zero-seed defect. The 32-bit branch calls runtime_memhash with uintptr(uint32(seed)) and uintptr(seed>>32). hash32 starts with result *= uint32(seed) (src/runtime/memhash_fnv.go:14), so a zero half gives 0 for all data, and the next state half comes from that same half, which keeps it at 0. The 64-bit branch has the same property with hash64. The probability is low, but the fix is the same one that comparableHash already uses.

  2. src/hash/maphash/maphash.go, comparableHash the value goes into any, so TinyGo boxes it. Upstream passes abi.NoEscape(unsafe.Pointer(&v)) and does not allocate. The runtime side goes through reflectlite, so escape analysis cannot remove the allocation. This gives Comparable and WriteComparable one heap allocation per call on microcontrollers. TinyGo maps do not pay this, because they use hashmapInterfacePtrHash with a pointer (src/runtime/hashmap.go:840). Is a pointer-based runtime entry possible here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants