Validate serialized sketches and preserve capacity in a versioned format

CHANGELOG.md

@@ -11,4 +11,12 @@
 
 ### Release compatibility
 
-Runtime minimum becomes Node 6 because Buffer.alloc is now used; test development requires modern Node (CI: 22/24/26). Dropping previously advertised Node 0.6 support requires a major release. No version has been bumped yet. Top-k tie ordering is unspecified and may change. The existing binary format is retained; it does not record the original top-k capacity when a sketch is serialized before filling, so that capacity cannot be fully recovered. Malformed-input/deserialization validation remains a follow-up before a release is considered complete.
+Runtime minimum becomes Node 6 because Buffer.alloc is now used; test development requires modern Node (CI: 22/24/26). Dropping previously advertised Node 0.6 support requires a major release. No version has been bumped yet. Top-k tie ordering is unspecified and may change. The follow-up below adds versioned serialization and validates malformed inputs. See SERIALIZATION.md for legacy import/export and the major-release migration.
+
+## Release validation follow-up
+
+- Validate construction probabilities, allocation bounds, keys and exact serialized byte windows; reject malformed data before unbounded allocation.
+- Preserve CountMinSketch capacity in CMS2 output, read legacy bytes, and offer explicit legacy capacity/import and legacy export options. See SERIALIZATION.md.
+- Reject overflowing counters atomically and validate HLL merges before mutation.
+- Fix the signed-minimum hash bucket edge case and retain the existing mapping for all other hashes.
+- Add legacy golden fixtures and tests for truncation, malformed metadata, invalid Unicode, capacity preservation and overflow.

SERIALIZATION.md

@@ -0,0 +1,24 @@
+# Serialization and migration
+
+CountMinSketch writes the versioned **CMS2** format by default. Readers automatically accept the old untagged layout. Integers are unsigned 32-bit little-endian, except legacy key lengths (one byte).
+
+CMS2: magic bytes `CMS2`, original `maxEntries`, log2(width), depth, width, `depth * width` row-major counters, hash count (equal to depth), that many odd hash multipliers, entry count, then each entry's count, UTF-8 byte length and key bytes. Each key length is uint32. Legacy omits the first eight bytes and uses uint8 key lengths.
+
+To write bytes for an old reader, use `sketch.serialize({ legacy: true })`. This rejects keys longer than 255 UTF-8 bytes and cannot preserve capacity. New files cannot be read by old package versions.
+
+Old files do not contain capacity. Import them with `CountMinSketch.deserialize(buffer, start, length, { maxEntries: originalCapacity })` if known. If omitted, legacy capacity defaults to the stored entry count, or one for an empty sketch. A supplied capacity must be at least the stored entry count. Versioned files preserve capacity automatically and reject conflicting overrides.
+
+HyperLogLog retains its layout: uint32 `32 - log2(registerCount)`, float64 scale factor, then uint32 registers. Legacy files with 2/4/8 registers remain readable, although new counters allocate at least 16. Saturation at the limit of the 32-bit hash universe reports Infinity rather than NaN.
+
+Readers honor the exact `(start, length)` byte window; omitted length means the rest of the buffer after start. Trailing or truncated bytes, invalid dimensions, invalid UTF-8, duplicate keys, inconsistent entry counts and impossible registers reject synchronously. Buffers from concatenated streams must be passed with their exact length.
+
+## Bounds
+
+- Probabilities/error rates must be finite numbers strictly between zero and one. Factory defaults apply only when an argument is omitted.
+- CMS: 1–1,048,576 capacity; 1–64 rows; power-of-two width from 4–1,048,576; no more than 4,194,304 cells in total.
+- HLL: newly constructed register arrays contain 16–1,048,576 registers.
+- Keys must be valid Unicode strings, at most 1,048,576 UTF-8 bytes. Empty strings are allowed.
+- Serialized buffers are limited to 64 MiB. Limits are checked before data-dependent allocation.
+- Counts cannot exceed uint32. An increment that would overflow rejects before changing state.
+
+These validation rules and the default CMS2 output are breaking changes and belong in the next major release. The legacy hash mapping is retained, with the signed-minimum bucket calculation corrected so it cannot produce a negative array index.

index.js

@@ -1,4 +1,5 @@
 var HyperLogLog = require('./lib/hyperLogLog');
+var valid = require('./lib/validation');
 var CountMinSketch = require('./lib/countMinSketch');
 
 exports.createUniquesCounter = createUniquesCounter;
@@ -20,7 +21,7 @@ exports.PRNG = require('./lib/prng');
  *        tradeoff. 0.01 is the default.
  */
 function createUniquesCounter(stdError) {
-  return new HyperLogLog(stdError || 0.01);
+  return new HyperLogLog(stdError === undefined ? 0.01 : stdError);
 }
 
 /**
@@ -40,7 +41,7 @@ function createUniquesCounter(stdError) {
  *                 tradeoff. 0.0001 is the default.
  */
 function createViewsCounter(topEntryCount, errFactor, failRate) {
-  return new CountMinSketch(topEntryCount, errFactor || 0.002, failRate || 0.0001);
+  return new CountMinSketch(topEntryCount, errFactor === undefined ? 0.002 : errFactor, failRate === undefined ? 0.0001 : failRate);
 }
 
 /**
@@ -50,21 +51,19 @@ function createViewsCounter(topEntryCount, errFactor, failRate) {
  * numbers.
  */
 function getUniquesObjSize(stdError) {
-  var acc = 1.04 / stdError;
-  var k = Math.ceil(Math.log(acc * acc) / Math.LN2);
-  return 12 + Math.pow(2, k) * 4;
+  return 12 + valid.hllLayout(stdError === undefined ? 0.01 : stdError) * 4;
 }
 
 /**
  * Returns the serialized size of a views counter (CountMinSketch) object in
  * bytes given an errFactor and failRate. NOTE: This does not include the size
  * of the serialized MinHeap which includes the size of each unique ID (up to a
- * max of topEntryCount) plus 5 bytes overhead per entry. NOTE2: The memory
+ * max of topEntryCount) plus 8 bytes overhead per entry. NOTE2: The memory
  * usage will be higher than this number since we serialize 32-bit integers but
  * JavaScript uses 64-bit numbers.
  */
 function getViewsObjSize(errFactor, failRate) {
-  var depth = Math.max(Math.ceil(Math.log(1.0 / failRate)), 1);
-  var width = Math.pow(2, Math.ceil(Math.log(Math.ceil(Math.E / errFactor)) / Math.LN2));
-  return 4 + 8 + depth * width * 4 + 4 + depth * 4 + 4;
+  var layout = valid.cmsLayout(errFactor === undefined ? 0.002 : errFactor,
+                              failRate === undefined ? 0.0001 : failRate);
+  return 28 + layout.depth * layout.width * 4 + layout.depth * 4;
 }