Skip to content

fix(csv): reject malformed separators and trailing garbage in typed temporal parse - #422

Open
belowzeroff wants to merge 2 commits into
RayforceDB:devfrom
belowzeroff:fix/csv-temporal-typed-parse-strict
Open

fix(csv): reject malformed separators and trailing garbage in typed temporal parse#422
belowzeroff wants to merge 2 commits into
RayforceDB:devfrom
belowzeroff:fix/csv-temporal-typed-parse-strict

Conversation

@belowzeroff

Copy link
Copy Markdown
Contributor

Problem

Typed .csv.read [DATE|TIME|TIMESTAMP] silently coerces malformed cells into bogus values instead of rejecting them as null. The fast parsers only validated field prefixes — they never checked the separator characters, never validated that the digit positions were digits, and never rejected trailing bytes.

Repro (before this change):

(.csv.read [DATE]      "")  ;; 2024/01/02        -> 2024.01.02   (should be null)
                             ;; 2024x01x02        -> 2024.01.02   (should be null)
                             ;; 2024-01-02junk    -> 2024.01.02   (should be null)
(.csv.read [TIME]      "")  ;; 12-34-56          -> 12:34:56.000 (should be null)
(.csv.read [TIMESTAMP] "")  ;; 2024x01x02D01:02:03      -> valid ts  (should be null)
                             ;; 2024-01-02T01:02:03junk  -> valid ts  (should be null)

This is silent data corruption on import: a mistyped or wrong-format column reads back as plausible-but-wrong values rather than surfacing as nulls. The numeric/GUID CSV parsers already reject this class of input; only the temporal parsers were lenient.

Fix

fast_date / fast_time / fast_time_ns / fast_timestamp now:

  • validate the - and : field separators and the T|space date/time separator;
  • require the digit fields to actually be digits;
  • reject any unconsumed trailing bytes (other than a trailing UTC offset on a timestamp).

The accepted format is kept in lockstep with detect_type() (schema-less inference) and the CSV writers, so nothing that round-trips today regresses:

parse_tz_offset now reports how many bytes it consumed so fast_timestamp can require the offset to reach the end of the field.

Testing

  • make test — full suite green (3701/3702 pass, 1 pre-existing skip, 0 failed).
  • Added regression coverage in test/rfl/io/csv_types.rfl (section 18) pinning malformed-separator / trailing-garbage rejection for all three types, plus the valid signed/>=24h/offset forms that must still parse.

…emporal parse

.csv.read [DATE|TIME|TIMESTAMP] silently coerced malformed cells to bogus
values: fast_date / fast_time / fast_timestamp only validated field prefixes,
so "2024/01/02", "2024x01x02", "2024-01-02junk", "12-34-56" and
"2024x01x02D01:02:03" all parsed as if valid instead of null.

Validate the '-' and ':' field separators and the 'T'|' ' date/time separator,
require digit fields, and reject any unconsumed trailing bytes (bar a trailing
UTC offset on a timestamp). This mirrors detect_type() and the csv writers, so
signed / >=24h TIME durations and 'Z' / +-HH:MM timestamp offsets still
round-trip. parse_tz_offset now reports its consumed length so the timestamp
parser can enforce full consumption.
@singaraiona

Copy link
Copy Markdown
Collaborator

Reviewed (with an adversarial verification pass against both the PR head and the base). The strict parsers are internally correct, but detect_type was not tightened to match — and the row loops have no STR fallback — so the divergence converts entire columns to null silently on schema-less reads. Confirmed regressions:

1. Auto-inferred TIMESTAMP columns with suffixes go all-null (src/io/csv.c fast_timestamp). detect_type validates only chars 0–18, so "2024-01-02 01:02:03 UTC" still infers TIMESTAMP; strict fast_timestamp then fails parse_tz_offset on the suffix and nulls every row. Pre-PR the suffix was ignored and the wall-clock value stored. No fallback to STR ⇒ silent data loss.

2. Same for TIME (fast_time). detect_type checks only offsets 0–7, so "12:34:56 EST", "12:34:56 PM" — and even a trailing space, since scan_field never trims — infer TIME then null out. Pre-PR the tail was skipped and 12:34:56 returned.

3. Bare trailing dot"12:34:56." / "2024-01-02T01:02:03." / "...03.Z" previously parsed as .000; the new digits == 0 rejection nulls them while detect_type still classifies them as TIME/TIMESTAMP.

4. Lowercase 't' separator now rejected — the new p[10] != 'T' && p[10] != ' ' check refuses RFC 3339-valid "2024-01-02t01:02:03" (old code never inspected byte 10), even though parse_tz_offset accepts lowercase 'z'.

5. (Judgment call) fast_date changed len < 10 to len != 10, so typed [DATE] reads of timestamp-shaped cells ("2024-01-02T01:02:03") no longer prefix-parse to the date — all-null instead. Possibly intended strictness, but it's a silent regression for a plausible use; worth an explicit decision in the PR description either way.

The core issue behind 1–3: the strictness boundary moved in the parsers but not in the classifier. Either tighten detect_type to the same full-consumption grammar (so mismatched columns demote to STR and no data is lost), or keep the classifier authoritative and have parse failures on inferred columns fall back to STR. As-is, typed and untyped reads of the same file disagree, and the failure mode is invisible.

Smaller points:

  • fast_time_ns consumes-but-doesn't-verify trailing bytes; the full-consumption invariant lives only in fast_timestamp's off + tz_used == len check. A future standalone caller checking only *is_null re-accepts trailing garbage. Worth documenting in the header comment or moving the check inside.
  • The if (consumed) NULL-guards are dead — each function's single caller always passes a non-NULL pointer. Returning the consumed count directly (0 = failure) would be simpler.
  • The new digit checks re-implement, scalar-by-scalar, what src/core/numparse.h (already included by csv.c) exports as SWAR helpers (ray_is_4_digits/ray_parse_4_digits etc., whose header comment says they exist for exactly these parsers). On the per-cell ingest hot path that's both duplication and a measurable cost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants