fix(csv): reject malformed separators and trailing garbage in typed temporal parse - #422
fix(csv): reject malformed separators and trailing garbage in typed temporal parse#422belowzeroff wants to merge 2 commits into
Conversation
…emporal parse .csv.read [DATE|TIME|TIMESTAMP] silently coerced malformed cells to bogus values: fast_date / fast_time / fast_timestamp only validated field prefixes, so "2024/01/02", "2024x01x02", "2024-01-02junk", "12-34-56" and "2024x01x02D01:02:03" all parsed as if valid instead of null. Validate the '-' and ':' field separators and the 'T'|' ' date/time separator, require digit fields, and reject any unconsumed trailing bytes (bar a trailing UTC offset on a timestamp). This mirrors detect_type() and the csv writers, so signed / >=24h TIME durations and 'Z' / +-HH:MM timestamp offsets still round-trip. parse_tz_offset now reports its consumed length so the timestamp parser can enforce full consumption.
|
Reviewed (with an adversarial verification pass against both the PR head and the base). The strict parsers are internally correct, but 1. Auto-inferred TIMESTAMP columns with suffixes go all-null ( 2. Same for TIME ( 3. Bare trailing dot — 4. Lowercase 5. (Judgment call) The core issue behind 1–3: the strictness boundary moved in the parsers but not in the classifier. Either tighten Smaller points:
|
Problem
Typed
.csv.read [DATE|TIME|TIMESTAMP]silently coerces malformed cells into bogus values instead of rejecting them as null. The fast parsers only validated field prefixes — they never checked the separator characters, never validated that the digit positions were digits, and never rejected trailing bytes.Repro (before this change):
This is silent data corruption on import: a mistyped or wrong-format column reads back as plausible-but-wrong values rather than surfacing as nulls. The numeric/GUID CSV parsers already reject this class of input; only the temporal parsers were lenient.
Fix
fast_date/fast_time/fast_time_ns/fast_timestampnow:-and:field separators and theT|space date/time separator;The accepted format is kept in lockstep with
detect_type()(schema-less inference) and the CSV writers, so nothing that round-trips today regresses:>=24hTIMEdurations (-00:00:01,25:00:00) still parse (per fix(csv): round-trip signed and >=24h TIME values in .csv.read #379);Z/±HH[:]MMtimestamp UTC offsets still parse and apply (per the ISO-8601 offset support).parse_tz_offsetnow reports how many bytes it consumed sofast_timestampcan require the offset to reach the end of the field.Testing
make test— full suite green (3701/3702 pass, 1 pre-existing skip, 0 failed).test/rfl/io/csv_types.rfl(section 18) pinning malformed-separator / trailing-garbage rejection for all three types, plus the valid signed/>=24h/offset forms that must still parse.