feat(pg-functions): implement PostgreSQL string functions - #407
Open
sunng87 wants to merge 3 commits into
Open
Conversation
Implement 14 string functions from the PostgreSQL built-in catalog as
DataFusion ScalarUDFs, organized into six modules under src/string/:
- convert: to_bin(int), to_oct(int) — integer-to-text base conversion
- quote: quote_literal(text), quote_nullable(text) — SQL quoting
- unicode: normalize(text[,form]), casefold(text), unicode_assigned(text),
unistr(text) — Unicode normalization and escape decoding
- format: format(fmt,...), sprintf(fmt,...) — PG %s/%I/%L text formatting
- regexp: regexp_substr(text,pattern,...), regexp_split_to_array(text,pat)
— regex extraction and splitting
- encoding: pg_client_encoding(), to_ascii(text) — encoding utilities
Each UDF follows the conventions from functions.md: ScalarUDFImpl with
PartialEq/Eq/Hash derives, NULL propagation, and unit tests covering
boundary cases. A string.slt integration test file exercises all functions
through the SQL → plan → execute path.
The 'string' Cargo feature now pulls in 'regex' and 'unicode-normalization'
as optional dependencies. functions.md is updated to mark 14 entries as 🔧.
Address every finding from the two-axis review of the string functions.
Spec (Postgres semantics):
- regexp_substr: 'start' is a 1-based CHARACTER position; resolve it via
char_indices so a start landing inside a multibyte UTF-8 char no longer
panics (was a crash on valid input).
- regexp_split_to_array: was implemented but never registered; now wired into
the string register() and given a working array/column path.
- to_bin/to_oct: negatives now render two's-complement (matching the to_hex
family) instead of sign-magnitude ('-1101').
- casefold: full Unicode case folding (CaseFolding.txt C+F), so casefold('ß')
-> 'ss' and long-s 'ſ' -> 's' (previously Rust to_lowercase, which kept 'ß').
- unicode_assigned: use ICU4X general-category tables via icu_properties;
Private-Use-Area chars (category Co) now correctly return true, and reserved
codepoints (Cn) correctly return false (was inverted on PUA, blind to Cn).
- quote_literal/quote_nullable: stop doubling backslashes to match
standard_conforming_strings = on (backslash is an ordinary char).
- to_ascii: transliterate Latin accented chars to their ASCII base (cafe,
Munchen) instead of replacing with '?'.
- format/sprintf: implement the exact Postgres grammar (%[position]s|I|L and
%%); reject width/flag specifiers, which the spec does not support.
Standards:
- Rename all UDF structs to the documented *UDF suffix (was *Udf).
- Add the previously-unregistered regexp_split_to_array to register().
- Document Postgres compatibility + link the manual at the top of each file.
- Add row-wise vectorized-batch unit tests for each function (convention datafusion-contrib#4).
- Replace 'Arc::new(...finish()) as _' with explicit 'as ArrayRef'.
- Clear all clippy warnings (borrowed-expression, map_or, match->?).
Add icu_properties behind the 'string' feature for the general-category
lookup. All 42 unit tests, the sqllogictest suite, and clippy pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the String Functions category (the catalog's 2nd part after the math functions) as DataFusion
ScalarUDFs, covering 14 PostgreSQL built-ins listed as 🚧 P2 in `datafusion-pg-functions/functions.md`.Organized into six modules under `src/string/`:
The `string` Cargo feature now pulls in `regex`, `unicode-normalization`, and `icu_properties` (optional). `functions.md` is updated to flip these rows to 🔧.
Postgres compatibility notes
Each UDF matches documented Postgres semantics:
Testing
Follows the conventions in `functions.md`: `*UDF` structs, `ScalarUDFImpl` impls, PG manual links + compatibility docs at each file header, constructors wired into `register()`.