feat: Implement BQL support (as discussed in #387) - #415
Open
ak-finccam wants to merge 7 commits into
Open
Conversation
Member
|
Thanks for sending this our way, it looks reasonably careful and complete. I appreciate the added tests. However, I will not be in a position to fully review and test this as I do not currently have access to a Bbg terminal. |
Author
|
I appreciate the quick response! Does this mean you are currently unable to merge any PRs or is there another maintainer that has a terminal? I don't want to pressure you, it's just if you are unable to test / review / merge at this time then we will internally maintain a fork and install from that fork. |
Member
|
@johnlaing may engage with this too and provide feedback. In the meantime I would absolutely dog-food my own PR if I were you and test the living daylight out of it by running it. That's a common pattern for most of us methinks. |
Contributor
|
I do have a terminal so let me see what I can do. It may take some time to get to. |
Author
|
Thank you! We will dogfood this internally in the meantime. |
Member
|
In case anybody (besides @johnlaing) follows along here and has a working Bbg terminal, we would still appreciate a test and a basic 'yep, it works as advertised' ... |
Implements BQL support as discussed in Rblp#387. The C++ layer sends a 'sendQuery' request to the //blp/bqlsvc service and returns the JSON document the service responds with; the R layer parses it into properly-typed data.frames (one per data item in the query's get() clause) using the column types the response itself declares, with jsonlite as an optional (Suggests) dependency. bql(..., parse=FALSE) returns the raw JSON for queries whose shape the parser cannot handle. A response larger than 4 MiB arrives in several messages that cut the JSON mid-token, so the fragments are joined before parsing. Without this, queries above that size -- a PX_LAST history for 84 tickers over three years is enough -- fail with 'parse error: premature EOF'. The 'NaN'/'NA' missing-value sentinels are applied to numeric columns only so STRING columns keep legitimate 'NA' values (e.g. the ticker of 'NA US Equity'). Item-level responseExceptions (type PARTIAL, accompanying usable data) surface as warnings while the data is returned; top-level exceptions raise errors. Offline unit tests cover parsing, type mapping (including undeclared types such as ENUM), NA handling, fragmented responses, grouped aggregations (composite group ids, INT columns, ORIG_IDS nulls), and error propagation via synthetic fixtures whose structure was verified against live //blp/bqlsvc responses; no captured Bloomberg data is included. A live test gated on RunRblpapiUnitTests exercises the service end to end. Verified live against terminal API 3.24.6.1, including all documented example queries of the polars-bloomberg package (screens, SRCH results, segments, axes, return series), and with a 23 MB history response arriving in six fragments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ak-finccam
force-pushed
the
feature/bql
branch
from
September 1, 2026 13:34
5956231 to
1f7e3cb
Compare
Author
|
One small update: We discovered a problem for large requests where the JSON response was chunked and needed to be put together before parsed. |
Member
|
BTS I still have reaching out to a friend via whom I may get Bbg access for a bit of time on the TODO list but as you can see I have not yet gotten around to that yet. Pinging @johnlaing again in case he can find a minute or two... |
Either RcppSimdJson or jsonlite can now parse a BQL response, with RcppSimdJson preferred when both are installed as suggested in Rblp#415. Both are asked not to simplify, so they return the same structure and therefore the same result; the option 'Rblpapi.bqlParser' selects one explicitly, which lets the tests exercise every installed parser and assert that they agree. The parser was not the bottleneck though. .bqlColumn converted the values one element at a time with vapply(), and routed every value through character. It now uses lengths() to find the JSON nulls and unlist() to flatten, so a column of JSON numbers stays numeric, and it converts only the distinct strings of a DATE or DATETIME column. On a live 1.1 MiB response of 21930 rows this takes parsing from 0.170s to 0.030s with jsonlite and to under 0.005s with RcppSimdJson. Keeping numeric columns out of character also makes them lossless. Bloomberg sends float-derived prices such as 230.66000366210938, which the as.character() round trip truncated to 230.66000366210901. Since unlist() would flatten a nested value instead of failing, the one row per value invariant is now checked explicitly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A response above 4 MiB arrives as several messages, cut at a byte boundary in the middle of a token, and only the joined fragments form one document. Reaching that path with a real query needs megabytes of data, so the fixtures are instead chunked into pieces far smaller than 4 MiB, which reproduces the same condition. The chunking is on bytes rather than on characters, because that is what the service does. A boundary can therefore fall inside a multi-byte UTF-8 character, which leaves that one fragment invalid UTF-8 on its own; a case is included for this, with the characters written as escapes so the file stays ASCII. Verified against a live 5.56 MiB response of 111792 rows, which the service sent as 4.00 MiB plus 1.56 MiB. Its first fragment alone gives the 'parse error: premature EOF' originally reported. Breaking .bqlJoin in four ways (first fragment only, extra separator, reversed order, last fragment dropped) makes these tests fail, so they do test the joining rather than merely pass alongside it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Corrects an overstated claim in cbf06e1: keeping a numeric column out of character only preserved the full precision of its values when the array held no "NaN", "NA" or "" placeholder. One such string promoted the whole column to character and the numbers came back from their 15 significant digit form, so precision depended on whether the service happened to send one. The placeholders are now blanked in the list before it is flattened, which keeps the column numeric in either case. Only the placeholders are blanked, as any other string is a number written as a string and still has to be converted. This costs a pass per element, but only for a numeric column which really does contain a placeholder; that case is now on a par with the code before this branch, while a column without one keeps its much larger gain. .bqlNumericNA becomes unreachable and is removed. Also fixed: * .bqlFromJSON had no default branch, so an unrecognised parser name returned NULL and the caller received an empty result rather than a diagnosis. * .bqlParser used match.arg(), which accepted an abbreviation such as "j" and silently dropped an unknown name given beside a known one. The option is now validated, and a failure names only the parser actually asked for instead of both. * The claim that the parsers agree is narrowed to the documents the service returns: they differ on 1e999, on integers above 2^64, on nesting thousands deep and on an embedded NUL. * A dead line, and a comment which described the non-scalar guard as stronger than it is: a value flattening to exactly one element is kept, as it was before. The tests were passing for the wrong reasons in several places, which mutation testing showed. All nine mutants are now caught, where five survived before: * A permuted date column was undetectable, because the only order-sensitive DATE fixture was checked for its column name alone and every other one holds duplicates. Values are asserted now, with columns whose distinct dates are neither sorted nor unique. * The placeholder assertions used expect_equal, and all.equal() treats NaN as equal to NA_real_, so they passed with the placeholder handling removed entirely. They use expect_identical now. * The parser preference took its expectation from .bqlParsers, so it agreed with any order that variable held. The name is written out. * Nothing asserted the parsers return the same intermediate structure, so max_simplify_lvl and the two 'empty' arguments could all be dropped without a failure. * A fixed 1000 byte chunk size left three fixtures in one piece, comparing a document with itself. The sizes derive from each document now, and the split is asserted. Verified against the saved 5.56 MiB two-fragment live response: both parsers identical, non-numeric columns identical to the previous code, and the DOUBLE column now exact rather than differing by 4.3e-15. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
No behaviour change beyond the two noted below.
R/bql.R:
* The placeholder pre-pass no longer calls vapply() once per element to
find the strings. The flattened vector is already the character form
at that point and is index-aligned, so one vectorised %in% gives the
same mask; measured at 1.7ms against 22.2ms on a column of 111792
values. It also stops re-reading and re-scanning the original list.
* substr() for a DATE column now runs inside .bqlByUnique, so it
truncates the distinct strings rather than every row: 32% off that
column and 11% off the whole parse of the 5.56 MiB live response.
* .bqlChar() is gone. as.character() already returns the same object
for a character vector, and the conversion is only needed in the
default branch, since substr() and as.POSIXct() both accept the
logical vector an all-null column flattens to.
* .bqlByUnique() takes ... , which removes the inline function wrapper
the DATETIME branch needed.
* The anyNA() clause in the parser validation could never fire, as an
NA matches no known parser name and the %in% test already rejects it.
* The option is renamed 'bqlParser' and exposed as a 'parser' argument,
following the house pattern of every other option in the package
(returnAs=getOption("bdhType"), simplify=getOption("blpSimplify")).
All fifteen existing option names are flat blp/bdh camelCase, and all
are argument defaults; bql() was the only entry point whose knob
could not be set per call. That argument is the behaviour change.
* Comments trimmed where they narrated the code or, after the change
above, described a per-element pass which no longer happens.
inst/tinytest/test_bql.R:
* Dropped the fragment test which split one fixture into three pieces
by character. The byte-level chunking loop covers every fixture at
five sizes, and splitting by character cannot land inside a
multi-byte character, which is the case worth testing.
* The join is now also asserted directly against .bqlJoin, so a failure
says whether the join or the parse broke rather than only that one of
them did.
* .with() is hoisted, so all four loops label assertions with the
parser in use; two assertions had lost the label and reported
identically on every parser. One .oneItemDoc() builder replaces two
hand-written document skeletons, one .outcome() replaces three copies
of the same tryCatch, and one .withOption() replaces two ways of
restoring an option, one of which leaked it on a failure.
* The three double-precision blocks become one loop covering eight
combinations per parser instead of five.
The mutation harness was rebuilt as well: it silently stopped applying
four of its mutants when the code above moved, and reported them as
surviving. It now fails loudly if a mutant does not change anything. All
twelve are killed, against 346 assertions passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Most of these are in .bqlItemToDataFrame and the C++ event loop, which
this branch had not touched; they are on feature/bql already, but a
reviewer meets them here.
* An item whose idColumn and valuesColumn are both null, with no
secondary columns, gave 'names' must be a character vector from
make.unique(NULL) instead of the empty result the surrounding
warn-and-continue path intends.
* A repeated column name dropped a column. Assigning cols[[nm]] by
name replaced the earlier column of that name rather than adding
one, which also left make.unique() with nothing to rename: two
DATE secondary columns produced three columns holding only the
second one's values. The columns are collected in order and named
at the end now, so both survive as DATE and DATE.1.
* Columns of unequal length were baked into a corrupt data.frame,
which reported the wrong number of rows. They are rejected.
* src/bql.cpp ignored REQUEST_STATUS, which a rejected or timed-out
request sends instead of a RESPONSE, so nextEvent() would have
blocked for good. Handled as bdh.cpp does.
* A session which ended before the response arrived left bql_Impl
returning nothing, and the JSON parser then reported a truncated
document. bql() now says what actually happened.
Two are about this branch's own work:
* bql()'s signature named .bqlParsers, which is not exported, so
?bql showed users an object they cannot reference. 'parser' now
defaults to NULL and .bqlParser() resolves it.
* The comment claiming a numeric column keeps the values exactly as
the service sent them holds for JSON numbers and placeholders, but
not when a number arrives as a string: that column still has to
come back from character. Narrowed, and pinned by tests using
expect_identical, as expect_equal's tolerance hid it.
Not changed: a BOOLEAN column whose values are JSON numbers is all NA,
but it was before too, since as.logical("0") is NA. Only a boolean
sharing an array with a number differs from the old code, which is the
coercion order already documented above .bqlColumn.
354 assertions pass and all twelve mutants are caught.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three functions defaulted the parser, but .bqlParse always passes it explicitly, so .bqlFromJSON's default was never taken. bql() is now the one place which decides, and the decision is passed down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Author
|
I have worked on some performance improvements and also on integrating RcppSimdJson. The improvements really only show for very large datasets but they are real (10x, so mostly from 1000ms to 100ms). We will run some more tests internally and let people at our firm try this out. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This work implements BQL support for Rblpapi. This code is inspired by polars-bloomberg (python package) and also the matthewgilbert/blp package.
Before implementing this, I have read issue #387 and picked up on the ideas and limitations discussed there.
jsonliteto parse the result from the BQL query. I did not vendor the C++ json library..bqlColumn.AI disclosure: This work was done using Claude Fable. I have limited knowledge of C++ and have only ever used Rblpapi but never looked too deep into the source code.