Isolate Antfly ANN benchmark from default full-text work

[ajroetker]

Summary

• remove Antfly's automatically created full-text index through the public API before an ANN-only benchmark
• retain the default full-text index when ANTFLY_VDBBENCH_KEEP_DEFAULT_FULL_TEXT=1
• cover the default removal and explicit opt-in behavior

sync_level=write does not wait for full-text indexing, but the full-text worker still consumes replay, storage, CPU, and memory. Removing it through the same public API used by a client isolates the vector benchmark; it does not change batch size or bypass Antfly's server.

Validation

• exercised by successful upstream public-API Antfly loads for OpenAI 50K and Cohere 1M, batch 100 with four load workers
• 50K: 25.05 s insert + 2.47 s readiness = 27.53 s
• 1M: 1,099.85 s insert + 10.81 s readiness = 1,110.66 s

The repository venv does not currently include pytest, so the focused pytest file could not be invoked locally. Both full E2E runs exercised the removal path and received a successful public API response.

Main Antfly optimization: antflydb/antfly#512

[dovinmu]

We do sometimes need that full-text index

[codex]

Blocker for filtered runs: #13 deletes an index that older Antfly needs for filters

VectorDBBench #13 (#13) deletes
full_text_index_v0 unconditionally unless an environment variable opts out.

For ordinary unfiltered ANN, that is appropriate. For filtered ANN, it is unsafe:

• The adapter still advertises NumGE and StrEqual.
• Circus has a honed LabelFilterPerformanceCase.
• Antfly v0.1.3 parses filter_query into a Bleve query and passes it into fused vector search;
  see the historical api_query.go
  (https://github.com/antflydb/antfly/blob/v0.1.3/src/metadata/api_query.go).
• Competitors retain their required scalar/payload indexes for filtered workloads.

[INFERENCE] On older Antfly, deleting the only full-text index can make filtered ANN fail or
lose the intended filter execution path. The PR only validated unfiltered 50K and 1M loads.

Recommendation: delete the default index only for an explicitly unfiltered ANN case. At
minimum:

if not self.with_scalar_labels and not self._keep_default_full_text_index():
   self._remove_default_full_text_index(client)

A workload/capability flag would be cleaner than inferring solely from with_scalar_labels,
because numeric-filter workloads may not use label data.

Add an end-to-end filtered test against every supported Antfly generation before allowing
deletion there.

[dovinmu]

[wrong PR]

[dovinmu]

Looks like this PR addresses this concern and we can just set the flag. Merging.