New way to input website keywords

[ppinchuk]

Users are no longer required to specify arbitrary keyword values. COMPASS can calculate the keyword scores just base don tiered keyword input

[Copilot]

Pull request overview

This PR updates COMPASS plugin configuration handling so users can provide ordered website keyword tiers (rather than manually assigned numeric weights), and COMPASS computes deterministic keyword scores (including URL-encoded variants) to drive website document prioritization during retrieval.

Changes:

• Added normalize_website_keywords() utility to accept either tiered keyword lists (preferred) or legacy keyword→score mappings, and to generate URL-encoded variants.
• Updated one-shot plugin keyword handling and multiple built-in/example plugin configs to use tier lists instead of explicit numeric weights.
• Added unit tests covering tier weight computation, backwards compatibility, and validation/warnings.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
compass/plugin/utilities.py	Introduces tier-to-weight normalization, URL variant augmentation, and ordinance-term warning behavior.
compass/plugin/__init__.py	Exposes normalize_website_keywords via the plugin package namespace.
compass/plugin/one_shot/base.py	Switches one-shot plugin website keyword handling to the new normalization function and updates config docs.
compass/extraction/solar/plugin.py	Converts built-in solar website keywords to tiers and normalizes to computed scores.
compass/extraction/wind/plugin.py	Converts built-in wind website keywords to tiers and normalizes to computed scores.
compass/extraction/small_wind/plugin.py	Converts built-in small-wind website keywords to tiers and normalizes to computed scores.
tests/python/unit/plugin/test_plugin_utilities.py	Adds tests for tier weight computation, validation rules, warnings, and legacy mapping support.
tests/python/unit/plugin/test_plugin_one_shot_base.py	Adds one-shot plugin test ensuring tiered YAML input produces computed keyword scores.
examples/one_shot_schema_extraction/README.rst	Updates documentation wording from keyword weights to ordered keyword tiers.
examples/one_shot_schema_extraction/plugin_config.yaml	Updates example one-shot config to provide tiered website keywords.
examples/water_rights_demo/one-shot/plugin_config.yaml	Updates demo one-shot config to provide tiered website keywords.
compass/extraction/oil_gas_wells/plugin_config.yaml	Migrates oil/gas website keyword config from explicit weights to tiers.
compass/extraction/natural_gas_pipelines/plugin_config.yaml	Migrates natural gas pipelines website keyword config from explicit weights to tiers.
compass/extraction/ghp/plugin_config.yaml	Migrates GHP website keyword config from explicit weights to tiers and updates heuristic exclusions.
compass/extraction/geothermal_electricity/plugin_config.yaml	Migrates geothermal electricity website keyword config from explicit weights to tiers.
.github/skills/plugin-config-setup/SKILL.md	Updates the skill documentation to describe tiered keyword inputs and legacy support.

Suppressed comments (3)

compass/extraction/solar/plugin.py:76

• The WEBSITE_KEYWORDS attribute is now a dict of keyword scores (after normalize_website_keywords), but the attribute docstring still describes a list and mentions a wind ordinance. This is misleading for API consumers and for maintainers.

    WEBSITE_KEYWORDS = normalize_website_keywords(
        BEST_SOLAR_ORDINANCE_WEBSITE_URL_KEYWORDS
    )
    """list: List of keywords

    Keywords indicate links which should be prioritized when performing
    a website scrape for a wind ordinance document.
    """

compass/extraction/wind/plugin.py:75

• The WEBSITE_KEYWORDS attribute is now a dict of keyword scores (after normalize_website_keywords), but the attribute docstring still describes a list. This is misleading for API consumers and for maintainers.

    WEBSITE_KEYWORDS = normalize_website_keywords(
        BEST_WIND_ORDINANCE_WEBSITE_URL_KEYWORDS
    )
    """list: List of keywords

    Keywords indicate links which should be prioritized when performing
    a website scrape for a wind ordinance document.
    """

compass/extraction/small_wind/plugin.py:76

• The WEBSITE_KEYWORDS attribute is now a dict of keyword scores (after normalize_website_keywords), but the attribute docstring still describes a list and refers generically to a wind ordinance. Updating it avoids confusion when debugging crawl prioritization.

    WEBSITE_KEYWORDS = normalize_website_keywords(
        BEST_SMALL_WIND_ORDINANCE_WEBSITE_URL_KEYWORDS
    )
    """list: List of keywords

    Keywords indicate links which should be prioritized when performing
    a website scrape for a wind ordinance document.
    """

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 85.00000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.82%. Comparing base (714c408) to head (65a1ea2).

Files with missing lines	Patch %	Lines
compass/plugin/utilities.py	85.91%	5 Missing and 5 partials ⚠️
compass/plugin/one_shot/base.py	33.33%	2 Missing ⚠️

Additional details and impacted files

@@                Coverage Diff                 @@
##           pp/oil_and_gas     #537      +/-   ##
==================================================
+ Coverage           64.42%   64.82%   +0.40%     
==================================================
  Files                  78       79       +1     
  Lines                7496     7553      +57     
  Branches              755      767      +12     
==================================================
+ Hits                 4829     4896      +67     
+ Misses               2510     2495      -15     
- Partials              157      162       +5     

Flag	Coverage Δ
unittests	64.82% <85.00%> (+0.40%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.