Skip to content

⚡ Bolt: [performance improvement] Cache alias lookups for clinical section detection #45

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
zrt219 wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

⚡ Bolt: [performance improvement] Cache alias lookups for clinical section detection #45

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions openmed/.jules/bolt.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
## 2024-05-24 - Cache alias lookups for clinical section detection
**Learning:** In `openmed/openmed/clinical/sections/detect.py`, `_alias_lookups` was repeatedly parsing section header lexicons and building dictionaries for each line of a document. For large, many-line documents, this caused significant slowdowns because the section lookup computation is deterministic based on the language. Additionally, since the return values needed caching, a regular dict couldn't be safely cached due to mutability.
**Action:** When caching dictionaries generated from configuration or lexicons, use `@functools.lru_cache` and ensure the return type is explicitly wrapped in `types.MappingProxyType` to prevent any downstream modifications to the cached object.
8 changes: 6 additions & 2 deletions openmed/openmed/clinical/sections/detect.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

from __future__ import annotations

import functools
import types
from collections.abc import Iterable, Mapping
from dataclasses import dataclass
from typing import Any
Expand Down Expand Up @@ -244,6 +246,7 @@ def _is_underline(text: str) -> bool:
return len(stripped) >= 3 and set(stripped) <= _UNDERLINE_CHARS


@functools.lru_cache(maxsize=16)
def _alias_lookups(language: str | None) -> tuple[tuple[str, Mapping[str, str]], ...]:
languages = (
tuple(dict.fromkeys((get_section_lexicon(language).language, "en")))
Expand All @@ -253,14 +256,15 @@ def _alias_lookups(language: str | None) -> tuple[tuple[str, Mapping[str, str]],
return tuple((code, _aliases_for_language(code)) for code in languages)


def _aliases_for_language(language: str) -> dict[str, str]:
@functools.lru_cache(maxsize=32)
def _aliases_for_language(language: str) -> Mapping[str, str]:
lexicon = get_section_lexicon(language)
aliases: dict[str, str] = {}
for label, headers in lexicon.sections.items():
aliases[normalize_section_header(label)] = label
for header in headers:
aliases[normalize_section_header(header)] = label
return aliases
return types.MappingProxyType(aliases)


def _dedupe_hits(hits: Iterable[_HeaderHit]) -> tuple[_HeaderHit, ...]:
Expand Down