Add opt-in ~repair to recover almost-XHTML feeds - #17
Open
cuihtlauac wants to merge 1 commit into
Open
Conversation
Some feeds declare <content type="xhtml"> (which RFC 4287 requires to be well-formed XML) but embed HTML void elements that are not self-closed (<img ...>) or are closed with a redundant end tag (<img ...></img>). xmlm, and therefore Syndic, then rejects the entire feed and River.fetch fails with "Neither Atom nor RSS2 feed". Add an opt-in `?repair` flag to `River.fetch`. When set, and only when the document fails to parse as-is, River retries after rewriting HTML void elements into self-closing form (`sanitize_void_elements`). Feeds that already parse are never touched, so the default behaviour is unchanged. Also add `River.of_string` (parse an in-memory document, no HTTP — useful for testing and callers that fetch themselves) and expose `River.sanitize_void_elements`. Includes tests covering the unclosed, redundantly-closed and already-self-closed shapes, plus a well-formed feed left untouched. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cuihtlauac
added a commit
to ocaml/ocaml.org
that referenced
this pull request
Aug 26, 2026
The MirageOS feed is valid Atom but its type="xhtml" content embeds unclosed HTML void elements (<img>), which strict XML parsing rejects, so the whole feed fails with "Neither Atom nor RSS2 feed". river gained an opt-in ~repair flag (tarides/river#17) that self-closes void elements before parsing, only when the feed fails to parse as-is. Wire it through the planet scraper: - planet-sources.yml: new per-source `repair:` key; set on mirage. - blog_scraper: parse the key and pass ~repair to River.fetch (alongside the existing ~timeout/~user_agent). The flag is a scrape-time concern, so it is kept out of the published Blog.source data model (carried in a local `resolved` record instead). - blog_parser: accept and ignore the new key so the site build still reads the file. - repoint the existing river git pin to a fork branch that is river 0.5 (timeout) plus ~repair, until tarides/river#17 is released. repair is opt-in per feed, so it only affects feeds we have verified are recoverable this way. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Some feeds declare
<content type="xhtml">— which RFC 4287 requires to be well-formed XML — but embed HTML void elements that aren't self-closed. A real example ishttps://mirage.io/feed.xml, which mixes three<img>shapes:<img ... />— correct ✅<img ...>— bare/unclosed ❌<img ...></img>— redundant end tag (valid XML, but common)xmlm(and thereforeSyndic) rejects the entire feed on the unclosed ones, soRiver.fetchfails withFailure "Neither Atom nor RSS2 feed"even though the feed is otherwise perfectly good Atom. (Context: ocaml/ocaml.org#3753.)Change
Add an opt-in
?repairflag toRiver.fetch(defaultfalse). When set, and only if the document fails to parse as-is, River retries after rewriting HTML void elements into self-closing form. Feeds that already parse are never modified, so default behaviour is unchanged and the fast path pays nothing.sanitize_void_elements— two passes: collapse<v ...></v>→<v ... />, then self-close bare<v ...>, for the 13 HTML void elements. Conservative: only raw occurrences are touched; escapedtype="html"content has no literal<, so it's never affected.River.of_string ?repair source xml— parse an in-memory document with no HTTP request (used by the tests; handy for callers that fetch themselves).sanitize_void_elementsis exposed for testing/reuse.Tests
test/test_repair.mlcovers the unclosed / redundantly-closed / already-self-closed shapes, asserts the raw feed is rejected without~repairand recovered with it, and checks a well-formed feed is left byte-for-byte untouched. Verified locally against the real mirage.io feed (78 entries recovered).Notes
strwas already a lib dependency).make fmtuses--auto-promoteso CI reconciles it.