Reduce front-end database queries from CMS templates, theme data and request logs

[LukeTowers]

Problem

Production query stats from a site taking heavy bot/crawler traffic:

Query	Calls	Total	Avg
select * from cms_theme_templates where source = ? and deleted_at is null and path = ? limit 1	717.8K	2h 01m	10.08ms
select * from cms_theme_data where (theme = ?) limit 1	358.9K	47m 03s	7.86ms
select * from system_request_logs where (url = ? and status_code = ?) limit 1	120.4K	12h 35m	376.02ms
update system_request_logs set count = count + 1, updated_at = ? where id = ?	119.4K	17m 35s	8.84ms

Reading the ratios: 717.8K / 358.9K is exactly 2.00 and theme data is memoised per process, so this is ~358.9K front-end requests, each doing 2 template lookups and 1 theme-data lookup — with ~33% of all requests being 404s (bots probing /wp-login.php, /.env, etc. all funnel through the CMS catch-all route).

system_request_logs alone is 80% of the listed database time despite being 25% of the calls.

Changes

1. CMS templates — the object cache removed no queries

CmsObject::loadCached() caches parsed objects, but Halcyon\Builder::getCached() validates every warm cache hit via isCacheBusted() → datasource->lastModified(), so each database-backed template still cost a query per request.

AutoDatasource already holds a forever-cached, correctly-invalidated manifest of which paths live in which datasource. Paired with wintercms/storm#238 that manifest now carries modification times, so AutoDatasource::lastModified() answers from memory.

Filesystem datasources are untouched — they report true in the manifest and fall through to live filemtime(), so editing templates on disk still takes effect immediately. Deleted paths report false and keep raising the existing "is deleted" exception.

Behaviour tradeoff

Template content edits made by direct SQL, outside the application, are no longer picked up until the paths cache is cleared. Edits through the backend, theme:sync, or any datasource API call call populateCache(true) and remain immediate. The paths manifest already had exactly these semantics for added/removed templates, so this widens an existing characteristic rather than introducing a new class of staleness.

2. Theme data — no persistent cache

ThemeData::forTheme() was memoised only in a static array, so it was one uncached query per request. The dominant caller is not {{ theme.* }} in templates — it is the asset combiner: cms.combiner.getCacheKey is registered for front-end requests only and fires inside CombineAssets::getCacheKey(), which runs before the combiner cache is consulted, so a warm combiner cache did not avoid it.

The record is now cached following the existing SettingsModel pattern, invalidated in afterSave() and a new afterDelete() — Theme::removeCustomData() deletes the record and had no hook. Theme::resetCache() also clears the in-memory instances, which it previously left stale for long-running workers (Octane, queue workers).

3. Request logs — no index at all

system_request_logs had no index beyond its primary key, so every 404 full-scanned a table that only ever grows. Since add() is what grows it, the cost accelerated on its own. Adds the (url, status_code) index matching what RequestLog::add() looks up.

Also adds a composite (source, path, deleted_at) index to cms_theme_templates, replacing the now-redundant standalone source index — every datasource query filters on source and path together, but the two independent single-column indexes meant only one could be used.

Tests

• AutoDatasourceTest (had one test) — mtime served from the path cache with zero queries, filesystem fall-through returns live filemtime(), deleted paths still throw, path-cache value shapes (int / false / true), stability for null updated_at, and that an update through the datasource is reflected immediately.
• ThemeDataTest (new) — cached across requests with zero queries, record created only once, defaults applied to cached rows, afterSave()/afterDelete() invalidation, and dynamic (jsonable) attributes surviving the cache.
• RequestLogTest (new — there was no coverage) — the index exists (driver-aware introspection), create/increment/status-code separation, and the log_requests setting being honoured.
• ThemeTest — resetCache() clears theme data instances.
• New fixture theme modules/cms/tests/fixtures/themes/themedata with a form: section; no existing fixture declared customization fields.

All new assertions were verified as genuine regression guards by reverting each fix and confirming the corresponding test fails.

Full suite result

711 tests. The 13 errors that remain are pre-existing and unrelated — Assetic\Filter\CssImportFilter::setImportValidator() is undefined with the currently-installed assetic, following the asset combiner security change. Confirmed identical with these changes stashed.

Companion PR

wintercms/storm#238 — merged. The manifest mtimes this PR consumes come from there.

Summary by CodeRabbit

• Performance

  • Improved loading speed for theme data and cached content paths.
  • Reduced unnecessary database queries when retrieving modification timestamps and theme settings.
  • Improved request log lookup performance through optimized indexing.

• Bug Fixes

  • Theme cache resets now consistently clear related cached data.
  • Modification timestamps remain accurate after content updates and support deleted or filesystem-based paths.
  • Theme data changes are reflected immediately after saving or deleting.

[coderabbitai]

Walkthrough

The CMS now caches theme data and datasource modification timestamps. Cache invalidation occurs after theme-data saves, deletes, and theme resets. A composite index supports theme-template lookups. New tests cover cache persistence, invalidation, datasource timestamp behavior, and filesystem fallback. The system module adds a composite request-log index and tests request aggregation, status separation, index presence, and disabled logging.

Estimated code review effort: 4 (Complex) | ~45 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 14.29% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main objective: reducing front-end database queries across CMS templates, theme data, and request logs.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/reduce-frontend-db-queries

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}