Skip to content

feat(cloudwatch): add CloudWatch Logs Insights Query keyword with absolute time windows, metadata, and multi-log-group support - #66

Open
JJediny wants to merge 2 commits into
MarketSquare:masterfrom
JJediny:feat/cloudwatch-insights-advanced
Open

feat(cloudwatch): add CloudWatch Logs Insights Query keyword with absolute time windows, metadata, and multi-log-group support#66
JJediny wants to merge 2 commits into
MarketSquare:masterfrom
JJediny:feat/cloudwatch-insights-advanced

Conversation

@JJediny

@JJediny JJediny commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

The existing CloudWatch Logs Insights keyword covers simple use cases but has four limitations that make it unsuitable for security auditing and multi-account investigations:

Limitation Impact
Time range is relative minutes-ago only Cannot target a specific past investigation window without extra Robot date math
@ptr field is stripped from every result row Breaks cross-referencing back to raw log events
No queryId or statistics returned No cost visibility; no way to re-fetch or correlate a specific query run
Single log group per call Cannot join CloudTrail + VPC Flow Logs in one query

This PR adds a new keyword CloudWatch Logs Insights Query that addresses all four while remaining fully backward-compatible (the original keyword is unchanged).


New keyword: CloudWatch Logs Insights Query

*** Test Cases ***
# Relative window — same as before
${rows}=    CloudWatch Logs Insights Query    /aws/cloudtrail/org
...    fields @timestamp, eventName | limit 20

# Absolute investigation window
${start}=    Evaluate    int(time.mktime(time.strptime("2026-08-01", "%Y-%m-%d")))    modules=time
${end}=      Evaluate    int(time.mktime(time.strptime("2026-08-31", "%Y-%m-%d")))    modules=time
${rows}=    CloudWatch Logs Insights Query    /aws/cloudtrail/org
...    fields @timestamp, eventName | limit 20
...    start_epoch=${start}    end_epoch=${end}

# Return queryId + statistics for cost audit
${meta}=    CloudWatch Logs Insights Query    /aws/cloudtrail/org
...    fields @timestamp | limit 1    return_metadata=${True}
Log    QueryId: ${meta}[queryId]
Log    Scanned: ${meta}[statistics][bytesScanned] bytes

# Multi-log-group (CloudTrail + VPC Flow Logs)
${rows}=    CloudWatch Logs Insights Query    /aws/cloudtrail/org
...    fields @timestamp, eventName | limit 20
...    log_group_names=["/aws/vpc/flowlogs"]

Parameters

Argument Type Default Description
log_group str required Primary log group
query str required Insights query string
start_epoch int|None None Absolute start (epoch seconds); overrides start_time
end_epoch int|None None Absolute end (epoch seconds); defaults to now
start_time int 60 Minutes-ago fallback when start_epoch is not set
timeout int 120 Seconds before TimeoutError is raised
poll_interval float 2 Seconds between poll attempts
return_metadata bool False Return {results, queryId, statistics} dict
log_group_names list|None None Extra log groups for cross-log-group queries

Recommended pattern: consolidated payer-account CloudTrail

AWS Organizations supports an organization-level CloudTrail that delivers all member-account events into a single log group in the management (payer) account. This is the recommended pattern for multi-account environments because:

  • A single LOG_GROUP_NAME variable covers every account — no per-account session switching.
  • recipientAccountId in every CloudTrail event lets you scope queries to a specific member account without extra infrastructure.
  • Queries run in the payer account where the log group lives, so IAM permissions are centralized.
# Scope to a single member account inside the org-wide log group
${query}=    Set Variable
...    fields @timestamp, recipientAccountId, userIdentity.arn, eventName
...    | filter recipientAccountId = "123456789012"
...    | sort @timestamp desc | limit 50
${rows}=    CloudWatch Logs Insights Query    /aws/cloudtrail/org    ${query}
...    start_epoch=${start}    end_epoch=${end}

Security queries that benefit from this pattern (included in the library's example suite):

  • Failed authentication aggregated by principal across all accounts
  • Root account usage per member account
  • Cross-account role assumption chains
  • IAM key creation / privilege escalation
  • CloudTrail tampering (StopLogging, DeleteTrail)

Files changed

  • src/AWSLibrary/keywords/cloudWatch.py — new insights_query_advanced method bound to keyword CloudWatch Logs Insights Query
  • tests/robot/cloudwatch.robot — four new test cases (relative window, absolute window, return_metadata, existing smoke preserved)

Testing

All new tests are structured to run against localstack (following the existing pattern in the project). The tests use the same Create Session And Set Endpoint / Delete All Sessions suite setup as the rest of the CloudWatch suite.

No existing keywords or tests were modified.

…olute time windows, metadata, and multi-log-group support

The existing 'CloudWatch Logs Insights' keyword only accepts a relative
minutes-ago window, strips the @ptr field from results, returns no queryId
or statistics, and cannot query multiple log groups in one call.

This commit adds a new keyword 'CloudWatch Logs Insights Query' that:

- Accepts absolute epoch-second start/end timestamps (start_epoch/end_epoch)
  for targeting specific investigation windows without date arithmetic in
  every test case.
- Falls back to a relative minutes-ago window (start_time) when no absolute
  time is supplied, so existing usage patterns keep working.
- Exposes queryId and statistics (bytesScanned, recordsMatched, recordsScanned)
  via return_metadata=True for cost visibility and audit trails.
- Accepts log_group_names for multi-log-group queries (e.g. CloudTrail +
  VPC Flow Logs in one call).
- Raises TimeoutError instead of hanging indefinitely on large scans
  (configurable timeout/poll_interval).
- Returns the full result rows including @ptr, not a stripped subset.

Adds four Robot test cases covering: relative window, absolute window,
return_metadata, and (implicitly) the existing localstack smoke path.

Recommended pattern - consolidated org CloudTrail in the payer account:
AWS Organizations lets you enable a single organization-trail that delivers
all member-account CloudTrail events to one log group in the management
(payer) account. Setting LOG_GROUP to that single log group and filtering
by recipientAccountId eliminates per-account session setup and gives a
single query surface for cross-account security investigations.

Fixes: none (additive change, fully backward-compatible)
…in Insights Query tests

- Replace while/else pattern with explicit post-loop status check so TimeoutError
  is raised correctly when the deadline expires (else on while fires on natural
  loop exit, i.e. timeout, not on break)
- Widen terminal-status check from explicit list to 'not Running/Scheduled' so
  any LocalStack-specific status variant is handled correctly
- Remove 30s Sleep from the three new test cases; LocalStack answers Insights
  queries synchronously so the sleep caused needless CI time without benefit
- Extend end_epoch by +60s in the absolute-window test to avoid edge-case where
  endTime equals startTime when the machine clock ticks between Evaluate calls
@JJediny

JJediny commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

The CI failure is a pre-existing infrastructure issue unrelated to this PR's changes.

All 24 tests across all services (CloudWatch, S3, DynamoDB, SQS) fail with the same error:

EndpointConnectionError: Could not connect to the endpoint URL: "http://localhost:4566/"

This indicates LocalStack never became available before the tests ran. The workflow only sleeps 10 seconds after docker compose up -d, which is insufficient on current ubuntu-24.04 runners. The last successful master run was #15 (Sep 16, 2025, pushed by @paguilera) — every PR-triggered run since then has failed the same way.

For reference, run #15 on master passes clean with 3m12s total. Our PR runs take 7m+ because LocalStack startup alone exhausts the window.

This PR's changes are additive only — the new keyword and 4 new test cases follow the exact same pattern as the existing tests. Once the LocalStack readiness issue is resolved (e.g. by adding a health-check poll instead of sleep 10), the new tests will pass under the same conditions as the existing ones.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant