feat(cloudwatch): add CloudWatch Logs Insights Query keyword with absolute time windows, metadata, and multi-log-group support - #66
Conversation
…olute time windows, metadata, and multi-log-group support The existing 'CloudWatch Logs Insights' keyword only accepts a relative minutes-ago window, strips the @ptr field from results, returns no queryId or statistics, and cannot query multiple log groups in one call. This commit adds a new keyword 'CloudWatch Logs Insights Query' that: - Accepts absolute epoch-second start/end timestamps (start_epoch/end_epoch) for targeting specific investigation windows without date arithmetic in every test case. - Falls back to a relative minutes-ago window (start_time) when no absolute time is supplied, so existing usage patterns keep working. - Exposes queryId and statistics (bytesScanned, recordsMatched, recordsScanned) via return_metadata=True for cost visibility and audit trails. - Accepts log_group_names for multi-log-group queries (e.g. CloudTrail + VPC Flow Logs in one call). - Raises TimeoutError instead of hanging indefinitely on large scans (configurable timeout/poll_interval). - Returns the full result rows including @ptr, not a stripped subset. Adds four Robot test cases covering: relative window, absolute window, return_metadata, and (implicitly) the existing localstack smoke path. Recommended pattern - consolidated org CloudTrail in the payer account: AWS Organizations lets you enable a single organization-trail that delivers all member-account CloudTrail events to one log group in the management (payer) account. Setting LOG_GROUP to that single log group and filtering by recipientAccountId eliminates per-account session setup and gives a single query surface for cross-account security investigations. Fixes: none (additive change, fully backward-compatible)
…in Insights Query tests - Replace while/else pattern with explicit post-loop status check so TimeoutError is raised correctly when the deadline expires (else on while fires on natural loop exit, i.e. timeout, not on break) - Widen terminal-status check from explicit list to 'not Running/Scheduled' so any LocalStack-specific status variant is handled correctly - Remove 30s Sleep from the three new test cases; LocalStack answers Insights queries synchronously so the sleep caused needless CI time without benefit - Extend end_epoch by +60s in the absolute-window test to avoid edge-case where endTime equals startTime when the machine clock ticks between Evaluate calls
|
The CI failure is a pre-existing infrastructure issue unrelated to this PR's changes. All 24 tests across all services (CloudWatch, S3, DynamoDB, SQS) fail with the same error: This indicates LocalStack never became available before the tests ran. The workflow only sleeps 10 seconds after For reference, run #15 on master passes clean with 3m12s total. Our PR runs take 7m+ because LocalStack startup alone exhausts the window. This PR's changes are additive only — the new keyword and 4 new test cases follow the exact same pattern as the existing tests. Once the LocalStack readiness issue is resolved (e.g. by adding a health-check poll instead of |
Summary
The existing
CloudWatch Logs Insightskeyword covers simple use cases but has four limitations that make it unsuitable for security auditing and multi-account investigations:@ptrfield is stripped from every result rowThis PR adds a new keyword
CloudWatch Logs Insights Querythat addresses all four while remaining fully backward-compatible (the original keyword is unchanged).New keyword:
CloudWatch Logs Insights QueryParameters
log_groupquerystart_epochstart_timeend_epochstart_timestart_epochis not settimeoutTimeoutErroris raisedpoll_intervalreturn_metadata{results, queryId, statistics}dictlog_group_namesRecommended pattern: consolidated payer-account CloudTrail
AWS Organizations supports an organization-level CloudTrail that delivers all member-account events into a single log group in the management (payer) account. This is the recommended pattern for multi-account environments because:
LOG_GROUP_NAMEvariable covers every account — no per-account session switching.recipientAccountIdin every CloudTrail event lets you scope queries to a specific member account without extra infrastructure.Security queries that benefit from this pattern (included in the library's example suite):
StopLogging,DeleteTrail)Files changed
src/AWSLibrary/keywords/cloudWatch.py— newinsights_query_advancedmethod bound to keywordCloudWatch Logs Insights Querytests/robot/cloudwatch.robot— four new test cases (relative window, absolute window, return_metadata, existing smoke preserved)Testing
All new tests are structured to run against localstack (following the existing pattern in the project). The tests use the same
Create Session And Set Endpoint/Delete All Sessionssuite setup as the rest of the CloudWatch suite.No existing keywords or tests were modified.