Skip to content

Multiple issues causing the benchamarks to be invalid and unrealistic. #3

Description

@Ekultek

A signal defined as evidence of an out of membrane effect is captured, and not included in the branch that determines the call verdict. When that signal fires and the normal effect collection is empty, the trial resolves as prevented. That is the exact condition the tripwire exists to detect.

The same release still appears to:

  • use proposal side data as ground truth for what executed;
  • evaluate authorization from run state rather than effect time;
  • ignores identity when correlating telemetry;
  • normalize successful calls and exceptions the same way;
  • assign metadata rather than measure it;
  • retain the unresolved problems documented in issue This benchmark cannot be trusted at face value still. #2

Please trace the actual execution path and explain how a bypass only egress event produces an unauthorized verdict. If it cannot, then the benchmark is not measuring the failure that it claims to measure. This cannot be fixed with doc changes and will require an implementation change and a full rerun.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions