“The model passed the evaluation.”
No, it didn’t. A system passed or failed. The same model behaves completely differently depending on the harness, the tools it can call, the skills loaded, the memory it carries, the permissions it holds, and the policy constraining it. Naming only the model in an assessment is like crash-testing a car and reporting only the engine.
CLI 0.4.0 fixes attribution. It gives every agent system a portable identity manifest, so results attach to the complete system that was actually tested.
What’s new
safe2 evidence system builds a system-identity manifest with eight component categories:
- Model
- Harness
- Tools
- Skills
- Memory
- Environment
- Policy
- Evaluator
Each component is versioned, the relationships between them are mapped explicitly (uses, invokes, governs, evaluates, runs-on), and the system’s authority is inventoried: read, write, execute, network, delegate, approve, admin. You can see not just what was tested, but what it was allowed to do.
Two details make this serious engineering rather than a spreadsheet:
- Dual hashes. Every manifest carries an exact-source SHA-256 and a normalized composition fingerprint that’s stable across formatting changes. Reformat the config file, and you can still prove it’s the same system.
- Honest states. Missing, partial, and not-applicable are distinguishable. A manifest that can’t see a component says so instead of pretending the component doesn’t exist.
Most assessments name only the model. The same model behaves differently across harness, tools, skills, and permissions. 0.4.0 makes the invisible parts of the system visible and attributable.
Why upgrade
- Results become re-identifiable. Six months later, you can prove the system you tested then is (or isn’t) the system running now.
- Blame gets precise. When something fails, you know which component changed, not just which model was in the room.
- Authority is on the record. The permission inventory shows what the system could do, which is half of any risk assessment.
- Everything downstream binds to this. Failure diagnosis (0.5.0), change attribution (0.6.0), and operational truth (0.7.0) all anchor to the identity 0.4.0 establishes. Skip it and the later releases have nothing to hold onto.
Takeaways
- 0.4.0 gives agent systems a portable, versioned, hashed identity.
- Eight component categories; authority inventoried; relationships mapped.
- This is the foundation every later CLI release builds on.
Bottom line: Stop assessing models. Start assessing systems. 0.4.0 is how.
Release notes: AI SAFE² CLI 0.4.0 on GitHub, tag 2026-09-14_CLI_0.4.0_Know_What_You_Actually_Tested.
