scrutiny.cloud: Open, transparent, end-user verifiable and LLM-ready security testing framework
Security products are typically tested against a range of currently known weaknesses and good design practices. While a dedicated evaluation laboratory (e.g., a FIPS 140 or CC lab) will typically perform the testing with a high level of expertise and specialized equipment, such testing is typically done only once, does not publicly disclose all testing details, and is thus not independently replicable by the product end-user. Independent researchers offer vital alternative analysis, but their results are scattered across blogs and papers, hard to discover, and frequently vulnerable to censorship or removal.
This talk introduces the scrutiny.cloud project, aimed at solving the following problems:
- End-user verifiability (knowing what was tested, how, and ideally having the possibility to replicate it),
- Discoverability (how to find relevant testing information produced by others),
- Censorship problem (vendors refusing to disclose reported vulnerabilities),
- Spam/trust problem (inherently associated with permissionless systems), and
- Data interpretation problem (extracting actionable information from a diverse range of partial metadata data points).
The idea is to use existing open analysis tools (some provided by us, but not limited to those) to produce signed metadata from testing, then upload and share it via scrutiny.cloud, a thin metadata overlay built atop the open nostr protocol. The uploaded metadata can later be discovered and interpreted by humans and/or LLMs. As all metadata blobs are signed by their producers, filtering input information to include only trustworthy entities is already a native feature inherited from the nostr protocol.