Evidence standard

What this page is

MatterCheck does not run a hardware test lab and does not buy devices. When a matrix cell needs a hands-on confirmation to move from Med to High confidence, we accept dated third-party tests — but only when they meet the requirements in methodology rule 8. This page is the public statement of that standard, and it replaces the old in-house test protocol.

Every rating still has to trace to a cited source. A third-party test can be the hands-on half of a High rating; it can never be the whole rating on its own, because High also requires an official platform statement covering the same behaviour.

The two halves of a High rating

A cell reaches High confidence only when it has both:

• An official platform statement covering the specific field — for example, a platform's Matter supported-device list for pairs, or its automation/routines help for automation_trigger or automation_action.
• At least one independent, dated hands-on test of that device type on that platform, using a native (directly commissioned) Matter device.

Each field rated High needs an official statement covering that field. One dated native hands-on of the device type on that platform is enough to cover the whole cell — we don't require a separate hands-on per field or per sub-feature (a lock's auto-lock or a thermostat's fan mode). A statement that a device can be an automation action is not enough to rate the trigger field High.

Rule 8: what a third-party test must have

To be accepted as the hands-on half of a High rating, a third-party test must be:

1. Dated. We need to know when the test happened so staleness can be checked. Undated write-ups are corroboration at best, never primary evidence.
2. Independent. The test must be done by someone with no interest in the result — a review site, forum user, or community tester. Vendor guides, vendor-published tests, and sponsored content without disclosure do not count as independent.
3. Native Matter path. The device must be commissioned directly over Matter — paired with its Matter setup code into the platform's own Matter path. A test done through a vendor cloud account, a proprietary hub, or a bridge is not a native Matter test, and it cannot isolate what Matter is doing from what the maker's cloud is doing.
4. Video or screenshot proof. A claim without a reproducible, visible demonstration is treated as an anecdote, not a test. A dated native report that only shows the device being added to the platform, with no control or automation shown, does not count.

Primary evidence vs corroboration

Corroboration is evidence that supports a rating but cannot carry it to High. Vendor guides and tables, aggregators, and official pages the platform labels as AI-generated (for example Google Home release notes "generated with the help of Gemini") are all corroboration only. A field that rests on such a page alone stays Med.

Community reports sit below hands-on tests: with no official source, two or more independent community reports that agree support Med, and a single report supports Low at most. Anecdotes alone are never High.

The evidence hierarchy

1. Platform Matter supported-device list (official)
2. Platform automation / routines documentation (official)
3. Platform release notes / blogs (release-note)
4. Reproducible public test write-up (test)
5. Independent community reports (community) — Med/Low only

Every cell that can cite a URL does, and those sources are shown as clickable links on the matrix, filter and type pages.

What does not count

• Cloud or bridge tests. If the maker's cloud account or a proprietary hub was involved, the native Matter path isn't isolated, so the test can't confirm Matter behaviour.
• Sponsored content without disclosure. A paid placement or an unrevealed affiliate review is not independent.
• Pairing-only reports. Adding a device to the platform, with no control or automation shown, does not demonstrate the field being rated.
• Omission-only "No" ratings. An official list leaving a type out supports N at Med at most. N at High needs explicit evidence such as a hands-on "not supported / not shown" report or a platform mapping table.
• AI-generated official pages. Treating a platform page the platform itself labels as AI-generated as a primary source would overstate the evidence.

Special cases

Lights. A dated native test of a more capable Matter light (dimmable, white-spectrum or colour) counts for the on/off light row only if it shows that light being switched on or off, because every Matter light includes On/Off. The reverse does not hold: the dimmable/colour light row needs a test showing colour being changed on a colour-capable bulb, and a white-spectrum-only bulb does not count. The colour temperature light row needs a test showing the white tone being changed.

Conflicting tests. If independent native hands-on reports disagree on a field, confidence stays Med until a newer independent native test resolves it.

Unknown fields. A cell cannot be High while any field is Unknown unless an official platform statement covers that field. For an Unknown automation trigger, that means a statement about the device type starting automations — a statement that it can be an action is not enough.

Script Editor only. A starter or action that works only in Google's Script Editor, not the Google Home app's automation editor, stays Partial.

How to submit a test

If you've run a dated, independent hands-on test of a Matter device on a platform — with the native Matter path and video or screenshot proof — you can submit it through the corrections and submissions form (linked from the site footer once live). Submissions that meet the rule 8 requirements are reviewed and may move a cell from Med to High; tests that use a cloud or bridge path, or that come from the device maker without disclosure, are not accepted. See our editorial policy for how submissions and corrections are handled.

Last reviewed 2026-10-23. This standard applies to every High-confidence cell in the matrix; the full rules live in the methodology.