How we test
The same rules as our Indus script project: a good-looking result must not be able to come from trying until something works.
Five rules
- Seal it before you look. Each measure, its preprocessing, its threshold and its pass mark are written down and SHA-256 hashed before we see any outcome.
- Test on people it never saw. Data is split by subject, and data from the same lab counts as a near-copy when we test transfer.
- Plant an answer first. A pipeline must recover an injected effect of known size, and must not call spectrum-matched noise 'conscious', before its output on real data counts.
- Let someone else try to break it. Every positive result goes to a red team that did not produce it.
- Publish it either way. Failed routes are written up with the size of effect the test could have caught.
Traps we watch for
- Report is the only ground truth. Every label is inferred from someone's report or behaviour, so we grade each label by what it rests on.
- Unresponsive is not unconscious. Ketamine and dreaming show the gap, which is why they are our hardest tests.
- The spectrum hides everything. Many complexity measures move with slow-wave power; we compare every measure against it.
- Analytic flexibility. The same COGITATE data has already been read both ways on offset ignition.
- Small samples. Most open datasets have 10-30 people; every negative states what it could have detected.