How we audited three real errors in Hormuz status verification

Why publish an error audit?

An automated monitor can fail in ways a conventional news story does not reveal: confusing outlet names, reading a question as a statement, or preserving an old status after the supporting evidence is no longer valid. Documenting these cases is more valuable than pretending the system is infallible. This page summarises three controls built from real project incidents.

Case 1: a short alias inside another word

An official source could be identified by the alias “ONA”. If the program searched for those letters as a substring, they also appeared inside “national”. The potential result was to attribute a National Security Journal or The National Interest story to Oman News Agency, unjustifiably increasing its authority.

Fix: short aliases must match whole words or a verified domain. A contradiction between the outlet in the headline and the detected publisher is no longer treated as a harmless warning.

Case 2: a question containing the word “open”

A headline such as “Why can’t X keep the Strait open?” contains “open” but does not state that the Strait is operational. It is an analytical question. A keyword-only classifier can produce a false positive.

Fix: question, opinion and “keeps insisting” headlines cannot confirm opening or closure. A separate factual description may still provide evidence when it contains an explicit operational statement.

Case 3: inheriting an invalid confirmation

Even after the classifier is fixed, a previously published status may persist through a continuity rule. If the last confirmation was invalid, the error can continue without a new story.

Fix: every cycle revalidates that the confirmation matches the status, signal, source and time window. If it fails, the system downgrades to UNCERTAIN rather than preserving inherited certainty.

What “human review” means here

Automation runs queries, normalises sources and applies rules. Human editorial responsibility means defining thresholds, investigating false positives, correcting attribution, documenting changes and not hiding uncertainty. It does not mean a person approves every hourly cycle; it means the rules and exceptions are auditable and correctable.

Change log and reproducibility

The repository retains tests and commits. The public status exposes check time, confidence, evidence and the last valid confirmation. Together they allow readers to reconstruct why a classification was published and whether a later change altered the outcome.

Why these controls improve, rather than weaken, credibility

Publishing a correction can appear uncomfortable, but hiding a known failure is more damaging. A transparent audit makes it possible to distinguish an error in source attribution from an error in the underlying maritime event. It also shows whether the correction changed only a label, removed a piece of evidence or altered the published status.

Each incident is converted into a regression test. The value of that test is not that software can never fail again; it is that the same known failure should not silently return after a later update. New edge cases may still appear, which is why the project keeps a corrections log and a conservative fallback to UNCERTAIN when supporting evidence no longer passes validation.

Primary sources

• IMO: shipping route and Traffic Separation Scheme
• UKMTO/JMIC advisory archive and traffic assessments

Explore more research View current status