Anomaly Detection

Overview

Anomaly Detection continuously monitors your project's key metrics for statistically significant deviations. When a metric moves far enough from its typical value — measured with a robust (median/MAD) z-score and confirmed by a false-discovery-rate test — the system flags it as an anomaly and can trigger alerts.

The detector is completeness-aware: it knows the difference between "your visibility dropped" and "the worker only captured a fraction of a normal day's snapshots." Partial-capture days are excluded entirely, so sparse data can never trigger false alerts.

Monitored Metrics

The system analyzes three categories of metrics in a single pass:

Visibility Score Anomalies

Detects unusual spikes or drops in your daily visibility scores. A sudden drop could indicate that an AI model has changed how it references your brand, while a spike might signal a successful optimization or a temporary ranking artifact.

Sentiment Shift Anomalies

Aggregates sentiment scores by day (not per-snapshot, which would double-count) and watches for sudden moves — especially toward negative — that signal AI models are presenting your brand in a less favorable light.

Brand Mention Rate Anomalies

Tracks the daily rate at which your brand is mentioned across snapshots. A declining mention rate means AI models are responding to industry prompts without referencing your brand, which is a direct visibility concern.

Data Quality Safeguards

This is what separates a production-grade detector from a noisy one. Every run applies four safeguards: two gate the data before testing (completeness, excluding today), and two govern significance (false-discovery control, persistence).

Completeness Gate

Each day's count is compared to your project's typical daily volume for that metric — the median of the metric's positive daily counts across the lookback window:

typical = median(positive daily counts in the window)

A day is analyzed only if its capture count meets both thresholds:

• ≥ 80% of typical (the completeness ratio)
• ≥ 10 snapshots absolute (to prevent tiny projects from running detection on statistically meaningless samples)

We compare against the observed typical volume rather than a fixed active_prompts × active_models expectation, so a project that grew its prompt set over time isn't penalised — each historical day is judged against the volume that was normal in its own window, not today's denser expectation. The median is robust: when most days are complete it equals a full day's volume, and genuine partial-capture days fall below the cut.

Example: If the median full day in the window has ~100 captures, days with fewer than 80 are skipped. A day with 15 is clearly a partial capture — it gets excluded, not flagged.

Today Is Excluded

Today's date is always skipped. Today's captures are still running — treating a half-complete day as a full day would guarantee false drops at 9 AM.

Benjamini-Hochberg False-Discovery Control

Three metrics are tested simultaneously (visibility, sentiment, mention rate), each over its recent points. Testing many things at once inflates false positives. Rather than the overly-strict Bonferroni correction (which controls the chance of any false positive and suppresses real signal), the detector pools every candidate's p-value and applies the Benjamini-Hochberg procedure, which controls the false-discovery rate — the expected fraction of flagged anomalies that are false — at 5%. This is the modern standard for monitoring: it keeps detection power high while still bounding false alarms.

Persistence Check

Every flagged anomaly is marked with a persistent flag. An anomaly is persistent when the previous day's point is also anomalous — a single spike is noise; two in a row is signal. Use the persistent flag to filter out one-off events in your alert rules.

How Detection Works

Robust Z-Score (Hampel Identifier)

For each metric, the system measures how far each recent point sits from the typical value over the lookback period (default: 30 days, excluding partial-capture days) — using robust statistics, the median and the Median Absolute Deviation (MAD):

robust z = (actual_value - median) / (c_n x MAD)

Why not the ordinary mean and standard deviation? Because the very anomalies we are hunting distort them: one extreme day inflates both the mean and the SD, which masks that day and raises the bar so genuine anomalies elsewhere slip through (the "masking effect"). The median and MAD have a ~50% breakdown point, so the baseline is unmoved by a handful of outliers. The c_n factor makes the scaled MAD a consistent estimator of the standard deviation, so the robust z reads on the familiar scale. We use the finite-sample-corrected constant c_n = 1.4826 x n / (n - 0.8) (Croux & Rousseeuw, 1992) rather than the bare asymptotic 1.4826: on the small daily windows monitoring produces (often 14-30 days), the raw MAD slightly under-estimates the true spread, which would inflate the z-score and surface false anomalies. The correction inflates the scale just enough at small n (about 6% at 14 days) and converges to 1.4826 as the window grows. Every recent point is then turned into a p-value, and the Benjamini-Hochberg procedure (below) decides which are significant — points are not pre-filtered by a fixed z-threshold, because selecting the test family from the observed values would invalidate the false-discovery-rate guarantee.

Requirements

Before any metric is tested:

• ≥ 7 clean data points in the lookback window (after completeness filtering)
• Non-zero robust scale (there must be actual variation, not a flat series)
• Only the 3 most recent data points are checked on each run

Severity Levels

Severity is assigned from the robust-z magnitude after a point passes the false-discovery-rate test:

Anomaly Types

Segment Filtering

Anomaly detection can run for any prompt segment:

• All prompts (default)
• Branded — Spot shifts in how your brand is referenced when explicitly asked.
• Non-Branded — Spot shifts in organic discoverability (the most sensitive segment).
• Competitor — Spot shifts in head-to-head positioning.

Pass the ?segment= parameter to the API or use the segment toggle on the analytics page.

Alert Integration

Detected anomalies are stored in the anomaly_events table with full context:

1. The event is recorded with expected value, actual value, z-score, persistent flag, and detection date.
2. If alert rules are configured for anomaly events, notifications fire via email or webhooks.
3. Anomalies remain in the active queue until acknowledged.
4. New runs deduplicate against existing unacknowledged events — you don't get alerted twice for the same anomaly.

Configuration

How to Use

1. Navigate to Analytics → Anomalies, or use the MCP tool detect_anomalies.
2. Focus on persistent anomalies first — these have the strongest signal.
3. Check severity: Critical (robust z ≥ 4.5) is truly unusual; investigate immediately.
4. Correlate with recent changes — deploys, content launches, competitor activity, LLM model updates.
5. Acknowledge anomalies once investigated to clear them.
6. Configure alert rules for automatic notifications.

Plan Requirements

Anomaly Detection requires the Advanced AI Insights feature, available on Pro-Individual plans and above.