In this post I describe the framework for a promotion/relegation system, where inference workers on the Allora Network can be promoted to and from mainnet topics. This will be the intended pathway for workers to demonstrate their performance on mirrored testnet topics and gain access to mainnet.
Promotion and relegation will be evaluated on a per worker, per topic basis. In other words, a worker may qualify for promotion on one topic while failing on another, because performance, participation, forecast horizon, etc., are all topic-specific.
The system is intentionally asymmetric. Long-horizon topics require weaker statistical significance for promotion (otherwise extremely long timeframes would be required), while relegation requires strong significant of failure in any metric.
Evaluation metrics
Inference workers are evaluated on 7 metrics in log return (or return-like) space:
- Directional accuracy (DA): Fraction of observations where the predicted and true changes have the same sign. An invalid prediction counts as a directional miss when the actual change is valid.
- Pearson correlation: Whether prediction magnitudes move positively with true outcomes.
- WRMSE improvement: Magnitude-weighted RMSE improvement over a zero-change baseline. Positive values are better than the baseline. Magnitude weighting ensures larger movements receive larger weights, so a model cannot pass merely by performing well during quiet periods.
- WCZAR improvement: Magnitude-weighted CZAR (a direction-sensitive robust loss) improvement. Positive values are better than the baseline.
- Log aspect ratio:
log10(std(prediction) / std(actual)). Zero means correctly scaled variation; −1 means forecasts vary ten times too little; +1 means ten times too much. - Normalized mean offset: Mean forecast error divided by the standard deviation of actual outcomes. This detects extreme systematic bias.
- Participation: Fraction of available forecasting opportunities on which the worker submitted, since the first submission (and allowing for whitelisting periods).
For each of the metrics we evaluate workers using their confidence intervals to take into account differing numbers of submissions. In topics where the epoch length is shorter than the ground truth lag, adjacent epochs are not independent due to their overlapping ground truth windows. Therefore, we use autocorrelation of returns to calculate an effective sample size n_eff and correct the confidence intervals.
An effective sample size is calculated for each inferer on the true returns for epochs with submitted inferences. Summing significant, positive autocorrelation values for epoch lags up to ground_truth_lag / epoch_length - 1, the effective sample size is n_eff = n_inferences / (1 + 2 * correlation_sum). For 100% participation we find n_eff ~ n_inferences * epoch_length / ground_truth_lag, while sparse or otherwise inconsistent participation is naturally handled by the autocorrelation correction.
Price, log-return, and volatility topics
The policy thresholds are the same across topic types, but the forecast representation and preprocessing differ between topic types.
Price topics
A price forecast is converted into a log return relative to the price known at the forecast time. The zero baseline therefore represents an unchanged price. Price forecasts must be positive to produce a valid log return. A zero or negative price forecast is invalid.
Log-return topics
The worker already submits the predicted log return, so no conversion is required. Otherwise, the resulting skill metrics are directly comparable with price topics.
Volatility topics
Volatility ground truth is calculated from one-minute log-price returns over trailing windows:
- Base volatility
V(t)covers the window ending at forecast time. - Target volatility
V(t+h)covers the following horizon beginning at the forecast time. - The evaluated change is the log ratio of target volatility to base volatility (analogous to log returns of prices).
However, volatility topics contain predictable mean reversion which inflates the directional accuracy relative to price and log-return topics. We consider the mean reversion expectation as the baseline that workers on volatility topics should pass. The evaluation therefore estimates a causal, trailing expectation
m(t) = a + b log[V(t)]
using a recency-weighted least-absolute-deviations fit. m(t) is subtracted from both predicted and actual log-volatility changes before calculating the evaluation metrics. Only information already available at forecast time is used. We use a trailing window of W horizons that depends on the horizon time h in minutes as
W = max(30, ceil(50 * sqrt(60 / h)))
since longer horizons are more more stable because they are calculated over a larger number of candles. This formula gives 100 windows for 15 min topics, 50 windows for 1 hour topics, and 30 windows for topics >= 4 hours.
Promotion to mainnet
Promotion asks for broad, statistically supported evidence of skill, reasonable calibration, and reliable participation. A worker must pass every promotion metric on a topic:
- Raw effective sample size must be at least 20.
- The one-sided 95% DA lower bound must be greater than 0.50.
- The two-sided 95% Pearson lower bound must be greater than zero.
- The adaptive WRMSE-improvement lower bound must be greater than 0%.
- The adaptive WCZAR-improvement lower bound must be greater than 0%.
- The Log-AR confidence interval must overlap the range [-0.5, +0.5].
- Participation must be strictly greater than 90%.
Forecast-horizon adjustment
Long-horizon topics generate independent evidence more slowly. For the DA, WRMSE, and WCZAR promotion tests, the implementation applies:
scale = max(1, horizon_minutes / 20)^0.5
The DA test uses the effective sample size n_eff × scale which reduces the confidence level from 0.95 at a 20 min horizon to ~ 0.71 at a 24 hour horizon. WRMSE and WCZAR divide their reconstructed standard errors by sqrt(scale) before calculating a one-sided 95% lower bound. The minimum n_eff ≥ 20 gate is checked before this adjustment. Pearson and Log AR do not receive the horizon adjustment.
A worker’s directional accuracy at a given horizon can then be mapped to an expected promotion time:
Relegation from mainnet
Relegation tolerates ordinary metric noise but removes workers showing jointly poor predictive performance, severe calibration defects, extreme bias, or persistent non-participation. In contrast to promotion, relegation only requires failure of a single condition (composed of one or multiple metrics), but at a fixed confidence level of 0.95. In other words, relegation requires strong evidence of failure, even for long horizon topics.
Any one of the following conditions produces a relegation candidate.
1. Combined predictive underperformance
All four conditions must hold simultaneously:
- DA two-sided upper bound < 0.49
- Pearson upper bound < −0.01
- WRMSE-improvement upper bound < 0%
- WCZAR-improvement upper bound < 0%
Requiring all four protects workers from relegation because of one noisy or weak metric.
2. Severe forecast-scale mismatch
The complete Log-AR interval must lie outside [-1, +1].
This represents strong evidence that prediction variation is more than roughly ten times too small or too large.
3. Extreme systematic bias
A worker may pass the aspect ratio test while still submitting predictions with a consistent large directional bias. A worker triggers this relegation criterion when its absolute mean prediction bias exceeds:
abs(mean(pred − true) / std(true)) > 7.5
This criterion uses the point estimate rather than a confidence interval.
4. Inactivity
A worker is eligible for removal after more than 14 days without a submission. The calculation counts forecasting opportunities after the last submission rather than raw wall-clock time, excluding periods when the network did not produce usable opportunities (e.e outages or upgrades).
Participation below 50% over the previous 30 days also triggers relegation. For newly or recently re-whitelisted workers, the window begins at the latest whitelist activation.
Grace period
The default grace period is 30 days after the latest active whitelist interval:
- Predictive-performance, Log-AR, and normalized-offset cuts are skipped during this period.
- The inactivity cut is also skipped for 30 days by default.
- Chronic 30-day participation is still evaluated, but its window is shortened to the post-whitelist period.
Minimum worker floor
If executing every proposed relegation would leave fewer than five workers, all relegations for that topic are cancelled and marked as floor-protected. Floor-protected topics are highlighted as priorities for promotion review.
Future topics
The current promotion and relegation criteria are designed for topics whose predictions and ground truth can be expressed as a log return format. The evaluation framework may be extended in the future with topic-specific metrics and thresholds for other task types, such as classification.