# Losses in returns prediction topics

**URL:** <https://research.allora.network/t/losses-in-returns-prediction-topics/120>\
**Category:** Applications\
**Tags:** loss-functions\
**Created:** [June 2, 2025, 10:39am UTC](https://research.allora.network/t/losses-in-returns-prediction-topics/120 "2025-06-02T10:39:37Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![joel](https://yyz1.discourse-cdn.com/flex003/user_avatar/research.allora.network/joel/32/351_2.png) [@joel](https://research.allora.network/u/joel)\
**Post date:** [June 2, 2025, 10:39am UTC](https://research.allora.network/t/losses-in-returns-prediction-topics/120/1 "2025-06-02T10:39:37Z")

</div>

I’ve noticed some strange behaviour of the losses in topics predicting log-returns. Some workers occasionally provide extremely large inferences (up to ~ 10^12) compared to typical returns values (\<0.1), but still have reasonable values for their losses:

 ![image](https://canada1.discourse-cdn.com/flex003/uploads/allora/original/1X/23f668f8fe4993d407b612f3f467383283973d7e.png)  
So at some point the losses flatten out, and inferences very far from the true value can have a lower loss than those close to the true value. That seems like unintended behaviour to me. Could it be an issue with the loss function?

---

<div class="post-metadata">

**Author:** ![Apollo11](https://yyz1.discourse-cdn.com/flex003/user_avatar/research.allora.network/apollo11/32/354_2.png) [@Apollo11](https://research.allora.network/u/Apollo11)\
**Post date:** [June 2, 2025, 2:34pm UTC](https://research.allora.network/t/losses-in-returns-prediction-topics/120/2 "2025-06-02T14:34:49Z")

</div>

Yeah we originally took the loss function for log-returns topics from the OpenGradient page [here](https://docs.opengradient.ai/modelthon/#scoring-criteria). It could indeed be related to the functional form of the (M)ZTAE loss function, because it flattens at large absolute deviations. It uses a tanh, which is symmetric and saturates at ±1.

Probably we don’t want it to saturate, but exhibit continued but shallower power law behaviour. The knee should then be the same sigma where it is now, and the power law slope should be a parameter we control. But all of this depends on whether this hypothesis is correct. What would the above figure look like for a standard MSE loss?

---

<div class="post-metadata">

**Author:** ![joel](https://yyz1.discourse-cdn.com/flex003/user_avatar/research.allora.network/joel/32/351_2.png) [@joel](https://research.allora.network/u/joel)\
**Post date:** [June 3, 2025, 2:31am UTC](https://research.allora.network/t/losses-in-returns-prediction-topics/120/3 "2025-06-03T02:31:55Z")

</div>

I see. I made a quick figure of the ZTAE loss function for different true values with mean=0 and standard deviation=1. So the function becomes more asymmetric as the true value increases from the mean. But because it flattens off “infinite” values in the correct direction (relative to the mean) can receive quite low losses, which is what we’re seeing above.

 ![image](https://canada1.discourse-cdn.com/flex003/uploads/allora/original/1X/e42d416dfa8f5d74c414acfd811f0005ec07c562.png)

This is how losses look for a topic with an MSE loss function. The scatter around the expected loss function is I think related to uncertainty on how the ground truth is defined (see [here](https://research.allora.network/t/handling-ground-truth-granularity/116)), plus differences between data providers. So the “true” values I’ve used (from Tiingo, rounded to the nearest minute) might be slightly different from what the reputers used, which is why the scatter increases as the difference from ground truth decreases. Still, inferences with much larger differences than the ground truth uncertainty should largely be unaffected, and indeed they have the largest losses.

 ![image](https://canada1.discourse-cdn.com/flex003/uploads/allora/original/1X/fdcf161a1fca2eefdedd5d62d9a7768b78a8fbac.png)

---

<div class="post-metadata">

**Author:** ![Apollo11](https://yyz1.discourse-cdn.com/flex003/user_avatar/research.allora.network/apollo11/32/354_2.png) [@Apollo11](https://research.allora.network/u/Apollo11)\
**Post date:** [June 3, 2025, 8:15am UTC](https://research.allora.network/t/losses-in-returns-prediction-topics/120/4 "2025-06-03T08:15:36Z")

</div>

Oh wow, that’s a great visualisation: so if `x_true = 1`, then `x = +infinity` receives a better loss than `x < 0.5` in this case. That’s the somewhat ridiculous consequence of the tanh saturation.

Sounds like a PL modification of the tanh could do wonders. For instance, if we use this:

```auto
def smooth_power_tanh_general(x, alpha=0.5, beta=2.0, x0=1.0):
    y = x / (1 + (x / x0)**beta)**((1 - alpha) / 2)
    return y

xtest = np.linspace(-5, 5, 1000)
plt.plot(xtest, np.tanh(xtest), ':k', lw=2, label='tanh')
plt.ylim((-5,5))
for alpha_test in np.array([0.25, 0.5, 1]):
    ytest = smooth_power_tanh_general(xtest, alpha=alpha_test)
    plt.plot(xtest, ytest, label='alpha = '+str(alpha_test))
plt.legend()

```

then the function looks like:

 ![image](https://canada1.discourse-cdn.com/flex003/uploads/allora/original/1X/52e8bd3ff72d213e4ea181673fb4a21f0ba3ca9c.png)  
which gets rid of the saturation.

Maybe less important, but then we can also change the transition point (would keep this at `x0 = 1` tbh), here shown adopting `alpha = 0.5`:

 ![image](https://canada1.discourse-cdn.com/flex003/uploads/allora/original/1X/a82ff8d2d13be6ce610d21f11c6e07a763c3d9ef.png)

I guess something like this could be better? Then we’d just need to decide on the value of `alpha`.

---

<div class="post-metadata">

**Author:** ![joel](https://yyz1.discourse-cdn.com/flex003/user_avatar/research.allora.network/joel/32/351_2.png) [@joel](https://research.allora.network/u/joel)\
**Post date:** [June 4, 2025, 2:05am UTC](https://research.allora.network/t/losses-in-returns-prediction-topics/120/5 "2025-06-04T02:05:39Z")

</div>

Nice, that could be a really simple solution!

Here are some tests replacing tanh in the ZTAE loss function with a power-tanh function.  
Regarding the best value for alpha, it seems there’s a trade-off between larger loss values for inferences far from the ground truth (higher alpha) and keeping the asymmetric behaviour of the tanh function (lower alpha).

 ![image](https://canada1.discourse-cdn.com/flex003/uploads/allora/original/1X/7d9bef13f9582896570b6a842d7b13c748ed37f7.png)

Another way is to look at a version of the above figure that’s ‘folded’ about the true value, so for each function the lower line is in the same direction as the true value.  
Based on this I think alpha=0.1 is too close to the initial tanh function, so will still have much of the same issues because it is still quite flat past the ‘knee’ of the function. alpha=0.5 might be too close to a power law function, there’s not much difference in the positive and negative directions so may not sufficiently reward predictions in the right direction.  
That leaves alpha ~= 0.25 as a nice middle ground?

 ![image](https://canada1.discourse-cdn.com/flex003/uploads/allora/original/1X/2ab68164c817fbb55fc5a0176a66bd128798cd1e.png)

If we apply this power-tanh loss function to the initial returns topic data we get the following median losses, which I think is a good improvement to the current tanh function!

 ![image](https://canada1.discourse-cdn.com/flex003/uploads/allora/original/1X/064e6ce94acd9926eb563ddee2bd5d8f5dee8044.png)

---

<div class="post-metadata">

**Author:** ![Apollo11](https://yyz1.discourse-cdn.com/flex003/user_avatar/research.allora.network/apollo11/32/354_2.png) [@Apollo11](https://research.allora.network/u/Apollo11)\
**Post date:** [June 4, 2025, 7:39am UTC](https://research.allora.network/t/losses-in-returns-prediction-topics/120/6 "2025-06-04T07:39:54Z")

</div>

Perfect! We can keep `alpha` (and plausibly some other moving parts) as free parameters, set `alpha = 0.25` as default, and keep a close eye on how this improves the network inference. Glad we got this sorted so quickly!

---

<div class="post-metadata">

**Author:** ![Apollo11](https://yyz1.discourse-cdn.com/flex003/user_avatar/research.allora.network/apollo11/32/354_2.png) [@Apollo11](https://research.allora.network/u/Apollo11)\
**Post date:** [June 4, 2025, 7:59am UTC](https://research.allora.network/t/losses-in-returns-prediction-topics/120/7 "2025-06-04T07:59:20Z")

</div>

So, for future reference, then the final form of the ZPTAE loss (inserting P for power law) is:

```auto
def smooth_power_tanh_general(x, alpha=0.25, beta=2.0, x0=1.0):
    y = x / (1 + (x / x0)**beta)**((1 - alpha) / 2)
    return y

```

---

<div class="post-metadata">

**Author:** ![joel](https://yyz1.discourse-cdn.com/flex003/user_avatar/research.allora.network/joel/32/351_2.png) [@joel](https://research.allora.network/u/joel)\
**Post date:** [July 16, 2025, 2:25am UTC](https://research.allora.network/t/losses-in-returns-prediction-topics/120/8 "2025-07-16T02:25:53Z")

</div>

Looking into the ZPTAE loss function again, we decided to slightly modify it by adding a “penalty” term for outliers to the loss function. This leaves the main functionality unchanged for reasonable inferences, but further penalises extremely large outliers (obviously unrealistic values). The aim is to make outliers more obvious in losses/regrets which should help with inference synthesis and allow the forecasters to better take outliers into account.

```auto
def power_tanh(x, alpha=0.25, beta=2):
    return x / (1 + np.abs(x)**beta)**((1 - alpha) / beta)

def loss_zptae(y_true, y_pred, sigma, mean, alpha=0.25, beta=2, gamma=4, penalty_norm=0.01):
    # Z power-tanh absolute error

    z_true = (y_true - mean) / sigma
    z_pred = (y_pred - mean) / sigma

    pt_true = power_tanh(z_true, alpha=alpha, beta=beta)
    pt_pred = power_tanh(z_pred, alpha=alpha, beta=beta)
    
    main_term = np.abs(pt_pred - pt_true)
    penalty_term = (penalty_norm * np.abs(z_pred - z_true))**gamma

    return main_term + penalty_term

```

Visualisation of the ZPTAE loss function with (solid lines) and without (dotted lines) a penalty term.

 ![image](https://canada1.discourse-cdn.com/flex003/uploads/allora/original/1X/3e2dff0456edf0cc77d1fdd4f414326a686eb662.png)

---

<div class="post-metadata">

**Author:** ![Apollo11](https://yyz1.discourse-cdn.com/flex003/user_avatar/research.allora.network/apollo11/32/354_2.png) [@Apollo11](https://research.allora.network/u/Apollo11)\
**Post date:** [November 4, 2025, 10:12am UTC](https://research.allora.network/t/losses-in-returns-prediction-topics/120/9 "2025-11-04T10:12:43Z")

</div>

Maybe useful to add now, because we use it so much these days: we also evaluate models by calculating the weighted ZPTAE improvement relative to predicting zero log-returns.

Using the above function `power_tanh`, this works as follows:

```auto
def wzptae_improvement(y_true, y_pred, alpha=0.25, beta=2):

    stdev = np.std(y_true)
    weights = np.abs(y_true)

    pt_true = power_tanh(y_true/stdev) # assuming mean=0 for log-returns
    pt_zero = power_tanh(0/stdev, alpha=0.25, beta=2)
    pt_pred = power_tanh(y_pred/stdev, alpha=0.25, beta=2)

    zptae_baseline = np.sum(weights*(np.abs(pt_true-pt_zero)))/np.sum(weights)
    zptae_model = np.sum(weights*(np.abs(pt_true-pt_pred)))/np.sum(weights)

    return (zptae_baseline - zptae_model) / zptae_baseline * 100 # in %

```

The analogous calculation for the weighted RMSE improvement relative to predicting zero log-returns is:

```auto
def wrmse_improvement(y_true, y_pred):

    weights = np.abs(y_true)

    mse_zero = y_true**2
    mse_pred = (y_true - y_pred)**2

    wrmse_baseline = np.sqrt(np.sum(weights * mse_zero) / np.sum(weights))
    wrmse_model = np.sqrt(np.sum(weights * mse_pred) / np.sum(weights))

    return (wrmse_baseline - wrmse_model) / wrmse_baseline * 100 # in %

```