The goal is to replicate the metric for the inferers on the forecast implied inferences. I modified the topic simulator to have a duplicate number of forecasters and to take an input to control the forecaster volatility. I am looking at the MDM of the FIIs, and normalizing by the MAE of the network combined inference. We perform simulations using 100 inferers and 30 forecasters. I will modulate three parameters in this simulation: the number of duplicate forecasters, the volatility of the inferers, and the volatility of the target.
Here are the results. Modulating inferer volatility:
Modulating forecaster volatility:
Modulating target volatility:
In general the MDMs are much smaller on the FIIs than they were for the raw inferences. In the last plot the MDM for 5 duplicate forecasters is smaller than the MDM for 7 duplicate forecasters. This seems odd. As a result (left most plot) the metric is not monotonically decreasing as duplicate forecasters increases. The second plots are a mess. It looks like those plots have better monotonicity but MDM/MAE does not maintain a constant ratio like it did for the raw inferences.
It is not immediately clear to me what is happening here. I will think about this more.