typing.Metric

A metric maps (pred, truth) forecast samples and ground truth to a scalar array.

typing.Metric=Callable[[Array, Array], Array]

pred has the sample axis first, shape (sample, *batch); truth has shape (*batch); the result is a 0-d array. Metrics must be pure JAX functions (jit- and vmap-compatible): host floats appear only at result boundaries (evaluate_forecast(), BacktestResult), never inside metrics, so backtest_vectorized() can vmap any metric over the window axis. Parametrize by closure: functools.partial for keywords (e.g. partial(eval_coverage, alpha=0.5)) or a factory like make_mase().

Every driver checks each result on its own, so the outcome for one metric never depends on the other metrics in the mapping: a 0-d bool or integer result, or a float8 or float4 one, is converted to the default float dtype (a float6 one keeps its dtype: XLA on CPU cannot convert it), and any other shape or dtype (for example the per-cell values of crps_empirical(), whose metric counterpart is eval_crps()) raises ValueError with the name, shape and dtype of the metric. evaluate_forecast() and backtest() check the value they score; backtest_vectorized() checks the value of one window inside its vmap, and traces the scoring on the window shapes before the fit, so the error comes before any window is fitted.

The built-in metrics and evaluate_forecast() check that truth.shape is pred.shape[1:] and raise ValueError otherwise. With the runtime type hook on (the default), the jaxtyping annotations reject a mismatched array first and raise jaxtyping.TypeCheckError. A Python or NumPy scalar truth passes the annotations, so the explicit check raises ValueError for it. The metric that make_mase() returns is a nested function, which the hook does not check, so it always raises the ValueError. In every case a mismatched truth never broadcasts into a wrong value.

A NaN is a valid result: the built-in metrics return NaN when a truth value or a forecast draw is NaN (see DEFAULT_METRICS), and the drivers pass it through without an error. A custom metric should follow the same rule, so that a NaN in a backtest table marks a window with missing input for every metric.

pred and truth may have any real dtype. A count model (a discrete observation family in predict()) needs integer data, so its forecast draws are integers. The built-in metrics cast an integer operand to the default float dtype (float32, or float64 with x64 enabled) before they compute, and keep a floating operand in its own dtype. float32 holds every integer up to 2^{24} exactly; larger counts round unless x64 is enabled. evaluate_forecast() (and with it backtest()) and backtest_vectorized() cast once before they call the metrics, so every metric receives floating arrays and can compute jnp.abs(pred - truth) without integer wraparound; a custom metric that needs the integer draws must read them from the kept predictions (keep_predictions=True) or cast back inside the metric. The built-in metrics reject boolean and complex arrays.