TimesX: Rethinking Multimodal Time-Series Forecasting Evaluation

Overview

A benchmark built for realistic multimodal forecasting

TimesX studies how pretrained time-series foundation models and language models behave when forecasts require both historical numeric signals and natural-language context such as metadata, calendar effects, covariates, and time-stamped events.

190 core variables

19 domains

4 context types

13 evaluated methods

312k+ LLM inferences

2018-2025 refreshable time span

Comparison of multimodal time-series forecasting benchmarks. — **Table 1.** Comparison of representative multimodal TSF benchmarks. TimesX is the only benchmark that is simultaneously real-world, leakage-free, covers all four context types, and reaches benchmark scale with 190 variables.

Design Principles: Real, Refreshable, Comprehensive and High-Quality

Real-World Data, Refreshable (Continuously Mitigating Leakage), and Comprehensive High-Quality Context

Three principles separate TimesX from prior benchmarks. Each is backed by a controlled experiment showing where existing evaluations can be misleading.

Synthetic versus real-world performance under the same setup. — **Real data.** Under the same setup, the method ordering on synthetic CiK (CodeRev > Gemini > TimesFM) completely flips on real-world TimesX. Synthetic generation can bias rankings toward instruction-following LLMs and coding.

Performance before and after the LLM knowledge cutoff. — **Strict time isolation.** Across the June-2024 knowledge cutoff, both LLMs degrade by ~13%, while leakage-free TFMs stay within 2% — direct evidence of the contamination that TimesX's time isolation is designed to prevent.

Controlled replacement of textual context on Time-MMD. — **Detailed context.** Replacing Time-MMD's original context with TimesX-quality context lowers geometric-mean MASE from 0.906 to 0.840 (7.3% relative).

Construction Pipeline

An automatic dataset agent driven by multi-agent collaboration and adversarial verification

The textual event pipeline uses constrained LLM agents and time-bounded retrieval to discover candidate events, verify sources and timestamps, enrich missing details, and synthesize grounded event narratives.

Deep research enriches the details. An Enricher agent runs iterative, time-bounded web searches and merges multiple sources to fill in missing event information.
Adversarial verification and collaboration mitigate hallucination. A Hypothesizer and a Verifier act adversarially to filter out fabricated or leaked content, then collaborate with the Enricher to raise event quality.
Time-bounded search and LLM date correction prevent leakage. Programmatic date-range constraints on Google search, combined with LLM timestamp correction, keep every annotated event strictly before its forecast window.

The four-role multi-agent dataset agent workflow. — **Figure 2.** The four-role dataset agent — Hypothesizer, Verifier, Enricher, and Synthesizer — for automatic, leakage-free event construction.

Data Sample

One variable, paired with its four context types

GAS_PRICE time-series example from TimesX. — **Time series.** A `GAS_PRICE` example from TimesX: daily gasoline price (USD/Gal), with the forecast horizon window highlighted and the event announcement/effect dates marked.

MetadataGasoline price (USD/Gal) in the Commodity Price domain, at daily frequency. Prediction target period: 2024-09-01 to 2024-09-15.

DateHistorical data as (timestamp, value): (2024-06-02, 2.4116), (2024-06-03, 2.3279), … Upcoming holidays in the prediction window: Labor Day (2024-09-02).

CovariatesFrom 2024-06-01 to 2024-08-31: (1) Brent Crude Oil (USD/BBL): maximum 87.43 on July 4, …, with an overall downward trend; (2) …

Events(1) On June 2, 2024, OPEC+ agreed to extend deep oil output cuts; the 2.2 million bpd cut would be extended until September 2024, then gradually phased out. Sources: [1] [2] [3]; (2) …

Empirical Findings

What our experiments on TimesX reveal

Simple ensemble methods are currently the most effective composition.
Training-free agentic solutions are often unstable.
Providing in-context samples is a promising direction.
We look forward to future fine-tuning work that addresses the problem more thoroughly.

Overall benchmark results of the 13 selected methods. — **Table 4.** Overall benchmark results (mean over 10 runs) of the 13 methods. The simple average ensemble *AvgEns: TimesFM-2.5 + Gemini-2.0-Flash* ranks first on both MASE and rank, ahead of every agentic revision method.

Boxplot of per-instance method performance. — **Figure 5.** Per-instance error spread. Agentic revision methods (CodeRev / FuncRev) show a wider spread and more severe outliers, which explains their unstable geometric means.

In-context learning probe results. — **Table 7.** An in-context-learning probe: a single retrieved demonstration makes TextRev surpass strong zero-shot baselines, suggesting that training-based methods can help.

Citation

BibTeX

@inproceedings{liu2026timesx,
  title     = {Rethinking Multimodal Time-Series Forecasting Evaluation},
  author    = {Liu, Haoxin and Zhou, Yichen and Sen, Rajat and
               Prakash, B. Aditya and Das, Abhimanyu},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning},
  year      = {2026}
}