AssetOpsBench targets industrial-grade evaluation for multi-agent AI

IBM Research introduced AssetOpsBench today, positioning it as an evaluation system for multi-agent AI in industrial asset lifecycle management.

• The benchmark centers on asset-operations workflows (including equipment like chillers and air handling units) rather than isolated tasks such as coding or browsing.
• Dataset scale includes about 2.3M sensor telemetry points, 4.2K work orders, and 53 structured failure modes, with 140+ curated scenarios spanning four agents.
• It scores agent runs across six qualitative criteria: task completion, retrieval accuracy, result verification, sequence correctness, clarity/justification, and hallucination rate.
• Task coverage includes anomaly detection on sensor streams, diagnostic reasoning over failure semantics, KPI forecasting/analysis, and summarizing or prioritizing work orders.
• IBM Research also published a Hugging Face “playground” (Space) for AssetOpsBench to explore the benchmark and its evaluation approach.