The evolution of the Internet of Things accelerated the development of Cyber-Physical Systems. Among them, Networked Control Systems (NCS) gained notable attention thanks to their application to industrial operations. Experimental NCS require expertise from control, computation, and communication disciplines. This requirement, together with the fragmentation of implementation platforms and experimental investigations, represents a challenge for the reproducibility and comparison of research results. In this paper, we tackle this problem by proposing a novel NCS benchmarking methodology that aids the reproducibility of NCS experiments. Relying on a novel approach to model the architectural elements and the delays of NCS, the methodology defines the experiment parameters and the relevant Key Performance Indicators (KPIs) that need to be observed during its execution. Furthermore, we detail the implementation of the first reproducible benchmarking platform for NCS. The proposed platform is open-source and designed to be easily reproducible and extensible by anyone. Finally, we replicate and evaluate the platform following the proposed NCS benchmarking methodology. The experimental results evaluate and compare the KPIs during the execution of the platform in different benchmarking scenarios, proving the validity of the proposed benchmarking methodology.

Authors: Samuele Zoppi, Onur Ayan, Fabio Molinari, Zenit Music, Sebastian Gallenmüller, Georg Carle, Wolfgang Kellerer

Authors: Charles Shelbourne, Leonardo Linguaglossa, Aldo Lipani, Tianzhu Zhang, Fabien Geyer

Authors: Johannes Naab, Patrick Sattler, Jonas Jelten, Oliver Gasser, Georg Carle

Authors: Stephan M. Günther, Nicolas Appel, Georg Carle

Authors: Paul Emmerich, Simon Ellmann, Fabian Bonk, Alex Egger, Esaú García Sánchez-Torija, Thomas Günzel, Sebastian Di Luzio, Alexandru Obada, Maximilian Stadlmeier, Sebastian Voit, Georg Carle

Authors: Paul Emmerich, Maximilian Pudelko, Simon Bauer, Stefan Huber, Thomas Zwickl, Georg Carle

Authors: Dominik Scholz, Andreas Oeldemann, Fabien Geyer, Sebastian Gallenmüller, Henning Stubbe, Thomas Wild, Andreas Herkersdorf, Georg Carle

Authors: Marcel von Maltitz

Authors: Quirin Scheitle

Authors: Oliver Gasser