InDepth: A Distributed Data Collection System for Modern Computer Networks

Abstract

Cybersecurity researchers and security analysts rely heavily on data to train and test network threat detection models, and to conduct post-breach forensic analyses. Comprehensive data-including network traces, host telemetry, and contextual information-are crucial for these tasks. However, widely used public datasets often suffer from outdated network traffic and features, statistical anomalies, and simulation artifacts. Furthermore, existing data collection systems frequently face architectural and computational limitations, necessitating workarounds that result in incomplete or disconnected data. Currently, no framework provides comprehensive data collection from all network segments without requiring specialized or proprietary hardware or software agents. This paper introduces InDepth, a scalable system employing a distributed, data-link layer architecture that enables comprehensive data acquisition across entire networks. We also present a model cyber range capable of dynamically generating datasets for evaluation. We demonstrate the effectiveness of InDepth using real-world network data.