Optimizing Traffic Management Systems: A Comparative Study of Apache Spark Deployments
March 2024
In today's fast-changing technological world, it is important to understand data quickly, especially when it comes to recognising and responding to incidents. Our recent study addresses this topic, focusing on how a powerful tool called Apache Spark performs in two different setups: one with containers and one directly on physical hardware, referred to as 'bare metal'.
Apache Spark is a kind of super-intelligent data processor. We tested how well it performs when set up in containers (like virtual boxes) with tools like Docker and Kubernetes, and how well it performs when installed directly on the computer hardware without virtual layers ('bare metal'). We wanted to understand this in order to decide how we should set up the technology, especially in situations where rapid incident detection and handling is extremely important.
We found that bare metal configurations provide direct access to the inner workings of the computer and are therefore faster. On the other hand, containerised setups managed by Kubernetes have the advantage of being scalable, resource efficient and easy to use. This difference is crucial for traffic management systems. The ability to change and improve our models quickly is key to keeping our traffic management system work well. Quick model changes mean quicker responses to unexpected traffic incidents, so our management solutions are accurate and timely.
As businesses and researchers work with large amounts of data, the way we set up Apache Spark becomes increasingly important. Using containers helps us to create really efficient dashboards that use smart algorithms to visualise data. This synergy between smart computing and container technologies is particularly useful and opens the door for future dashboard developments and emphasises the need to choose the right setup for different computing needs.
To summarise, our recent study conducted by Frontier Innovations researchers comparing how Apache Spark works in container configurations with bare-metal environments has shown us how to best set up the CONDUCTOR management system. By choosing the right setup that allows us to quickly change and improve our models, we can ensure that our incident detection technology works really well and provides fast and accurate responses. This not only helps us to decide how best to set up our technology, but also helps to improve traffic management technology. It shows how important it is to choose the right technical setup for fast and efficient data analysis and response systems.