The One Minute Manager: Lightweight On-Demand Overlays for Distributed Application Management

Abstract

The emergence of large-scale distributed computing clusters such as PlanetLab and Utility Grids has fueled the development of applications ranging from content distribution to name service to large-scale prototype experiments. However, the management of such applications when they are deployed in a real world, wide area environment remains a challenging problem. In this paper, we present MON (Management Overlay Networks), a simple, scalable and lightweight system for distributed application management. At the most basic level, MON builds short-lived, on-demand overlays that can be used to execute management commands such as status query and control. To further address the coverage, reliability and performance issues of on-demand overlays, we exploit techniques such as incremental overlay construction, overlay adjustment and opportunistic DAG (directed acyclic graph) based aggregation, which greatly improve the practicality of on-demand overlays. Our extensive experiments on the PlanetLab show that for a large group of more than 300 nodes, on-demand overlays can be built to (1) cover more than 95% of the nodes (2) last for tens of minutes even without failure repairs; and (3) achieve an end-to-end response time of just a couple of seconds. Further, we demonstrate the utility of MON by showing how it can be used to query the aggregate state of a real application (Pastry) deployed in a real world environment.