Natjam: Eviction Policies For Supporting Priorities and Deadlines in Mapreduce Clusters
Author(s)
Gupta, Indranil
Cho, Brian
Rahman, Muntasir Raihan
Chajed, Tej
Abad, Cristina L.
Roberts, Nathan
Lin, Philbert
Issue Date
2013-06-06
Keyword(s)
Cloud Computing
Mapreduce
Scheduling
Abstract
This paper presents Natjam, a system that supports arbitrary job priorities, hard real-time scheduling, and efficient preemption for Mapreduce clusters that are resource-constrained. Our contributions include: i) smart eviction policies for jobs and for tasks, based on resource usage, task runtime, and job deadlines; and ii) a work-conserving task preemption mechanism. We incorporated Natjam into the Hadoop YARN scheduler framework (in Hadoop 0.23). We present experiments from deployments on a test cluster, Emulab and a Yahoo! commercial cluster, using both synthetic traces as well as Hadoop cluster traces we obtained from Yahoo!. Our results reveal that Natjam incurs overheads of under 7%. Under real Hadoop workloads, Natjam performs better than existing techniques.
Use this login method if you
don't
have an
@illinois.edu
email address.
(Oops, I do have one)
IDEALS migrated to a new platform on June 23, 2022. If you created
your account prior to this date, you will have to reset your password
using the forgot-password link below.
