Ziv Scully

Dispatching systems, where arriving jobs are immediately assigned to one of multiple queues, are ubiquitous in computer systems and service systems. A natural and practically relevant model is one in which each queue serves jobs in FCFS (First-Come First-Served) order. We consider the case where the dispatcher is size-aware, meaning it learns the size (i.e. service time) of each job as it arrives; and state-aware, meaning it always knows the amount of work (i.e. total remaining service time) at each queue. While size- and state-aware dispatching to FCFS queues has been extensively studied, little is known about optimal dispatching for the objective of minimizing mean delay. A major obstacle is that no nontrivial lower bound on mean delay is known, even in heavy traffic (i.e. the limit as load approaches capacity). This makes it difficult to prove that any given policy is optimal, or even heavy-traffic optimal. In this work, we propose the first size- and state-aware dispatching policy that provably minimizes mean delay in heavy traffic. Our policy, called CARD (Controlled Asymmetry Reduces Delay), keeps all but one of the queues short, then routes as few jobs as possible to the one long queue. We prove an upper bound on CARD's mean delay, and we prove the first nontrivial lower bound on the mean delay of any size- and state-aware dispatching policy. Both results apply to any number of servers. Our bounds match in heavy traffic, implying CARD's heavy-traffic optimality. In particular, CARD's heavy-traffic performance improves upon that of LWL (Least Work Left), SITA (Size Interval Task Assignment), and other policies from the literature whose heavy-traffic performance is known.

Looking Beyond Mean Delay in Queue Scheduling A central aim of queueing theory is to design scheduling policies that reduce delays. Much of literature focuses on minimizing mean delay, but, in practice, it can be more important to reduce the delay tail—that is, the probability of especially large delays. There is a subtle interplay between these objectives. For instance, the SRPT policy (Shortest Remaining Processing Time) prioritizes short jobs over long jobs, which optimizes mean delay. Sometimes, SRPT also has optimal delay tail, but other times, its delay tail is pessimal. Thus, good mean delay need not imply good delay tail. The article When Does the Gittins Policy Have Asymptotically Optimal Response Time Tail in the M/G/1? explores the interplay between mean delay and delay tail for systems with unknown job durations. The article studies the Gittins policy, which optimizes mean delay in this setting. Curiously, its delay tail can be optimal, pessimal, or in between, depending on the job duration distribution. Fortunately, the worst case can be avoided: the article shows how to modify Gittins to avoid pessimal delay tail while maintaining near-optimal mean delay.