Frederic T. Chong, Rajeev Barua, Fredrik Dahlgren, John D. Kubiatowicz, and Anant Agarwal. The Sensitivity of Communication Mechanisms to Bandwidth and Latency. Proceedings of 4th Int'l Symposium on High Performance Computer Architecture(HPCA), Las Vegas, NV, Feb 1-4, 1998.
(pdf, compressed postscript)

Abstract:

The goal of this paper is to gain insight into the relative performance of communication mechanisms as bisection bandwidth and network latency vary. We compare shared memory with and without prefetching, message passing with interrupts and with polling, and bulk transfer via DMA. We present two sets of experiments involving four irregular applications on the MIT Alewife multiprocessor. First, we introduce I/O cross-traffic to vary bisection bandwidth. Second, we change processor clock speeds to vary relative network latency.

We establish a framework from which to understand a range of results. On Alewife, shared memory provides good performance, even on producer-consumer applications with little data-reuse. On machines with lower bisection bandwidth and higher network latency, however, message-passing mechanisms become important. In particular, the high communication volume of shared memory threatens to become difficult to support on future machines without expensive, high-dimensional networks. Furthermore, the round-trip nature of shared memory may not be able to tolerate the latencies of future networks.