Parallelization framework for Java on distributed-memory clusters

I’ve been participating in this fascinating discussion with Rich Hickey, the author of Clojure about Software Transactional Memory.  I decided to cleanup and echo the conversation here, but the original can be found here.

"This paper describes a new QR factorization algorithm which is especially designed for massively parallel platforms combining parallel distributed multi-core nodes. These platforms make the present and the foreseeable future of high-performance computing. Our new QR factorization algorithm falls in the category of the tile algorithms which naturally enables good data locality for the sequential kernels executed by the cores (high sequential performance), low number of messages in a parallel distributed setting (small latency term), and fine granularity (high parallelism)."

For as long as I’ve been in computing, the subject of concurrency has always induced a kind of thinking man’s hysteria. When I was coming up, the name of the apocalypse was symmetric multiprocessing — and its arrival was to be the Day of Reckoning for software. There seemed to be no end of doomsayers, even among those who putatively had the best understanding of concurrency. (Of note was a famous software engineer who — despite substantial experience in SMP systems at several different computer companies — confidently asserted to me in 1995 that it was “simply impossible” for an SMP kernel to “ever” scale beyond 8 CPUs. Needless to say, several of his past employers have since proved him wrong…)

In this post I don’t want to focus on strategies, though. What has been bugging me about strategies is that, while the abstraction has some nice properties (compositionality, modularity), it does that at the expense of transparency. The problem is that it can be quite hard to reason about performance when using strategies, because you have to think quite carefully about when things are evaluated. In order to use a strategy, the input to the strategy has to be a data structure with some unevaluated components, that the strategy can exploit to create parallelism. Using laziness in this way is the key trick that allows strategies to be modular – indeed it’s an instance of the modularity insight from John Hughes’ famous “Why Functional Programming Matters” – but it can render the resulting program somewhat opaque to understanding the parallel execution.

Multi-core chips are a new paradigm!" "We are entering the age of parallelism!" These are today's faddish rallying cries for new lines of research and commercial development. Is this really the first time when computing professionals seriously engaged with parallel computation? Is parallelism new? Is parallelism a new paradigm?

I believe that concurrency, via mobile code and tierless architectures, will become the most significant opportunity for scalable parallelism. To a lesser extent, I believe this is already the case, but we need to improve certain aspects of our languages and code-distribution architecture to take fine-grained advantage of it.

Recently I've been doing some presentations as well as just general sketches of some things I've been thinking about regarding optimization, concurrency and data design. I've been posting them on Twitter to gather feedback from my pals there. A couple have caused a little controversy, but remember that all of them are given in the simple spirit of sharing ideas among peers. And don't forget it's all in good fun!

I attended the EuroPython conference in Birmingham last week. Nice place and nice meeting overall. There were lots of interesting talks on many subjects. I want to focus on the talks about concurrency here. We had a keynote by Russel Winder about the "multicore revolution" and various talks about different approaches to concurrency (Python-CSP, Twisted, stackless, etc). Since this is a hot topic in Python (and in other languages) and everybody wants to have his saying, I will take the occasion to make a comment.

Superpy distributes python programs across a cluster of machines or across multiple processors on a single machine. This is a coarse-grained form of parallelism in the sense that remote tasks generally run in separate processes and do not share memory with the caller.

"Ultimately, the advice I’ll offer is that these developers should start thinking about tens, hundreds, and thousands of cores now in their algorithmic development and deployment pipeline. This starts at a pretty early stage of development

"The story of how GPU came to be used for high-performance computation is pretty cool. Hardware heavily optimized for graphics turned out to be useful for another use: certain types of high-performance computations. In this article, I will explore how and why this happened, and summarize the state of general computation on GPUs today."

"It's actually surprising what you can learn about computer science from cooking and vice versa."

Netlog, datalog, graph queries, first-order logic, fixed-point logic.

"Fork/Join is similar to MapReduce in that they are both algorithms for parallelizing tasks." Gah! Okay, look. Fork/Join looks pretty damn cool. Neat use of reflection, and Doug Lea is an uber-smart guy (and good writer!), and automatic ways of detecting and implementing parallelism are teh hottness. But! I think the quoted line gives a good example of what Stonebraker and DeWitt were complaining about, when they wrote their criticism of MapReduce. First of all, it's not even clear in what sense MapReduce is "an algorithm." But even if you grant that, it's this reflexive comparison of "parallel things" to MapReduce, as if the latter were any kind of gold standard for doing things in parallel... it's just that, look, MR gets its power from the fact that it can rely on the programmer having already arranged his or her data into independent key-value chunks. *That's* what's doing the heavy lifting, algorithmically, in MR!

Tips on locking and concurrency, from the ACM Queue book, "The Concurrency Problem."