Feature #11817: map.parallel

• Status: Closed
• Assignee: -
• Target version: -

[eike.rb (Eike Dierks)]

I'd like to suggest a new method: #parallel

That method should take care of executing workloads in parallel across multiple cores.

I believe that #map can sometimes be distributed to many simultaneous threads.

So we would spread out the map blocks into parallel processing,
and then collect the results in order.

I believe this would need some new infrastructure like apple did with great_central,
so I'm talking along these lines.
(Aka we only want to open up as many threads as there are cores)

This would not help much with a simple map (because we need to collect and serialize the results in the end)
but it could be cool for more complex mappings.

Maybe this should have been a rails topic first,
because there it might make a lot of sense to render partials in parallel,
this could really speed up rails response rendering ;-)

But I believe this is a topic for ruby in general.
Given that we now go to really many cores,
it should be at the heart of ruby to use that.

For a start we could just declare #parallel as -> return self.
and than improve on it.

A naive implementation would just open a new thread for every parallel map block,
but that would not work well -- it actually would be very slow.

So we actually need a thread management,
along the lines of great central.

I believe this should be built into the very core libs of the ruby language.

I know this is hard stuff, but it needs to be done.
Let's go parallel from the heart of the language.

---

Please let us discuss if adding the #parallel method would be a good idea.
Let's come up with some implementations to find the problems.

I'm explicitly referring to map#parallel here.
In rails that might be cache.parallel (if you followed)

for some math tasks it might also be
parallel do {}

[matz (Yukihiro Matsumoto)]

Show me your imaginary code example, so we can understand your concrete intention.
There's lot of issues around concurrency in Ruby. Simply adding map.parallel may not solve the problem.
But we are interested in your idea.

Matz.

[Mordeccai (Martin Wein)]

I’d like to revive this proposal with a concrete use case I ran into in a Rails application.

I frequently need to perform the same I/O-bound operation concurrently over a collection — for example, generating multiple documents through an external API — wait until all operations complete, and collect the results in order.

In JavaScript this is straightforward:

await Promise.all(items.map(process))

In Ruby, this still generally requires a gem and additional machinery. I think Ruby would benefit from a simple standard abstraction such as:

items.concurrently { process(it) }

or:

items.concurrent_map { process(it) }

The implementation could use Ruby’s existing async/non-blocking I/O capabilities rather than requiring one thread per item.

Nearly 11 years after this issue was opened, has Ruby’s concurrency model evolved enough that something like this could now be reconsidered?

[rkh (Konstantin Haase)]

Nearly 11 years after this issue was opened, has Ruby’s concurrency model evolved enough that something like this could now be reconsidered?

Would you expect the blocks to run in Threads, Fibers, or Ractors? A fixed or dynamic pool of Threads or Ractors? How is that managed? Or a true system thread for each value? How are exceptions being dealt with?
For Ractors, should the collection values be copied or moved? For Fibers, how would this be handled if no scheduler is set up (Ruby doesn't ship with a built-in one)?

I do think such functionality is either done explicitly (using Thread.new { ... }.value, Ractor.new(it) { ... }.value, or Fiber.schedule { ... }) or provided by one of the concurrency gems (async or concurrent-ruby), which come with built-in thread pools.

[matz (Yukihiro Matsumoto)]

• Status changed from Open to Closed

Thank you for bringing a concrete use case.

For I/O-bound work like yours, this already works today:

results = items.map { |item| Thread.new { process(item) } }.map(&:value)

Beyond that, a built-in method has to decide the things @rkh (Konstantin Haase) listed: which execution unit to use, how many run at once, and what happens to the others when one fails. I don't see a single answer that fits everyone, so for now I think this belongs to gems such as async, concurrent-ruby or parallel.

Matz.