Feature #10267: Number of processors

• Status: Closed
• Assignee: -
• Target version: -

[akr (Akira Tanaka)]

How about a method to obtain number of processors?

Several committers discussed this feature at RubyKaigi 2014.

The number is important to use all available processing power.
Too few processes doesn't use all processors.
Too much processes wastes processing power.

For example, test/lib/test/unit has -j N option for
parallel tests and N is the number of processes to run simultaneously.
But we must specify N explicitly.
I think it should have reasonable default.

I propose a primitive method to obtain number of processors for such purpose:
Etc.nprocessors_online.

It is implemented using sysconf(_SC_NPROCESSORS_ONLN).
It is not standardized but available on many platforms:
GNU/Linux, NetBSD, FreeBSD, OpenBSD, DragonFly BSD,
OpenIndiana, Mac OS X and AIX.

usa-san said he will implement on Windows (after this feature is implemented).

kosaki-san said he will improve the method for container on GNU/Linux.
(If a process is run in a container, usable number of processors may be restricted.
So sysconf(_SC_NPROCESSORS_ONLN) may be not approprate.)

matz accepted this feature including the method name,
Etc.nprocessors_online.

However kosaki-san said he will try to find better name
(by survey container technology).
So I'm waiting.

[normalperson (Eric Wong)]

[e-mail] wrote:

matz accepted this feature including the method name,
Etc.nprocessors_online.

I think the name is too long. How about naming it after the GNU
coreutils nproc(1) command?

Etc.nproc(:online)

I think it is important to be able to get all CPUs, not just
online CPUs, too (to workaround old SMP bugs in the kernel):

Etc.nproc(:all)

[normalperson (Eric Wong)]

KOSAKI Motohiro [e-mail] wrote:

On Sat, Sep 20, 2014 at 1:03 PM, Eric Wong [e-mail] wrote:

[e-mail] wrote:

matz accepted this feature including the method name,
Etc.nprocessors_online.

I think the name is too long. How about naming it after the GNU
coreutils nproc(1) command?

    Etc.nproc(:online)

This is an option. But "proc" is not unclear on ruby context. It is sometimes
procedure, and it is sometimes process. I like processor or cpu.

OK. I mainly want the ability to specify type as arg.

I think it is important to be able to get all CPUs, not just
online CPUs, too

(to workaround old SMP bugs in the kernel):

What bug?

Anything which may involve synchronizing with softirq or workqueues, I
think... I suppose my fix in 128dd1759d96ad36c379240f8b9463e8acfd37a1
(missing barrier in EPOLL_CTL_MOD) is one example. I was never able to
reproduce that bug on my HW, but it worries me because some users are
still on old kernels and potentially affected.

    Etc.nproc(:all)

I don't think this is useful. I'd like to explain some kernel internal.
Now, Linux has three type of number of cpus.

1. online
2. present
3. possible

I think these names are fine for Ruby.

"all" is not clear to me.

If you elaborate why you need nproc(:all), I may be able to suggest better
name.

I just chose "all" because it matched the nproc(1) --all option
and didn't want to use Linux-specific naming.
But online/present/possible is fine, for now, too.

We could also have extra options like:

:cores - exclude HyperThread (probably useful for Intel users)
:physical - number of sockets (maybe for affinity)

[akr (Akira Tanaka)]

• File etc-nprocessors2.patch etc-nprocessors2.patch added

Eric Wong wrote:

[e-mail] wrote:

matz accepted this feature including the method name,
Etc.nprocessors_online.

I think the name is too long. How about naming it after the GNU
coreutils nproc(1) command?

Etc.nproc(:online)

I feel "proc" as process.

I think it is important to be able to get all CPUs, not just
online CPUs, too (to workaround old SMP bugs in the kernel):

Etc.nproc(:all)

I (and kosaki-san) agreed that the most important number of processors is
the number of online processors.

I'm not sure that other numbers (all, cores, etc.) is actually useful for
Ruby programming.

However I think the method name can be changed to
Etc.nprocessors.

If someone succeeds to pursuade matz that other numbers are also important,
we can add an optional argument as Etc.nprocessors(type).

[normalperson (Eric Wong)]

[e-mail] wrote:

However I think the method name can be changed to
Etc.nprocessors.

If someone succeeds to pursuade matz that other numbers are also important,
we can add an optional argument as Etc.nprocessors(type).

OK, I think Etc.nprocessors gives room for future changes in case
other types are accepted.

In your patch2, the test is still using the old name.

[akr (Akira Tanaka)]

• File etc-nprocessors3.patch etc-nprocessors3.patch added

Eric Wong wrote:

In your patch2, the test is still using the old name.

Oops. Fixed.

[kosaki (Motohiro KOSAKI)]

• File etc-nprocessors-kosaki.patch etc-nprocessors-kosaki.patch added

kosaki-san said he will improve the method for container on GNU/Linux.
(If a process is run in a container, usable number of processors may be restricted.
So sysconf(_SC_NPROCESSORS_ONLN) may be not approprate.)

I wrote an incrementa patch on top of akr's etc-nprocessors3.patch for this purpose.
(see attached etc-nprocessors-kosaki.patch file.)

Comments are welcome!

btw, in akr-san's etc-nprocessor3.patch, Changelog still reference an old name.

[normalperson (Eric Wong)]

[e-mail] wrote:

I wrote an incrementa patch on top of akr's etc-nprocessors3.patch for this purpose.
(see attached etc-nprocessors-kosaki.patch file.)

Comments are welcome!

Minor: we need to call CPU_FREE in the unlikely case sched_getaffinity
fails.

[kosaki (Motohiro KOSAKI)]

• File etc-nprocessors-kosaki2.patch etc-nprocessors-kosaki2.patch added

Thanks Eric.
I also found a design issue in my patch. So, I wrote v2 patch from scratch.

[akr (Akira Tanaka)]

• Status changed from Open to Closed
• % Done changed from 0 to 100

Applied in changeset r47761.

---

• ext/etc/etc.c (etc_nprocessors): New method.
  Accepted by matz at RubyKaigi 2014.
  [ruby-core:65142] [Feature #10267]

[akr (Akira Tanaka)]

Ok.

It seems that there is no problem with the method name, Etc.nprocessors.

So I committed my patch (etc-nprocessors3.patch).

Please commit your patch, kosaki-san.
(I didn't commit the patch just because separate commits makes "svn ann" more clearer.)

[akr (Akira Tanaka)]

It seems kosaki-san is busy.

I committed etc-nprocessors-kosaki2.patch at r47939.