Project

General

Profile

Feature #10267

Number of processors

Added by akr (Akira Tanaka) about 5 years ago. Updated about 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
[ruby-core:65142]

Description

How about a method to obtain number of processors?

Several committers discussed this feature at RubyKaigi 2014.

The number is important to use all available processing power.
Too few processes doesn't use all processors.
Too much processes wastes processing power.

For example, test/lib/test/unit has -j N option for
parallel tests and N is the number of processes to run simultaneously.
But we must specify N explicitly.
I think it should have reasonable default.

I propose a primitive method to obtain number of processors for such purpose:
Etc.nprocessors_online.

It is implemented using sysconf(_SC_NPROCESSORS_ONLN).
It is not standardized but available on many platforms:
GNU/Linux, NetBSD, FreeBSD, OpenBSD, DragonFly BSD,
OpenIndiana, Mac OS X and AIX.

usa-san said he will implement on Windows (after this feature is implemented).

kosaki-san said he will improve the method for container on GNU/Linux.
(If a process is run in a container, usable number of processors may be restricted.
So sysconf(_SC_NPROCESSORS_ONLN) may be not approprate.)

matz accepted this feature including the method name,
Etc.nprocessors_online.

However kosaki-san said he will try to find better name
(by survey container technology).
So I'm waiting.


Files

etc-nprocessors.patch (1.91 KB) etc-nprocessors.patch akr (Akira Tanaka), 09/20/2014 02:45 AM
etc-nprocessors2.patch (2.31 KB) etc-nprocessors2.patch akr (Akira Tanaka), 09/22/2014 03:07 AM
etc-nprocessors3.patch (2.29 KB) etc-nprocessors3.patch akr (Akira Tanaka), 09/22/2014 03:22 AM
etc-nprocessors-kosaki.patch (3.06 KB) etc-nprocessors-kosaki.patch kosaki (Motohiro KOSAKI), 10/01/2014 08:56 PM
etc-nprocessors-kosaki2.patch (3.68 KB) etc-nprocessors-kosaki2.patch kosaki (Motohiro KOSAKI), 10/01/2014 09:48 PM

Associated revisions

Revision 7b25bd8b
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): New method. Accepted by matz at RubyKaigi 2014. [ruby-core:65142] [Feature #10267]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47761 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 47761
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): New method. Accepted by matz at RubyKaigi 2014. [ruby-core:65142] [Feature #10267]

Revision 47761
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): New method. Accepted by matz at RubyKaigi 2014. [ruby-core:65142] [Feature #10267]

Revision 47761
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): New method. Accepted by matz at RubyKaigi 2014. [ruby-core:65142] [Feature #10267]

Revision 47761
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): New method. Accepted by matz at RubyKaigi 2014. [ruby-core:65142] [Feature #10267]

Revision 47761
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): New method. Accepted by matz at RubyKaigi 2014. [ruby-core:65142] [Feature #10267]

Revision 47761
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): New method. Accepted by matz at RubyKaigi 2014. [ruby-core:65142] [Feature #10267]

Revision 56fa18c9
Added by usa (Usaku NAKAMURA) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): Windows support. see [Feature #10267]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47764 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 47764
Added by usa (Usaku NAKAMURA) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): Windows support. see [Feature #10267]

Revision 47764
Added by usa (Usaku NAKAMURA) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): Windows support. see [Feature #10267]

Revision 47764
Added by usa (Usaku NAKAMURA) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): Windows support. see [Feature #10267]

Revision 47764
Added by usa (Usaku NAKAMURA) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): Windows support. see [Feature #10267]

Revision 47764
Added by usa (Usaku NAKAMURA) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): Windows support. see [Feature #10267]

Revision 47764
Added by usa (Usaku NAKAMURA) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors): Windows support. see [Feature #10267]

Revision 7d9e04de
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors_affinity): use sched_getaffinity
    for getting precious number of available cpus.

  • ext/etc/etc.c (etc_nprocessors): use etc_nprocessors_affinity if
    possible.

[Feature #10267] etc-nprocessors-kosaki2.patch

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@47939 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 47939
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors_affinity): use sched_getaffinity
    for getting precious number of available cpus.

  • ext/etc/etc.c (etc_nprocessors): use etc_nprocessors_affinity if
    possible.

[Feature #10267] etc-nprocessors-kosaki2.patch

Revision 47939
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors_affinity): use sched_getaffinity
    for getting precious number of available cpus.

  • ext/etc/etc.c (etc_nprocessors): use etc_nprocessors_affinity if
    possible.

[Feature #10267] etc-nprocessors-kosaki2.patch

Revision 47939
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors_affinity): use sched_getaffinity
    for getting precious number of available cpus.

  • ext/etc/etc.c (etc_nprocessors): use etc_nprocessors_affinity if
    possible.

[Feature #10267] etc-nprocessors-kosaki2.patch

Revision 47939
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors_affinity): use sched_getaffinity
    for getting precious number of available cpus.

  • ext/etc/etc.c (etc_nprocessors): use etc_nprocessors_affinity if
    possible.

[Feature #10267] etc-nprocessors-kosaki2.patch

Revision 47939
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors_affinity): use sched_getaffinity
    for getting precious number of available cpus.

  • ext/etc/etc.c (etc_nprocessors): use etc_nprocessors_affinity if
    possible.

[Feature #10267] etc-nprocessors-kosaki2.patch

Revision 47939
Added by akr (Akira Tanaka) about 5 years ago

  • ext/etc/etc.c (etc_nprocessors_affinity): use sched_getaffinity
    for getting precious number of available cpus.

  • ext/etc/etc.c (etc_nprocessors): use etc_nprocessors_affinity if
    possible.

[Feature #10267] etc-nprocessors-kosaki2.patch

History

Updated by normalperson (Eric Wong) about 5 years ago

akr@fsij.org wrote:

matz accepted this feature including the method name,
Etc.nprocessors_online.

I think the name is too long. How about naming it after the GNU
coreutils nproc(1) command?

Etc.nproc(:online)

I think it is important to be able to get all CPUs, not just
online CPUs, too (to workaround old SMP bugs in the kernel):

Etc.nproc(:all)

Updated by normalperson (Eric Wong) about 5 years ago

KOSAKI Motohiro kosaki.motohiro@gmail.com wrote:

On Sat, Sep 20, 2014 at 1:03 PM, Eric Wong normalperson@yhbt.net wrote:

akr@fsij.org wrote:

matz accepted this feature including the method name,
Etc.nprocessors_online.

I think the name is too long. How about naming it after the GNU
coreutils nproc(1) command?

    Etc.nproc(:online)

This is an option. But "proc" is not unclear on ruby context. It is sometimes
procedure, and it is sometimes process. I like processor or cpu.

OK. I mainly want the ability to specify type as arg.

I think it is important to be able to get all CPUs, not just
online CPUs, too

(to workaround old SMP bugs in the kernel):

What bug?

Anything which may involve synchronizing with softirq or workqueues, I
think... I suppose my fix in 128dd1759d96ad36c379240f8b9463e8acfd37a1
(missing barrier in EPOLL_CTL_MOD) is one example. I was never able to
reproduce that bug on my HW, but it worries me because some users are
still on old kernels and potentially affected.

    Etc.nproc(:all)

I don't think this is useful. I'd like to explain some kernel internal.
Now, Linux has three type of number of cpus.

1) online
2) present
3) possible

I think these names are fine for Ruby.

"all" is not clear to me.

If you elaborate why you need nproc(:all), I may be able to suggest better
name.

I just chose "all" because it matched the nproc(1) --all option
and didn't want to use Linux-specific naming.
But online/present/possible is fine, for now, too.

We could also have extra options like:

:cores - exclude HyperThread (probably useful for Intel users)
:physical - number of sockets (maybe for affinity)

Updated by akr (Akira Tanaka) about 5 years ago

Eric Wong wrote:

akr@fsij.org wrote:

matz accepted this feature including the method name,
Etc.nprocessors_online.

I think the name is too long. How about naming it after the GNU
coreutils nproc(1) command?

Etc.nproc(:online)

I feel "proc" as process.

I think it is important to be able to get all CPUs, not just
online CPUs, too (to workaround old SMP bugs in the kernel):

Etc.nproc(:all)

I (and kosaki-san) agreed that the most important number of processors is
the number of online processors.

I'm not sure that other numbers (all, cores, etc.) is actually useful for
Ruby programming.

However I think the method name can be changed to
Etc.nprocessors.

If someone succeeds to pursuade matz that other numbers are also important,
we can add an optional argument as Etc.nprocessors(type).

Updated by normalperson (Eric Wong) about 5 years ago

akr@fsij.org wrote:

However I think the method name can be changed to
Etc.nprocessors.

If someone succeeds to pursuade matz that other numbers are also important,
we can add an optional argument as Etc.nprocessors(type).

OK, I think Etc.nprocessors gives room for future changes in case
other types are accepted.

In your patch2, the test is still using the old name.

Updated by akr (Akira Tanaka) about 5 years ago

Eric Wong wrote:

In your patch2, the test is still using the old name.

Oops. Fixed.

Updated by kosaki (Motohiro KOSAKI) about 5 years ago

kosaki-san said he will improve the method for container on GNU/Linux.
(If a process is run in a container, usable number of processors may be restricted.
So sysconf(_SC_NPROCESSORS_ONLN) may be not approprate.)

I wrote an incrementa patch on top of akr's etc-nprocessors3.patch for this purpose.
(see attached etc-nprocessors-kosaki.patch file.)

Comments are welcome!

btw, in akr-san's etc-nprocessor3.patch, Changelog still reference an old name.

Updated by normalperson (Eric Wong) about 5 years ago

kosaki.motohiro@gmail.com wrote:

I wrote an incrementa patch on top of akr's etc-nprocessors3.patch for this purpose.
(see attached etc-nprocessors-kosaki.patch file.)

Comments are welcome!

Minor: we need to call CPU_FREE in the unlikely case sched_getaffinity
fails.

Updated by kosaki (Motohiro KOSAKI) about 5 years ago

Thanks Eric.
I also found a design issue in my patch. So, I wrote v2 patch from scratch.

Updated by akr (Akira Tanaka) about 5 years ago

  • Status changed from Open to Closed
  • % Done changed from 0 to 100

Applied in changeset r47761.


  • ext/etc/etc.c (etc_nprocessors): New method. Accepted by matz at RubyKaigi 2014. [ruby-core:65142] [Feature #10267]

Updated by akr (Akira Tanaka) about 5 years ago

Ok.

It seems that there is no problem with the method name, Etc.nprocessors.

So I committed my patch (etc-nprocessors3.patch).

Please commit your patch, kosaki-san.
(I didn't commit the patch just because separate commits makes "svn ann" more clearer.)

Updated by akr (Akira Tanaka) about 5 years ago

It seems kosaki-san is busy.

I committed etc-nprocessors-kosaki2.patch at r47939.

Also available in: Atom PDF