Feature #3187

Allow dynamic Fiber stack size

Added by Mike Perham almost 4 years ago. Updated 7 months ago.

[ruby-core:29734]
Status:Feedback
Priority:Normal
Assignee:Koichi Sasada
Category:core
Target version:next minor

Description

=begin
I'd like a way to increase the size of the Fiber stack dynamically so when my program starts, I can set it to whatever value I need for the code I'm running. 4KB is too easy to run into problems when running recursive code but settling on any arbitrary static value seems pointless to me.
=end

fiber_stacksize.patch Magnifier (5.89 KB) Tomoyuki Chikanaga, 10/19/2011 11:26 PM

History

#1 Updated by Mike Perham almost 4 years ago

=begin
To be clear, I'm not asking to extend the stack of a currently executing Fiber, just a one-time configuration when my program starts. Something like this:

 require 'fiber'
 Fiber.stack_size = 16 * 1024

 Fiber.new do
   parse_some_big_xml
 end.resume

=end

#2 Updated by Yusuke Endoh almost 4 years ago

  • Target version set to 2.0.0

=begin

=end

#3 Updated by Robert Pankowecki over 2 years ago

I would also welcome such improvement.

#4 Updated by Tomoyuki Chikanaga over 2 years ago

Hi,

I've written a patch. This patch adds some methods Fiber.defaultvmstacksize, Fiber.defaultvmstacksize=, Fiber#vm_stacksize and add an optional Hash argument to Fiber#initialize, and tests for them.
You can specify default VM stack size of Fiber created afterward and/or specify individually when create a Fiber.

ex)
Fiber.defaultvmstacksize = 16 * 1024

Fiber.new do
do_something
end

or

Fiber.new(:vmstacksize => 16 * 1024) do
do
simething
end

Note that the size of VM stack is number of Objects and memsize of stack is vmstacksize * sizeof(VALUE) bytes.
Also note that this patch enable to change only VM stack size, but not machine stack size. I think when Fiber is implemented based on makecontext/swapcontext (FIBER
USENATIVE=1), machine stack size (default: 64KB) can be configurable. I wonder if procedures like `parsesomebigxml' in Mike's example also need larger machine stack size? Does anyone have such a testcase?

#5 Updated by Motohiro KOSAKI over 2 years ago

Note that the size of VM stack is number of Objects and memsize of stack is vmstacksize * sizeof(VALUE) bytes.
Also note that this patch enable to change only VM stack size, but not machine stack size. I think when Fiber is implemented based on makecontext/swapcontext (FIBER
USE_NATIVE

#6 Updated by Tomoyuki Chikanaga over 2 years ago

I don't think we should export vm_stack. It's purely implementation
detail. Just expose "stack size"
knob and it should change both vm-stack and machine-stack size.
You're right. It's better to provide more abstract way to tune memory usage of Fiber.
How about like the following?

Fiber.stacksize = 2.0 # => twice the size of default stack size

or maybe we have to examine how VM/machine stack are consumed.

And even tough we provide such a unified interface, I think more low-level,
implementation specific methods are convenient when users need fine-tuned parameter set for
their applications. Is it a bad idea?
for example, ObjectSpace.count_objects etc.. tightly depend on MRI implementation,
and are useful just because of it.

#7 Updated by B Kelly over 2 years ago

Tomoyuki Chikanaga wrote:

I've written a patch. This patch adds some methods Fiber.defaultvmstacksize, Fiber.defaultvmstacksize=, Fiber#vm_stacksize and add an optional Hash argument to Fiber#initialize, and tests for them.
You can specify default VM stack size of Fiber created afterward and/or specify individually when create a Fiber.

ex)
Fiber.defaultvmstacksize = 16 * 1024

Fiber.new do
do_something
end

or

Fiber.new(:vmstacksize => 16 * 1024) do
do
simething
end

Very nice!

My application uses fibers extensively, and it began to exceed the
default fiber stack during recursive traversal of relatively shallow
trees. (Not surprising, given the 4K default stack size.)

I had patched cont.c locally as follows:

#define FIBERSTACKSIZE_SCALE 8 /* need more fiber stack space */

#define FIBERMACHINESTACKALLOCATIONSIZE (0x10000 *
FIBERSTACKSIZE_SCALE)

#define FIBERVMSTACKSIZE ((4 * 1024) * FIBERSTACKSIZESCALE)

Note that the size of VM stack is number of Objects and memsize of stack is vmstacksize * sizeof(VALUE) bytes.
Also note that this patch enable to change only VM stack size, but not machine stack size. I think when Fiber is implemented based on makecontext/swapcontext (FIBER
USENATIVE=1), machine stack size (default: 64KB) can be configurable. I wonder if procedures like `parsesomebigxml' in Mike's example also need larger machine stack size? Does anyone have such a testcase?

Can anyone comment on the relationship between the VM stack size and the
machine stack size?

I scaled them both equally to be "safe".

But I don't know how they are related, so I'm not sure if the
corresponding increase to the machine stack was necessary?

Thanks,

Bill

#8 Updated by Koichi Sasada over 2 years ago

Hi,

I agree with this proposal. Also add same parameter setting feature for
Thread.

However, I can't make good API to specify stack (VM and machine) size
(and other parameters if there are).

There are several proposals.

Type 1: Thread creating argument (ex: Thread.new(stacksize: 1024))
Type 2: Thread global parameter (ex: Thread.stack
size = 1024)

I think Type 2 is not good (to set default value is okay. However it
conflicts other usage. Typically thread-unsafe).

I also think Type 1 has also problem. If you want to pass keyword
argument "stack_size" to fiber or thread, it should be conflict.

My idea is creating new Thread (Fiber) class with new parameter:

MyThread = Thread.newtemplate(stacksize: 1024)
MyThread.new{
...
}

# Of course "new_template" is bad name.

Any ideas?

(2011/10/12 21:42), Robert Pankowecki wrote:

Issue #3187 has been updated by Robert Pankowecki.

I would also welcome such improvement.

Feature #3187: Allow dynamic Fiber stack size
http://redmine.ruby-lang.org/issues/3187

Author: Mike Perham
Status: Open
Priority: Normal
Assignee:
Category:
Target version: 1.9.x

=begin
I'd like a way to increase the size of the Fiber stack dynamically so when my program starts, I can set it to whatever value I need for the code I'm running. 4KB is too easy to run into problems when running recursive code but settling on any arbitrary static value seems pointless to me.
=end

--
// SASADA Koichi at atdot dot net

#9 Updated by Charles Nutter over 2 years ago

Quick comments.

  • JVM can specify per-thread stack size but always in bytes. Any API that
    exposes stack size should be abstract rather than require a knowledge of
    stack frame sizes from impl to impl. The "stack size factor" version is
    pretty good.

  • I agree it would be nice to specify per-thread stack sizes too. We must
    do this for JRuby on Android, and have to use JRuby- specific mechanisms
    for it.

  • Charlie (mobile)

#10 Updated by Hiroshi Nakamura about 2 years ago

  • Assignee set to Koichi Sasada

How about;

t = Thread.new(stacksize: 1024)
t.stack
size = 1024
t[:initialtlshash] = {}
t.run {
...
}

#11 Updated by Shyouhei Urabe about 2 years ago

  • Status changed from Open to Assigned

#12 Updated by Koichi Sasada almost 2 years ago

Any other idea about it? It's only API design.

I'm not sure nahi-san's idea is good for Ruby or not.
I feel that it is too different from current style.

#13 Updated by Koichi Sasada almost 2 years ago

I make another ticket about it:
https://bugs.ruby-lang.org/issues/6694

Thanks,
Koichi

(2012/06/26 4:20), ko1 (Koichi Sasada) wrote:

Issue #3187 has been updated by ko1 (Koichi Sasada).

Any other idea about it? It's only API design.

I'm not sure nahi-san's idea is good for Ruby or not.
I feel that it is too different from current style.


Feature #3187: Allow dynamic Fiber stack size
https://bugs.ruby-lang.org/issues/3187#change-27423

Author: mperham (Mike Perham)
Status: Assigned
Priority: Normal
Assignee: ko1 (Koichi Sasada)
Category: core
Target version: 2.0.0

=begin
I'd like a way to increase the size of the Fiber stack dynamically so when my program starts, I can set it to whatever value I need for the code I'm running. 4KB is too easy to run into problems when running recursive code but settling on any arbitrary static value seems pointless to me.
=end

--
// SASADA Koichi at atdot dot net

#14 Updated by Koichi Sasada over 1 year ago

I'm considering it because no progress on https://bugs.ruby-lang.org/issues/6694 (sorry, it is naming issue, I think).

Can I add an environment variable (such as RUBYFIBERMACHINESTACKSIZE) to avoid this issue temporarily?

#15 Updated by Yusuke Endoh over 1 year ago

  • Target version changed from 2.0.0 to next minor

#16 Updated by Koichi Sasada 7 months ago

  • Status changed from Assigned to Feedback

Ruby 2.0 already has
RUBYVMFIBERVMSTACKSIZE
RUBY
FIBERMACHINESTACK_SIZE

is it enough?

Also available in: Atom PDF