Project

General

Profile

Actions

Feature #6047

open

read_all: Grow buffer exponentially in generic case

Added by MartinBosslet (Martin Bosslet) over 10 years ago. Updated about 7 years ago.

Status:
Assigned
Priority:
Normal
Assignee:
-
Target version:
-
[ruby-core:42748]

Description

In the general case, read_all grows its buffer linearly by just the amount that is currently read from the underlying source. This results in a linear number of reallocs, It might turn out beneficial if the buffer were grown exponentially by multiplying with a constant factor (e.g. 1.5 or 2), thus resulting in only a logarithmic numver of reallocs.

I will provide a patch and benchmarks, but I'm already opening this issue so I won't forget.

See also https://bugs.ruby-lang.org/issues/5353 for more details.

Updated by ko1 (Koichi Sasada) almost 10 years ago

ping. status?
Do you need helps or comments?

Updated by MartinBosslet (Martin Bosslet) almost 10 years ago

ko1 (Koichi Sasada) wrote:

ping. status?
Do you need helps or comments?

Thanks for your help, to be honest, I haven't tried so far. Can we leave it at 2.0.0 target for now? If I run into problems, I'll ask here!

Updated by normalperson (Eric Wong) almost 10 years ago

Martin Bosslet wrote:

In the general case, read_all grows its buffer linearly by just the
amount that is currently read from the underlying source. This results
in a linear number of reallocs, It might turn out beneficial if the
buffer were grown exponentially by multiplying with a constant factor
(e.g. 1.5 or 2), thus resulting in only a logarithmic numver of
reallocs.

I think growing the buffer exponentially makes sense.

I would enforce a hard limit (probably <= 8 MB) for each growth,
to:

  1. discourage read_all() for large files, it's very wasteful and
    usually hurts performance

  2. prevent memory exhaustion for edge cases (especially on 32-bit)

Updated by mame (Yusuke Endoh) almost 10 years ago

  • Target version changed from 2.0.0 to 2.6

My experience also shows that it is useless to open a ticket for a reminder to myself :-)

I'm setting to next minor tentatively, but if it is really just a performance improvement (i.e., it affects no external modules), you can commit it to 2.0.0 before code freeze.

--
Yusuke Endoh

Actions #5

Updated by zzak (Zachary Scott) about 7 years ago

  • Assignee changed from MartinBosslet (Martin Bosslet) to 7150
Actions

Also available in: Atom PDF