Feature #17056: Array#index: Allow specifying the position to start search as in String#index

• Status: Open
• Assignee: -
• Target version: -

[TylerRick (Tyler Rick)]

I have a use case of finding the first matching line within a given section in a file. After finding the line number of the start of the section, I want to find the first match after that line.

My workaround for now is to use with_index:

lines = pathname.read.lines
section_start_line = lines.index {|line| line.start_with?(/#* #{section_name}/) }
lines.index.with_index {|line, i| i > section_start_line && line.include?(sought) }

I'd like to do it in a more concise way using a feature of Array#index that I propose here, which is analogous to String#index.

If the second parameter of String#index is present, it specifies the position in the string to begin the search:

'abcabc'.index('a') # => 0
'abcabc'.index('a',2) # => 3

I would expect to also be able to do:

'abcabc'.chars.index('a') # => 0
'abcabc'.chars.index('a', 2)

Using such feature, I would be able to do:

lines.index(sought, section_start_line)

This would give Ruby better parity with other programming languages like Python:

>>> list('abcabc')
['a', 'b', 'c', 'a', 'b', 'c']
>>> list('abcabc').index('a')
0
>>> list('abcabc').index('a', 2)
3

End index

We can further think of an optional parameter to specify the position to end the search. The following languages allow specifying both start and end indexes:

• Python
• C#

Ruby's String#index does not have one, so we could make a separate proposal to add end to both methods at the same time.

[TylerRick (Tyler Rick)]

• Description updated (diff)

[TylerRick (Tyler Rick)]

• Description updated (diff)

[sawa (Tsuyoshi Sawada)]

• Subject changed from Array#index: Allow specifying start index to search like String#index does to Array#index: Allow specifying the position to start search as in String#index
• Description updated (diff)

[marcandre (Marc-Andre Lafortune)]

👍

I'd like to have optional start and stop arguments for find_index, find, bsearch and bsearch_index.

As mentionned, a typical usecase is to repeat a lookup, but another one is to lookup a range of indices (e.g. which elements of a sorted array are between 10 and 20).

I've had to iterate on the indices instead but it is not elegant and is less performant.

[fatkodima (Dima Fatko)]

I have implemented an offset parameter for Array#index - https://github.com/ruby/ruby/pull/3448
Will adjust to more methods if asked.

[matz (Yukihiro Matsumoto)]

Accepted.

How do you think about end index? Do we need it? If so, should we add end index to String#index as well?

Matz.

[Eregon (Benoit Daloze)]

What if a block is given, and one want to use a start index? (for efficiency and not run the block for the first start elements).

ary.index(start) { |i| ... } seems confusing.

Probably keyword arguments are better:
ary.index(from: start) { |i| ... } or ary.index(start: start) { |i| ... }

Although personally I'm not convinced we need these complications.
One can do 'abcabc'.chars[2..].index('a') + 2 instead of 'abcabc'.chars.index('a', 2).
And the [2..] is quite cheap considering that arrays use copy-on-write.

It can also be done with ary = 'abcabc'.chars; (2...ary.size).find { |i| ary[i] == 'a' }.
That's a little bit more complicated, but it's also usable in many more situations than just index.
I would expect it's fairly rare to need a start offset, so I think there is no need for a shortcut.

[fatkodima (Dima Fatko)]

Eregon (Benoit Daloze) wrote in #note-7:

What if a block is given, and one want to use a start index? (for efficiency and not run the block for the first start elements).

ary.index(start) { |i| ... } seems confusing.

Probably keyword arguments are better:
ary.index(from: start) { |i| ... } or ary.index(start: start) { |i| ... }
ary.index(from: start) { |i| ... } or ary.index(start: start) { |i| ... }

Agreed.

Eregon (Benoit Daloze) wrote in #note-7:

It can also be done with ary = 'abcabc'.chars; (2...ary.size).find { |i| ary[i] == 'a' }.
That's a little bit more complicated, but it's also usable in many more situations than just index.
I would expect it's fairly rare to need a start offset, so I think there is no need for a shortcut.

Personally, I had a need for start index a couple of times. What I have seen most of the times, developers just slice an array (allocating a new array; is there a CoW here?) in needed range. And it will be convenient to have a start index argument. And it will be consistent with String#index.

matz (Yukihiro Matsumoto) wrote in #note-6:

Accepted.

How do you think about end index? Do we need it? If so, should we add end index to String#index as well?

Matz.

As for end index, I think this is truly would be rarely needed and can be simulated with something like ...with_index ... { |..., index| ... break if index > end_index ... }

As @marcandre (Marc-Andre Lafortune) pointed out, other methods would probably also benefit from such method arguments, but to avoid updating all of them, I would prefer just add start index argument to Array#index, for consistency with String#index, and it can be used in user code for emulating other methods, like

# find with start index
index = array.index(start: 10) { |e| e % 10 == 0 }
item = array[index] if index

[mame (Yusuke Endoh)]

Hi,

fatkodima (Dima Fatko) wrote in #note-8:

to avoid updating all of them, I would prefer just add start index argument to Array#index, for consistency with String#index,

I agree with your approach. However, your PR changes not only Array#index but also Array#find_index. This brings another inconsistency: Enumerable#find_index does not accept "start", but Array#find_index does.

We discussed this ticket at today's dev-meeting, and @ko1 (Koichi Sasada) proposed removing Array#find_index so that ary.find_index invokes Enumerable#find_index instead of keeping it as an alias to Array#index, and matz agreed with the removal.

[byroot (Jean Boussier)]

• Related to Feature #19177: optional offset for Array#index added

[Earlopain (Earlopain _)]

I often use the offset parameter from String#byteindex and was a bit surprised there is not yet something for arrays. It's a bit unfortunate with the ambiguity between block/noblock usage but a keyword argument offset seems like a perfectly fine solution for that.

[matz (Yukihiro Matsumoto)]

I think it's good to add an offset: keyword argument to Array#index and Array#rindex, for consistency with String#index and String#rindex.

As for Array#find_index, I think we should add offset: there as well, keeping it in sync with Array#index. We will leave Enumerable#find_index untouched, since adding an offset to a lazy enumerable feels unnatural.

We have no plan to add an end index for now.

Matz.

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|33db313e855dfa83d7c66c2de6d63b9b401c32a0.

---

[Feature #17056] Add offset specs for Array#index and rindex

[nobu (Nobuyoshi Nakada)]

• Status changed from Closed to Open

I think we should reopen this issue and clarify the API before fixing the implementation.

There are several API details that I think need to be decided explicitly.

Issues to clarify

1. Object, block, and Enumerator forms

The discussion in #note-7 and #note-8 explicitly considered using an offset with the block form:

array.index(offset: 10) { |element| ... }

However, the reverted implementation only handled offset: when an object was given. The following calls raised ArgumentError, despite the documented Enumerator signatures:

array.index(offset: 2)
array.index(offset: 2) { |element| ... }

array.rindex(offset: -2)
array.rindex(offset: -2) { |element| ... }

Should all three forms accept offset:?

array.index(object, offset: 2)
array.index(offset: 2) { |element| ... }
array.index(offset: 2) # => Enumerator

The same question applies to Array#rindex. Since Array#find_index is an alias of Array#index, it should have exactly the same behavior.

If the block form accepts offset:, the Enumerator form should retain the keyword argument so that calling each performs the same search.

2. Compatibility with searching for a Hash

Before this feature, the following searches for the Hash { offset: 2 }:

[{ offset: 2 }].index(offset: 2) # => 0

If a keyword-only call now means the block/Enumerator form, this behavior changes.

Searching for the Hash would still be possible with explicit braces:

[{ offset: 2 }].index({ offset: 2 }) # => 0

Is this compatibility change acceptable?

The same ambiguity exists when a block is given. Previously the keyword-like Hash was the object and the block was ignored with a warning.

3. Semantics of Array#rindex

The reverted implementation did not behave like String#rindex.

It also documented the signature as rindex(object, offset: nil), but an explicit nil was not accepted:

array.rindex(object, offset: nil)
# => TypeError: no implicit conversion of nil into Integer

In the implementation, nil was only used internally to represent an omitted keyword. Documenting offset: nil makes it look like nil is an accepted default value. For an Array, offset: -1 would express the actual default starting position while keeping an explicit nil invalid.

String#rindex(pattern, offset) starts searching backward at offset. A non-negative offset is an absolute position. A negative offset is relative to the end. An offset greater than the string length is clamped to the end.

For example:

"ababa".rindex("a", 2)   # => 2
"ababa".rindex("a", -2)  # => 2
"ababa".rindex("a", 100) # => 4

I think the corresponding Array behavior should be:

[:a, :b, :a, :b, :a].rindex(:a, offset: 2)   # => 2
[:a, :b, :a, :b, :a].rindex(:a, offset: -2)  # => 2
[:a, :b, :a, :b, :a].rindex(:a, offset: 100) # => 4

The reverted implementation instead treated a non-negative offset as the lower bound of a search starting at the end:

[:a, :b, :a, :b, :a].rindex(:a, offset: 2) # => 4

This makes positive and negative offsets have different meanings and is not consistent with String#rindex.

Should Array#rindex follow String#rindex exactly?

4. Boundary and conversion behavior

Should Array#index and Array#rindex follow the corresponding String methods for all of these cases?

• Negative offsets relative to the end
• An offset smaller than -array.size
• An offset equal to array.size
• An offset greater than array.size
• Objects accepted through numeric conversion
• Explicit nil
• Integers too large for the internal index type

These should be specified and tested for both forward and backward searches.

Proposed signatures

If the above behavior is intended, the signatures would be:

index(object, offset: 0) -> integer or nil
index(offset: 0) { |element| ... } -> integer or nil
index(offset: 0) -> Enumerator

find_index(object, offset: 0) -> integer or nil
find_index(offset: 0) { |element| ... } -> integer or nil
find_index(offset: 0) -> Enumerator

rindex(object, offset: -1) -> integer or nil
rindex(offset: -1) { |element| ... } -> integer or nil
rindex(offset: -1) -> Enumerator

The implementation has been reverted. Once these points are agreed, I think we should add specs for the complete behavior before reimplementing it.

[Dan0042 (Daniel DeLorme)]

1. Object, block, and Enumerator forms

Imho there are two options here

a) If block form is supported, the offset is a keyword argument.
It was not requested by the OP (also not in my duplicate #19177), but it's more versatile and future-proof.
Although it introduces incompatibility due to keyword extension (described below)

index(object, offset: 0)
index(offset: 0){ |element| ... }
index(offset: 0)

b) If block form is not supported, the offset should be a positional argument, for consistency and duck-typing with String#index
It's more limited, and makes it hard to add offset to block form in the future.

index(object, offset = 0)

Overall a) seems better to me, even if not fully consistent with String#index

2. Compatibility with searching for a Hash

Ah yes, this is the standard problem with keyword extension

What would happen with this code?

[{ key: 2 }].index(key: 2)

Will it now raise "unknown keyword: :key" ?

3. Semantics of Array#rindex

I think the corresponding Array behavior should be:

[:a, :b, :a, :b, :a].rindex(:a, offset: 2)   # => 2
[:a, :b, :a, :b, :a].rindex(:a, offset: -2)  # => 2
[:a, :b, :a, :b, :a].rindex(:a, offset: 100) # => 4

100% agree.

4. Boundary and conversion behavior

Should Array#index and Array#rindex follow the corresponding String methods for all of these cases?

In general I would say yes, simply because having consistency for this reduces gotchas and footguns.
But I'm not sure about this behavior of String:

"0aaaaaaaaaa9".rindex("a", 999) #=> 10
"0aaaaaaaaaa9".index("a", -999) #=> nil

To me, these two results are inconsistent and I feel like the last case should return 1.
Should Array#index follow the same pattern or not?

[:a, :b, :a, :b, :a].index(:a, offset: -100)  #=> nil (like String#index) or 0 ?