Feature #14602: Version of dig that raises error if a key is not present

• Status: Open
• Assignee: -
• Target version: -

[amcaplan (Ariel Caplan)]

Currently, if I have a hash like this:

{
    :name => {
        :first => "Ariel",
        :last => "Caplan"
    }
}

and I want to navigate confidently and raise a KeyError if something is missing, I can do:

hash.fetch(:name).fetch(:first)

Unfortunately, the length of the name, combined with the need to repeat the method name every time, means most programmers are more likely to do this:

hash[:name][:first]

which leads to many unexpected errors.

The Hash#dig method made it easy to access methods safely from a nested hash; I'd like to have something similar for access without error protection, and I'd think the most natural name would be Hash#dig!. It would work like this:

hash = {
    :name => {
        :first => "Ariel",
        :last => "Caplan"
    }
}
hash.dig!(:name, :first) # => Ariel
hash.dig!(:name, :middle) # raises KeyError (key not found: :middle)
hash.dig!(:name, :first, :foo) # raises TypeError (String does not have #dig! method)

[shevegen (Robert A. Heiler)]

I think this may be somewhat problematic since it does not appear
to fit to other methods that end with a "!", such as .chop() versus
.chop!() for a String or .map() versus .map!() for an Array.

In the past I thought that "!" would mean mostly "modify in place",
but matz wrote somewhere on the bug tracker that it is more meant
as a "caution" indicator to the ruby user.

Another problem is, I think, that your suggestion of .dig!() does
something that .dig() itself isn't doing (raising multiple
different errors). But that may just be me, perhaps others see
no problem - at the end of the day you'd only have to convince
matz anyway. :)

[duerst (Martin Dürst)]

Would a keyword parameter to dig work for you?

E.g. hash.dig!(:name, :middle, raise_error: true) or something similar.

[amcaplan (Ariel Caplan)]

shevegen (Robert A. Heiler) wrote:

I think this may be somewhat problematic since it does not appear
to fit to other methods that end with a "!", such as .chop() versus
.chop!() for a String or .map() versus .map!() for an Array.

In the past I thought that "!" would mean mostly "modify in place",
but matz wrote somewhere on the bug tracker that it is more meant
as a "caution" indicator to the ruby user.

Another problem is, I think, that your suggestion of .dig!() does
something that .dig() itself isn't doing (raising multiple
different errors). But that may just be me, perhaps others see
no problem - at the end of the day you'd only have to convince
matz anyway. :)

You have a good point about the bang methods often signifying an in-place operation rather than an error-prone one; the latter is more of a Rails convention than a Ruby one, and I mixed things up. Thank you for pointing that out. Mea culpa.

Ideally, we'd just have Hash#fetch take an arbitrary number of arguments, each representing another layer of depth. However, that option is closed off due to backwards compatibility issues, since it already takes an optional second argument representing a default return value in case the item isn't found.

So it sounds like we need a new method name. Perhaps something like #deep_fetch would work, but I'll have to think on it more to see if I can come up with something better. EDIT: Apparently there's already a deep_fetch gem that does exactly this, which might indicate that the name #deep_fetch would be understood intuitively by the community.

In terms of the multiple errors: This isn't actually new, and in fact I copied the current behavior of Hash#dig with the exception of adding the KeyError:

hash = {
    :name => {
        :first => "Ariel",
        :last => "Caplan"
    }
}

hash.dig(:name, :first) # => Ariel
hash.dig(:name, :middle) # => nil
hash.dig(:name, :first, :foo) # raises TypeError (String does not have #dig method)

[amcaplan (Ariel Caplan)]

duerst (Martin Dürst) wrote:

Would a keyword parameter to dig work for you?

E.g. hash.dig!(:name, :middle, raise_error: true) or something similar.

I appreciate the thought. I personally would be more likely to do hash.fetch(:name).fetch(:middle) instead of adding a keyword argument to #dig, unless the list was extremely long (probably at least 4 consecutive keys), which I'd suspect is unusual enough that it's not worth adding to Ruby core for that unusual case.

[k0kubun (Takashi Kokubun)]

• Is duplicate of Feature #12282: Hash#dig! for repeated applications of Hash#fetch added

[k0kubun (Takashi Kokubun)]

• Has duplicate Feature #15563: #dig that throws an exception if a key doesn't exist added

[robb (Robb Shecter)]

amcaplan (Ariel Caplan) wrote:

The Hash#dig method made it easy to access methods safely from a nested hash; I'd like to have something similar for access without error protection, and I'd think the most natural name would be Hash#dig!.

FYI, I've implemented this as a gem: https://github.com/dogweather/digbang

[jaredbeck (Jared Beck)]

I personally don't mind dig!. I interpret the ! as a general sign of caution, rather than some meaning specific to data structures (ie. self-modification). But, if we can't have dig!, how about fetch_dig or dig_fetch?

Whatever we call it, it's a good idea and should be in standard ruby. IMO it's too small to be a gem. We don't want to live in a "left-pad" world. :)

[fpsvogel (Felipe Vogel)]

For me this is a nice shortcut to safely access values in a large config hash. So I would use it if it became part of Ruby core.

I like the name dig! because it's short, but if that has too much of a Rails flavor rather than Ruby, then deep_fetch and dig_fetch seem like fine names too.

I recently wrote about some alternatives for this with benchmarks and other considerations, including the two gems mentioned above as well as custom implementations, at https://fpsvogel.com/posts/2022/ruby-hash-dot-syntax-dig-performance-benchmarks#dig-with-errors

[zverok (Victor Shepelev)]

Just a bit of "design space" analysis:

1. I think dig! is unusual for core Ruby. A lot of Rubyists are used that in Rails pairs like find_by/find_by! are raising/non-raising, but I don't remember any Ruby core API using this convention
2. I don't believe the keyword argument is expressive enough. The "visual structure" of the dig signature includes multiple values of user data (and the list of values might be of arbitrary length), so the option at the end of arguments is a) not visible enough and b) not immediately intuitively obvious if it isn't part of user's data
3. In Hash, we already have at least two examples of using fetch in a sense "get the value or fail": #[] vs #fetch and #values_at vs #fetch_values. It seems like it gives enough precedent to look at the "fetch"-based naming, and it seems like fetch_dig, while grammatically not ideal, would be guessable enough, based on existing experience.

[jeremyevans0 (Jeremy Evans)]

duerst (Martin Dürst) wrote in #note-2:

Would a keyword parameter to dig work for you?

E.g. hash.dig!(:name, :middle, raise_error: true) or something similar.

Keyword approach is not backwards compatible, because keywords are currently treated as positional arguments:

{:name=>{:middle=>{{:raise_error=>true}=>2}}}.dig(:name, :middle, raise_error: true)
# => 2

[nobu (Nobuyoshi Nakada)]

[:name, :middle].inject(hash, :fetch) # raises KeyError (key not found: :middle)

[shyouhei (Shyouhei Urabe)]

nobu (Nobuyoshi Nakada) wrote in #note-12:

[:name, :middle].inject(hash, :fetch) # raises KeyError (key not found: :middle)

Doesn't interface with Arrays.

[mame (Yusuke Endoh)]

shyouhei (Shyouhei Urabe) wrote in #note-13:

Doesn't interface with Arrays.

[0, 0, 0].inject([[[:foo]]], :fetch) #=> :foo ?

I don't think @nobu (Nobuyoshi Nakada) is serious about the idiom, though.

[shyouhei (Shyouhei Urabe)]

mame (Yusuke Endoh) wrote in #note-14:

shyouhei (Shyouhei Urabe) wrote in #note-13:

Doesn't interface with Arrays.

[0, 0, 0].inject([[[:foo]]], :fetch) #=> :foo ?

I don't think @nobu (Nobuyoshi Nakada) is serious about the idiom, though.

Hmm, sorry. I was confusing.

[Eregon (Benoit Daloze)]

There is also this blog post discussing about dig_fetch, which seems very similar: https://shopify.engineering/dig-fetch-truffleruby

message = data.dig_fetch(:response, :message) { IdentityObject.new }
# instead of
message = data.fetch(:response, {}).fetch(:message, IdentityObject.new)

So it still allows a default value if any part is missing when provided a block, and raises an exception otherwise.

[matz (Yukihiro Matsumoto)]

It seems to be a nice idea. But is deep_fetch is the best name for it?

Matz.

[byroot (Jean Boussier)]

A couple other ideas I had:

• lookup
• pick
• traverse

[st0012 (Stan Lo)]

How about dig_for? If we’re digging for something, it kinda makes sense to raise an exception if it’s not there.

[mollemoll (Jonas Molander)]

Some ideas:

reveal
uncover
unfold
dive
enlight
scoop
reel

[p8 (Petrik de Heus)]

Maybe:

fetch_each(:name, :first)
fetch_all(:name, :first)
fetch_tail(:name, :first)
fetch_end(:name, :first)

# or with dig_
dig_each(:name, :first)
dig_tail(:name, :first)
dig_all(:name, :first)
dig_end(:name, :first)
dig_bottom(:name, :first)
dig_deep(:name, :first)
dig_down(:name, :first)

# or maybe
delve(:name, :first)
drill(:name, :first) # as in drill down 

[olivierlacan (Olivier Lacan)]

Considering dig as an operation that can be done manually (with fingers) and address exceptional situations (unexpected objects) more subtly, I would suggest shovel which is a more blunt instrument which tends to raise a notable exception sound (clang!) when it hits an unexpected object.

hash = {
  name: {
    first: "Ariel",
    last: "Caplan"
  }
}

hash.dig(:name, :first) # => Ariel
hash.dig(:name, :middle) # => nil

hash.shovel(:name, : first) # => Ariel
hash.shovel(:name, :middle) # => KeyError (key not found: :middle)

There's a slight semantic overlap with the Hash shovel operator (<<) which could be an issue.

[amcaplan (Ariel Caplan)]

We can think of this as either a variation of fetch or a variation of dig. Ultimately it's both, of course, just depends how you look at it.

If we think of it as fetch-based, deep_fetch would be OK but we also might go with something that really describes quite literally what it does, which is fetch recursively. So, fetch_recursive or rfetch (think of Array#bsearch as prior art - though of course it's not 100% comparable) might be the way to go.

If we take the dig-based perspective, it's dig but non-permissive. So dig_strict might be the most literal way of explaining what it does.

Rather than advocating strongly for 1 specific word, I'd just gently recommend that we avoid introducing more dig-like verbs. While seasoned Rubyists will know the difference, newcomers won't have any obvious reason to assume that dig differs from traverse in strictness. At least there's a convention for fetch vs [] which already exists and we can use it to avoid introducing more new language/concepts.

[duerst (Martin Dürst)]

amcaplan (Ariel Caplan) wrote in #note-23:

We can think of this as either a variation of fetch or a variation of dig. Ultimately it's both, of course, just depends how you look at it.

Or maybe we can think it as a combination of dig with fetch. Then what about dig_fetch or fetch_dig? These names don't look very natural, but it's easy to understand what they are about.

[ufuk (Ufuk Kayserilioglu)]

My humble suggestion would be Hash#retrieve. The operation tries to retrieve a value in the depths of a hash and if it comes back empty handed, that is an error.

hash = {
  name: {
    first: "Ariel",
    last: "Caplan"
  }
}

hash.dig(:name, :first) # => Ariel
hash.dig(:name, :middle) # => nil

hash.retrieve(:name, :first) # => Ariel
hash.retrieve(:name, :middle) # => KeyError (key not found: :middle)

The word "retrieve" is a synonym for "fetch" in English, and has deep roots in CS terminology related to "data retrieval" and similar.

[zverok (Victor Shepelev)]

I fully agree with @duerst (Martin Dürst) in #14602#note-24:

maybe we can think it as a combination of dig with fetch. Then what about dig_fetch or fetch_dig? These names don't look very natural, but it's easy to understand what they are about.

First, we already have examples of fetch-based naming: not only #fetch itself as a variation of #[], but also #fetch_values as a variation of #values_at, so there is a precedent for recognizability

Second, I value short one-word names, so all the witty options like #shovel and #retrieve are nice, but I am afraid that when we have a variation of a known method in an API established long ago, introducing completely new word into Ruby would be a false move. Imagine you started to read code and met with #retrieve (or #shovel) for the first time. There is nothing that might help you to understand what it does; one verb that "a bit resembles dig" is not suggestive enough.

Third, deep_fetch is somewhat suggestive, but the problem "it behaves like dig, but the name logic is nothing like dig" stands. Maybe if it would a pair of, IDK, #deep_fetch and #deep_get it might've been tolerable, but now is too late for that, everybody has used to #dig.

fetch_dig, OTOH, is reasonably short, clearly suggests the meaning, and follows the logic of other methods existing.

[sawa (Tsuyoshi Sawada)]

What about simply allowing fetch to take multiple arguments?

If there are more than one argument, always interpret the last one as the default value unless when there is a block, in which case, the block is evaluated if a key is missing somewhere in the path.

You would need to explicitly write a block if you want to raise an error, but I believe your intention is not to let the raised error go through all the way to the top level to end the program; you intend to catch that error somewhere, and do something with it, right? Then, you can instead write that routine in the block from the beginning.

(1) No change to present behavior:

hash.fetch(:name, :first) # => {:first => "Ariel", :last => "Caplan"}

(2) Perhaps error prone use cases, but these are suited for dig, so do not use fetch in practice in such cases, and it would not be a problem:

hash.fetch(:name, :first, nil) # => "Ariel"
hash.fetch(:name, :middle, nil) # => `nil`

(3) Explicitly raise an error in a block:

hash.fetch(:name, :first){|key| raise KeyError} # => "Ariel"
hash.fetch(:name, :middle){|key| raise KeyError} # !> KeyError

(4) Or, write a routine:

hash.fetch(:name, :middle){|key| process_missing_key(key)}

[sinsoku (Takumi Shotoku)]

I agree with #14602#note-23.
I think a fetch-based or dig-based name would be better, since the function can be inferred from the method name.

Personally, I feel that deep_fetch is a simple and nice name.
I often use deep_dup and deep_merge in Rails apps and are familiar with the method name deep_*.

I found fetch_dig and dig_fetch to be a little strange in that they have verbs lined up.
I think dig_strict is a good name, but I prefer deep_fetch because deep has fewer characters and is simpler than strict.

[Eregon (Benoit Daloze)]

I like deep_fetch too.

[mame (Yusuke Endoh)]

We discussed at the dev meeting but did not reach a conclusion. But there has been some progress.

• @akr (Akira Tanaka) said we shouldn't add new vocabulary, and Matz agreed.
• There is Array#fetch which throws an exception when it is not found, so we want to use this vocabulary: fetch_* or *_fetch
  • Matz didn't like any of the proposed fetch_each fetch_all fetch_tail fetch_end
• Matz prefers dig or recursive to deep to represent this behavior.
  • deep sounds to be an operation on the entire data structure, due to the influence of deep_copy.
• Matz was interested in how dig_fetch or fetch_dig would sound to English native speakers.
• Matz also mentioned recursive_fetch, but said it seemed a bit long.

[Eregon (Benoit Daloze)]

mame (Yusuke Endoh) wrote in #note-30:

• deep sounds to be an operation on the entire data structure, due to the influence of deep_copy.

It actually is a deep operation, on the entire data structure/multiple objects, exactly like deep_copy would.
So it seems a perfect match, no?

[mame (Yusuke Endoh)]

As far as I understand, Matz expects that, for a tree structure:

• dig_* touches only nodes on a specific path,
• deep_* touches all nodes.

[edmz (Ed Mz)]

Following what @mame (Yusuke Endoh) explained about Matz expectations, I would like to (humbly) suggest another option:

• dig_expected

(and a my +1 to dig_strict)

[jordan-brough (Jordan Brough)]

Matz was interested in how dig_fetch or fetch_dig would sound to English native speakers.

As a native english speaker, "dig_fetch" sounds natural and makes sense to me.