Project

General

Profile

Actions

Bug #17779

closed

特定の順序でHashのkeyを削除した場合に Hash#first が遅くなる

Added by tompng (tomoya ishida) about 1 year ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
ruby -v:
ruby 3.1.0dev (2021-04-06T03:02:46Z :detached: ff91b97c83) [x86_64-linux]
[ruby-dev:51046]

Description

再現コード

require'benchmark'
hash = 1000000.times.to_h { [rand, true]}
p Benchmark.realtime { 100000.times { hash.first } }
#=> 0.03949428700025237

hash.keys[1..100000].each { hash.delete _1 }
hash.delete hash.keys.first
p Benchmark.realtime { 100000.times { hash.first } }
#=> 12.849613588000011

原因と思われる部分
entriesのentries_start+1番目がdeletedな状態でentries_start番目を削除すると、それ以降entries_startが更新されなくなる
その結果、entriesの先頭からdeletedな要素が連続して続く状態が作られるようになり、その状態ではdeletedではない最初の要素を探すのにコストがかかるようになる

patchを添付しました

patch当てる前と後のベンチマーク

hash = 1000000.times.to_h { [rand, true]}
t=Time.now
100000.times { hash.first }
p Time.now-t
# => before: 0.047851
# => after: 0.047678

hash.keys[1..100000].each { hash.delete _1 }
hash.delete hash.keys.first
t=Time.now
100000.times { hash.first }
p Time.now-t
# => before: 24.724466
# => after: 0.051886

patch内容について気になる点

  • st_tableのdeleteが少し遅くなるとruby全体の速度に影響するのかどうか
  • とはいえ、元の tab->entries_start = n + 1; はfirstが高速に動作することを意図したものだと思うなので、このpatchにより意図通りの挙動をするようになるはず

このpatchが不利になりそうなケースのベンチマーク

original_hash = 100000.times.to_h { [rand, true]}
keys = original_hash.keys.take(10000)
times = 4001.times.map {
  hash = original_hash.dup
  t = Time.now
  # case1
  keys.each { hash.delete _1 }
  # case2
  # keys[1..].each { hash.delete _1 }; hash.delete(keys.first)
  Time.now - t
}
p min: times.min, avg: times.sum / times.size, mid: times.sort[times.size/2], max: times.max

# case1 先頭を削除するときのオーバーヘッドの測定
# master:  {:min=>0.001074, :avg=>0.0016952556860784804, :mid=>0.0016, :max=>0.01718}
# patched: {:min=>0.001088, :avg=>0.001468765058735316, :mid=>0.001236, :max=>0.02119}

# case2 先頭から2番目以降を削除(このpatchで変わらないはず)した後、最後に先頭を削除しentries_startが一気に更新されるパターン
# master: {:min=>0.00108, :avg=>0.0016331899525118718, :mid=>0.001615, :max=>0.007555}
# patched: {:min=>0.001087, :avg=>0.0016653474131467132, :mid=>0.001376, :max=>0.041265}

# minはこのpatchで1~3%くらい悪化していそう、avg mid maxは結果が安定しない(逆転することが多かったり)

Files

st.patch (730 Bytes) st.patch patch for st.c tompng (tomoya ishida), 04/06/2021 05:40 PM
Actions #1

Updated by tompng (tomoya ishida) about 1 year ago

  • Description updated (diff)
Actions #2

Updated by tompng (tomoya ishida) about 1 year ago

  • Description updated (diff)
Actions #3

Updated by tompng (tomoya ishida) about 1 year ago

  • Status changed from Open to Closed

Applied in changeset git|9f9045123efefbd11dd397b4d59596290765feec.


st.c: skip all deleted entries [Bug #17779]

Update the start entry skipping all already deleted entries.
Fixes performance issue of Hash#first in a certain case.

Actions #4

Updated by nobu (Nobuyoshi Nakada) about 1 year ago

  • Backport changed from 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN to 2.6: REQUIRED, 2.7: REQUIRED, 3.0: REQUIRED

Updated by nagachika (Tomoyuki Chikanaga) about 1 year ago

  • Backport changed from 2.6: REQUIRED, 2.7: REQUIRED, 3.0: REQUIRED to 2.6: REQUIRED, 2.7: REQUIRED, 3.0: DONTNEED

I think this is a performance thing.
If there are any real world application affected by this issue, please let me know.

Actions #6

Updated by nagachika (Tomoyuki Chikanaga) about 1 year ago

  • Backport changed from 2.6: REQUIRED, 2.7: REQUIRED, 3.0: DONTNEED to 2.6: REQUIRED, 2.7: REQUIRED, 3.0: WONTFIX
Actions

Also available in: Atom PDF