Bug #13536: Ripper fails to tokenize nested squiggly heredoc

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 2.5.0dev (2017-05-02 trunk 58544) [x86_64-darwin15]
• Backport: 2.2: REQUIRED, 2.3: DONE, 2.4: DONE

[shyouhei (Shyouhei Urabe)]

require 'ripper'

str = \
<<~'end'
<<~"do"
#{
<<~"begin"
this must be a valid ruby
begin
}
do
end
puts eval(str)
str == Ripper.tokenize(str)

This program fails at the last line.

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset trunk|r58545.

---

ripper/lexer.rb: nested indented heredoc

• ext/ripper/lib/ripper/lexer.rb (on_heredoc_dedent): fix for
  nested indedented here documents, where Elems are nested too.
  [ruby-core:80977] [Bug #13536]

[shyouhei (Shyouhei Urabe)]

• Status changed from Closed to Open

Thank you, the exception disappeared.

But it seems incomplete. Look at this slightly different example:

require 'ripper'

str = \
<<~'end'
<<~"do"
#{
<<~"begin"
 this must be a valid ruby
begin
}
do
end
puts eval(str)
str == Ripper.tokenize(str).join

The last line does not raise exception, but evaluates false.

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset trunk|r58584.

---

ripper/lexer.rb: nested indented heredoc

• ext/ripper/lib/ripper/lexer.rb (on_heredoc_dedent): insert
  stripped leading spaces as on_ignored_sp elements, so that the
  original source can be reconsructed.
  [ruby-core:80977] [Bug #13536]

[usa (Usaku NAKAMURA)]

• Backport changed from 2.2: REQUIRED, 2.3: REQUIRED, 2.4: REQUIRED to 2.2: REQUIRED, 2.3: DONE, 2.4: REQUIRED

ruby_2_3 r59214 merged revision(s) 58545,58584.

[nagachika (Tomoyuki Chikanaga)]

• Backport changed from 2.2: REQUIRED, 2.3: DONE, 2.4: REQUIRED to 2.2: REQUIRED, 2.3: DONE, 2.4: DONE

ruby_2_4 r59247 merged revision(s) 58545,58584.