Actions
Bug #19838
closedRipper nested heredocs
Description
I'm seeing some odd behavior on nested heredocs in the ripper lexer.
<<~H1
1
#{<<~H2}
2
H2
3
H1
$ cat test.rb | ruby -rripper -e 'pp Ripper.lex(ARGF.read)'
[[[1, 0], :on_heredoc_beg, "<<~H1", BEG],
[[1, 5], :on_nl, "\n", BEG],
[[2, 0], :on_ignored_sp, " ", BEG],
[[2, 2], :on_tstring_content, "1\n", BEG],
[[3, 0], :on_ignored_sp, " ", BEG],
[[3, 2], :on_embexpr_beg, "\#{", BEG],
[[3, 4], :on_heredoc_beg, "<<~H2", BEG],
[[3, 9], :on_embexpr_end, "}", END],
[[3, 10], :on_tstring_content, "\n", BEG],
[[4, 0], :on_ignored_sp, " ", BEG],
[[4, 4], :on_tstring_content, "2\n", BEG],
[[5, 0], :on_heredoc_end, " H2\n", BEG],
[[6, 0], :on_ignored_sp, " ", BEG],
[[6, 2], :on_tstring_content, "3\n", BEG],
[[7, 0], :on_heredoc_end, "H1\n", BEG]]
This seems totally correct. But if you change it to a non-indenting heredoc:
<<-H1
1
#{<<~H2}
2
H2
3
H1
$ cat test.rb | ruby -rripper -e 'pp Ripper.lex(ARGF.read)'
[[[1, 0], :on_heredoc_beg, "<<-H1", BEG],
[[1, 5], :on_nl, "\n", BEG],
[[2, 0], :on_tstring_content, " 1\n" + " ", BEG],
[[3, 2], :on_embexpr_beg, "\#{", BEG],
[[3, 4], :on_heredoc_beg, "<<~H2", BEG],
[[3, 9], :on_embexpr_end, "}", END],
[[3, 10], :on_tstring_content, "\n" + " 3\n", BEG],
[[4, 0], :on_ignored_sp, " ", BEG],
[[4, 4], :on_tstring_content, "2\n", BEG],
[[5, 0], :on_heredoc_end, " H2\n", BEG],
[[7, 0], :on_heredoc_end, "H1\n", BEG]]
Now the newline on line 3 gets concatenated with the content from line 6, and that token is dropped from the list entirely. This seems unintentional, but I'm not sure.
Updated by nobu (Nobuyoshi Nakada) 9 months ago
- Status changed from Open to Closed
Applied in changeset git|e5e1f9813e23c427cf45a25181b2e6d3cf97411a.
[Bug #19838] Flush delayed token nonconsecutive with the next token
Actions
Like0
Like0