Bug #19838: Ripper nested heredocs

• Status: Open
• Priority: Normal
• Assignee: -
• Target version: -
• Backport: 3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN

[kddnewton (Kevin Newton)]

I'm seeing some odd behavior on nested heredocs in the ripper lexer.

<<~H1
  1
  #{<<~H2}
    2
  H2
  3
H1

$ cat test.rb | ruby -rripper -e 'pp Ripper.lex(ARGF.read)'

[[[1, 0], :on_heredoc_beg, "<<~H1", BEG],
 [[1, 5], :on_nl, "\n", BEG],
 [[2, 0], :on_ignored_sp, "  ", BEG],
 [[2, 2], :on_tstring_content, "1\n", BEG],
 [[3, 0], :on_ignored_sp, "  ", BEG],
 [[3, 2], :on_embexpr_beg, "\#{", BEG],
 [[3, 4], :on_heredoc_beg, "<<~H2", BEG],
 [[3, 9], :on_embexpr_end, "}", END],
 [[3, 10], :on_tstring_content, "\n", BEG],
 [[4, 0], :on_ignored_sp, "    ", BEG],
 [[4, 4], :on_tstring_content, "2\n", BEG],
 [[5, 0], :on_heredoc_end, "  H2\n", BEG],
 [[6, 0], :on_ignored_sp, "  ", BEG],
 [[6, 2], :on_tstring_content, "3\n", BEG],
 [[7, 0], :on_heredoc_end, "H1\n", BEG]]

This seems totally correct. But if you change it to a non-indenting heredoc:

<<-H1
  1
  #{<<~H2}
    2
  H2
  3
H1

$ cat test.rb | ruby -rripper -e 'pp Ripper.lex(ARGF.read)'

[[[1, 0], :on_heredoc_beg, "<<-H1", BEG],
 [[1, 5], :on_nl, "\n", BEG],
 [[2, 0], :on_tstring_content, "  1\n" + "  ", BEG],
 [[3, 2], :on_embexpr_beg, "\#{", BEG],
 [[3, 4], :on_heredoc_beg, "<<~H2", BEG],
 [[3, 9], :on_embexpr_end, "}", END],
 [[3, 10], :on_tstring_content, "\n" + "  3\n", BEG],
 [[4, 0], :on_ignored_sp, "    ", BEG],
 [[4, 4], :on_tstring_content, "2\n", BEG],
 [[5, 0], :on_heredoc_end, "  H2\n", BEG],
 [[7, 0], :on_heredoc_end, "H1\n", BEG]]

Now the newline on line 3 gets concatenated with the content from line 6, and that token is dropped from the list entirely. This seems unintentional, but I'm not sure.