Bug #14046: Ripper loses DATA

• Status: Closed
• Priority: Normal
• Assignee: -
• Target version: -
• ruby -v: ruby 2.5.0dev (2017-10-19 trunk 60213) [x86_64-darwin13]
• Backport: 2.3: UNKNOWN, 2.4: UNKNOWN
• Tags: ext

[mjago (Martyn Jago)]

Since the following is a valid self-contained script:

p DATA.read # comment
__END__
 abc

should the __END__ suffix data be preserved with :on___end__ ?
Alternatively should there be an :on_ignored___end token to preserve DATA?
Notably the comment content is preserved:

[[[1, 0], :on_ident, "p", EXPR_CMDARG],
 [[1, 1], :on_sp, " ", EXPR_CMDARG],
 [[1, 2], :on_const, "DATA", EXPR_ARG],
 [[1, 6], :on_period, ".", EXPR_DOT],
 [[1, 7], :on_ident, "read", EXPR_ARG],
 [[1, 11], :on_sp, " ", EXPR_ARG],
 [[1, 12], :on_comment, "# comment\n", EXPR_ARG],
 [[2, 0], :on___end__, "__END__\n", EXPR_BEG]]

[jeremyevans0 (Jeremy Evans)]

• Status changed from Open to Closed

Ruby stops parsing the file when __END__ is reached. Since the parser doesn't see the rest of the file, it is expected that Ripper doesn't either. If you want to get DATA content, you can handle that using the result of the lex:

require 'ripper'

s = "p DATA.read # comment\n__END__\n abc\n"

lex = Ripper.lex(s)

if lex[-1][1] == :on___end__
  off = -1
  lex[-1][0][0].times{off = s.index("\n", off+1)}
  data = s[(off+1)..-1]
end