Project

General

Profile

Bug #11485

Ripper parser generates nil tokens for ignored_nl event during parse errors

Added by lsegal (Loren Segal) over 2 years ago.

Status:
Open
Priority:
Normal
Assignee:
-
Target version:
-
[ruby-core:<unknown>]

Description

This seems to be a regression in the Ripper parser where the on_ignored_nl event is passed a nil value for the token parameter in the event when a syntax error occurs. I'm not entirely sure of the details, but this is the most minimal reproduction example I could produce:

require 'ripper'

class Parser < Ripper
  def on_ignored_nl(*args) p args end
  def on_parse_error(msg) $stderr.puts "ERROR! #{msg}" end
end

Parser.new(DATA, 'stdin').parse

__END__
something # comment
...

Note the "..." which causes the parse error, but the comment on the line before causes the ignored_nl event to get called with nil as the argument. I'm not sure if it's specific to the ... token or if there are other ways to trigger this parse error. I tried with other invalid syntaxes and could not reproduce with those.

Running the above produces the following output (in 2.0.0p481 and 2.3.0-dev respective):

~$ ruby -v
ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb 
ERROR! syntax error, unexpected ..., expecting end-of-input

~$ ruby -v
ruby 2.3.0dev (2015-08-24 trunk 51672) [x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb 
[nil]
ERROR! syntax error, unexpected ..., expecting end-of-input

I would not expect the token parameter to ever be nil, since a token event should always have a token to pass in if it is triggered. Note that this event typically produces a "\n" token value (and still does in normal cases). The expectation should be to either provide a valid token or not trigger the event at all.

(This behavior is present in 2.2.0p95 as well.)

Also available in: Atom PDF