Bug #11485
closed
Ripper parser generates nil tokens for ignored_nl event during parse errors
Description
This seems to be a regression in the Ripper parser where the on_ignored_nl event is passed a nil value for the token parameter in the event when a syntax error occurs. I'm not entirely sure of the details, but this is the most minimal reproduction example I could produce:
require 'ripper'
class Parser < Ripper
def on_ignored_nl(*args) p args end
def on_parse_error(msg) $stderr.puts "ERROR! #{msg}" end
end
Parser.new(DATA, 'stdin').parse
__END__
something # comment
...
Note the "..." which causes the parse error, but the comment on the line before causes the ignored_nl event to get called with nil as the argument. I'm not sure if it's specific to the ... token or if there are other ways to trigger this parse error. I tried with other invalid syntaxes and could not reproduce with those.
Running the above produces the following output (in 2.0.0p481 and 2.3.0-dev respective):
~$ ruby -v
ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb
ERROR! syntax error, unexpected ..., expecting end-of-input
~$ ruby -v
ruby 2.3.0dev (2015-08-24 trunk 51672) [x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb
[nil]
ERROR! syntax error, unexpected ..., expecting end-of-input
I would not expect the token parameter to ever be nil, since a token event should always have a token to pass in if it is triggered. Note that this event typically produces a "\n" token value (and still does in normal cases). The expectation should be to either provide a valid token or not trigger the event at all.
(This behavior is present in 2.2.0p95 as well.)
Files
Updated by 
Updated by