Bug #11485: Ripper parser generates nil tokens for ignored_nl event during parse errors

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: 2.3.0-dev
• Backport: 2.4: REQUIRED, 2.5: REQUIRED, 2.6: REQUIRED

[lsegal (Loren Segal)]

This seems to be a regression in the Ripper parser where the on_ignored_nl event is passed a nil value for the token parameter in the event when a syntax error occurs. I'm not entirely sure of the details, but this is the most minimal reproduction example I could produce:

require 'ripper'

class Parser < Ripper
  def on_ignored_nl(*args) p args end
  def on_parse_error(msg) $stderr.puts "ERROR! #{msg}" end
end

Parser.new(DATA, 'stdin').parse

__END__
something # comment
...

Note the "..." which causes the parse error, but the comment on the line before causes the ignored_nl event to get called with nil as the argument. I'm not sure if it's specific to the ... token or if there are other ways to trigger this parse error. I tried with other invalid syntaxes and could not reproduce with those.

Running the above produces the following output (in 2.0.0p481 and 2.3.0-dev respective):

~$ ruby -v
ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb 
ERROR! syntax error, unexpected ..., expecting end-of-input

~$ ruby -v
ruby 2.3.0dev (2015-08-24 trunk 51672) [x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb 
[nil]
ERROR! syntax error, unexpected ..., expecting end-of-input

I would not expect the token parameter to ever be nil, since a token event should always have a token to pass in if it is triggered. Note that this event typically produces a "\n" token value (and still does in normal cases). The expectation should be to either provide a valid token or not trigger the event at all.

(This behavior is present in 2.2.0p95 as well.)

[jeremyevans0 (Jeremy Evans)]

• File ripper-ignored-nl-nil.patch ripper-ignored-nl-nil.patch added

This bug is still present on master. Attached is a patch that fixes it, by not calling dispatch_delayed_token when ripper is used if the dispatched value is nil.

[nobu (Nobuyoshi Nakada)]

• Description updated (diff)
• Backport changed from 2.0.0: DONTNEED, 2.1: UNKNOWN, 2.2: UNKNOWN to 2.4: REQUIRED, 2.5: REQUIRED, 2.6: REQUIRED

Thank you for the patch, somehow I've missed this issue at all.

dispatch_delayed_token is an empty expression unless ripper, so it is useless inside #ifndef RIPPER.
And delayed tokens should not be nil generally.

[nobu (Nobuyoshi Nakada)]

[ruby-core:<unknown>]

This seems not having sent to the ML, and it happens sometimes, but I'm not sure why.

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|f19e048d244c709e2990ddbb0f986e8b51357bd2.

---

Do not dispatch a nil token in ripper

As a comment token includes the newline, so delayed newline token
just follows it should not be dispatched. [Bug #11485]

Co-Authored-By: Jeremy Evans [e-mail]