Project

General

Profile

Bug #11485

Ripper parser generates nil tokens for ignored_nl event during parse errors

Added by lsegal (Loren Segal) about 4 years ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
[ruby-core:<unknown>]

Description

This seems to be a regression in the Ripper parser where the on_ignored_nl event is passed a nil value for the token parameter in the event when a syntax error occurs. I'm not entirely sure of the details, but this is the most minimal reproduction example I could produce:

require 'ripper'

class Parser < Ripper
  def on_ignored_nl(*args) p args end
  def on_parse_error(msg) $stderr.puts "ERROR! #{msg}" end
end

Parser.new(DATA, 'stdin').parse

__END__
something # comment
...

Note the "..." which causes the parse error, but the comment on the line before causes the ignored_nl event to get called with nil as the argument. I'm not sure if it's specific to the ... token or if there are other ways to trigger this parse error. I tried with other invalid syntaxes and could not reproduce with those.

Running the above produces the following output (in 2.0.0p481 and 2.3.0-dev respective):

~$ ruby -v
ruby 2.0.0p481 (2014-05-08 revision 45883) [universal.x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb 
ERROR! syntax error, unexpected ..., expecting end-of-input

~$ ruby -v
ruby 2.3.0dev (2015-08-24 trunk 51672) [x86_64-darwin14]
~$ ruby ripper-bug-2.2.0.rb 
[nil]
ERROR! syntax error, unexpected ..., expecting end-of-input

I would not expect the token parameter to ever be nil, since a token event should always have a token to pass in if it is triggered. Note that this event typically produces a "\n" token value (and still does in normal cases). The expectation should be to either provide a valid token or not trigger the event at all.

(This behavior is present in 2.2.0p95 as well.)


Files

ripper-ignored-nl-nil.patch (1.42 KB) ripper-ignored-nl-nil.patch jeremyevans0 (Jeremy Evans), 07/03/2019 08:31 PM

Associated revisions

Revision f19e048d
Added by nobu (Nobuyoshi Nakada) 4 months ago

Do not dispatch a nil token in ripper

As a comment token includes the newline, so delayed newline token
just follows it should not be dispatched. [Bug #11485]

Co-Authored-By: Jeremy Evans code@jeremyevans.net

History

Updated by jeremyevans0 (Jeremy Evans) 4 months ago

This bug is still present on master. Attached is a patch that fixes it, by not calling dispatch_delayed_token when ripper is used if the dispatched value is nil.

Updated by nobu (Nobuyoshi Nakada) 4 months ago

  • Backport changed from 2.0.0: DONTNEED, 2.1: UNKNOWN, 2.2: UNKNOWN to 2.4: REQUIRED, 2.5: REQUIRED, 2.6: REQUIRED
  • Description updated (diff)

Thank you for the patch, somehow I've missed this issue at all.

dispatch_delayed_token is an empty expression unless ripper, so it is useless inside #ifndef RIPPER.
And delayed tokens should not be nil generally.

Updated by nobu (Nobuyoshi Nakada) 4 months ago

[ruby-core:<unknown>]

This seems not having sent to the ML, and it happens sometimes, but I'm not sure why.

#4

Updated by nobu (Nobuyoshi Nakada) 4 months ago

  • Status changed from Open to Closed

Applied in changeset git|f19e048d244c709e2990ddbb0f986e8b51357bd2.


Do not dispatch a nil token in ripper

As a comment token includes the newline, so delayed newline token
just follows it should not be dispatched. [Bug #11485]

Co-Authored-By: Jeremy Evans code@jeremyevans.net

Also available in: Atom PDF