Bug #15924: Ripper::Lexer RuntimeError - Ripper FATAL - master/trunk

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 2.7.0dev (2019-06-15T03:04:16Z master 5d79054906) [x64-mingw32]
• Backport: 2.4: UNKNOWN, 2.5: UNKNOWN, 2.6: UNKNOWN

[MSP-Greg (Greg L)]

YARD based docs systems use Ripper for highlighting both source code and 'code snippets' in comments and rdoc/md files.

Using the build shown, I have an error as shown:

lib/ruby/2.7.0/ripper/lexer.rb:101:in `lex'
lib/ruby/2.7.0/ripper/lexer.rb:123:in `parse'
lib/ruby/2.7.0/ripper/lexer.rb:123:in `parse': [Ripper FATAL] unknown token -1 (RuntimeError)

A quick look shows that it was parsing the rdoc files in the doc folder. Sorry for not being able to check further tonite, I'll look more tomorrow...

[MSP-Greg (Greg L)]

Found the issue, below is a repo, which in MinGW master, generates RuntimeError...

require 'ripper'

temp = "<<\"EOS\n\" # This has been warned since 2.4\nEOS"

puts '', temp, ''

lexer = Ripper::Lexer.new temp
lexer.lex.each { |t| pp t }
puts "lexer.error? #{lexer.error?}"

[MSP-Greg (Greg L)]

• Subject changed from Ripper FATAL - master/trunk to Ripper::Lexer RuntimeError - Ripper FATAL - master/trunk

[nobu (Nobuyoshi Nakada)]

It is an example of syntax error, cannot be fallen back?

YARD itself just warns it but succeeds.

$ echo $'<<\"EOS\n\"\nEOS' > yardtest.rb
$ ~/.gem/ruby/2.7.0/bin/yard doc yardtest.rb 
[warn]: Syntax error in `yardtest.rb`:(1,0): unterminated here document identifier
Files:           0
Modules:         0 (    0 undocumented)
Classes:         0 (    0 undocumented)
Constants:       0 (    0 undocumented)
Attributes:      0 (    0 undocumented)
Methods:         0 (    0 undocumented)
 100.00% documented

[MSP-Greg (Greg L)]

@nobu (Nobuyoshi Nakada)

Thanks. Actually, YARD doesn't use the Lexer, but I switched to using it a long time ago, and all the repos shown at https://msp-greg.github.io/ are created using it. Conversely, Lexer is subclassed from Ripper. So what's using what is a little vague...

Until recently, all 'bad' code snippets have flipped Ripper#error? to true with no RuntimeError. I guess it is a edge case. Ok to close...

[jeremyevans0 (Jeremy Evans)]

• Status changed from Open to Closed