Bug #15924

Ripper::Lexer RuntimeError - Ripper FATAL - master/trunk

Added by MSP-Greg (Greg L) about 1 year ago. Updated almost 1 year ago.

Target version:
ruby -v:
ruby 2.7.0dev (2019-06-15T03:04:16Z master 5d79054906) [x64-mingw32]


YARD based docs systems use Ripper for highlighting both source code and 'code snippets' in comments and rdoc/md files.

Using the build shown, I have an error as shown:

lib/ruby/2.7.0/ripper/lexer.rb:101:in `lex'
lib/ruby/2.7.0/ripper/lexer.rb:123:in `parse'
lib/ruby/2.7.0/ripper/lexer.rb:123:in `parse': [Ripper FATAL] unknown token -1 (RuntimeError)

A quick look shows that it was parsing the rdoc files in the doc folder. Sorry for not being able to check further tonite, I'll look more tomorrow...

Updated by MSP-Greg (Greg L) about 1 year ago

Found the issue, below is a repo, which in MinGW master, generates RuntimeError...

require 'ripper'

temp = "<<\"EOS\n\" # This has been warned since 2.4\nEOS"

puts '', temp, ''

lexer = temp
lexer.lex.each { |t| pp t }
puts "lexer.error? #{lexer.error?}"

Updated by MSP-Greg (Greg L) about 1 year ago

  • Subject changed from Ripper FATAL - master/trunk to Ripper::Lexer RuntimeError - Ripper FATAL - master/trunk

Updated by nobu (Nobuyoshi Nakada) about 1 year ago

It is an example of syntax error, cannot be fallen back?

YARD itself just warns it but succeeds.

$ echo $'<<\"EOS\n\"\nEOS' > yardtest.rb
$ ~/.gem/ruby/2.7.0/bin/yard doc yardtest.rb 
[warn]: Syntax error in `yardtest.rb`:(1,0): unterminated here document identifier
Files:           0
Modules:         0 (    0 undocumented)
Classes:         0 (    0 undocumented)
Constants:       0 (    0 undocumented)
Attributes:      0 (    0 undocumented)
Methods:         0 (    0 undocumented)
 100.00% documented

Updated by MSP-Greg (Greg L) about 1 year ago

nobu (Nobuyoshi Nakada)

Thanks. Actually, YARD doesn't use the Lexer, but I switched to using it a long time ago, and all the repos shown at are created using it. Conversely, Lexer is subclassed from Ripper. So what's using what is a little vague...

Until recently, all 'bad' code snippets have flipped Ripper#error? to true with no RuntimeError. I guess it is a edge case. Ok to close...


Updated by jeremyevans0 (Jeremy Evans) almost 1 year ago

  • Status changed from Open to Closed

Also available in: Atom PDF