Project

General

Profile

Bug #15924

Ripper::Lexer RuntimeError - Ripper FATAL - master/trunk

Added by MSP-Greg (Greg L) about 1 month ago. Updated about 1 month ago.

Status:
Open
Priority:
Normal
Assignee:
-
Target version:
-
ruby -v:
ruby 2.7.0dev (2019-06-15T03:04:16Z master 5d79054906) [x64-mingw32]
[ruby-core:93155]

Description

YARD based docs systems use Ripper for highlighting both source code and 'code snippets' in comments and rdoc/md files.

Using the build shown, I have an error as shown:

lib/ruby/2.7.0/ripper/lexer.rb:101:in `lex'
lib/ruby/2.7.0/ripper/lexer.rb:123:in `parse'
lib/ruby/2.7.0/ripper/lexer.rb:123:in `parse': [Ripper FATAL] unknown token -1 (RuntimeError)

A quick look shows that it was parsing the rdoc files in the doc folder. Sorry for not being able to check further tonite, I'll look more tomorrow...

History

Updated by MSP-Greg (Greg L) about 1 month ago

Found the issue, below is a repo, which in MinGW master, generates RuntimeError...

require 'ripper'

temp = "<<\"EOS\n\" # This has been warned since 2.4\nEOS"

puts '', temp, ''

lexer = Ripper::Lexer.new temp
lexer.lex.each { |t| pp t }
puts "lexer.error? #{lexer.error?}"
#2

Updated by MSP-Greg (Greg L) about 1 month ago

  • Subject changed from Ripper FATAL - master/trunk to Ripper::Lexer RuntimeError - Ripper FATAL - master/trunk

Updated by nobu (Nobuyoshi Nakada) about 1 month ago

It is an example of syntax error, cannot be fallen back?

YARD itself just warns it but succeeds.

$ echo $'<<\"EOS\n\"\nEOS' > yardtest.rb
$ ~/.gem/ruby/2.7.0/bin/yard doc yardtest.rb 
[warn]: Syntax error in `yardtest.rb`:(1,0): unterminated here document identifier
Files:           0
Modules:         0 (    0 undocumented)
Classes:         0 (    0 undocumented)
Constants:       0 (    0 undocumented)
Attributes:      0 (    0 undocumented)
Methods:         0 (    0 undocumented)
 100.00% documented

Updated by MSP-Greg (Greg L) about 1 month ago

nobu (Nobuyoshi Nakada)

Thanks. Actually, YARD doesn't use the Lexer, but I switched to using it a long time ago, and all the repos shown at https://msp-greg.github.io/ are created using it. Conversely, Lexer is subclassed from Ripper. So what's using what is a little vague...

Until recently, all 'bad' code snippets have flipped Ripper#error? to true with no RuntimeError. I guess it is a edge case. Ok to close...

Also available in: Atom PDF