Bug #17345: ripper: nothing raised when assigning to keyword variables

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: ruby 3.0.0dev (2020-11-25T04:36:39Z master 00f046ef57) [x86_64-linux]
• Backport: 2.5: REQUIRED, 2.6: REQUIRED, 2.7: REQUIRED

[no6v (Nobuhiro IMAI)]

require "ripper"

Ripper.tokenize("retry=1", raise_errors: true)
# => SyntaxError (syntax error, unexpected '=', expecting end-of-input)

Ripper.tokenize("nil=1", raise_errors: true) # => ["nil", "=", "1"]
lexer = Ripper::Lexer.new("nil=1")
lexer.tokenize # => ["nil", "=", "1"]
lexer.error? # => true
lexer.errors # => []

Lexer recognizes there was an error, but nothing is set to errors.

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|f5ca3ff4dbcf5c140a77d1de5ff3fe3eed2d558d.

---

Store all kinds of syntax errors [Bug #17345]

[no6v (Nobuhiro IMAI)]

Thanks for the quick fix, however *_error parser events except parse_error seem not to set an error message properly.

$ ruby -rripper -ve 'Ripper.lex("nil=1", raise_errors: true)'
ruby 3.0.0dev (2020-11-26T11:14:34Z master f5ca3ff4db) [x86_64-linux]
/home/nov/.rvm/rubies/ruby-head/lib/ruby/3.0.0/ripper/lexer.rb:134:in `parse': #<struct Ripper::Lexer::Elem pos=[1, 0], event=:on_kw, tok="nil", state=END, message=nil> ;#<struct Ripper::Lexer::Elem pos=[1, 3], event=:on_op, tok="=", state=BEG, message=nil> ;#<struct Ripper::Lexer::Elem pos=[1, 4], event=:on_int, tok="1", state=END, message=nil> (SyntaxError)

[no6v (Nobuhiro IMAI)]

no6v (Nobuhiro IMAI) wrote in #note-2:

Thanks for the quick fix, however *_error parser events except parse_error seem not to set an error message properly.

$ ruby -rripper -ve 'Ripper.lex("nil=1", raise_errors: true)'
ruby 3.0.0dev (2020-11-26T11:14:34Z master f5ca3ff4db) [x86_64-linux]
/home/nov/.rvm/rubies/ruby-head/lib/ruby/3.0.0/ripper/lexer.rb:134:in `parse': #<struct Ripper::Lexer::Elem pos=[1, 0], event=:on_kw, tok="nil", state=END, message=nil> ;#<struct Ripper::Lexer::Elem pos=[1, 3], event=:on_op, tok="=", state=BEG, message=nil> ;#<struct Ripper::Lexer::Elem pos=[1, 4], event=:on_int, tok="1", state=END, message=nil> (SyntaxError)

Won't to fix this?
This breaks irb --colorize which uses Ripper::Lexer#scan and which expects Ripper::Lexer::Elem#message to return a string.
https://github.com/ruby/ruby/blob/master/lib/irb/color.rb#L161-L163

While typing nil=, the following exception raises immediately on pressing = key.

$ docker run --rm -it rubylang/all-ruby:latest ./bin/ruby-3.0.0-preview2 -rirb -eIRB.start
irb(main):001:0> nil/build-all-ruby/3.0.0-preview2/lib/ruby/3.0.0/irb/color.rb:163:in `block (2 levels) in scan': no implicit conversion of Array into String (TypeError)
...

[no6v (Nobuhiro IMAI)]

Sharing error messages between ruby core and ripper is fine, thank you!
Now, the '#tok' on some error events are Elem instead of string.

$ ./ruby -rripper -ve 'pp Ripper::Lexer.new("alias $x $1").scan'
ruby 3.0.0dev (2020-12-16T07:17:54Z master c668772b14) [x86_64-linux]
[#<Ripper::Lexer::Elem: on_kw@1:0 FNAME|FITEM token: "alias">,
 #<Ripper::Lexer::Elem: on_sp@1:5 FNAME|FITEM token: " ">,
 #<Ripper::Lexer::Elem: on_gvar@1:6 END token: "$x">,
 #<Ripper::Lexer::Elem: on_sp@1:8 END token: " ">,
 #<Ripper::Lexer::Elem: on_backref@1:9 END token: "$1">,
 #<Ripper::Lexer::Elem:
   on_alias_error@1:11                                               # <-
   END
   token: #<Ripper::Lexer::Elem: on_gvar@1:6 END token: "$x">        # <-
   message: can't make alias for the number variables>]

A quick patch is below (also fixes lineno and column as well):

diff --git a/ext/ripper/lib/ripper/lexer.rb b/ext/ripper/lib/ripper/lexer.rb
index 6a9a9cb39a..26a53a00ce 100644
--- a/ext/ripper/lib/ripper/lexer.rb
+++ b/ext/ripper/lib/ripper/lexer.rb
@@ -190,8 +190,9 @@ def _push_token(tok)
       e
     end
 
-    def on_error(mesg, tok = token())
-      @errors.push Elem.new([lineno(), column()], __callee__, tok, state(), mesg)
+    def on_error(mesg, tok = nil)
+      lineno, column, token = tok ? [*tok.pos, tok.tok] : [lineno(), column(), token()]
+      @errors.push Elem.new([lineno, column], __callee__, token, state(), mesg)
     end
     PARSER_EVENTS.grep(/_error\z/) do |e|
       alias_method "on_#{e}", :on_error

In particular, the #tok on on_alias_error may be wrong.
It should be $1 in above example.

$ ./ruby -e 'alias $x $1' 2>&1 | cat
-e:1: can't make alias for the number variables
alias $x $1
         ^~

[nobu (Nobuyoshi Nakada)]

• Status changed from Closed to Open
• Backport changed from 2.5: UNKNOWN, 2.6: UNKNOWN, 2.7: UNKNOWN to 2.5: REQUIRED, 2.6: REQUIRED, 2.7: REQUIRED

[nobu (Nobuyoshi Nakada)]

• Status changed from Open to Closed

Applied in changeset git|47328ad217ecaf240f0faedac89723dcd6a917ac.

---

Ripper: Fixed erred token on wrong alias [Bug #17345]

[no6v (Nobuhiro IMAI)]

Now, the '#tok' on some error events are Elem instead of string.

pull request: https://github.com/ruby/ruby/pull/3936

[nagachika (Tomoyuki Chikanaga)]

MEMO: I tried to backport f5ca3ff4dbcf5c140a77d1de5ff3fe3eed2d558d,e0bdd54348514ff06df88a9bac88fa56058235a4,47328ad217ecaf240f0faedac89723dcd6a917ac,9c859f4b3c2098af590d3e0eadc2d1011fb32bb8,e33eb09b76e9070fd49837e8b896319636501b38 but I run into compile error on ripper.

[nobu (Nobuyoshi Nakada)]

• Related to Bug #20649: Ripper fails to tokenize `def f; (x)::A =` added