Bug #3765: Ripper::Lexer missed out tokens after '=>' operator

• Status: Rejected
• Assignee: -
• Target version: -
• ruby -v: ruby 1.9.1p376 (2009-12-07 revision 26041) [x86_64-linux]

[ngtzeyang (TzeYang Ng)]

=begin
Given the following ripper use case:

pp Ripper.lex(':x => 1')

I'm getting:

[[[1, 0], :on_symbeg, ":"],
[[1, 1], :on_ident, "x"],
[[1, 2], :on_sp, " "],
[[1, 3], :on_op, "=>"]]

I'm thinking the output should be:

[[[1, 0], :on_symbeg, ":"],
[[1, 1], :on_ident, "x"],
[[1, 2], :on_sp, " "],
[[1, 3], :on_op, "=>"],
[[1, 4], :on_sp, " "],
[[1, 5], :on_int, "1"]]

Somehow the trailing ' 1' is missed out. Seems like a bug ?
=end

[eitoball (Eito Katagiri)]

=begin
I think that it is because the provided statement was illegal and Ripper stopped parsing it.

% ruby -ve ":x => 1"
ruby 1.9.1p378 (2010-01-10 revision 26273) [i386-darwin9.8.0]
-e:1: syntax error, unexpected tASSOC, expecting $end
:x => 1
^

=end

[mame (Yusuke Endoh)]

• Status changed from Open to Rejected

=begin
Eito is right.

Ruby lexer has very complex states.
Such an incomplete code fragment may not parse.
You should write complete code:

$ ruby -rripper -e 'p *Ripper.lex("foo :x => 1")'
[[1, 0], :on_ident, "foo"]
[[1, 3], :on_sp, " "]
[[1, 4], :on_symbeg, ":"]
[[1, 5], :on_ident, "x"]
[[1, 6], :on_sp, " "]
[[1, 7], :on_op, "=>"]
[[1, 9], :on_sp, " "]
[[1, 10], :on_int, "1"]

--
Yusuke Endoh [e-mail]
=end