Bug #19848
closed
Ripper BOM behavior
Description
When there is a byte-order mark in a file, the first token in the file usually begins at -3. For example:
Ripper.lex("\xEF\xBB\xBF[]")
# => [[[1, -3], :on_lbracket, "[", BEG|LABEL], [[1, 1], :on_rbracket, "]", END]]
The rest of the tokens appear as if the byte-order-mark never existed. This is consistent except for the case where the file starts with a global variable, an instance variable, or a class variable. In those cases the first token begins at 0. For example:
Ripper.lex("\xEF\xBB\xBF@foo")
# => [[[1, 0], :on_ivar, "@foo", END]]
Ripper.lex("\xEF\xBB\xBF@@foo")
# => [[[1, 0], :on_cvar, "@@foo", END]]
Ripper.lex("\xEF\xBB\xBF$foo")
# => [[[1, 0], :on_gvar, "$foo", END]]
Additionally, when there is a byte-order mark it usually does not appear as part of the first token, unless the token is a magic encoding comment. If it's a magic encoding comment, then it's part of the value:
Ripper.lex("\xEF\xBB\xBF# encoding: us-ascii")
# => [[[1, -3], :on_comment, "\xEF\xBB\xBF# encoding: us-ascii", BEG]]
For solutions - when there is a byte-order mark I think the column information should either always start at 0, or always start at -3. Then for the encoding comment, it should probably not show up as part of the value, or it should show up for all comments.
Updated by 
Updated by