Bug #22600
closedparse.y corrupts line-1 heredoc delimiter/resume offset by 3 bytes after a UTF-8 BOM
Description
When a Ruby script or eval string starts with a UTF-8 BOM (\xEF\xBB\xBF) and declares a heredoc on line 1 (e.g. \xEF\xBB\xBF<<EOF\nEOF\n, \xEF\xBB\xBF(<<EOF)\nEOF\n, \xEF\xBB\xBFp(<<EOF)\nEOF\n, or \xEF\xBB\xBF[<<EOF]\nEOF\n), CRuby parse.y miscomputes the heredoc identifier/resume offset on line 1 by 3 bytes (the length of the UTF-8 BOM), looking up "..." shifted 3 bytes to the left (such as "(<<", "[<<", or "\xBF<<") and failing with SyntaxError: can't find string ... anywhere before EOF followed by unexpected constant.
If a newline separates the BOM from the heredoc (\xEF\xBB\xBF\n(<<EOF)\nEOF\n), both parse.y and --parser=prism succeed.
Discovered by differential fuzzing of prism against parse.y.
Reproduction¶
Tested on CRuby 4.1.0dev (2026-11-18 HEAD ed3e747aa4) and ruby 4.0.9 (2026-10-23 revision 3e93b23afc) +PRISM [arm64-darwin27].
Updated by nobu (Nobuyoshi Nakada) about 13 hours ago
- Status changed from Open to Closed
Applied in changeset git|9036f16dc369aaf28ecdc9686293af1dbf88d087.
[Bug #22600] Count BOM just before here document