Misc #17662: The heredoc pattern used in tests does not syntax highlight correctly in many editors

• Status: Assigned
• Assignee: nobu (Nobuyoshi Nakada)

[Eregon (Benoit Daloze)]

This heredoc pattern

    assert_ruby_status([], "#{<<-"begin;"}\n#{<<-'end;'}", bug)
    begin;
      exit("1" == Thread.start(1, &:to_s).value)
    end;

completely breaks syntax highlighting in at least:

• GitHub: there are many examples
• Atom
• RubyMine (and IntelliJ)
• Likely many more editors based on TextMate grammars

Could another pattern be used in tests inside the ruby/ruby repository (at least for test/ruby)?

Due to this issue, it is very annoying and inconvenient to look at/read/investigate many tests.

I think this pattern is also very complicated to understand (and using ; is quite weird for this).
I suggest to replace it with this obvious and simple pattern many people use:

    assert_ruby_status([], <<~'RUBY', bug)
      exit("1" == Thread.start(1, &:to_s).value)
    RUBY

This syntax highlights correctly in most (all?) editors, and as an added bonus the code inside the heredoc is also highlighted in some editors (due to the label being RUBY).

[Eregon (Benoit Daloze)]

• Subject changed from The herdoc pattern used in tests does not syntax highlight correctly in many editors to The heredoc pattern used in tests does not syntax highlight correctly in many editors

[xtkoba (Tee KOBAYASHI)]

In the suggested pattern, the here document seems not highlighted nor auto-indented by Emacs Ruby Mode. Would you please also suggest the configuration of Emacs Ruby Mode for the suggested pattern?

[Eregon (Benoit Daloze)]

I'm not an Emacs user, so I don't know.
I think indentation doesn't matter much, it could be this too:

    assert_ruby_status([], <<~'RUBY', bug)
    exit("1" == Thread.start(1, &:to_s).value)
    RUBY

Highlighting the Ruby code inside the heredoc seems a nice extra, but the issue of not highlighting the entire file due to the pattern seems much worse.

I guess the best would be if the Emacs Ruby Mode implements highlighting if the heredoc label is RUBY.
I'm open to see other patterns that might work better on Emacs and still don't destroy syntax highlighting of the file in other editors.

[xtkoba (Tee KOBAYASHI)]

Emacs Ruby Mode behaves well with the current pattern. I wonder why the other editors do not simulate it.

[marcandre (Marc-Andre Lafortune)]

It's a really complex formulation: a double HEREDOC inside string interpolatiosn, with a delimiter of the HEREDOC using double quotes but clearly that's not the end of the string because it's inside a #{}...
I can barely makes sense of it myself.

[mame (Yusuke Endoh)]

• Status changed from Open to Assigned
• Assignee set to nobu (Nobuyoshi Nakada)

My vim works great with the code :-) But breaking syntax highligt in GitHub is not so great.

AFAIK the style is mainly written by @nobu (Nobuyoshi Nakada). Nobu, what do you think?

[nobu (Nobuyoshi Nakada)]

I don't mind to transfer to <<~RUBY if ruby-mode.el supporting it gets released.
BTW, what editors support it?

[nobu (Nobuyoshi Nakada)]

Also, does auto indentation work inside <<~RUBY as begin;...end; hack?

[mame (Yusuke Endoh)]

Okay, the style is an intentional hack to trick the editors into syntax-highlinting and auto-indenting the code within the heredoc. I didn't know, but it makes sense to me.

I investigated the syntax highlighting behavior of GitHub. They does not correctly support an empty heredoc in an indent:

https://gist.github.com/mame/90c1427163266ba73fab4cf7d96edd0a

I think they should fix it, but I found one workaround: adding a new line in <<"begin;" heredoc solves the issue.

https://github.com/ruby/ruby/blob/d10b535806227d24a3f8f5c7f425be3a39c2ac74/test/ruby/test_string.rb#L697

[mame (Yusuke Endoh)]

mame (Yusuke Endoh) wrote in #note-9:

I investigated the syntax highlighting behavior of GitHub. They does not correctly support an empty heredoc in an indent:

https://gist.github.com/mame/90c1427163266ba73fab4cf7d96edd0a

Wow, it looks like that GitHub fixed the issue. The code is now colored correctly as a heredoc. Great.

Eregon (Benoit Daloze) wrote:

completely breaks syntax highlighting in at least:

• GitHub: there are many examples

These lines are also syntax-highlighted correctly.

For the record: when this ticket was created, all the code after the heredoc was not colored at all in GitHub file view.

[marcandre (Marc-Andre Lafortune)]

Great for Github!

FWIW, both VS Code and Sublime Text get very confused by this pattern.

[Eregon (Benoit Daloze)]

Here is a particularly weird/complex example usage of heredocs in tests:
https://github.com/ruby/ruby/blob/eef3c08edc2fc74496cdc1e4d89798d053e32561/test/ruby/test_thread.rb#L1380-L1394

It does not syntax highlight, it is very hard to read, and the {#1 nesting seems there for no reason after 2 lines of code?
The inner script is actually just:

code = <<~RUBY
tpid = #{Process.pid}
sig = :#{sig}
STDOUT.sync = true
while gets
  puts
  Process.kill(sig, tpid)
end
RUBY

I think it would be great to consolidate to a single style of heredoc in tests,
and if possible one that does not involve twice the number of heredocs than needed.

[nobu (Nobuyoshi Nakada)]

Eregon (Benoit Daloze) wrote in #note-12:

and if possible one that does not involve twice the number of heredocs than needed.

It is intentional to indent inside here-docs and separate interpolation-enabled and -disabled parts.