Feature #19015: Language extension by a heredoc

• Status: Open
• Assignee: -
• Target version: -

[ko1 (Koichi Sasada)]

This propose new heredoc extension with <<!LANG like

doc = <<!LANG
  # description written in lang LANG
  foo bar
LANG

and it is translated to:

doc = heredoc_extension_LANG(heredoc_text, binding)

Example

require 'erb'

def heredoc_extension_erb str, b
  ERB.new(str).run(b)
end

name = 'ko1'

html = <<!erb
<div>Hello <%= name %></div>
erb

puts html #=> <div>Hello ko1</div>

Background / considerations

• Sometimes we write Ruby syntax string with <<RUBY and this proposal inspired by it.
• it is similar to shebang (#!LANG in shell)
• Elixir's custom sigil translates ~u(...) translates to sigil_u(...). This is why it translated to heredoc_extension_LANG(...) private method call.
• JavaScript has JSX but I don't think it is fit to the Ruby language.
• Heredoc is Ruby's chaos part and already confusing a lot. Additional chaos doesn't matter.
• <<!foo is valid syntax but now I don't think it is not used. gem codesearch doesn't find the usage.
• Sorry I couldn't wait 1st/Apr.

Implementation

I attached the experimental implementation which only supports erb (because I couldn't find how to get delimiter to determine a method name :p).

[ko1 (Koichi Sasada)]

• File deleted (tmp.YubJ1EHi0O-xyzzy)

[ko1 (Koichi Sasada)]

• File heredoc_extension.patch heredoc_extension.patch added

[ko1 (Koichi Sasada)]

Related I heard:

• Template literals (Template strings) - JavaScript | MDN Tagged templates

[retro (Josef Šimánek)]

Would you mind to add also what's the output to the description? If I understand it well, following will be printed.

<div>Hello ko1</div>

Any plans already what to do when method is not implemented?

[zverok (Victor Shepelev)]

I am not sure how serious this is (considering the "Apr 1" notice), but I have somewhat adjacent thought:

In many modern code editors, highlighting of several different languages in the same file is supported. Namely, SublimeText (I am not sure about the others, but I suppose the idea is not unique) understands this:

query = <<~SQL
  SELECT * FROM users WHERE status='active
SQL

DB.execute(query)

...and highlights the code inside a heredoc as SQL.

I am thinking that maybe some way of preserving the "tag" it was surrounded (in String metadata?.. Or, making HEREDOC-produced object a different class, a descendant of String with extra functionality) would be generically useful. It will allow implementing the @ko1 (Koichi Sasada) 's example just like this:

require 'erb'

def heredoc_extension_erb str, b
  ERB.new(str).run(b)
end

name = 'ko1'

html = <<~erb
<div>Hello <%= name %></div>
erb

puts execute_heredoc(html, binding)

# where...
def execute_heredoc(str, binding)
  case str.__tag__
  when 'erb'
    ERB.new(str).run(binding)
  # ....
  end
end

The idea can even be expanded to provide additional metadata (currently invalid syntax, so no compatibility would be broken):

html = <<~erb(trim_mode="%>")
<div>Hello <%= name %></div>
erb

WDYT?

[ko1 (Koichi Sasada)]

• Description updated (diff)

[yugui (Yuki Sonoda)]

one bikeshedding..

Usually syntax-suger in Ruby are mapped into methods whose names are very consistent with the syntax. e.g. `, []=, or foo=. Therefore, it is more consistent to map the new syntax into <<!LANG method.
It seems possible to make the lexer distinguish such a notation from << symbol.

[ko1 (Koichi Sasada)]

This is a tiny example of SQL.
Note that I'm newbe of SQL (I goooooooogled the syntax/api and the example is never tested).

module SQLite3HeredocExtension
  def compile str
    vs = []
    compiled = str.gsub(/\?([a-z_][a-zA-Z\d]+)/){|m|
      vs << $1
      ??
    }
    [vs, compiled]
  end

  # syntax
  # * SQL statement
  # * but ?var is replaced with the Ruby's variable

  def heredoc_exntesion_SQL str, b
    vs, compiled_str = compile str
    stmt = @db.prepare compiled_str
    @db.execute stmt, vs.map{|v| b.local_variable_get(v).to_s}
  end
end

require 'sqlite3'

@db = SQLite3::Database.new "test.db"
include SQLite3HeredocExtension

results = <<!SQL
  select person.name from people where person.age > ?lowest_age
SQL

## hmm, "SQL" seems to return a SQL statement...?

# AR
results = People.select(:name).where("age >= ?", lowest_age)

# AR/execute
results = ActiveRecord::Base.connection.execute("select person.name from perople where person.age > ?", lowest_age)

[estum (Anton (estum))]

Wow, I am not the only such geek %)
My solution of the similar goal is 7 yo and it'll go to school soon.

$ git log lib/osascript.rb
commit 1f39d1d42b499d1424af1fa5a109ecd6ab219563 (HEAD -> master)
Author: Anton
Date:   Thu Jun 11 08:47:12 2015 +0300

# @example Simple
#   Osascript.new(<<~SCPT.freeze).()
#     activate application "Finder"
#   SCPT
#
# @example JSC with args
#   # The script takes 2 arguments: directory path & image path
#   # to set a folder icon to the given directory.
#   script = Osascript.new(<<-JS.freeze, lang: 'JavaScript')
#     ObjC.import("Cocoa");
#     function run(input) {
#       var target_path = input[0].toString();
#       var source_image = $.NSImage.alloc.initWithContentsOfFile(input[1].toString());
#       var result = $.NSWorkspace.sharedWorkspace.setIconForFileOptions(source_image, target_path, 0);
#       return target_path;
#     }
#   JS
#   script.(target_dir, folder_icon)
class Osascript
  attr_accessor :script

  def initialize(script = nil, lang: "AppleScript")
    @script = block_given? ? yield : script
    @lang = lang
  end

  def call(*other)
    handle_errors do
      cmd = ["/usr/bin/env", "osascript", *params(*other)]

      IO.popen cmd, "r+", 2 => %i(child out) do |io|
        io.write script
        io.close_write
        io.readlines
      end
    end
  end

  def params(*args)
    ["-l", @lang].tap { |e| e.concat(args.unshift(?-)) unless args.empty? }
  end

  ERROR_PATTERN = /(?<=execution error: )(.+?)(?=$)/
  USER_CANCELLED_PATTERN = /user canceled/i
  NL = "\n"

  private

  def handle_errors
    yield().each_with_object([]) do |line, buf|
      line.match(ERROR_PATTERN) { |m| raise error_for(m[0]), m[0], caller(4) }
      buf << line.strip
    end.join(NL)
  end

  def error_for(msg)
    USER_CANCELLED_PATTERN.match?(msg) ? UserCanceled : ExecutionError
  end

  class ExecutionError < RuntimeError
    CAPTURE_MSG_AND_CODE = /(.+?) \((-?\d+?)\)$/

    attr_reader :code

    def initialize(msg)
      msg.match(CAPTURE_MSG_AND_CODE) { |m| msg, @code, * = m.captures }
      super(msg)
    end
  end

  UserCanceled = Class.new(ExecutionError)
end

I've wrote it when I've known that cool syntax hook at the first time — an ability to pass only the opening heredoc word in closed parenthesis on single line and you can ducktype it infinitely.

Oh, and I just called in mind one more thing about heredoc: there is some tricky heredoc syntax in core source file forwardable.rb which brakes my brain when I try to understand it:

    if _valid_method?(method)
      loc, = caller_locations(2,1)
      pre = "_ ="
      mesg = "#{Module === obj ? obj : obj.class}\##{ali} at #{loc.path}:#{loc.lineno} forwarding to private method "
      method_call = "#{<<-"begin;"}\n#{<<-"end;".chomp}"
        begin;
          unless defined? _.#{method}
            ::Kernel.warn #{mesg.dump}"\#{_.class}"'##{method}', uplevel: 1
            _#{method_call}
          else
            _.#{method}(*args, &block)
          end
        end;
    end

    _compile_method("#{<<-"begin;"}\n#{<<-"end;"}", __FILE__, __LINE__+1)
    begin;
      proc do
        def #{ali}(*args, &block)
          #{pre}
          begin
            #{accessor}
          end#{method_call}
        end
      end
    end;

Pretty cryptic, isn't it?

[duerst (Martin Dürst)]

This proposal sounds interesting, but the naming looks like behind-the-scenes metaprogramming; it may be better to use a more flexible approach that doesn't fix the function name.

[Eregon (Benoit Daloze)]

This seems nice.
Would it also remove leading indentation like <<~HEREDOC?

[ko1 (Koichi Sasada)]

Would it also remove leading indentation like <<~HEREDOC?

Sure.

[dsisnero (Dominic Sisneros)]

It reminds me of javascript template literals and tagged templates. Instead of passing in bindings they pass the interpolation points and the first argument is an array containing thestring split at interpolation points, and the second is the interplation values.

require 'cgi'

def heredoc_extension_erb strs_separated_array_at, **interpolation_points
    result = strs_separated_array.zip(interpolation_points).each_with_object([]) do |s, v, ar|
         ar << s
         ar << v if v
   end
   result.join(" ")
end

name = 'ko1'

html = <<!html
<div class="#{clsname}">Hello #{name}</div>
erb

I could see where this could be used to provide cmd interpolation similar to julia

files = ["/etc/password", "/Volumes/External HD/data.csv"]

def heredoc_ext_cmd(strs_ar, values)
   # shellescape values - 
   # if current_values is a array and curr_str ends in space and next str is nonexistant or ends in space join values with a space
   # 
   #collect cmd and cmd_args
  result = Process.new(cmd, cmd_args)
end

mycmd = <<!cmd
  grep foo #{files}
cmd

mycmd.run

[pyromaniac (Arkadiy Zabazhanov)]

Hey folks. I'm actually wondering, why don't support Elixir-like sigils in Ruby? We have a ton of them already: %w[], %r//, so why don't just add a support for custom ones?
I'm, for once, tired of writing Date.new or Time.parse, I'd like to have it %D[2023-05-05] or Money gem can introduce something like %M(100 USD) instead of the long Money.from_amount(100, "USD") version.

[duerst (Martin Dürst)]

@pyromaniac (Arkadiy Zabazhanov) I think the main problem would be how to handle namespacing. With single letters, the chance of collision is very high. How does Elixir handle this?

[pyromaniac (Arkadiy Zabazhanov)]

@duerst (Martin Dürst) It is funny you asked cause I just found this

[Kernel] Support for multi-letter uppercase sigils

here https://hexdocs.pm/elixir/main/changelog.html
We can definitely start doing this right from the beginning.
Also, from my experience, even 1-letter sigils are not causing any issues but simplifying code reading/writing a lot.

Sigils are going especially improve the readability in specs where you need to initialize a lot of VO/DTO literals.