Misc #10312: Give people more control over how the ruby parser sees code and lexical code elements (valid/invalid - toggle options) + macros

• Status: Open
• Assignee: -

[shevegen (Robert A. Heiler)]

Hi,

I am aware that this proposal has most likely not a chance for
implementation, but I'd still like to make it - this is why I
put it here into misc rather than bugs or features. It's misc -
like ideas!

So first allow me to introduce how I came up with this idea at
all.

I have built a "web framework", which is pretty ugly, but
useful to me. It is actually less of a web framework and
more of a way to describe a "web app" - right now using
valid ruby syntax, but hopefully one day I can transition
into something that is even terser than what it is right
now, and eventually becomes valid ruby (or css or javascript
etc... a bit like HaXe http://haxe.org/ but with a more
elegant syntax resembling ruby rather than java)

I describe a web-page currently such as that way:

w {
title 'My little page'
css_style '
body {
padding: 0.20em;
}'
favicon 'foo/bar.png'
font_size '20px'
use_jquery
}

Ok, w is simply a method call w() returning an instance of class
WebObject, and we pass it a block. The block I use for a DSL-like
approach to describe the page in question. For instance, use_jquery
simply is in fact w.use_jquery - so it is a method call on that
web object.

Before that, I was using this line here:

jquery :+

The + reminds me of "yes, enable jquery".

Obviously, to disable jquery, I would do:

jquery :-

Then I thought - "Hey, it would be cool if I could do this:

jquery +

instead. In other words, get rid of the : symbol identifier.

But here the ruby parser rightfully chokes because it does not
understand what is meant with jquery +. Fair enough, this is
not valid ruby.

But to my human eyes, jquery + reads nicer than jquery :+

I could get away doing this instead and treat it as a String:

'jquery +'

And simply parse that with ruby. But I already use:

use_jquery

above, so that is nicer IMO than using the two ' characters.

So now, sorry for that long introduction but perhaps you can
understand my line of thought here.

So I was thinking it would be nice if people could get more
control over the Ruby Parser itself.

For instance, in the above example and only for that file/class,
I would like to tell the ruby parser "hey dude, don't mind if
that specific block has invalid syntax, I'd parse on my own.

Possibly it would be enough if those invalid syntax elements
could become strings - then I can parse those strings on my
own.

Of course the net gain between:

jquery :+

and

jquery +

is minimal, but I love ruby's terse syntax.

Anyway, I am aware that this has no real chance for the ruby
2.x era but perhaps considering all the ideas in regards to
optional static type information and what-not for 3.x ruby era,
more control over the ruby parser itself would be nice
(lisp-like macros!).

Regards.

[jeremyevans0 (Jeremy Evans)]

Here are a couple of options for your use case, if you want terse syntax that is valid ruby:

jquery.+
jquery.-

or:

+jquery
-jquery

[shevegen (Robert A. Heiler)]

Hi,

The problem is that they are extremely similar to the other solutions
such as:

  jquery :+

Like

  jquery.+

Not counting the ' ' character, that one has the same amount of
tokens as the above variant with a Symbol.

  +jquery

May be an option but I am not allowed to use a ' ' character?

  + jquery

?

So preferentially I'd prefer it to be

  jquery +

[shevegen (Robert A. Heiler)]

I would also like to point out that there are already some
limited or perhaps indirect ways to "control" the parser,
in at least one regards, which is the

__END__

construct.

"Invalid" arbitrary syntax can be put after that, and it
is even available for ruby scripts through DATA.read,
which is pretty nifty.

The only complaint I would have is that it must happen
as the last part of a script - it would be nice if it
could happen in the middle of a .rb file too, and
then there would be another toggle switch to allow me
to tell the ruby parser "ok, continue to evaluate the
rest of that .rb file as if it were valid ruby code
again".

I only wanted to point this out, that we already have
a few ways to control the parser - at least by telling
it to ignore anything that comes after END.

[shevegen (Robert A. Heiler)]

Sorry, the __END__ seemed to become bold - that was not
intentionally, I hope you guys still understand what I
was trying to say.

[avit (Andrew Vit)]

That sounds like heredoc syntax, which ruby already does.

[nobu (Nobuyoshi Nakada)]

Robert A. Heiler wrote:

May be an option but I am not allowed to use a ' ' character?

  + jquery

?

Of course, you can.

So preferentially I'd prefer it to be

  jquery +

That "dangling operator" looks very strange to me.

[shevegen (Robert A. Heiler)]

That sounds like heredoc syntax, which ruby already does.

Hmm I guess you are right - I can get it into a string
that way. But the problem is - I can already get it into
a String if I do this:

'jquery +'

so your suggestion would not be a real advantage
compared to if I already turn it into a 'string'.

But then I'd also lose the advantage of terseness in
a mini-DSL.

jquery +

That "dangling operator" looks very strange to me.

Yes, precisely - if it were valid ruby. But of course it
is invalid ruby, so it is not a dangling operator instead,
but should be interpretable differently in some other way. :)

After thinking about this, I guess what I would ideally like
is a way to have multiple END like constructs and
additionally a way to force the ruby parser to want to
continue in case of seemingly errors like the above.

In the example above, END is probably not a good example
either, as I need 7 characters for it, and probably again
7 characters to untoggle it.

Perhaps I should come with some other more generic way hmm.

[jwmittag (Jörg W Mittag)]

I think some form of macros, DSL embedding, and/or compile-time metaprogramming would indeed be cool to have. I have, however, no idea what they would look like.

Here are some pointers to non-homoiconic languages (i.e. not Lisp) with some of those features.

• Converge combines dynamically-typed OO semantics and Pythonic syntax with embedded DSLs and CTMP, the author chose Pythonic syntax specifically, because it makes macros/DSL embedding extra hard.
• PLOT – Programming Language for Old Timers is in some sense a Lisp, but also very different: it is not homoiconic, it is not based around a single all-powerful datatype, it is (kind-of) object-oriented, it has rich Pythonic/Rubyish syntax, and yet it has hygienic macros, and powerful reflective metaprogramming; it can be used exactly like a Lisp while at the same time being much more like Ruby.
• Scala has naturally evolved macros along a simple path:
  1. getting good IDE support for Scala requires a fair bit of type inference and implicit resolution to be done, so instead of each IDE vendor having to implement that over and over again, the Scala developers refactored the compiler so that it can be used inside an IDE. That's actually nothing special, as languages become more complex and IDE users become more demanding, that's the way a lot of implementations are moving: Rubinius has powerful program comprehension features which it exposes via APIs, JRuby has split off the JRubyParser project for usage in IDEs and editors, Ripper can be seen as a simple step in that direction, LLVM and Clang were explicitly designed for IDE embedding, Roslyn is designed to replace all the custom-built C# and VB.NET tools within Visual Studio.NET, Kotlin is even specifically designed by an IDE vendor within an IDE.
  2. Anyway, after making the compiler available to the IDE, it turned out it only took some small features to also make the compiler available as a Toolbox to be called from user code, and thus the Scala REPL was born.
  3. Then, the developers realized that the job of a reflection and the job of a compiler are actually really similar: both need to understand the code deeply. So, they decided to build a completely new reflection library on top of the compiler. That meant that they had to move the compiler into the runtime.
  4. And once they did that, they realized that moving the compiler into the runtime is pretty much the same thing as moving the runtime into the compiler, and in fact the same APIs that are used for reflection then almost automatically also give you macros almost for free.

There's a lot of stuff in Scala Macros related to how to make them type-safe, which is irrelevant in Ruby, but the interesting thing is: moving from reflection to macros is actually quite straightforward, and Ruby already is a highly reflective language.