Feature #8110

Regex methods not changing global variables

Added by Ilya Vorontsov about 1 year ago. Updated 12 months ago.

[ruby-core:53498]
Status:Assigned
Priority:Normal
Assignee:Yukihiro Matsumoto
Category:core
Target version:next minor

Description

It is useful to have methods allowing pattern matching without setting global variables. It can be very hard to understand where the problem is when you for example insert a string like puts pat === my_str and your program fails in a place which is far-far away from inserted place. This can happen due to replacing global variables of previous pattern match. I caught to this when placed pattern-match inside case-statement and shadowed global vars which were initially filled by match in when-statement.
For now one can extract pattern matching into another method thus defining method-scope for that variables. But sometimes it looks like an overkill. May be simple method like #match_globalsafe can prevent that kind of errors. At least when a programmer see such a method in a list of methods, he's warned that usual match can cause such problems.


Related issues

Related to ruby-trunk - Feature #8206: Should Ruby core implement String#blank? Open 04/03/2013

History

#1 Updated by Yui NARUSE about 1 year ago

  • Category set to core
  • Status changed from Open to Assigned
  • Assignee set to Yukihiro Matsumoto
  • Target version set to next minor

It sounds reasonable.

The API can be some of following:
* new API like Regexp#matchwithoutbackref
* new option for Regexp like Regexp.new("foo", Regexp::NO_BACKREF) or /foo/B
* new syntax

#2 Updated by Sam Saffron about 1 year ago

@naruse

There is a perf implication that really needs addressing here that would help elsewhere:

in re.c, there is a whole bunch of work that can be avoided when NO_BACKREF is passed in for the match:

In particular:

 match = match_alloc(rb_cMatch);
onig_region_copy(RMATCH_REGS(match), regs);
onig_region_free(regs, 0);
}
else {
if (rb_safe_level() >= 3)
    OBJ_TAINT(match);
else
    FL_UNSET(match, FL_TAINT);
}

RMATCH(match)->str = rb_str_new4(str);
RMATCH(match)->regexp = re;
RMATCH(match)->rmatch->char_offset_updated = 0;
rb_backref_set(match);

OBJ_INFECT(match, re);
OBJ_INFECT(match, str);

This in turn should improve the performance of regex matching with the /B option quite a lot.

I have been looking at this recently due to some performance issues I noticed on Active Supports String#blank?

The c implementation of:

def blank?
self !~ /[[:space:]]/
end

is the somewhat crazy:

https://github.com/SamSaffron/fast_blank/blob/master/ext/fast_blank/fast_blank.c#L16-L55

This implementation is 5 to 8x faster.

I vote for:

  • new option for Regexp like Regexp.new("foo", Regexp::NO_BACKREF) AND /foo/B

You can then feature detect if its available by looking for Regexp::NO_BACKREF

I do wonder how much faster this will be for my micro benchmark vs the native c implementation, when you are done can you ping me so I can bench it? (at sam.saffron@gmail.com)

#3 Updated by Sam Saffron about 1 year ago

A slight concern here is naming, since:

rbdefinevirtualvariable("$~", matchgetter, matchsetter);
rb
definevirtualvariable("$&", lastmatchgetter, 0);
rbdefinevirtualvariable("$`", prematchgetter, 0);
rbdefinevirtualvariable("$'", postmatchgetter, 0);
rbdefinevirtualvariable("$+", lastparenmatchgetter, 0);

rb_define_virtual_variable("$=", ignorecase_getter, ignorecase_setter);
rb_define_virtual_variable("$KCODE", kcode_getter, kcode_setter);
rb_define_virtual_variable("$-K", kcode_getter, kcode_setter);

even though core uses the term backref quite extensively, often people can confuse it with:

"round brackets also create a "backreference". A backreference stores the part of the string matched by the part of the regular expression inside the parentheses."

see: http://www.regular-expressions.info/brackets.html

I wonder if a different term all together should leak out Regexp::SKIP_GLOBALS and /foo/S , this is far more explicit and clear to explain.

#4 Updated by Sam Saffron about 1 year ago

another slight note, I wonder how far this can stretch into onigaruma itself, can it be smart enough to avoid uneeded allocations when in a no backref mode?

#5 Updated by Yui NARUSE about 1 year ago

sam.saffron (Sam Saffron) wrote:

another slight note, I wonder how far this can stretch into onigaruma itself, can it be smart enough to avoid uneeded allocations when in a no backref mode?

Oniguruma supplies the way to regexp search without backref.
see also my patch in https://bugs.ruby-lang.org/issues/8206#note-4

#6 Updated by Yura Sokolov about 1 year ago

+1 for skip globals: when String#match used, there is no need to set globals, but no way to avoid it.
Equally String#[]. And even sometime with =~ and ===. So that //S and Regexp::SKIP_GLOBALS will be very useful.

#7 Updated by Sam Saffron about 1 year ago

Has anyone given any thought at how to make this friendly with older versions of Ruby ... say I have

def is_foo?(val)
val =~ /foo/
end

And now I want this code to work in both 1.9.3 and master.

ugly and slow

def isfoo?(val)
if defined? Regexp::SKIP
GLOBALS
val =~ /foo/G
else
val =~ /foo/
end
end

will not work on 1.9.3

def is_foo?(val)
val =~ /foo/G
end

could work, risky perf

def is_foo?(val)
val =~ _G(/foo/)
end

least horribly imho

ISFOO = _G(/foo/)
def is
foo?(val)
val =~ IS_FOO
end


So I wonder, is the plan to backport this? Are there any other ways to keep code compatible and clean?

#8 Updated by Matthew Kerwin about 1 year ago

=begin
sam.saffron (Sam Saffron) wrote:

Has anyone given any thought at how to make this friendly with older
versions of Ruby ... say I have

def is_foo?(val)
val =~ /foo/
end

And now I want this code to work in both 1.9.3 and master.

ugly and slow

def isfoo?(val)
if defined? Regexp::SKIP
GLOBALS
val =~ /foo/G
else
val =~ /foo/
end
end

[snip]

least horribly imho

ISFOO = _G(/foo/)
def is
foo?(val)
val =~ IS_FOO
end


So I wonder, is the plan to backport this? Are there any other ways to
keep code compatible and clean?

Defining a (({_G})) method in 1.9.* is no more or less a backport than allowing (and possibly ignoring) a /G modifier, and is pretty ugly to boot. I see no harm in adding some amount of /G support to 1.9.x and 2.0.0, once (if) the functionality is added to trunk, however I also think it is reasonable to expect developers to either
(1) target 2.0.1 by using language features only supported by 2.0.1, or
(2) target <=2.0.0 and 2.0.1 by only using language features that haven't changed, or
(3) go to lengths to explicitly polyfill the older language versions.

Similar things happened with parser changes from 1.8 to 1.9 when adding the new (({{a: 1}})) hash syntax (which makes new code not work in old ruby) and removing the (({if x: y; end})) syntax (which makes old code not work in new ruby). At least adding a new pattern modifier doesn't break old code. In fact, it doesn't change the behaviour of the old code at all.

Also note that your "ugly and slow" code won't work because the parser still attempts (and fails) to parse (({/foo/G})) even if the condition is false.

To make your code fully backwards-compatible you'd probably use something like:

ISFOO = Regexp.new('foo', defined?(Regexp::SKIPGLOBALS) ? Regexp::SKIPGLOBALS : 0)
def is
foo? val
val =~ IS_FOO
end

I also note you've changed /S to /G in your examples.
=end

#9 Updated by Sam Saffron about 1 year ago

sorry, I really did not mean to say the language should ship a crazy _G macro it was just a simple polyfill in the app. even with the polyfill it is way verbose.

#app code not ruby
def G(re)
Regexp.new(re.to
s, re.options | defined?(Regexp::SKIPGLOBALS) ? Regexp::SKIPGLOBALS : 0)
end

#10 Updated by Matthew Kerwin about 1 year ago

=begin
sam.saffron (Sam Saffron) wrote:

sorry, I really did not mean to say the language should ship a crazy _G
macro it was just a simple polyfill in the app. even with the polyfill
it is way verbose.

#app code not ruby
def G(re)
Regexp.new(re.to
s, re.options | defined?(Regexp::SKIPGLOBALS) ?
Regexp::SKIP
GLOBALS : 0)
end

Why not just do the following (evil, wicked, untenable) hack?

class Regexp; SKIPGLOBALS = 0 unless defined? SKIPGLOBALS; end

You still can't use (({//})) or (({%r()})) literals, but it means you can use
Regexp.new('foo|bar(baz)?', Regexp::SKIP_GLOBALS)
without fear.
=end

#11 Updated by Charles Nutter about 1 year ago

Crazy idea: what if in the future you needed to set $~ to nil in order for it to be settable by downstream calls (e.g. regexp match)? It would eliminate a great deal of magic and treat those calls the same way we treat closures: if the variable has not been instantiated outside the nested scope/call, it's not available to be set.

e.g.

def foo(regexp)
  "Hello, world?" =~ regexp # does not set backtrace
  $~ = nil
  "Hello, world?" =~ regexp # does set backtrace, same as a closure setting a non-local variable.
end

Is this unreasonable?

#12 Updated by Yui NARUSE about 1 year ago

headius (Charles Nutter) wrote:

Crazy idea: what if in the future you needed to set $~ to nil in order for it to be settable by downstream calls (e.g. regexp match)? It would eliminate a great deal of magic and treat those calls the same way we treat closures: if the variable has not been instantiated outside the nested scope/call, it's not available to be set.

e.g.

def foo(regexp)
  "Hello, world?" =~ regexp # does not set backtrace
  $~ = nil
  "Hello, world?" =~ regexp # does set backtrace, same as a closure setting a non-local variable.
end

Is this unreasonable?

It breaks compatibility like following code:
def foo
regexp =~ "foo"
p $&
end

#13 Updated by Yui NARUSE about 1 year ago

You may misunderstand, unlike Perl, Ruby's setting global variable cost is small.
Ruby only set a MatchData object to its scope.
$~ (Regexp.lastmatch) gets it.
The implementation of $& (Regexp.last
match[0]), $` (Regexp.lastmatch.prematch), and $' (Regexp.lastmatch.postmatch)
are get $~ and call [0], prematch, or postmatch.
So setting cost is very small (0.2 second for 1,000,000 times).

And if it doesn't set global variable, it means that it can't recycle previous MatchData object.
So it allocates new MatchData object each time, it costs both allocation and GC.
On following case, its cost is beyond the setting cost.
r = Regexp.new(foo, Regexp::SKIP_GLOBALS); 1000000.times{r=~"foo"}

Therefore if you want speed up, you must remove making MatchData object.
String#match won't speed up so much because its API need creating MatchData object.
(moreover its current implementation uses $~)

#14 Updated by Sam Saffron about 1 year ago

@naruse @charlie should this be moved to ruby common?

#15 Updated by Charles Nutter about 1 year ago

naruse (Yui NARUSE) wrote:

You may misunderstand, unlike Perl, Ruby's setting global variable cost is small.
Ruby only set a MatchData object to its scope.

That ignores the fact that without $~, the scope wouldn't need to be allocated either. In JRuby, when we know there's no $~ use, we allocate no scope; JVM can then inline our methods and avoid all allocation, putting locals in registers and speeding things up tremendously.

As an example of how much it helps... MRI 2.0.0 was changed to not allocate frames for core class methods, a change we made a couple years ago for JRuby 1.6. This had a massive impact on performance. If MRI could do this for Ruby methods as well, it would improve things further, but $~ and its implicit nature prevent that from being feasible right now.

$~ (Regexp.lastmatch) gets it.
The implementation of $& (Regexp.last
match[0]), $` (Regexp.lastmatch.prematch), and $' (Regexp.lastmatch.postmatch)
are get $~ and call [0], prematch, or postmatch.
So setting cost is very small (0.2 second for 1,000,000 times).

The scope cost is the hidden cost.

And if it doesn't set global variable, it means that it can't recycle previous MatchData object.
So it allocates new MatchData object each time, it costs both allocation and GC.

There are other ways to reduce the cost of allocating MatchData. In the end the MatchData object isn't as big as the matcher structures from the regexp engine anyway, right?

On following case, its cost is beyond the setting cost.
r = Regexp.new(foo, Regexp::SKIP_GLOBALS); 1000000.times{r=~"foo"}

The cost here is as much the closure binding as it is the setting of $~. If =~ did not set $~, no binding at all would be required for the closure and it would boil down to just the cost of calling =~ and creating the literal string.

Therefore if you want speed up, you must remove making MatchData object.
String#match won't speed up so much because its API need creating MatchData object.
(moreover its current implementation uses $~)

String#match would be known to not need $~, and implementations could avoid allocating the memory used to store $~ (not the MatchData but the method scope).

I will grant that since MRI does not have a JIT compiler, you need artificial scopes/frames anyway, but for implementations with optimizing JITs (JRuby, Rubinius) $~ is one of the biggest barriers to optimization.

#16 Updated by Charles Nutter about 1 year ago

sam.saffron (Sam Saffron) wrote:

@naruse @charlie should this be moved to ruby common?

If CommonRuby officially becomes the host project for features...yes, it should. I'm not sure we've had a final decision yet.

#17 Updated by Yui NARUSE 12 months ago

headius (Charles Nutter) wrote:

naruse (Yui NARUSE) wrote:

You may misunderstand, unlike Perl, Ruby's setting global variable cost is small.
Ruby only set a MatchData object to its scope.

That ignores the fact that without $~, the scope wouldn't need to be allocated either. In JRuby, when we know there's no $~ use, we allocate no scope; JVM can then inline our methods and avoid all allocation, putting locals in registers and speeding things up tremendously.

On such case, Regexp::SKIP_GLOBALS is useless.
Without it JRuby would optimize to skip globals.

As an example of how much it helps... MRI 2.0.0 was changed to not allocate frames for core class methods, a change we made a couple years ago for JRuby 1.6. This had a massive impact on performance. If MRI could do this for Ruby methods as well, it would improve things further, but $~ and its implicit nature prevent that from being feasible right now.

If so, ko1 should implement some way to treat $~ without frames.

$~ (Regexp.lastmatch) gets it.
The implementation of $& (Regexp.last
match[0]), $` (Regexp.lastmatch.prematch), and $' (Regexp.lastmatch.postmatch)
are get $~ and call [0], prematch, or postmatch.
So setting cost is very small (0.2 second for 1,000,000 times).

The scope cost is the hidden cost.

This doesn't include the cost to create a new scope because they are the same scope on $~ context.

And if it doesn't set global variable, it means that it can't recycle previous MatchData object.
So it allocates new MatchData object each time, it costs both allocation and GC.

There are other ways to reduce the cost of allocating MatchData. In the end the MatchData object isn't as big as the matcher structures from the regexp engine anyway, right?

My String#include?(regexp) patch in Feature #8206 is an example.

On following case, its cost is beyond the setting cost.
r = Regexp.new(foo, Regexp::SKIP_GLOBALS); 1000000.times{r=~"foo"}

The cost here is as much the closure binding as it is the setting of $~. If =~ did not set $~, no binding at all would be required for the closure and it would boil down to just the cost of calling =~ and creating the literal string.

block doesn't make scope for $~.
And the time I show is compared between original and simply commented out rbbackrefset version.

Therefore if you want speed up, you must remove making MatchData object.
String#match won't speed up so much because its API need creating MatchData object.
(moreover its current implementation uses $~)

String#match would be known to not need $~, and implementations could avoid allocating the memory used to store $~ (not the MatchData but the method scope).

Setting $~ itself doesn't cause memory allocation because it is only setting the same object to VM.

I will grant that since MRI does not have a JIT compiler, you need artificial scopes/frames anyway, but for implementations with optimizing JITs (JRuby, Rubinius) $~ is one of the biggest barriers to optimization.

I know $~ prevents optimization because I used Perl before.
But it doesn't relate to Regexp::SKIPGLOBALS because $~ is still here even if you usually use Regexp::SKIPGLOBALS.

Also available in: Atom PDF