Bug #21503: \p{Word} does not match on \p{Join_Control} while docs say it does

• Status: Closed
• Assignee: -
• Target version: -
• ruby -v: 3.4.4
• Backport: 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: DONE

[procmarco (Marco Concetto Rudilosso)]

in the docs it is mentioned that \p{Word} matches the equivalent of: [\p{M}\p{Nd}\p{Pc}\p{Alpha}\p{Join_Control}] as it's also defined in the unicode spec

the issue is that it does not seem to be the case

irb(main):018> REGEX = /\p{Word}/u
=> /\p{Word}/
irb(main):019> "\u200D".gsub(REGEX, "-")
=> "‍"
irb(main):020> REGEX2 = /\p{Join_Control}/u
=> /\p{Join_Control}/
irb(main):021> "\u200D".gsub(REGEX2, "-")
=> "-"

There's 2 solutions here, either we change the docs or the code.

[procmarco (Marco Concetto Rudilosso)]

What I mean is that the current implementation of \p{Word} does not seem to match \p{Join_Control} even though it should and it also says so in the docs

[mame (Yusuke Endoh)]

• Related to Bug #19417: Regexp \p{Word} and [[:word:]] do not match Unicode Other_Number character added

[mame (Yusuke Endoh)]

There is already a PR for that: https://github.com/ruby/ruby/pull/7711

Can you take a look? @duerst (Martin Dürst) @naruse (Yui NARUSE)

[naruse (Yui NARUSE)]

It looks \p{Word} is updated in TR#18 Version 15.
https://www.unicode.org/reports/tr18/tr18-15.html
The fix looks good.

[hsbt (Hiroshi SHIBATA)]

• Status changed from Open to Closed
• Backport changed from 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: UNKNOWN to 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: REQUIRED

https://github.com/ruby/ruby/pull/7711 has been merged

[k0kubun (Takashi Kokubun)]

The patch to master modified CR_Word on enc/unicode/16.0.0/name2ctype.h, but Ruby 3.4 uses enc/unicode/15.0.0/name2ctype.h that has a different content in CR_Word. I'm not sure how to backport this properly. Could @procmarco or anybody else have a look at making a backport PR to ruby_3_4 branch on GitHub?

[naruse (Yui NARUSE)]

@k0kubun (Takashi Kokubun) name2ctype.h is generated by tool/enc-unicode.rb. You can run it on ruby_3_4.

[alanwu (Alan Wu)]

• Backport changed from 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: REQUIRED to 3.2: UNKNOWN, 3.3: UNKNOWN, 3.4: DONE

Backport for 3.4 done in 5a42d267bfabc86f86cae2e83de24b1b86bc316a