Project

General

Profile

Bug #15337

String#each_grapheme_cluster wrongly splits "\r\n"

Added by duerst (Martin Dürst) 7 months ago. Updated 7 months ago.

Status:
Closed
Priority:
Normal
Target version:
ruby -v:
ruby 2.6.0dev (2018-11-24 trunk 65950) [x86_64-linux]
[ruby-core:90031]

Description

The method each_grapheme_cluster on the String class wrongly splits CRLF into two clusters, as follows:

[duerst@stazersee ruby2]$ ./ruby -e 'puts "\r\n".each_grapheme_cluster.to_a.inspect'
["\r", "\n"]
[duerst@stazersee ruby2]$ ./ruby -v
ruby 2.6.0dev (2018-11-24 trunk 65950) [x86_64-linux]

The expected result in the above case is:

["\r\n"]

Except for the test cases that contain (unpaired) surrogates, which don't apply for us, this is the only test case that fails when testing with the data at http://www.unicode.org/Public/10.0.0/ucd/auxiliary/GraphemeBreakTest.txt. I have a test script using all that data that I plan to commit as soon as this bug is dealt with.

I would like to make sure that this is correct for Unicode 10.0.0 before moving to Unicode 11.0.0. In both specifications, the CRLF case is listed explicitly first in the specification.

I will try to find out how to fix this by myself, but would definitely appreciate help.


Related issues

Blocks Ruby trunk - Feature #14802: Update Unicode data to Unicode Version 11.0.0ClosedActions

Associated revisions

Revision 7850586a
Added by naruse (Yui NARUSE) 7 months ago

Don't use single byte optimization on grapheme clusters

Unicode Text Segmentation considers CRLF as a character. [Bug #15337]

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65954 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 65954
Added by naruse (Yui NARUSE) 7 months ago

Don't use single byte optimization on grapheme clusters

Unicode Text Segmentation considers CRLF as a character. [Bug #15337]

Revision 65954
Added by naruse (Yui NARUSE) 7 months ago

Don't use single byte optimization on grapheme clusters

Unicode Text Segmentation considers CRLF as a character. [Bug #15337]

Revision 7599b3f6
Added by duerst (Martin Dürst) 7 months ago

add tests using Unicode test data for grapheme clusters

Add file test/ruby/enc/test_grapheme_breaks.rb to test String#each_grapheme_cluster
and \X extended grapheme cluster matcher in regular expressions against test data
provided by Unicode (ucd/auxiliary/GraphemeBreakTest.txt).

Some lines in the data file are ignored, as follows:

  • Lines with a surrogate, because Ruby doesn't handle these
  • The case of "\r\n", because there is a bug (#15337) in the implementation

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65955 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 65955
Added by duerst (Martin Dürst) 7 months ago

add tests using Unicode test data for grapheme clusters

Add file test/ruby/enc/test_grapheme_breaks.rb to test String#each_grapheme_cluster
and \X extended grapheme cluster matcher in regular expressions against test data
provided by Unicode (ucd/auxiliary/GraphemeBreakTest.txt).

Some lines in the data file are ignored, as follows:

  • Lines with a surrogate, because Ruby doesn't handle these
  • The case of "\r\n", because there is a bug (#15337) in the implementation

Revision 65955
Added by duerst (Martin Dürst) 7 months ago

add tests using Unicode test data for grapheme clusters

Add file test/ruby/enc/test_grapheme_breaks.rb to test String#each_grapheme_cluster
and \X extended grapheme cluster matcher in regular expressions against test data
provided by Unicode (ucd/auxiliary/GraphemeBreakTest.txt).

Some lines in the data file are ignored, as follows:

  • Lines with a surrogate, because Ruby doesn't handle these
  • The case of "\r\n", because there is a bug (#15337) in the implementation

Revision 33afc9e0
Added by duerst (Martin Dürst) 7 months ago

remove guard against bug #15337, because it is fixed

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@65958 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 65958
Added by duerst (Martin Dürst) 7 months ago

remove guard against bug #15337, because it is fixed

Revision 65958
Added by duerst (Martin Dürst) 7 months ago

remove guard against bug #15337, because it is fixed

Revision 29eae8b1
Added by nagachika (Tomoyuki Chikanaga) 7 months ago

merge revision(s) 65954,65955,65958: [Backport #15337]

    Don't use single byte optimization on grapheme clusters

    Unicode Text Segmentation considers CRLF as a character. [Bug #15337]

    add tests using Unicode test data for grapheme clusters

    Add file test/ruby/enc/test_grapheme_breaks.rb to test String#each_grapheme_cluster
    and \X extended grapheme cluster matcher in regular expressions against test data
    provided by Unicode (ucd/auxiliary/GraphemeBreakTest.txt).

    Some lines in the data file are ignored, as follows:
    - Lines with a surrogate, because Ruby doesn't handle these
    - The case of "\r\n", because there is a bug (#15337) in the implementation

    remove guard against bug #15337, because it is fixed

git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/branches/ruby_2_5@66073 b2dd03c8-39d4-4d8f-98ff-823fe69b080e

Revision 66073
Added by nagachika (Tomoyuki Chikanaga) 7 months ago

merge revision(s) 65954,65955,65958: [Backport #15337]

Don't use single byte optimization on grapheme clusters

Unicode Text Segmentation considers CRLF as a character. [Bug #15337]

add tests using Unicode test data for grapheme clusters

Add file test/ruby/enc/test_grapheme_breaks.rb to test String#each_grapheme_cluster
and \X extended grapheme cluster matcher in regular expressions against test data
provided by Unicode (ucd/auxiliary/GraphemeBreakTest.txt).

Some lines in the data file are ignored, as follows:
- Lines with a surrogate, because Ruby doesn't handle these
- The case of "\r\n", because there is a bug (#15337) in the implementation

remove guard against bug #15337, because it is fixed

History

#1

Updated by duerst (Martin Dürst) 7 months ago

  • Blocks Feature #14802: Update Unicode data to Unicode Version 11.0.0 added
#2

Updated by naruse (Yui NARUSE) 7 months ago

  • Status changed from Open to Closed

Applied in changeset trunk|r65954.


Don't use single byte optimization on grapheme clusters

Unicode Text Segmentation considers CRLF as a character. [Bug #15337]

Updated by duerst (Martin Dürst) 7 months ago

Once we have a fix for this bug, it should be backported to all the versions that implement \X and that we still support.

#4

Updated by naruse (Yui NARUSE) 7 months ago

  • Backport changed from 2.3: UNKNOWN, 2.4: UNKNOWN, 2.5: UNKNOWN to 2.3: DONTNEED, 2.4: REQUIRED, 2.5: REQUIRED

Updated by nagachika (Tomoyuki Chikanaga) 7 months ago

  • Backport changed from 2.3: DONTNEED, 2.4: REQUIRED, 2.5: REQUIRED to 2.3: DONTNEED, 2.4: REQUIRED, 2.5: DONE

ruby_2_5 r66073 merged revision(s) 65954,65955,65958.

Also available in: Atom PDF