Project

General

Profile

Actions
Copy link

Feature #12831

closed

/\X/ (extended grapheme cluster) can't pass unicode.org's GraphemeBreakTest

Added by mtsmfm (Fumiaki Matsushima) over 7 years ago. Updated over 7 years ago.

Status:
Closed
Assignee:
naruse (Yui NARUSE)
Target version:
-
[ruby-core:77586]

Description

I'm trying to replace Rails's grapheme implementation (http://api.rubyonrails.org/classes/ActiveSupport/Multibyte/Unicode.html#method-i-unpack_graphemes) with Ruby's extended grapheme cluster (/X/).
https://github.com/rails/rails/issues/26743

I noticed that Ruby's grapheme cluster can't pass unicode.org's GraphemeBreakTest.

Following test script will fail on Ruby 2.2/2.3

require 'rubygems'
require 'open-uri'
require 'test/unit'

UNICODE_VERSION =
  if Gem::Version.new(RUBY_VERSION) >= Gem::Version.new("2.3.0")
    "8.0.0"
  else
    "7.0.0"
  end

class TestGrapheme < Test::Unit::TestCase
  # https://github.com/rails/rails/blob/v5.0.0.1/activesupport/test/multibyte_grapheme_break_conformance_test.rb#L37
  def test_breaks
    each_line_of_break_tests do |*cols|
      *clusters, comment = *cols
      string = clusters.map {|c| c.pack("U*") }.join
      assert_equal clusters, string.scan(/\X/).map(&:codepoints), comment
    end
  end

  def each_line_of_break_tests(&block)
    lines = 0
    max_test_lines = 0 # Don't limit below 21, because that's the header of the testfile
    URI.parse("http://www.unicode.org/Public/#{UNICODE_VERSION}/ucd/auxiliary/GraphemeBreakTest.txt").open do |f|
      until f.eof? || (max_test_lines > 21 && lines > max_test_lines)
        lines += 1
        line = f.gets.chomp!
        next if line.empty? || line.start_with?("#")

        cols, comment = line.split("#")
        # Cluster breaks are represented by ÷
        clusters = cols.split("÷").map { |e| e.strip }.reject { |e| e.empty? }
        clusters = clusters.map do |cluster|
          # Codepoints within each cluster are separated by ×
          codepoints = cluster.split("×").map { |e| e.strip }.reject { |e| e.empty? }
          # codepoints are in hex in the test suite, pack wants them as integers
          codepoints.map { |codepoint| codepoint.to_i(16) }
        end

        # The tests contain a solitary U+D800 <Non Private Use High
        # Surrogate, First> character, which Ruby does not allow to stand
        # alone in a UTF-8 string. So we'll just skip it.
        next if clusters.flatten.include?(0xd800)

        clusters << comment.strip

        yield(*clusters)
      end
    end
  end
end

https://gist.github.com/mtsmfm/38f46882c3d4ccde35c269594fc24ebc

I found an issue on Onigmo (https://github.com/k-takata/Onigmo/issues/46)
but I couldn't on bugs.ruby-lang.org so I created this ticket.

I'm unfamiliar with grapheme so please tell me if I get something wrong.

Actions
Copy link

Also available in: Atom PDF

Like0
Like0Like0Like0