Feature #19697: Resolv::DNS resolution for international domains fails with "Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT"

• Status: Open
• Assignee: -
• Target version: -

[clairity (claire c)]

i try to resolve a registered international domain, and instead of receiving DNS records, get an incompatible character encodings error:

# irb
3.2.2 :001 > require 'resolv'
3.2.2 :002 > res = Resolv::DNS.open { |dns|  dns.getresource '우편.닷컴', Resolv::DNS::Resource::IN::MX  }
~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:777:in `[]=': incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError)
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:777:in `sender'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:527:in `block in fetch_resource'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1125:in `block (3 levels) in resolv'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1123:in `each'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1123:in `block (2 levels) in resolv'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1122:in `each'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1122:in `block in resolv'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1120:in `each'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1120:in `resolv'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:521:in `fetch_resource'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:507:in `each_resource'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:488:in `getresource'
  from (irb):3:in `block in <top (required)>'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:298:in `open'
  from (irb):3:in `<main>'
  from ~/.rvm/gems/ruby-3.2.2/gems/irb-1.6.4/exe/irb:9:in `<top (required)>'
  ... 4 levels...

i also tried this command in 3.3.0-preview1 and 2.1.4, receiving the same error. the expected response is to get DNS records for 우편.닷컴 (or any registered international domain). dig 우편.닷컴 does indeed return a short but valid DNS record.

[byroot (Jean Boussier)]

While I think Ruby should offer a way to handle unicode domains, I'm not sure punycode should be handled at the Resolve layer though.

Either way, this looks more like a feature request to me.

[clairity (claire c)]

clairity (claire c) wrote:

i try to resolve a registered international domain, and instead of receiving DNS records, get an incompatible character encodings error:

# irb
3.2.2 :001 > require 'resolv'
3.2.2 :002 > res = Resolv::DNS.open { |dns|  dns.getresource '우편.닷컴', Resolv::DNS::Resource::IN::MX  }
~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:777:in `[]=': incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError)
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:777:in `sender'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:527:in `block in fetch_resource'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1125:in `block (3 levels) in resolv'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1123:in `each'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1123:in `block (2 levels) in resolv'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1122:in `each'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1122:in `block in resolv'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1120:in `each'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:1120:in `resolv'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:521:in `fetch_resource'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:507:in `each_resource'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:488:in `getresource'
  from (irb):3:in `block in <top (required)>'
  from ~/.rvm/rubies/ruby-3.2.2/lib/ruby/3.2.0/resolv.rb:298:in `open'
  from (irb):3:in `<main>'
  from ~/.rvm/gems/ruby-3.2.2/gems/irb-1.6.4/exe/irb:9:in `<top (required)>'
  ... 4 levels...

i also tried this command in 3.3.0-preview1 and 2.1.4, receiving the same error. the expected response is to get DNS records for 우편.닷컴 (or any registered international domain). dig 우편.닷컴 does indeed return a short but valid DNS record.

edit: i should add that unicode remaining unicode characters is strongly preferred over being translated into punycode, as the former is human- and machine-readable but the latter is only machine-readable (and as a result, can introduce social engineering vulnerabilities for instance).

[jeremyevans0 (Jeremy Evans)]

• Tracker changed from Bug to Feature
• ruby -v deleted (3.2.2)
• Backport deleted (3.0: UNKNOWN, 3.1: UNKNOWN, 3.2: UNKNOWN)

[chucke (Tiago Cardoso)]

I agree with byroot, this shouldn't be solved at the resolve layer.

FWIW I've touched on the potential feature request in the stdlib uri library which was not further discussed. Since then, I've released idnx, which uses libidn2 (or winnls) to encode IDNA2008 domains into punycode. I built it for httpx, the http library I'm the maintainer of.

The confusion may also come from the fact that getaddrinfo supports IDNA domain name resolution in macOS. However, that's not a DNS feature.

[byroot (Jean Boussier)]

FWIW I've touched on the potential feature request in the stdlib uri library which was not further discussed

That issue started as mentioning the removal of URI.escape, which probably prevented it from getting more attention.

I think there is potential for a feature request for IDN support in either URI or Resolv, but IMO a new clean feature request should be started, with a proposed API.

[chucke (Tiago Cardoso)]

byroot (Jean Boussier) wrote in #note-5:

FWIW I've touched on the potential feature request in the stdlib uri library which was not further discussed

That issue started as mentioning the removal of URI.escape, which probably prevented it from getting more attention.

I think there is potential for a feature request for IDN support in either URI or Resolv, but IMO a new clean feature request should be started, with a proposed API.

here: https://github.com/ruby/uri/issues/76