Bug #6266

encoding related exception with recent integrated psych

Added by Jon Forums about 2 years ago. Updated over 1 year ago.

[ruby-core:44163]
Status:Closed
Priority:Normal
Assignee:Aaron Patterson
Category:ext
Target version:1.9.3
ruby -v:ruby 1.9.3p176 (2012-04-04 revision 35238) [i386-mingw32] Backport:

Description

Newer versions of Psych integrated into both ruby19_3 and trunk create exceptions if the yaml file isn't perceived to be correctly encoded.

While I discovered this in my code that uses the 'logging' gem on Win7 32bit, this test snippet shows the same failure on both ruby19_3@r35238 and trunk@r35229:

# hello.yml uses LF-only line endings
C:\temp>type hello.yml
:greeting: hello yaml

# yamltest.rb uses LF-only
C:\temp>type yaml
test.rb
require 'yaml'
puts "YAML engine: #{YAML::ENGINE.yamler} #{'v'+Psych::VERSION if defined?(Psych)}"

# fix: 'rb'
# fix: 'r:utf-8'
# fix: Encoding.defaultexternal = Encoding::UTF8 if defined?(Encoding)
# fix: chcp 65001 in cmd.exe before running script
# fix: run script with ruby -Eutf-8
File.open('hello.yml', 'r') do |f|
puts YAML.load_stream(f)
end

# IBM437 codepage is a typical default for the cmd.exe shell
C:\temp>chcp
Active code page: 437

C:\temp>ruby -v yamltest.rb
ruby 1.9.3p176 (2012-04-04 revision 35238) [i386-mingw32]
YAML engine: psych v1.3.1
C:/ruby193/lib/ruby/1.9.1/psych.rb:206:in parse': YAML file must be UTF-8, UTF-16LE, or UTF-16BE, not IBM437 (ArgumentError)
from C:/ruby193/lib/ruby/1.9.1/psych.rb:206:in
parse
stream'
from C:/ruby193/lib/ruby/1.9.1/psych.rb:289:in load_stream'
from yaml_test.rb:10:in
block in '
from yaml_test.rb:9:in open'
from yaml_test.rb:9:in
'

# older versions of integrated psych do not cause exceptions
C:\temp>chcp
Active code page: 437

C:\temp>ruby -v yaml_test.rb
tcs-ruby 1.9.3p134 (2012-02-19, TCS patched 2012-02-25) [i386-mingw32]
YAML engine: psych v1.2.2
{:greeting=>"hello yaml"}

I've not tried on Arch with bash.

From the test_yaml.rb comments you can see some of the workarounds. None of these are good options because they require changes to either a users code/yaml, gem code that uses the psych library, or the ruby invocation command line.

For both trunk and ruby19_3, I'd prefer psych implement a fallback strategy that Just Works combined with an informative warning rather than a hard exception.

noname (500 Bytes) Anonymous, 04/07/2012 10:29 AM

History

#1 Updated by Jon Forums about 2 years ago

On Arch (3.2.14-1-ARCH) the sample also exceptions when I change my locale to enUS.iso88591 from enUS.utf8.

But on trunk it doesn't matter anymore :)

$ ruby -v yaml_test.rb
ruby 2.0.0dev (2012-04-07 trunk 35247) [i686-linux]
YAML engine: psych v1.3.1
{:greeting=>"hello yaml"}

...and trunk looks good on Win7 32bit

C:\temp>chcp
Active code page: 437

C:\temp>ruby -v yaml_test.rb
ruby 2.0.0dev (2012-04-07 trunk 35247) [i386-mingw32]
YAML engine: psych v1.3.1
{:greeting=>"hello yaml"}

Speedy...thanks! I'll fetch ruby19_3 later and double check.

#2 Updated by Anonymous about 2 years ago

On Sat, Apr 07, 2012 at 06:31:25AM +0900, jonforums (Jon Forums) wrote:

Issue #6266 has been updated by jonforums (Jon Forums).

On Arch (3.2.14-1-ARCH) the sample also exceptions when I change my locale to enUS.iso88591 from enUS.utf8.

But on trunk it doesn't matter anymore :)

$ ruby -v yaml_test.rb
ruby 2.0.0dev (2012-04-07 trunk 35247) [i686-linux]
YAML engine: psych v1.3.1
{:greeting=>"hello yaml"}

...and trunk looks good on Win7 32bit

C:\temp>chcp
Active code page: 437

C:\temp>ruby -v yaml_test.rb
ruby 2.0.0dev (2012-04-07 trunk 35247) [i386-mingw32]
YAML engine: psych v1.3.1
{:greeting=>"hello yaml"}

Speedy...thanks! I'll fetch ruby19_3 later and double check.

I need to backport the fix to ruby19_3 first. :-)

I'll put together the patches for the 1.9.3 branch and open a ticket to
backport. Thanks for testing!

--
Aaron Patterson
http://tenderlovemaking.com/

#3 Updated by Yusuke Endoh about 2 years ago

  • Status changed from Open to Assigned

#4 Updated by Jon Forums about 2 years ago

r35245 fixed this issue, and r35251 backported to ruby19_3

#5 Updated by Yusuke Endoh about 2 years ago

  • Status changed from Assigned to Closed

Okay, closing. Thanks!

Yusuke Endoh mame@tsg.ne.jp

Also available in: Atom PDF