https://bugs.ruby-lang.org/https://bugs.ruby-lang.org/favicon.ico?17113305112011-11-04T19:15:48ZRuby Issue Tracking SystemBackport193 - Backport #5564: 1.9.3 regression with encoding conversionhttps://bugs.ruby-lang.org/issues/5564?journal_id=218852011-11-04T19:15:48Zduerst (Martin Dürst)duerst@it.aoyama.ac.jp
<ul></ul><p>I can't explain why there's a bug, the three lines of code shouldn't produce one.</p>
<p>But if the code in question is really<br>
s = "hello \u{fc}mlat"<br>
s.force_encoding "ISO-8859-1"<br>
s.encode! 'UTF-8'<br>
then I suggest you either explain why you'd want to do that, or fix it.<br>
"\u{fc}mlaut" is an escape for "ümlaut", and the \u escape makes sure the string is encoded in UTF-8.<br>
Using force-encoding to label it as ISO-8859-1 then mislabels it.<br>
The last line is then converting it to UTF-8 again, resulting is what's called double-encoding.<br>
What you may have wanted is "hello \x{fc}mlaut"; then the next two lines would make much more sense.</p> Backport193 - Backport #5564: 1.9.3 regression with encoding conversionhttps://bugs.ruby-lang.org/issues/5564?journal_id=218862011-11-04T19:16:11Znaruse (Yui NARUSE)naruse@airemix.jp
<ul><li><strong>Tracker</strong> changed from <i>Bug</i> to <i>Backport</i></li><li><strong>Project</strong> changed from <i>Ruby master</i> to <i>Backport193</i></li><li><strong>Status</strong> changed from <i>Open</i> to <i>Assigned</i></li><li><strong>Assignee</strong> set to <i>yugui (Yuki Sonoda)</i></li></ul><p>This is the same problem as Bug <a class="issue tracker-1 status-5 priority-4 priority-default closed" title="Bug: $SAFEが3以上の時にString#encodeがSecurityErrorを発生させるケースがある (Closed)" href="https://bugs.ruby-lang.org/issues/5279">#5279</a> and fixed in r33328.<br>
"it can't load encoding library on $SAFE >= 3"</p>
<p>The workaround is just as you say load the library before set $SAFE.</p> Backport193 - Backport #5564: 1.9.3 regression with encoding conversionhttps://bugs.ruby-lang.org/issues/5564?journal_id=218882011-11-04T22:43:47Zjonleighton (Jon Leighton)j@jonathanleighton.com
<ul></ul><p>Hello,</p>
<p>Thanks Martin and Yui for your responses.</p>
<p>Martin:</p>
<p>I'm sorry, I made a mistake in my 'simplified' example. It should be the following:</p>
<p>s = "hello \xFCmlat"<br>
s.force_encoding "ISO-8859-1"<br>
s.encode! 'UTF-8'</p>
<p>In other words: taking a valid ISO-8859-1 string and converting it to UTF-8.</p>
<p>Yui:</p>
<p>I am not sure this is the same bug, because we are not setting $SAFE in Rails. So the $SAFE level is 0 throughout. Am I missing something?</p>
<p>Many thanks,<br>
Jon</p> Backport193 - Backport #5564: 1.9.3 regression with encoding conversionhttps://bugs.ruby-lang.org/issues/5564?journal_id=218892011-11-04T23:33:08Zjonleighton (Jon Leighton)j@jonathanleighton.com
<ul></ul><p>I have just tried applying r33328 to the ruby_1_9_3 branch, and after doing so I still experience this problem. However, I may have done it incorrectly (I don't really know what I am doing).</p>
<p>Regarding the work-around, it is problematic because we don't know until runtime what conversions we want to perform (so just doing "Encoding::Converter.new('ISO-8859-1', 'UTF-8')" won't help if we need to convert Shift_JIS to UTF-8 at runtime, for example). Is there a way around that problem?</p>
<p>Thank you.</p> Backport193 - Backport #5564: 1.9.3 regression with encoding conversionhttps://bugs.ruby-lang.org/issues/5564?journal_id=219002011-11-06T01:37:06Zjonleighton (Jon Leighton)j@jonathanleighton.com
<ul></ul><p>I have just tried to build trunk and still experience this bug:</p>
<p>$ ruby -v<br>
ruby 2.0.0dev (2011-11-06 trunk 33644) [x86_64-linux]<br>
$ ruby -Itest test/template/template_test.rb<br>
Run options:</p>
<a name="Running-tests"></a>
<h1 >Running tests:<a href="#Running-tests" class="wiki-anchor">¶</a></h1>
<p>.E..............</p>
<p>Finished tests in 0.052595s, 304.2143 tests/s, 399.2812 assertions/s.</p>
<ol>
<li>Error:<br>
test_default_external_works(TestERBTemplate):<br>
ActionView::Template::Error: code converter not found (ISO-8859-1 to UTF-8)<br>
/home/turnip/Code/rails/actionpack/lib/action_view/template/handlers/erb.rb:83:in <code>encode!' /home/turnip/Code/rails/actionpack/lib/action_view/template/handlers/erb.rb:83:in </code>call'<br>
/home/turnip/Code/rails/actionpack/lib/action_view/template.rb:252:in <code>compile' /home/turnip/Code/rails/actionpack/lib/action_view/template.rb:189:in </code>compile!'<br>
/home/turnip/Code/rails/actionpack/lib/action_view/template.rb:142:in <code>block in render' /home/turnip/Code/rails/activesupport/lib/active_support/notifications.rb:55:in </code>instrument'<br>
/home/turnip/Code/rails/actionpack/lib/action_view/template.rb:141:in <code>render' test/template/template_test.rb:53:in </code>render'<br>
test/template/template_test.rb:135:in <code>block in test_default_external_works' test/template/template_test.rb:174:in </code>with_external_encoding'<br>
test/template/template_test.rb:133:in <code>test_default_external_works' /usr/local/lib/ruby/gems/1.9.1/gems/mocha-0.10.0/lib/mocha/integration/mini_test/version_230_to_251.rb:28:in </code>run'</li>
</ol>
<p>16 tests, 21 assertions, 0 failures, 1 errors, 0 skips</p>
<p>ruby -v: ruby 2.0.0dev (2011-11-06 trunk 33644) [x86_64-linux]</p>
<p>So I do not think it's just a case of backporting an existing fix.</p> Backport193 - Backport #5564: 1.9.3 regression with encoding conversionhttps://bugs.ruby-lang.org/issues/5564?journal_id=219102011-11-06T16:36:35Zduerst (Martin Dürst)duerst@it.aoyama.ac.jp
<ul></ul><p>Hello Jon,</p>
<p>What I would do now is to try and get a C-level backtrace that shows what happens exactly between the call to encode! and the actual errro.</p>
<p>As for "we don't know what converter we might need at runtime", one solution would be to include all converters. When we designed the whole transcoding stuff, we were careful to save runtime memory by dynamically loading just the stuff that's really needed, but there may be some cases where it's easier to load everything in advance. On a clever OS, even if you run many instances of your application, all the data for the transcodings should be shared because it's all static. (I'm not saying that's the solution to your bug, but it's something that may come in handy in some situations.)</p>
<p>Another solution might be to introduce a way to 'bless' certain libraries in advance so that they can be loaded later under $SAFE >= 3 if they are needed. That would leave a security risk of a library being changed between being blessed and being loaded, but there should be enough ways to prevent that.</p> Backport193 - Backport #5564: 1.9.3 regression with encoding conversionhttps://bugs.ruby-lang.org/issues/5564?journal_id=219182011-11-06T18:43:44Zjonleighton (Jon Leighton)j@jonathanleighton.com
<ul></ul><p>Hi Martin,</p>
<p>Thanks for your reply. I don't know how to get a C-level backtrace. Where can I find documentation about how to do this?</p>
<p>Regarding $SAFE level - just to reiterate, I don't think that's the issue here, because in this code $SAFE == 0 throughout.</p>
<p>Thanks</p> Backport193 - Backport #5564: 1.9.3 regression with encoding conversionhttps://bugs.ruby-lang.org/issues/5564?journal_id=219232011-11-06T20:05:03Zduerst (Martin Dürst)duerst@it.aoyama.ac.jp
<ul></ul><p>To get a C-level backtrace:</p>
<ol>
<li>Make sure you compiled with -g</li>
<li>Run from gdb, or attach gdb to the process once it's running</li>
<li>Set a breakpoint just before the code that produces the exception (message)</li>
<li>Run. gdb will stop at the breakpoint</li>
<li>Get a backtrace from gdb.</li>
</ol>
<p>It's not that I do this every day, but the above should work.</p> Backport193 - Backport #5564: 1.9.3 regression with encoding conversionhttps://bugs.ruby-lang.org/issues/5564?journal_id=219262011-11-06T21:26:54Zjonleighton (Jon Leighton)j@jonathanleighton.com
<ul></ul><p>Ok, here you go: <a href="https://gist.github.com/1342809" class="external">https://gist.github.com/1342809</a></p>
<p>Is that what you need?</p>
<p>Thanks</p> Backport193 - Backport #5564: 1.9.3 regression with encoding conversionhttps://bugs.ruby-lang.org/issues/5564?journal_id=219582011-11-07T16:39:16Zduerst (Martin Dürst)duerst@it.aoyama.ac.jp
<ul></ul><p><a href="https://gist.github.com/1342809" class="external">https://gist.github.com/1342809</a> is indeed what I was asking for, but it doesn't seem to be enough.</p>
<p>Rather different question: Is this a problem just with your installation, or can this problem be replicated widely (different OSs,...)?</p> Backport193 - Backport #5564: 1.9.3 regression with encoding conversionhttps://bugs.ruby-lang.org/issues/5564?journal_id=219622011-11-07T19:29:20Zjonleighton (Jon Leighton)j@jonathanleighton.com
<ul></ul><p>My installation is Fedora 15. We have also reproduced on the Rails CI server (Ubuntu Lucid), and I just asked some people to test on a Mac, and they also can reproduce. So I do not think it's platform specific.</p>
<p>You should be able to reproduce on your own computer by setting up the Rails test suite. (This is a bit of a pain I know, I'm sorry I haven't found a simpler test case.)</p>
<ol>
<li>Clone git://github.com/rails/rails.git</li>
<li>bundle install</li>
<li>cd actionpack/</li>
<li>Open lib/action_view/template.rb and comment the conditional block starting with if defined?(RUBY_ENGINE) (this disables the workaround for the bug) [only necessary if you are running exactly 1.9.3p0, but this bug can be repro'd on trunk also]</li>
<li>ruby -Itest test/template/template_test.rb -n test_default_external_works</li>
</ol> Backport193 - Backport #5564: 1.9.3 regression with encoding conversionhttps://bugs.ruby-lang.org/issues/5564?journal_id=236112012-02-08T09:40:46Znaruse (Yui NARUSE)naruse@airemix.jp
<ul><li><strong>Status</strong> changed from <i>Assigned</i> to <i>Closed</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>This issue was solved with changeset r34465.<br>
Jon, thank you for reporting this issue.<br>
Your contribution to Ruby is greatly appreciated.<br>
May Ruby be with you.</p>
<hr>
<p>merge revision(s) 33201,33249,33328: [Backport <a class="issue tracker-4 status-5 priority-4 priority-default closed" title="Backport: 1.9.3 regression with encoding conversion (Closed)" href="https://bugs.ruby-lang.org/issues/5564">#5564</a>]</p>
<pre><code>* encoding.c (load_encoding): predefined encoding names are safe.
<a href="/issues/5279">[ruby-dev:44469]</a> [Bug #5279]
* transcode.c (load_transcoder_entry): ditto.
* encoding.c (require_enc): reject only loading from untrusted
load paths. <a href="/issues/5279">[ruby-dev:44541]</a> [Bug #5279]
* transcode.c (load_transcoder_entry): ditto.
</code></pre>