Bug #19196: The string saved to Tempfile from URI.open escapes "&" character

• Status: Rejected
• Assignee: -
• Target version: -
• Backport: 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN

[westoque (William Estoque)]

When I am reading the string response from a URI.open, the response is not equivalent to the response body.

How to reproduce:

url = "https://www.podcastone.com/podcast?categoryID2=1237"

handle = URI.open(url)
=> #<Tempfile:/path/to/tempfile>

puts handle.read
.... https://dts.podtrac.com/redirect.mp3/pdst.fm/e/chrt.fm/track/E2G895/aw.noxsolutions.com/launchpod/adswizz/1237/762-FeedbackFriday-249-V2_mzwq_b1dc1677.mp3?awCollectionId=1237&#38;awEpisodeId=ee01b21a-878d-4be4-974c-e504b1dc1677&#38;adwNewID3=true&#38;awNetwork=309...

In the browser, the actual string reads:

https://dts.podtrac.com/redirect.mp3/pdst.fm/e/chrt.fm/track/E2G895/aw.noxsolutions.com/launchpod/adswizz/1237/762-FeedbackFriday-249-V2_mzwq_b1dc1677.mp3?awCollectionId=1237&awEpisodeId=ee01b21a-878d-4be4-974c-e504b1dc1677&adwNewID3=true&awNetwork=309

Notice the characters #38;

My initial research is that it's because the Tempfile that gets created is in ascii-8bit, and in ascii-8bit, the amperstand is a "38".

I propose that we should have a way to force the encoding of the Tempfile to UTF8 so that this character is not escaped and the string encoding is preserved.

[westoque (William Estoque)]

• Subject changed from The string saved to Tempfile from URI.open escapes "&" characters to The string saved to Tempfile from URI.open escapes "&" character

[westoque (William Estoque)]

• Description updated (diff)

[westoque (William Estoque)]

• Description updated (diff)

[ufuk (Ufuk Kayserilioglu)]

The content you are reading is XML and & characters are there because of XML-escaping. They are not related to any kind of file encoding, ASCII-8BIT or UTF-8.

Moreover, they are there in the response from the server, which you can see by looking at the output of curl for the same resource:

$ curl -s "https://www.podcastone.com/podcast?categoryID2=1237" | grep "aw.noxsolutions.com/launchpod/adswizz/1237/762-"
...
<enclosure length="74614442" type="audio/mpeg" url="https://dts.podtrac.com/redirect.mp3/pdst.fm/e/chrt.fm/track/E2G895/aw.noxsolutions.com/launchpod/adswizz/1237/762-FeedbackFriday-249-V2_mzwq_b1dc1677.mp3?awCollectionId=1237&#38;awEpisodeId=ee01b21a-878d-4be4-974c-e504b1dc1677&#38;adwNewID3=true&#38;awNetwork=309"></enclosure>
...

So, this is not a Ruby problem at all. On the contrary, Ruby can help you unescape these characters:

require "cgi"
CGI.unescapeHTML("foo&bar") # => "foo&bar"

[Eregon (Benoit Daloze)]

• Status changed from Open to Rejected

[westoque (William Estoque)]

@ufuk (Ufuk Kayserilioglu) thank you for that explanation. I may have jumped to conclusions when checking that response in the browser (Chrome) vs curl which unescaped the characters.