Feature #2542: URI lib should be updated to RFC 3986

• Status: Closed
• Assignee: naruse (Yui NARUSE)
• Target version: 2.6

[marcandre (Marc-Andre Lafortune)]

=begin
RFC 2396 has been obsolete for nearly 5 years now.

It was replaced by RFC 3986 which aims at clarifying aspects that were not previously clear.
=end

[marcandre (Marc-Andre Lafortune)]

• Subject changed from URI lib should be updated to RFC 39886 to URI lib should be updated to RFC 3986

[naruse (Yui NARUSE)]

FYI, RFC 3986 will be obsoleted.
http://tools.ietf.org/html/draft-duerst-iri-bis-07

[duerst (Martin Dürst)]

No, RFC 3986 (URI) will NOT be updated. RFC 3987 (IRI), in due time,
will be updated. See also
http://www.ietf.org/ibin/c5i?mid=6&rid=49&gid=0&k1=934&k2=7294&tid=1262671757.

Regards, Martin.

---

#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp mailto:[e-mail]

[naruse (Yui NARUSE)]

Ah, Martin is right, it's RFC 3987 and RFC 3986 will be still alive.

Anyway Bob Aman introduces an alternative library named Addressable.
http://addressable.rubyforge.org/
It looks good but some incompatibilities for current URI library.
I'll suggest to bundle Addressable and obsolete current URI lib,
but I have to plan its migration path.

[mame (Yusuke Endoh)]

• Target version changed from 1.9.2 to 2.0.0

Hi,

I'll suggest to bundle Addressable and obsolete current URI lib,
but I have to plan its migration path.

This ticket seems to need much work.
I guess we can't make the deadline of spec freezing.
So I change the target to 1.9.x.
If you want 1.9.2 to include the feature, please discuss right now.

---

Yusuke Endoh [e-mail]

[marcandre (Marc-Andre Lafortune)]

• Target version changed from 2.0.0 to 1.9.2

I feel the spec for 1.9.2 has been quite clear for 5 years ... follow RFC 3986!

Integrating some of the features of the addressable gem can be discussed later.

Do we have to wait for Akira Yamada, the official maintainer of this library?

[naruse (Yui NARUSE)]

I object to target 1.9.2.
Following RFC 3986 makes some incompatibilities.
It shouldn't be done without consideration.

[duerst (Martin Dürst)]

Hello Yui,

Is there a list of incompatibilities, or can you make one?

Regards, Martin.

---

#-# Martin J. Dürst, Professor, Aoyama Gakuin University
#-# http://www.sw.it.aoyama.ac.jp mailto:[e-mail]

[naruse (Yui NARUSE)]

2010/3/25 "Martin J. Dürst" [e-mail]:

Is there a list of incompatibilities, or can you make one?

Some structures of the syntax is changed in RFC 3986.
This breaks URI::REGEXP::PATTERN::TOPLABEL and some constants.

---

NARUSE, Yui
[e-mail]

[znz (Kazuhiro NISHIYAMA)]

• Status changed from Open to Assigned
• Target version changed from 1.9.2 to 2.0.0

[hedgehog (Hedge Hog)]

Rather than reinvent anything. Consider employing an FFI interface to uriparser:

http://uriparser.sourceforge.net/

not sure if there is a port for windows or if an equivalent windows lib is available?

[luislavena (Luis Lavena)]

On Wed, May 12, 2010 at 8:01 PM, Hedge Hog [e-mail] wrote:

Issue #2542 has been updated by Hedge Hog.

Rather than reinvent anything.  Consider employing an FFI interface to uriparser:

http://uriparser.sourceforge.net/

not sure if there is a port for windows or if an equivalent windows lib is available?

we will not only depend on uriparser C library but also will require
libcpptest to be able to configure and compile uriparser.

---

Luis Lavena
AREA 17

---

Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry

[philo (Philippe Lucas)]

It depends of libcpptest only for test so you can build the package with '--disable-test'.

[naruse (Yui NARUSE)]

• Assignee changed from akira (akira yamada) to naruse (Yui NARUSE)

I come to think uri lib should move to RFC 3986 even if it breaks some compatibility.
But I don't want that new implementation/spec will be also a white box like now.
So I think:

• keep current URI::REGEXP, URI::Parser and so on.
• at least URI.parse doesn't use URI::Parser but use new implementation.

How about this?

[naruse (Yui NARUSE)]

• Target version changed from 2.0.0 to 1.9.4

Just for information or memo.
http://tools.ietf.org/html/draft-ietf-iri-3987bis-05
http://tools.ietf.org/html/draft-weber-iri-guidelines-00

[nikosd (Nikos Dimitrakopoulos)]

Are there any plans for actually fixing this? Not sure I can help, and no troll appetite - just asking :)

[mame (Yusuke Endoh)]

• Target version changed from 1.9.4 to 2.6

Naruse-san, could you please answer to Nikos?

I'm setting to next minor, but if you are willing to do anything to 2.0.0, and if the impact is so small, I may accept.

--
Yusuke Endoh [e-mail]

[naruse (Yui NARUSE)]

Just an experimental implementation:
http://github.com/nurse/url

[naruse (Yui NARUSE)]

• Status changed from Assigned to Closed
• % Done changed from 0 to 100

Applied in changeset r46491.

---

support RFC3986 [Feature #2542]

• lib/uri/common.rb (URI::REGEXP): move to lib/uri/rfc2396_parser.rb.

• lib/uri/common.rb (URI::Parser): ditto.

• lib/uri/common.rb (URI.split): use RFC3986_Parser.

• lib/uri/common.rb (URI.parse): ditto.

• lib/uri/common.rb (URI.join): ditto.

• lib/uri/common.rb (URI.extract): deprecated.

• lib/uri/common.rb (URI.regexp): ditto.

• lib/uri/rfc2396_parser.rb: added.

• lib/uri/rfc3986_parser.rb: added.

[tenderlovemaking (Aaron Patterson)]

r46491 broke this script:

require 'uri'

thing = URI.parse 'http://example.com'
thing.query = 'location[]=1&location[]=2&age_group[]=2'

Before r46491 it would set the query, after r46491, it raises an exception.

Is this a bug in the new implementation? Or should I be doing something different? (I pulled this from the Rails tests, so I'm not 100% sure what it is actually for)

[bitsweat (Jeremy Daer)]

In RFC 3986, square brackets are no longer allowed in the query part.

Source of the unescaped brackets, in this case: https://github.com/brynary/rack-test/blob/master/lib/rack/test/utils.rb

This may become a common issue since plenty of code uses URI.parse and expects its more permissive RFC 2396 parsing.

[zzak (zzak _)]

I think #9990 is related /cc @naruse (Yui NARUSE) @JK @tenderlove

[naruse (Yui NARUSE)]

I'm considering to change the error policy of URI library, for example:
BEFORE: raise error if invalid characters exist
AFTER: percent-escape them

[lengarvey (Leonard Garvey)]

I've implemented something similar to that policy in the following gist: https://gist.github.com/lengarvey/31983eac6664351ed16d

It's a very basic naive implementation but I believe it roughly does what we need URI.parse to do.

[nobu (Nobuyoshi Nakada)]

• Related to Bug #10402: URI regression in 2.2.0-preview1 (bad URI(is not URI?): URI::InvalidURIError) added

[nobu (Nobuyoshi Nakada)]

• Related to Bug #9990: URI.parse and URI.encode use different RFCs added