Feature #4071: support basic auth for Net::HTTP.get requests

• Status: Closed
• Assignee: -
• Target version: -

[coderrr (coderrr .)]

=begin
support basic auth for Net::HTTP.get(URI.parse("http://user:[e-mail]/asdf"))
=end

[naruse (Yui NARUSE)]

=begin
I'm negative because such user:pass@host formed URI is deprecated now.
=end

[coderrr (coderrr .)]

=begin
Yui, please clarify and point me to the deprecation you are referring to.
=end

[coderrr (coderrr .)]

=begin
just to clarify... the current way you would have to do this is:

Net::HTTP.start('whatever.com') {|h| g = Net::HTTP::Get.new('/asdf'); g.basic_auth('user', 'pass'); h.request(g).body }
=end

[naruse (Yui NARUSE)]

• Status changed from Open to Rejected

=begin
It is RFC3986, see also Feature #3848.

Anyway, you can following:

require 'open-uri'
URI("http://example.com/asdf").read(http_basic_authentication: ["user","pass"])
=end

[coderrr (coderrr .)]

=begin
Thanks for showing me that open-uri version, I didn't know that previously. I do think that it is not as concise as the example in my ticket though.

Here's the deprecation paragraph.

Use of the format "user:password" in the userinfo field is
deprecated.  Applications should not render as clear text any data
after the first colon (":") character found within a userinfo
subcomponent unless the data after the colon is the empty string
(indicating no password).  Applications may choose to ignore or
reject such data when it is received as part of a reference and
should reject the storage of such data in unencrypted form.  The
passing of authentication information in clear text has proven to be
a security risk in almost every case where it has been used.

I believe the sentiment of this deprecation is the last sentence: "The passing of authentication information in clear text has proven to be a security risk in almost every case where it has been used." Which doesn't apply at all to the example code in my ticket.

Even though this deprecation was stated in 2005 it's five years later and here's a list of applications which still support it: Firefox, Chrome, Opera, wget, curl, axel. Safari also supports it but displays an ominous warning. IE stopped supporting it after 6.0 for this reason:

A malicious user might use this URL syntax to create a hyperlink that appears to open a legitimate Web site but actually opens a deceptive (spoofed) Web site. For example, the following URL appears to open http://www.wingtiptoys.com but actually opens http://example.com: http://[e-mail]
[ http://support.microsoft.com/kb/834489 ]

This reason also doesn't apply to ruby. If you were building a web browser in ruby it'd be up to the developer to handle security issues such as this, not the stdlib.

Finally, URI already supports this format. If URI didn't support it I wouldn't have made this ticket or a ticket to add it to URI because that's a much bigger change philosophically. Net::HTTP.get already accepts a URI. URI.parse already accepts a user:pass. I see no reason not to pass that info down the chain.

If you feel strongly enough about that deprecation I think we should patch URI to no longer accept user:pass and see if people agree.

Net::HTTP.get(URI(..)) is a great concise way to fetch a single webpage. Having to muck the code up just because a site uses basic auth seems like a real shame to me. Especially when this is only a 4 line change.

I should probably also add that I'm not advocating everyone add user:pass to URIs everywhere. There are just certain situations where this is the simplest way to do things in your code with no downsides.
=end

[naruse (Yui NARUSE)]

=begin

Finally, URI already supports this format. If URI didn't support it
I wouldn't have made this ticket or a ticket to add it to URI because
that's a much bigger change philosophically. Net::HTTP.get already
accepts a URI. URI.parse already accepts a user:pass. I see no
reason not to pass that info down the chain.

Because a user:pass is deprecated, we have no reason to add
a new feature to support a user:pass.

If you feel strongly enough about that deprecation I think we should
patch URI to no longer accept user:pass and see if people agree.

URI.parse accepts a user:pass because it is still valid URI
even if it is deprecated.
We don't reject, but don't help such use case.

Net::HTTP.get(URI(..)) is a great concise way to fetch a single
webpage. Having to muck the code up just because a site uses basic
auth seems like a real shame to me. Especially when this is only a 4
line change.

I think URI(..).read is a more concise way.

Anyway, adding optional hash argumentto Net::HTTP.get like
Net::HTTP.get(uri_or_host, path = nil, port = nil, opt=nil)
and use like
Net::HTTP.get(uri, http_basic_authentication: ["user","pass"])
may be acceptable.

--
NARUSE, Yui [e-mail]

=end

[coderrr (coderrr .)]

=begin
It seems to me you're being very pedantic on the deprecation issue. The only way I see the deprecation's sentiment applying to Ruby is in the sense that if Ruby supports this, then it will be a small incentive for people to continue using URIs this way and in turn cause some security issues for someone somewhere. I don't think it's Ruby's job to take the lead in complying 100% to the URI RFC. The fact that many other big HTTP clients still support this says to me there is a benefit to continuing supporting it that outweighs the costs: it's a well known concise way of specifying basic auth info for a web site. If most of the other http clients had gone the way of removing the functionality from their code I wouldn't be arguing. But it seems to me it's still a somewhat informal standard. Also Ruby is such a small % of the HTTP client market, whether or not we support it is most likely not going to make any impact to how URIs are used as a whole.

I agree there are or could be other concise ways of doing this in Ruby (I personally prefer net/http over open-uri) but I don't think that means we shouldn't consider this one. The other thing is, in many cases, regardless of how you pass the user/password as basic auth, it's going to be coming from a plaintext source anyway which is what the RFC cites as its main reason for deprecation.

Why not just accept the patch and add a line to Net::HTTP.get rdoc mentioning the deprecation of using URI in this way?
=end

[naruse (Yui NARUSE)]

=begin

The fact that many other big HTTP clients still support this says to me there is a benefit to continuing supporting it that outweighs the costs: it's a well known concise way of specifying basic auth info for a web site.

I think, the benefit is compatibility.
Languages which needn't such compatibility, like Java's HttpServletRequest.getRequestURI()
and Python's urllib2 don't support it.
http://stackoverflow.com/questions/828076/authentication-using-uri-userinfo-component-http-userinfohostnameport-path-in
http://docs.python.org/library/urllib2.html

Why not just accept the patch and add a line to Net::HTTP.get rdoc mentioning the deprecation of using URI in this way?

The patch makes the use userinfo of URI easy.
I don't want to promote such deprecated feature.

Like this, Ruby encourage suitable way by adding utility methods,
and Ruby don't add utility methods for wrong way.
Feature request should be on this format.
=end

[coderrr (coderrr .)]

=begin
ok cool, I accept your decision :) .. thanks for your time
=end

[coderrr (coderrr .)]

=begin
sorry, just one more thing...

Net::HTTP.post_form currently acts exactly like my patch, using the user/pass from the URI and is documented as such in the Net::HTTP rdoc.
=end

[naruse (Yui NARUSE)]

=begin

Net::HTTP.post_form currently acts exactly like my patch, using the user/pass from the URI and is documented as such in the Net::HTTP rdoc.

Yes, if the compatibility allows I want to remove it.
=end

[meta (mathew murphy)]

=begin
Something I'd like to point out:

• RFC 1738 did not allow username and password in HTTP URLs at all; see section 3.3.

• RFC 2396 said:

Some URL schemes use the format "user:password" in the userinfo field.
This practice is NOT RECOMMENDED, because the passing of authentication
information in clear text (such as URI) has proven to be a security risk
in almost every case where it has been used.

RFC 2396 did not, however, state that HTTP was one of the schemes that used that information. Nor did it include the addition of username and password information to http URLs in section G3, which listed the modifications to RFC1738.

So in fact, username and password in HTTP URLs was never standard as per the RFCs.
=end

[coderrr (coderrr .)]

=begin
That's interesting but IMHO it's not Ruby's job/goal to comply as closely as possible with RFCs, but to provide programmers with the best possible experience.
=end

[naruse (Yui NARUSE)]

=begin

That's interesting but IMHO it's not Ruby's job/goal to comply as closely as possible with RFCs, but to provide programmers with the best possible experience.

Yes, we want to provide the best experience.
RFC is one of the criterion of good things,
but there are another criterions like extensibility, speed, size of code, easiness, and so on.
=end

[marcandre (Marc-Andre Lafortune)]

• Status changed from Rejected to Open

=begin
Hi!

If I understand correctly, Yui, your argument is that user & password should not have been part of the URI::HTTP class. That may be true, but I fear it is too late to change that.

Given the fact that URI may have user & password information, I would also prefer if Net::HTTP.get took it into account. Maybe there is something I don't understand, because I think it should also respect a https URI (I should open a different ticket for this).

If the patch won't be accepted, the documentation of either post_form or get should be made more explicit to state that one supports it, or that the other doesn't as clearly this situation can only be justified by historical reasons.
=end

[naruse (Yui NARUSE)]

=begin

If I understand correctly, Yui, your argument is that user & password should not have been part of the URI::HTTP class. That may be true, but I fear it is too late to change that.

No.

Given the fact that URI may have user & password information, I would also prefer if Net::HTTP.get took it into account. Maybe there is something I don't understand, because I think it should also respect a https URI (I should open a different ticket for this).

Did you read this thread?

If the patch won't be accepted, the documentation of either post_form or get should be made more explicit to state that one supports it, or that the other doesn't as clearly this situation can only be justified by historical reasons.

I agree.
=end

[naruse (Yui NARUSE)]

=begin
Hi,

2010/11/25 Marc-Andre Lafortune [e-mail]:

On Wed, Nov 24, 2010 at 11:55 PM, Yui NARUSE [e-mail] wrote:

Given the fact that URI may have user & password information, I would also prefer if Net::HTTP.get took it into account. Maybe there is something I don't understand, because I think it should also respect a https URI (I should open a different ticket for this).

Did you read this thread?

I'm not sure if you are seriously implying that I didn't read the
thread, but in case it needs to be said: yes, of course, I did, and I
would be grateful if, in the future, you always assumed that I read
the full thread before replying.

I was confused because your point is not on the context of this thread.
Or maybe I didn't understand your point.

If your question is not about the word "also" but about any other
word(s) in the two sentences, then I'll need some more explanation.

I want to say, I can't understand what you say.
Please explain some more.

--
NARUSE, Yui
[e-mail]

=end

[naruse (Yui NARUSE)]

=begin
Hi,

(2010/11/25 18:45), Marc-Andre Lafortune wrote:

The class URI::HTTP has two attributes named 'user' and 'password'. I
feel these can not be removed for compatibility reasons.

Partly yes, I think userinfo in HTTP URI is in RFC3986.
Different from mathew, I think RFC3986 syntactically allows userinfo in HTTP
URI because of compatibility to existing implementations.

If you
disagree, please say so. Otherwise, we have to accept that the
information is there (even if you/some RFC think it's a bad idea).

Note that setters exist too, so they can be specified in other ways
than with through the "user:pass@...":
u = URI("https://github.com")
u.user = "bob"
u.password = "foo"

The Net::HTTP.get and Net::HTTP.post_form both accept a URI::HTTP
argument. I feel they should deal with all the information provided by
the URI::HTTP class.

No to "they should deal with all the information provided by the URI::HTTP class".

This includes 'user' and 'password'. Currently, get doesn't, but
post_form does. I feel that changing get would make the situation
consistent, convenient and shouldn't impact compatibility.

Don't change get is from "correctness".

I feel that discouraging the use of user& password, or of basic
authentifcation over HTTP is not the role of the standard library.

I think it is the role of the standard library.

What next, a warning when we set request.basic_auth?

If it is needed.
But I think it isn't needed because basic auth with SSL is enough secure.

Moreover, if it was deemed important that URI didn't have a user&
password, then get is the wrong level to intervene.

I remark that a scheme set to 'https' is currently ignored by get
and post_form, which I believe to be erroneous, counter-intuitive
and insecure. Either I'm missing the ideology behind ignoring info
from URI, or it is simply a similar oversight as ignoring the user&
password. I believe it is related and in the right context, as
exemplified by the following line:

Net::HTTP.get(URI("https://bob:[e-mail]"))

Why is it the right context?
At least in github's context, it only includes the user name.

If get worked as I believe it should, the above line would work and
security would not be compromised (at least at the network level),
while currently the request is made in HTTP (on port 443) and no
credential is ever sent.

RFC's argument is mainly for the client side.
The network level is out of the scope.
(and your point is true for basic auth it self)

--
NARUSE, Yui [e-mail]

=end

[coderrr (coderrr .)]

=begin
Yui,

I should point out that the RFC actually does provide an "out" for allowing a password in the userinfo in section 7.5: "A password appearing within the userinfo component is deprecated and should be considered an error (or simply ignored) except in those rare cases where the 'password' parameter is intended to be public." Since Ruby of course can't know when a password is intended to be public you could make an argument that we should support that format and leave the decision to ignore it or not up to the developer.
=end

[naruse (Yui NARUSE)]

=begin
(2010/11/26 19:38), coderrr . wrote:

Yui,

I should point out that the RFC actually does provide an "out" for
allowing a password in the userinfo in section 7.5: "A password
appearing within the userinfo component is deprecated and should be
considered an error (or simply ignored) except in those rare cases
where the 'password' parameter is intended to be public." Since Ruby
of course can't know when a password is intended to be public you
could make an argument that we should support that format and leave
the decision to ignore it or not up to the developer.

Yes, Ruby can't know whether the password is intended to be public or not.
Moreover Ruby doesn't know whether the userinfo includes a password or not.
(of course, we may know by userinfo.split(":"))

Another reason why uri lib still supports userinfo is other URI implementations
like .NET Framework and Java also support it.
http://msdn.microsoft.com/ja-jp/library/system.uri_members(VS.80).aspx
http://download.oracle.com/javase/7/docs/api/java/net/URI.html

--
NARUSE, Yui [e-mail]

=end

[meta (mathew murphy)]

=begin
On Wed, Nov 24, 2010 at 22:12, Marc-Andre Lafortune
[e-mail]:

If the patch won't be accepted, the documentation of either post_form or
get should be made more explicit to state that one supports it, or that
the other doesn't as clearly this situation can only be justified by
historical reasons.

I just checked, and my original submitted documentation for net/http warned
that authentication information in the URI was not valid as per RFC 1738
section 3.3, even though the URI class accepted it and

res = Net::HTTP.post_form(URI.parse('

http://jack:[e-mail]/todo.cgi'),

                                        {'from'=>'2005-01-01',

'to'=>'2005-03-31'})

worked. So presumably at some point someone removed the warning from the
documentation.

mathew

URL:http://www.pobox.com/~meta/

On Wed, Nov 24, 2010 at 22:12, Marc-Andre Lafortune <[e-mail]> wrote:

If the patch won't be accepted, the documentation of either post_form or get should be made more explicit to state that one supports it, or that the other doesn't as clearly this situation can only be justified by historical reasons.

 

I just checked, and my original submitted documentation for net/http warned that authentication information in the URI was not valid as per RFC 1738 section 3.3, even though the URI class accepted it and

<title></title> <style type="text/css"> p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 12.0px Helvetica} </style>

res = Net::HTTP.post_form(URI.parse('http://jack:[e-mail]/todo.cgi'),

                                           {'from'=>'2005-01-01', 'to'=>'2005-03-31'})

worked. So presumably at some point someone removed the warning from the documentation.

mathew
--
<URL:http://www.pobox.com/~meta/>

=end

[naruse (Yui NARUSE)]

• Status changed from Open to Closed
• % Done changed from 0 to 100

=begin
This issue was solved with changeset r30155.
coderrr, thank you for reporting this issue.
Your contribution to Ruby is greatly appreciated.
May Ruby be with you.

=end