Bug #14071
closedHTTP Header requiring dual authorization fails with 'header field value cannot include CR/LF'
Description
Not sure if this is a bug or not but I know where it was introduced and when it worked.
ruby 2.3.1p112 (Code Works)
ruby 2.3.4p301 (Code Works)
ruby 2.3.5p376 (Code Fails)
ruby 2.4.1p111 (Code Works)
ruby 2.4.2p198 (Code Fails)
My code that works - (Depending on Ruby version - see above versions of ruby for pass fail status):
Start Working Code¶
url = my_url + "/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=" + safe
url += "&Keywords=" + keywords if ! keywords.nil?
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Get.new(url)
request["authorization"] = "Bearer #{pf_token}\r\nAuthorization: #{ck_token}"
request["oauth_clientid"] = pf_credentials['client_id']
request["content-type"] = 'application/json'
# Send the request
http.set_debug_output $stderr
res = http.request(request)
I am no expert and the code above may be a hack but it works on sites where dual authentication is required, at least with some versions of Ruby. I came to this solution by inspecting the http request by setting 'http.set_debug_output $stderr
' and saw that header elements are separate by '\r\n'
This curl command works:
curl -X GET 'https://xxxx/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=Safe1' -H 'authorization: Bearer xxxxxxxxxxxxxxxxxxx' -H 'authorization: YYYYYYYYYYY' -H 'content-type: application/json' -H 'oauth_clientid: clientid1'
The above code fails with 'header field value cannot include CR/LF' in:
ruby 2.3.5p376
ruby 2.4.2p198
This was most recently was re-introduced by this commit: https://github.com/ruby/ruby/commit/427f5b57135fa165990f87c93658fafbe070289f
I have tried the following on the newer failing version of Ruby but these also fail with #<Net::HTTPUnauthorized:0x0000000003183780> => "1012116 - Invalid token."
Start Failing Code¶
url = my_url + "/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=" + safe
url += "&Keywords=" + keywords if ! keywords.nil?
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Get.new(url)
request["authorization"] = ["Bearer #{pf_token}", ck_token]
request["oauth_clientid"] = pf_credentials['client_id']
request["content-type"] = 'application/json'
# Send the request
http.set_debug_output $stderr
res = http.request(request)
and this:
url = my_url + "/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=" + safe
url += "&Keywords=" + keywords if ! keywords.nil?
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
request = Net::HTTP::Get.new(url)
request.add_field("authorization", "Bearer #{pf_token}")
request.add_field("authorization", ck_token)
request.add_field("oauth_clientid", pf_credentials['client_id'])
request.add_field("content-type", 'application/json')
# Send the request
http.set_debug_output $stderr
res = http.request(request)
Another variation also fails in all versions with "undefined method
strip' for #Array:0x00000000034ad910"`
url = my_url + "/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=" + safe
url += "&Keywords=" + keywords if ! keywords.nil?
uri = URI.parse(url)
http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
header = {
'authorization' => ["Bearer #{pf_token}", "#{ck_token}"],
'oauth_clientid' => pf_credentials['client_id'],
'content-type' => 'application/json'
}
# Send the request
http.set_debug_output $stderr
res = http.request_get(uri.path, header)
Updated by phluid61 (Matthew Kerwin) about 7 years ago
From my understanding of the HTTP specs that define the Authorization header, there's no standards-compatible way to send multiple Authorization header fields in a single message. So I don't think it can be called Ruby's bug.
According to this Stack Overflow question, servers that require multiple Authorization fields apparently might accept a flattened, comma-separated list (which is the inline representation for any other standard multi-valued header (aside from cookies)).
Have you tried: request["authorization"] = "Bearer #{pf_token}, #{ck_token}"
?
Updated by nobu (Nobuyoshi Nakada) about 7 years ago
- Description updated (diff)
- Status changed from Open to Third Party's Issue
https://tools.ietf.org/html/rfc7230#section-3.2.2
A sender MUST NOT generate multiple header fields with the same field
name in a message unless either the entire field value for that
header field is defined as a comma-separated list [i.e., #(values)]
or the header field is a well-known exception (as noted below).
If that server doesn't accept comma-separated list but requires multiple headers, it seems a bug of the server.
Updated by naruse (Yui NARUSE) about 7 years ago
https://fetch.spec.whatwg.org/ also denies multiple Authorization header.
Please fix your server or RFC/WHATWG spec.
Updated by dgames (Dax Games) about 7 years ago
Thanks all for the responses.
Unfortunately I do not have the capability to ''fix ' the server and a header similar to the header resulting from my 2nd and 3rd failed examples works in a nodejs sample app but not in Ruby. Where the header is defined as a hash:
header = {
'authorization' => ["Bearer #{pf_token}", "#{ck_token}"],
'oauth_clientid' => pf_credentials['client_id'],
'content-type' => 'application/json'
}
Working NodeJS Sample¶
let options = {
method: 'GET',
url: 'https://xxxxxxx/v1/PasswordVault/WebServices/PIMServices.svc/Safes',
qs:
{
client_id: 'ClientId1',
client_secret: 'ClientSecret1',
grant_type: 'client_credentials',
scope: 'api'
},
headers:
{
'Authorization': ['Bearer ' + pftoken, cktoken],
'oauth_clientid': 'clientid1',
'content-type': 'application/json'
}
};
console.log(options.headers);
request(options, function (error, response, body) {
if (error) {
reject(error);
}
resolve(body);
});
So this leads me to believe the server is not the issue.
This was my main concern, wondering if there is some issue with the way ruby is dealing with it?
I will try the comma separated string. I think I already did that but will try again.
Updated by dgames (Dax Games) about 7 years ago
I found the code that is actually causing the issue I was wrong before. The actual code is in: class Net::HTTPGenericRequest
def write_header(sock, ver, path)
reqline = "#{@method} #{path} HTTP/#{ver}"
if /[\r\n]/ =~ reqline
raise ArgumentError, "A Request-Line must not contain CR or LF"
end
buf = ""
buf << reqline << "\r\n"
each_capitalized do |k,v|
buf << "#{k}: #{v}\r\n"
end
buf << "\r\n"
sock.write buf
end
I think this code is overly critical of '\r\n' since it is actually erroring out on what is valid to separate header elements. The method itself is using this same separator.
Updated by nobu (Nobuyoshi Nakada) about 7 years ago
dgames (Dax Games) wrote:
Thanks all for the responses.
Unfortunately I do not have the capability to ''fix ' the server and a header similar to the header resulting from my 2nd and 3rd failed examples works in a nodejs sample app but not in Ruby.
You should report it to the server's administrators.
As far as the standard specs prohibit the duplicate header, we can't change the correct behavior and violate the standards.
And it sounds that node.js has the bug.
Updated by phluid61 (Matthew Kerwin) about 7 years ago
dgames (Dax Games) wrote:
I think this code is overly critical of '\r\n' since it is actually erroring out on what is valid to separate header elements. The method itself is using this same separator.
That's like saying you should allow quotation marks in a string, because you're hacking the string to actually be two strings. It bans the "\r\n" characters specifically because they're separators, they can't be part of the value.
I think perhaps what you'd like to request, instead, is the ability to define the way multiple header fields with the same name are sent in a HTTP message. Here's a straw-man idea:
h1.always_concatenate!
h1['foo'] = 'bar, baz'
h1['foo'] = 'qux'
# =>
#
# Foo: bar, baz, qux
#
h2.never_concatenate!
h2['foo'] = 'bar, baz'
h2['foo'] = 'qux'
# =>
#
# Foo: bar, baz
# Foo: qux
#
That is definitely a separate feature request, though.
Updated by dgames (Dax Games) about 7 years ago
nobu (Nobuyoshi Nakada) wrote:
As far as the standard specs prohibit the duplicate header, we can't change the correct behavior and violate the standards.
And it sounds that node.js has the bug.
Node is passing a single header with an array of values as Ruby tries to do in the failing code samples but Node works Ruby fails. They are both trying to do the same thing one fails one doesn't.
For now I just overloaded the methods I need to behave the way I need them to behave and I have it working.
Updated by nobu (Nobuyoshi Nakada) about 7 years ago
dgames (Dax Games) wrote:
nobu (Nobuyoshi Nakada) wrote:
As far as the standard specs prohibit the duplicate header, we can't change the correct behavior and violate the standards.
And it sounds that node.js has the bug.Node is passing a single header with an array of values as Ruby tries to do in the failing code samples but Node works Ruby fails. They are both trying to do the same thing one fails one doesn't.
When what must fail don't fail, we call it a bug.
Updated by dgames (Dax Games) about 7 years ago
nobu (Nobuyoshi Nakada) wrote:
When what must fail don't fail, we call it a bug.
I'm sorry I do not understand. Maybe I wasn't clear, Node is doing EXACTLY what you say Ruby should do and works where Ruby fails when it does EXACTLY what you say it should do.
I am not arguing with you I am just trying to get a clear understanding of why Ruby does not work when using the 'correct' header based on the RFC using multiple auth methods in an array on a URI that works with Node using the 'correct' header based on the RFC using multiple auth methods in an array on the very same URI.
Updated by nobu (Nobuyoshi Nakada) about 7 years ago
dgames (Dax Games) wrote:
I'm sorry I do not understand. Maybe I wasn't clear, Node is doing EXACTLY what you say Ruby should do and works where Ruby fails when it does EXACTLY what you say it should do.
What is sent by NodeJS in that case?