Project

General

Profile

Actions

Bug #14071

closed

HTTP Header requiring dual authorization fails with 'header field value cannot include CR/LF'

Added by dgames (Dax Games) over 4 years ago. Updated over 4 years ago.

Status:
Third Party's Issue
Priority:
Normal
Assignee:
-
Target version:
-
ruby -v:
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]
[ruby-core:83626]

Description

Not sure if this is a bug or not but I know where it was introduced and when it worked.

ruby 2.3.1p112 (Code Works)
ruby 2.3.4p301 (Code Works)
ruby 2.3.5p376 (Code Fails)
ruby 2.4.1p111 (Code Works)
ruby 2.4.2p198 (Code Fails)

My code that works - (Depending on Ruby version - see above versions of ruby for pass fail status):

Start Working Code

        url = my_url + "/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=" + safe
        url += "&Keywords=" + keywords if ! keywords.nil?

        uri = URI.parse(url)

        http = Net::HTTP.new(uri.host, uri.port)
        http.use_ssl = true

        request = Net::HTTP::Get.new(url)

        request["authorization"] = "Bearer #{pf_token}\r\nAuthorization: #{ck_token}"
        request["oauth_clientid"] = pf_credentials['client_id']
        request["content-type"] = 'application/json'

        # Send the request
        http.set_debug_output $stderr
        res = http.request(request)

I am no expert and the code above may be a hack but it works on sites where dual authentication is required, at least with some versions of Ruby. I came to this solution by inspecting the http request by setting 'http.set_debug_output $stderr' and saw that header elements are separate by '\r\n'

This curl command works:

curl -X GET 'https://xxxx/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=Safe1' -H 'authorization: Bearer xxxxxxxxxxxxxxxxxxx' -H 'authorization: YYYYYYYYYYY' -H 'content-type: application/json' -H 'oauth_clientid: clientid1'

The above code fails with 'header field value cannot include CR/LF' in:

ruby 2.3.5p376
ruby 2.4.2p198

This was most recently was re-introduced by this commit: https://github.com/ruby/ruby/commit/427f5b57135fa165990f87c93658fafbe070289f

I have tried the following on the newer failing version of Ruby but these also fail with #<Net::HTTPUnauthorized:0x0000000003183780> => "1012116 - Invalid token."

Start Failing Code

        url = my_url + "/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=" + safe
        url += "&Keywords=" + keywords if ! keywords.nil?

        uri = URI.parse(url)

        http = Net::HTTP.new(uri.host, uri.port)
        http.use_ssl = true

        request = Net::HTTP::Get.new(url)

        request["authorization"] = ["Bearer #{pf_token}",  ck_token]
        request["oauth_clientid"] = pf_credentials['client_id']
        request["content-type"] = 'application/json'

        # Send the request
        http.set_debug_output $stderr
        res = http.request(request)

and this:

        url = my_url + "/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=" + safe
        url += "&Keywords=" + keywords if ! keywords.nil?

        uri = URI.parse(url)

        http = Net::HTTP.new(uri.host, uri.port)
        http.use_ssl = true

        request = Net::HTTP::Get.new(url)

        request.add_field("authorization", "Bearer #{pf_token}")
        request.add_field("authorization", ck_token)
        request.add_field("oauth_clientid", pf_credentials['client_id'])
        request.add_field("content-type", 'application/json')

        # Send the request
        http.set_debug_output $stderr
        res = http.request(request)

Another variation also fails in all versions with "undefined method strip' for #Array:0x00000000034ad910"`

        url = my_url + "/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=" + safe
        url += "&Keywords=" + keywords if ! keywords.nil?

        uri = URI.parse(url)

        http = Net::HTTP.new(uri.host, uri.port)
        http.use_ssl = true

        header = {
          'authorization' => ["Bearer #{pf_token}", "#{ck_token}"],
          'oauth_clientid' => pf_credentials['client_id'],
          'content-type' => 'application/json'
        }

        # Send the request
       
        http.set_debug_output $stderr
        res = http.request_get(uri.path, header)      
Actions #1

Updated by dgames (Dax Games) over 4 years ago

  • Description updated (diff)
Actions #2

Updated by dgames (Dax Games) over 4 years ago

  • Description updated (diff)
Actions #3

Updated by dgames (Dax Games) over 4 years ago

  • Description updated (diff)
Actions #4

Updated by dgames (Dax Games) over 4 years ago

  • Description updated (diff)
Actions #5

Updated by dgames (Dax Games) over 4 years ago

  • Description updated (diff)
Actions #6

Updated by dgames (Dax Games) over 4 years ago

  • Description updated (diff)

Updated by phluid61 (Matthew Kerwin) over 4 years ago

From my understanding of the HTTP specs that define the Authorization header, there's no standards-compatible way to send multiple Authorization header fields in a single message. So I don't think it can be called Ruby's bug.

According to this Stack Overflow question, servers that require multiple Authorization fields apparently might accept a flattened, comma-separated list (which is the inline representation for any other standard multi-valued header (aside from cookies)).

Have you tried: request["authorization"] = "Bearer #{pf_token}, #{ck_token}" ?

Updated by nobu (Nobuyoshi Nakada) over 4 years ago

  • Status changed from Open to Third Party's Issue
  • Description updated (diff)

https://tools.ietf.org/html/rfc7230#section-3.2.2

A sender MUST NOT generate multiple header fields with the same field
name in a message unless either the entire field value for that
header field is defined as a comma-separated list [i.e., #(values)]
or the header field is a well-known exception (as noted below).

If that server doesn't accept comma-separated list but requires multiple headers, it seems a bug of the server.

Updated by naruse (Yui NARUSE) over 4 years ago

https://fetch.spec.whatwg.org/ also denies multiple Authorization header.
Please fix your server or RFC/WHATWG spec.

Updated by dgames (Dax Games) over 4 years ago

Thanks all for the responses.

Unfortunately I do not have the capability to ''fix ' the server and a header similar to the header resulting from my 2nd and 3rd failed examples works in a nodejs sample app but not in Ruby. Where the header is defined as a hash:

header = {
          'authorization' => ["Bearer #{pf_token}", "#{ck_token}"],
          'oauth_clientid' => pf_credentials['client_id'],
          'content-type' => 'application/json'
        }

Working NodeJS Sample

            let options = {
                method: 'GET',
                url: 'https://xxxxxxx/v1/PasswordVault/WebServices/PIMServices.svc/Safes',
                qs:
                    {
                        client_id: 'ClientId1',
                        client_secret: 'ClientSecret1',
                        grant_type: 'client_credentials',
                        scope: 'api'
                    },
                headers:
                    {
                        'Authorization': ['Bearer ' + pftoken, cktoken],
                        'oauth_clientid': 'clientid1',
                        'content-type': 'application/json'
                    }
            };

            console.log(options.headers);
            request(options, function (error, response, body) {
                if (error) {
                    reject(error);
                }

                resolve(body);
            });

So this leads me to believe the server is not the issue.

This was my main concern, wondering if there is some issue with the way ruby is dealing with it?

I will try the comma separated string. I think I already did that but will try again.

Updated by dgames (Dax Games) over 4 years ago

I found the code that is actually causing the issue I was wrong before. The actual code is in: class Net::HTTPGenericRequest

  def write_header(sock, ver, path)
    reqline = "#{@method} #{path} HTTP/#{ver}"
    if /[\r\n]/ =~ reqline
      raise ArgumentError, "A Request-Line must not contain CR or LF"
    end
    buf = ""
    buf << reqline << "\r\n"
    each_capitalized do |k,v|
      buf << "#{k}: #{v}\r\n"
    end
    buf << "\r\n"
    sock.write buf
  end

I think this code is overly critical of '\r\n' since it is actually erroring out on what is valid to separate header elements. The method itself is using this same separator.

Updated by nobu (Nobuyoshi Nakada) over 4 years ago

dgames (Dax Games) wrote:

Thanks all for the responses.

Unfortunately I do not have the capability to ''fix ' the server and a header similar to the header resulting from my 2nd and 3rd failed examples works in a nodejs sample app but not in Ruby.

You should report it to the server's administrators.
As far as the standard specs prohibit the duplicate header, we can't change the correct behavior and violate the standards.
And it sounds that node.js has the bug.

Updated by phluid61 (Matthew Kerwin) over 4 years ago

dgames (Dax Games) wrote:

I think this code is overly critical of '\r\n' since it is actually erroring out on what is valid to separate header elements. The method itself is using this same separator.

That's like saying you should allow quotation marks in a string, because you're hacking the string to actually be two strings. It bans the "\r\n" characters specifically because they're separators, they can't be part of the value.

I think perhaps what you'd like to request, instead, is the ability to define the way multiple header fields with the same name are sent in a HTTP message. Here's a straw-man idea:

h1.always_concatenate!
h1['foo'] = 'bar, baz'
h1['foo'] = 'qux'

# =>
#
# Foo: bar, baz, qux
#

h2.never_concatenate!
h2['foo'] = 'bar, baz'
h2['foo'] = 'qux'

# =>
#
# Foo: bar, baz
# Foo: qux
#

That is definitely a separate feature request, though.

Updated by dgames (Dax Games) over 4 years ago

nobu (Nobuyoshi Nakada) wrote:

As far as the standard specs prohibit the duplicate header, we can't change the correct behavior and violate the standards.
And it sounds that node.js has the bug.

Node is passing a single header with an array of values as Ruby tries to do in the failing code samples but Node works Ruby fails. They are both trying to do the same thing one fails one doesn't.

For now I just overloaded the methods I need to behave the way I need them to behave and I have it working.

Updated by nobu (Nobuyoshi Nakada) over 4 years ago

dgames (Dax Games) wrote:

nobu (Nobuyoshi Nakada) wrote:

As far as the standard specs prohibit the duplicate header, we can't change the correct behavior and violate the standards.
And it sounds that node.js has the bug.

Node is passing a single header with an array of values as Ruby tries to do in the failing code samples but Node works Ruby fails. They are both trying to do the same thing one fails one doesn't.

When what must fail don't fail, we call it a bug.

Updated by dgames (Dax Games) over 4 years ago

nobu (Nobuyoshi Nakada) wrote:

When what must fail don't fail, we call it a bug.

I'm sorry I do not understand. Maybe I wasn't clear, Node is doing EXACTLY what you say Ruby should do and works where Ruby fails when it does EXACTLY what you say it should do.

I am not arguing with you I am just trying to get a clear understanding of why Ruby does not work when using the 'correct' header based on the RFC using multiple auth methods in an array on a URI that works with Node using the 'correct' header based on the RFC using multiple auth methods in an array on the very same URI.

Updated by nobu (Nobuyoshi Nakada) over 4 years ago

dgames (Dax Games) wrote:

I'm sorry I do not understand. Maybe I wasn't clear, Node is doing EXACTLY what you say Ruby should do and works where Ruby fails when it does EXACTLY what you say it should do.

What is sent by NodeJS in that case?

Actions

Also available in: Atom PDF