Bug #14071: HTTP Header requiring dual authorization fails with 'header field value cannot include CR/LF'

• Status: Third Party's Issue
• Assignee: -
• Target version: -
• ruby -v: ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]
• Backport: 2.3: UNKNOWN, 2.4: UNKNOWN

[dgames (Dax Games)]

Not sure if this is a bug or not but I know where it was introduced and when it worked.

ruby 2.3.1p112 (Code Works)
ruby 2.3.4p301 (Code Works)
ruby 2.3.5p376 (Code Fails)
ruby 2.4.1p111 (Code Works)
ruby 2.4.2p198 (Code Fails)

My code that works - (Depending on Ruby version - see above versions of ruby for pass fail status):

Start Working Code

        url = my_url + "/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=" + safe
        url += "&Keywords=" + keywords if ! keywords.nil?

        uri = URI.parse(url)

        http = Net::HTTP.new(uri.host, uri.port)
        http.use_ssl = true

        request = Net::HTTP::Get.new(url)

        request["authorization"] = "Bearer #{pf_token}\r\nAuthorization: #{ck_token}"
        request["oauth_clientid"] = pf_credentials['client_id']
        request["content-type"] = 'application/json'

        # Send the request
        http.set_debug_output $stderr
        res = http.request(request)

I am no expert and the code above may be a hack but it works on sites where dual authentication is required, at least with some versions of Ruby. I came to this solution by inspecting the http request by setting 'http.set_debug_output $stderr' and saw that header elements are separate by '\r\n'

This curl command works:

curl -X GET 'https://xxxx/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=Safe1' -H 'authorization: Bearer xxxxxxxxxxxxxxxxxxx' -H 'authorization: YYYYYYYYYYY' -H 'content-type: application/json' -H 'oauth_clientid: clientid1'

The above code fails with 'header field value cannot include CR/LF' in:

ruby 2.3.5p376
ruby 2.4.2p198

This was most recently was re-introduced by this commit: https://github.com/ruby/ruby/commit/427f5b57135fa165990f87c93658fafbe070289f

I have tried the following on the newer failing version of Ruby but these also fail with #<Net::HTTPUnauthorized:0x0000000003183780> => "1012116 - Invalid token."

Start Failing Code

        url = my_url + "/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=" + safe
        url += "&Keywords=" + keywords if ! keywords.nil?

        uri = URI.parse(url)

        http = Net::HTTP.new(uri.host, uri.port)
        http.use_ssl = true

        request = Net::HTTP::Get.new(url)

        request["authorization"] = ["Bearer #{pf_token}",  ck_token]
        request["oauth_clientid"] = pf_credentials['client_id']
        request["content-type"] = 'application/json'

        # Send the request
        http.set_debug_output $stderr
        res = http.request(request)

and this:

        url = my_url + "/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=" + safe
        url += "&Keywords=" + keywords if ! keywords.nil?

        uri = URI.parse(url)

        http = Net::HTTP.new(uri.host, uri.port)
        http.use_ssl = true

        request = Net::HTTP::Get.new(url)

        request.add_field("authorization", "Bearer #{pf_token}")
        request.add_field("authorization", ck_token)
        request.add_field("oauth_clientid", pf_credentials['client_id'])
        request.add_field("content-type", 'application/json')

        # Send the request
        http.set_debug_output $stderr
        res = http.request(request)

Another variation also fails in all versions with "undefined method strip' for #Array:0x00000000034ad910"`

        url = my_url + "/PasswordVault/WebServices/PIMServices.svc/Accounts?Safe=" + safe
        url += "&Keywords=" + keywords if ! keywords.nil?

        uri = URI.parse(url)

        http = Net::HTTP.new(uri.host, uri.port)
        http.use_ssl = true

        header = {
          'authorization' => ["Bearer #{pf_token}", "#{ck_token}"],
          'oauth_clientid' => pf_credentials['client_id'],
          'content-type' => 'application/json'
        }

        # Send the request
       
        http.set_debug_output $stderr
        res = http.request_get(uri.path, header)      

[dgames (Dax Games)]

• Description updated (diff)

[dgames (Dax Games)]

• Description updated (diff)

[dgames (Dax Games)]

• Description updated (diff)

[dgames (Dax Games)]

• Description updated (diff)

[dgames (Dax Games)]

• Description updated (diff)

[dgames (Dax Games)]

• Description updated (diff)

[phluid61 (Matthew Kerwin)]

From my understanding of the HTTP specs that define the Authorization header, there's no standards-compatible way to send multiple Authorization header fields in a single message. So I don't think it can be called Ruby's bug.

According to this Stack Overflow question, servers that require multiple Authorization fields apparently might accept a flattened, comma-separated list (which is the inline representation for any other standard multi-valued header (aside from cookies)).

Have you tried: request["authorization"] = "Bearer #{pf_token}, #{ck_token}" ?

[nobu (Nobuyoshi Nakada)]

• Description updated (diff)
• Status changed from Open to Third Party's Issue

https://tools.ietf.org/html/rfc7230#section-3.2.2

A sender MUST NOT generate multiple header fields with the same field
name in a message unless either the entire field value for that
header field is defined as a comma-separated list [i.e., #(values)]
or the header field is a well-known exception (as noted below).

If that server doesn't accept comma-separated list but requires multiple headers, it seems a bug of the server.

[naruse (Yui NARUSE)]

https://fetch.spec.whatwg.org/ also denies multiple Authorization header.
Please fix your server or RFC/WHATWG spec.

[dgames (Dax Games)]

Thanks all for the responses.

Unfortunately I do not have the capability to ''fix ' the server and a header similar to the header resulting from my 2nd and 3rd failed examples works in a nodejs sample app but not in Ruby. Where the header is defined as a hash:

header = {
          'authorization' => ["Bearer #{pf_token}", "#{ck_token}"],
          'oauth_clientid' => pf_credentials['client_id'],
          'content-type' => 'application/json'
        }

Working NodeJS Sample

            let options = {
                method: 'GET',
                url: 'https://xxxxxxx/v1/PasswordVault/WebServices/PIMServices.svc/Safes',
                qs:
                    {
                        client_id: 'ClientId1',
                        client_secret: 'ClientSecret1',
                        grant_type: 'client_credentials',
                        scope: 'api'
                    },
                headers:
                    {
                        'Authorization': ['Bearer ' + pftoken, cktoken],
                        'oauth_clientid': 'clientid1',
                        'content-type': 'application/json'
                    }
            };

            console.log(options.headers);
            request(options, function (error, response, body) {
                if (error) {
                    reject(error);
                }

                resolve(body);
            });

So this leads me to believe the server is not the issue.

This was my main concern, wondering if there is some issue with the way ruby is dealing with it?

I will try the comma separated string. I think I already did that but will try again.

[dgames (Dax Games)]

I found the code that is actually causing the issue I was wrong before. The actual code is in: class Net::HTTPGenericRequest

  def write_header(sock, ver, path)
    reqline = "#{@method} #{path} HTTP/#{ver}"
    if /[\r\n]/ =~ reqline
      raise ArgumentError, "A Request-Line must not contain CR or LF"
    end
    buf = ""
    buf << reqline << "\r\n"
    each_capitalized do |k,v|
      buf << "#{k}: #{v}\r\n"
    end
    buf << "\r\n"
    sock.write buf
  end

I think this code is overly critical of '\r\n' since it is actually erroring out on what is valid to separate header elements. The method itself is using this same separator.

[nobu (Nobuyoshi Nakada)]

dgames (Dax Games) wrote:

Thanks all for the responses.

Unfortunately I do not have the capability to ''fix ' the server and a header similar to the header resulting from my 2nd and 3rd failed examples works in a nodejs sample app but not in Ruby.

You should report it to the server's administrators.
As far as the standard specs prohibit the duplicate header, we can't change the correct behavior and violate the standards.
And it sounds that node.js has the bug.

[phluid61 (Matthew Kerwin)]

dgames (Dax Games) wrote:

I think this code is overly critical of '\r\n' since it is actually erroring out on what is valid to separate header elements. The method itself is using this same separator.

That's like saying you should allow quotation marks in a string, because you're hacking the string to actually be two strings. It bans the "\r\n" characters specifically because they're separators, they can't be part of the value.

I think perhaps what you'd like to request, instead, is the ability to define the way multiple header fields with the same name are sent in a HTTP message. Here's a straw-man idea:

h1.always_concatenate!
h1['foo'] = 'bar, baz'
h1['foo'] = 'qux'

# =>
#
# Foo: bar, baz, qux
#

h2.never_concatenate!
h2['foo'] = 'bar, baz'
h2['foo'] = 'qux'

# =>
#
# Foo: bar, baz
# Foo: qux
#

That is definitely a separate feature request, though.

[dgames (Dax Games)]

nobu (Nobuyoshi Nakada) wrote:

As far as the standard specs prohibit the duplicate header, we can't change the correct behavior and violate the standards.
And it sounds that node.js has the bug.

Node is passing a single header with an array of values as Ruby tries to do in the failing code samples but Node works Ruby fails. They are both trying to do the same thing one fails one doesn't.

For now I just overloaded the methods I need to behave the way I need them to behave and I have it working.

[nobu (Nobuyoshi Nakada)]

dgames (Dax Games) wrote:

nobu (Nobuyoshi Nakada) wrote:

As far as the standard specs prohibit the duplicate header, we can't change the correct behavior and violate the standards.
And it sounds that node.js has the bug.

Node is passing a single header with an array of values as Ruby tries to do in the failing code samples but Node works Ruby fails. They are both trying to do the same thing one fails one doesn't.

When what must fail don't fail, we call it a bug.

[dgames (Dax Games)]

nobu (Nobuyoshi Nakada) wrote:

When what must fail don't fail, we call it a bug.

I'm sorry I do not understand. Maybe I wasn't clear, Node is doing EXACTLY what you say Ruby should do and works where Ruby fails when it does EXACTLY what you say it should do.

I am not arguing with you I am just trying to get a clear understanding of why Ruby does not work when using the 'correct' header based on the RFC using multiple auth methods in an array on a URI that works with Node using the 'correct' header based on the RFC using multiple auth methods in an array on the very same URI.

[nobu (Nobuyoshi Nakada)]

dgames (Dax Games) wrote:

I'm sorry I do not understand. Maybe I wasn't clear, Node is doing EXACTLY what you say Ruby should do and works where Ruby fails when it does EXACTLY what you say it should do.

What is sent by NodeJS in that case?