Skip to content

Add tests for signature/Host header mismatch - #68

Merged
tedder merged 1 commit into
tedder:mainfrom
phillipberndt:main
May 15, 2023
Merged

tedder merged 1 commit into
tedder:mainfrom
phillipberndt:main

Conversation

@phillipberndt

Copy link
Copy Markdown
Contributor

This adds the unit tests we discussed in #65.

Pytest output:

================================================================================================================= FAILURES =================================================================================================================
_____________________________________________________________________________ AWS4Auth_GetCanonicalHeaders_Test.test_netloc_port_is_kept_for_non_standard_port _____________________________________________________________________________

self = <requests_aws4auth.test.test_requests_aws4auth.AWS4Auth_GetCanonicalHeaders_Test testMethod=test_netloc_port_is_kept_for_non_standard_port>

    def test_netloc_port_is_kept_for_non_standard_port(self):
        """
        When urllib3 is used, the Host header is not part of the prepared request,
        but generated later, and the port is kept in the header if it is not the
        standard HTTPS port. d190dcb has a bug that also strips non-standard ports
        from the signature, causing signature and host header to mismatch. This is
        a regression test for that bug.
    
        """
        req = requests.Request('GET', 'https://amazonaws.com:8443')
        preq = req.prepare()
        self.assertNotIn('host', preq.headers)
        result = AWS4Auth.get_canonical_headers(preq, include=['host'])
        cano_hdrs, signed_hdrs = result
        expected = 'host:amazonaws.com:8443\n'
>       self.assertEqual(cano_hdrs, expected)
E       AssertionError: 'host:amazonaws.com\n' != 'host:amazonaws.com:8443\n'
E       - host:amazonaws.com
E       + host:amazonaws.com:8443
E       ?                   +++++

requests_aws4auth/test/test_requests_aws4auth.py:977: AssertionError

..which is exactly the issue others complained about in #34, I fixed in 8e1417 and that it then turned out another user relied on in #65.

See tedder#65 and tedder#34. This adds a failing test that would be fixed by 8e1417.
@tedder

tedder commented May 15, 2023

Copy link
Copy Markdown
Owner

Phillip, this is great, thanks. It fails as we expect and communicates it very well.

@tedder

tedder commented May 15, 2023

Copy link
Copy Markdown
Owner

Merging but I won't cut a release for it.

@tedder
tedder merged commit 9e9e7bf into tedder:main May 15, 2023
tedder added a commit that referenced this pull request Jul 27, 2026
The Host header was signed with the port stripped unconditionally, while
the request going out keeps the port whenever it is not the default for
the scheme. The two disagreed, so the service computed a different
signature and rejected the request with SignatureDoesNotMatch.

This is #79, and before it #34. It was fixed once in 8e1417c (#63) and
reverted in b328b86 after #65 reported a regression.

Two changes, which together make the divergence unrepresentable:

- get_host_header() computes the value the client will send: strip the
  port only when it is the default for that scheme. 443 is default for
  https but not for http, so http://host:443 keeps its port. IPv6
  literals are no longer split on their first colon, which previously
  reduced https://[::1] to a single "[" regardless of any port.

- __call__ now sets the Host header instead of only signing it. The wire
  value is byte-identical either way, but signed value and sent value
  become the same string, so they cannot drift apart again if urllib3
  ever changes how it derives the header.

Verified against real SigV4 implementations on non-default ports, which
is something AWS itself cannot test: every AWS endpoint is on 443, so
this code path is unreachable there. RadosGW on 7480 and MinIO on 9111
both reject the old signature and accept the new one. Real AWS S3 is
unaffected, including URLs with an explicit :443.

Also flips the assertion added in #68, which was inverted: it asserted
the signature did not match the wire, documenting the bug rather than
testing the behaviour, while its httpx twin asserted the opposite for
the same URL.

Fixes #79
Fixes #34

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKqoiAcwVRjYpwpDKcPqDQ
tedder added a commit that referenced this pull request Jul 27, 2026
The Host header was signed with the port stripped unconditionally, while
the request going out keeps the port whenever it is not the default for
the scheme. The two disagreed, so the service computed a different
signature and rejected the request with SignatureDoesNotMatch.

This is #79, and before it #34. It was fixed once in 8e1417c (#63) and
reverted in b328b86 after #65 reported a regression.

Two changes, which together make the divergence unrepresentable:

- get_host_header() computes the value the client will send: strip the
  port only when it is the default for that scheme. 443 is default for
  https but not for http, so http://host:443 keeps its port. IPv6
  literals are no longer split on their first colon, which previously
  reduced https://[::1] to a single "[" regardless of any port.
  Credentials are stripped, since they sit in the netloc but are never
  sent in the Host header -- the old code signed everything before the
  first colon ('user'), and 8e1417c would have signed them along with
  the port. Neither matches the wire.

- __call__ now sets the Host header instead of only signing it. The wire
  value is byte-identical either way, but signed value and sent value
  become the same string, so they cannot drift apart again if urllib3
  ever changes how it derives the header.

Verified against real SigV4 implementations on non-default ports, which
is something AWS itself cannot test: every AWS endpoint is on 443, so
this code path is unreachable there. RadosGW on 7480 and MinIO on 9111
both reject the old signature and accept the new one. Real AWS S3 is
unaffected, including URLs with an explicit :443.

The host header table gains the credential-bearing URL shapes, and a new
test checks them against a real socket rather than trusting the oracle:
the oracle models requests stripping credentials before it opens the
connection, and that model should itself be tested. Without the fix it
reports "AssertionError: 'user' != '127.0.0.1:52809'".

Also flips the assertion added in #68, which was inverted: it asserted
the signature did not match the wire, documenting the bug rather than
testing the behaviour, while its httpx twin asserted the opposite for
the same URL.

Fixes #79
Fixes #34

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKqoiAcwVRjYpwpDKcPqDQ
tedder added a commit that referenced this pull request Oct 9, 2026
The Host header was signed with the port stripped unconditionally, while
the request going out keeps the port whenever it is not the default for
the scheme. The two disagreed, so the service computed a different
signature and rejected the request with SignatureDoesNotMatch.

This is #79, and before it #34. It was fixed once in 8e1417c (#63) and
reverted in b328b86 after #65 reported a regression.

Two changes, which together make the divergence unrepresentable:

- get_host_header() computes the value the client will send: strip the
  port only when it is the default for that scheme. 443 is default for
  https but not for http, so http://host:443 keeps its port. IPv6
  literals are no longer split on their first colon, which previously
  reduced https://[::1] to a single "[" regardless of any port.
  Credentials are stripped, since they sit in the netloc but are never
  sent in the Host header -- the old code signed everything before the
  first colon ('user'), and 8e1417c would have signed them along with
  the port. Neither matches the wire.

- __call__ now sets the Host header instead of only signing it. The wire
  value is byte-identical either way, but signed value and sent value
  become the same string, so they cannot drift apart again if urllib3
  ever changes how it derives the header.

Verified against real SigV4 implementations on non-default ports, which
is something AWS itself cannot test: every AWS endpoint is on 443, so
this code path is unreachable there. RadosGW on 7480 and MinIO on 9111
both reject the old signature and accept the new one. Real AWS S3 is
unaffected, including URLs with an explicit :443.

The host header table gains the credential-bearing URL shapes, and a new
test checks them against a real socket rather than trusting the oracle:
the oracle models requests stripping credentials before it opens the
connection, and that model should itself be tested. Without the fix it
reports "AssertionError: 'user' != '127.0.0.1:52809'".

Also flips the assertion added in #68, which was inverted: it asserted
the signature did not match the wire, documenting the bug rather than
testing the behaviour, while its httpx twin asserted the opposite for
the same URL.

Fixes #79
Fixes #34

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GKqoiAcwVRjYpwpDKcPqDQ
@tedder tedder mentioned this pull request Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants