SignatureDoesNotMatch when endpoint URL contains a port

Hi,

I am facing an issue when sending requests directly to a RadosGW instance running on the default port 7480.

All requests fail with the following error: SignatureDoesNotMatch

After some investigation, I found that the issue appears to be related to how the Host header is handled during signature calculation.

In requests-aws4auth, the port is stripped from the Host value when generating the signature:

requests-aws4auth/requests_aws4auth/aws4auth.py

Line 610 in 3dda9dc

headers['host'] = urlparse(str(req.url)).netloc.split(':')[0]

However, the actual HTTP request still follows RFC behavior and includes the port in the Host header. This results in a mismatch between the signed value and the value sent over the wire, causing the SignatureDoesNotMatch error.

I am wondering why the port is removed from the Host header during signature generation instead of using the value from the request URL directly, for example:

urlparse(str(req.url)).netloc

Adding the Host header manually before sending the request resolves the issue:

headers = {
    "Host": urlparse(url).netloc
}

response = requests.request(
    method=http_method.upper(),
    url=url,
    auth=self.auth,
    params=params,
    verify=self.verify,
    headers=headers
)

Could you please clarify whether stripping the port is intentional? If not, would it make sense to preserve the port from the request URL when calculating the signature?

It works in httpx but not in requests today. See #34 #63 #65 #68 for the history. I think the assertNotEqual in 8e1417c (from #68) is actually inverted. The real trick is if we can identify the actual errors (and create unit tests for them) and make sure the fix encompasses all of them. I'll put an AI summary next; if you've got time it would be a great project to read through the issues and PRs to make sure it is correct.

AI summary of this issue and its history (#34, #63, #65, #68), with everything verified by running it rather than inferred from the threads. All claims below are reproducible with the scripts at the bottom.

Short version

The diagnosis in this issue is correct, and this is a rediscovery of #34 (2017). The fix already exists — 8e1417c from #63 — and was reverted in b328b86 over the regression reported in #65. It has been stalled there since.

The assertNotEqual in #68's test_netloc_port_is_kept_for_non_standard_port is indeed inverted. Its httpx twin asserts assertEqual on the same URL, so the suite currently asserts that the two clients must disagree — which is why 78 green tests can't see this bug.

Reproduced against real servers

RadosGW (quay.io/ceph/demo), the same server and port as this issue:

test	result
list buckets with `requests`	FAIL — `SignatureDoesNotMatch`
list buckets with `httpx`	pass
both clients sign identically	FAIL
explicit `Host` header (the workaround)	pass

MinIO on port 9111, same story:

request	HTTP	code
HEAD behavior (library computes host)	403	`SignatureDoesNotMatch`
host including port	200	OK

These are independent implementations of the spec rejecting the signature — not our model of http.client disagreeing with itself. Neither needs AWS credentials, and both start in well under a minute.

Where HEAD is wrong

http.client (what requests/urllib3 uses underneath) is ground truth for what goes on the wire:

URL	on the wire	HEAD signs	`8e1417c` signs
`https://h`	`h`	✅ `h`	✅ `h`
`http://h`	`h`	✅ `h`	✅ `h`
`https://h:443`	`h`	✅ `h`	✅ `h`
`https://h:8443`	`h:8443`	❌ `h`	✅ `h:8443`
`http://h:80`	`h`	✅ `h`	✅ `h`
`http://h:7480`	`h:7480`	❌ `h`	✅ `h:7480`
`https://h:80`	`h:80`	❌ `h`	✅ `h:80`
`http://h:443`	`h:443`	❌ `h`	✅ `h:443`
`https://[::1]:8443`	`[::1]:8443`	❌ `[`	✅ `[::1]:8443`
`https://[::1]:443`	`[::1]`	❌ `[`	✅ `[::1]`
`https://[::1]`	`[::1]`	❌ `[`	✅ `[::1]`

HEAD is wrong on 7 of 11. 8e1417c is correct on all 11. Row 6 is this issue.

Why only `requests` is affected

aws4auth.py:610 only runs when 'host' not in headers. httpx's Request._prepare() populates Host itself, correctly, port and all — so the library passes it through untouched. requests' PreparedRequest leaves Host unset (urllib3 adds it at send time), so the library computes it, and .netloc.split(':')[0] drops the port unconditionally.

Same URL, different signature, depending on which HTTP client is used.

Two bugs in here that aren't in any issue

IPv6 literals are entirely broken, ports aside. .netloc.split(':')[0] on [::1] returns [ — a single open bracket. Even https://[::1] with no port signs host:[. 8e1417c fixes this incidentally.

Cross-scheme ports. https://h:80 and http://h:443 are non-default for their scheme, so the port does go on the wire. The one-liner proposed in #34's comments ("check if the port is 80 or 443") gets both wrong; 8e1417c's scheme-aware check gets them right. Worth flagging before anyone lands the obvious-looking patch.

Fixing it automatically, without a flag

Two mechanisms that compose, and together remove the need for an opt-in:

A. Compute the host the way the client will. This is 8e1417c: strip the port only when it is the scheme's default. Correct on all 11 rows above, and it changes nothing about what is sent over the wire. Its one durable weakness is that it's a reimplementation of http.client's rule, so it has to stay in sync with urllib3 indefinitely.

B. Then set the Host header rather than only signing it. Once the value is computed, write it onto the prepared request. Verified: the wire header is byte-identical either way, so this changes nothing about what is transmitted — but signature and wire can no longer diverge, because they become the same string by construction. This is what makes A's sync risk stop mattering, and it closes the whole class of bug rather than the three instances currently known.

The caveat on B is redirects: requests copies headers when rebuilding a redirected request, so an explicit Host would follow to the new target. SigV4 across redirects is already broken for other reasons, so this is likely theoretical, but it should be tested rather than assumed.

C. A flag, only if A+B actually breaks someone. If needed, sign_host_port= (explicit intent) or legacy_strip_port= (names it as legacy), either defaulting to the correct behavior and documented as a compatibility shim rather than a supported mode. Worth keeping in reserve rather than shipping preemptively — every flag here is a permanently supported second code path.

The open question: was #65 real?

This is what blocked the fix, so it deserves scrutiny — and from first principles the report looks inverted.

The setup in #65 was a tunnel from localhost:9200 to a managed OpenSearch endpoint on 443. Through a tunnel the server receives Host: localhost:9200 on the wire regardless of what gets signed, and SigV4 verification recomputes from the received headers. So signing host:localhost:9200 (post-fix) should have matched, and signing host:localhost (pre-fix) should have failed — the opposite of what was reported. Phillip Berndt reached the same read in #65 at the time.

One new piece of evidence, against real AWS S3 (read-only ListBuckets, us-east-1):

case	result
`https://s3.amazonaws.com/` (no port)	200
`https://s3.amazonaws.com:443/`, HEAD behavior	200
`https://s3.amazonaws.com:443/`, fixed behavior	200
`https://s3.amazonaws.com:443/`, signing `host:s3.amazonaws.com:443`	200

That last row is the informative one: S3 does not check that Host matches its own hostname. It only checks that the signature matches the Host actually sent. Any self-consistent Host is accepted. That is exactly the invariant A+B delivers, so the fix is safe against real AWS — and it is consistent with #65 having been misdiagnosed, since a self-consistent host:localhost:9200 should have been accepted.

Not proof: that was OpenSearch through a tunnel, not S3 directly. A SignedHeaders line from a failing request would settle it. A+B makes it moot either way, since it signs exactly what is sent.

Test approach: a wire-truth oracle

The reason a wrong expectation got committed in #68 is that each case was hand-written, so someone had to decide the right answer per case, and nothing cross-checked them.

The alternative is to not hardcode answers at all — derive the expected value by asking http.client what it would put on the wire for that URL, then assert the signature matches:

def test_signed_host_matches_wire_using_requests(self):
    for url in HOST_HEADER_URLS:
        with self.subTest(url=url):
            preq = requests.Request('GET', url).prepare()
            self.assertEqual(self.canonical_host(preq), wire_host_header(url))

This encodes the invariant that actually matters, and it can't go stale if http.client's default-port rule ever changes. Written against HEAD, over the 11 URLs above plus caller-supplied Host and malformed ports:

Ran 83 tests
FAILED (failures=14, skipped=2)

..._using_requests — 7 failures, exactly the ❌ rows above
..._using_httpx — passes all 11
test_requests_and_httpx_sign_the_same_host — the same 7
test_caller_supplied_host_is_used_verbatim — passes
test_malformed_ports_are_rejected_before_signing — passes

All 78 pre-existing tests still pass, so this is purely additive. The 14 failures are the enumerated bug list; A+B should turn all of them green along with the RadosGW and MinIO failures above.

On malformed ports: urlparse().port raises ValueError on :notaport and :99999, which looked like it could surface from inside the auth handler. In practice both clients reject those URLs while building the request, before auth ever runs, so the signing path never sees them. Worth a defensive guard, but not a live defect.

Integration testing in CI

Real AWS can't catch this class of bug — every endpoint is on 443, so the non-default-port path is unreachable, and test_live.py hardcodes 'https://' + path_qs with no port. That's a large part of why this survived nine years.

A matrix job running MinIO and RadosGW as containers on non-default ports covers it, with no credentials and no cost. The tests skip when AWS4_TEST_S3_ENDPOINT is unset, so local runs are unaffected.

One trap worth writing down: Ceph's demo entrypoint seds the secret key into a config file, so CEPH_DEMO_SECRET_KEY must not contain /. The standard AWS example key does, and the container exits 1 with sed: -e expression #1, char 44: unknown option to 's' after it looks like it booted successfully.

Repro

Wire truth vs HEAD vs 8e1417c, no dependencies

import http.client, io
from urllib.parse import urlparse

class FakeSock(io.BytesIO):
    def sendall(self, b): self.write(b)
    def makefile(self, *a, **k): return io.BytesIO(b"")
    def close(self): pass

def wire_host(url):
    p = urlparse(url)
    cls = http.client.HTTPSConnection if p.scheme == "https" else http.client.HTTPConnection
    c = cls(p.netloc); c.sock = FakeSock()
    c.putrequest("GET", "/"); c.endheaders()
    for line in c.sock.getvalue().decode().split("\r\n"):
        if line.lower().startswith("host:"): return line[6:]

def head(url):   # aws4auth.py:610 as it stands
    return urlparse(url).netloc.split(':')[0]

def fixed(url):  # 8e1417c
    p = urlparse(url); netloc = p.netloc
    if (p.port == 80 and p.scheme == 'http') or (p.port == 443 and p.scheme == 'https'):
        netloc = netloc.rsplit(":", 1)[0]
    return netloc

for u in ["https://h", "http://h", "https://h:443", "https://h:8443",
          "http://h:80", "http://h:7480", "https://h:80", "http://h:443",
          "https://[::1]:8443", "https://[::1]:443", "https://[::1]"]:
    w = wire_host(u)
    print(f"{u:22} wire={w:14} head={head(u):14}{'' if head(u)==w else ' <BAD'}"
          f"  fixed={fixed(u):14}{'' if fixed(u)==w else ' <BAD'}")

End-to-end against a local socket on port 7480

import threading, http.server, requests
from requests_aws4auth import AWS4Auth

received = {}
class H(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        received['host'] = self.headers.get('Host')
        self.send_response(200); self.end_headers(); self.wfile.write(b'ok')
    def log_message(self, *a): pass

srv = http.server.HTTPServer(('127.0.0.1', 7480), H)
threading.Thread(target=srv.serve_forever, daemon=True).start()

url = 'http://127.0.0.1:7480/bucket'
requests.get(url, auth=AWS4Auth('AKIDEXAMPLE', 'secret', 'us-east-1', 's3'))
preq = requests.Request('GET', url).prepare()

print("wire Host header      :", received['host'])
print("library signs host as :", AWS4Auth.get_canonical_headers(preq, include=['host'])[0].strip())
srv.shutdown()

Against RadosGW — the server in this issue

docker run -d --rm --name rgw -p 7480:8080 \
  -e CEPH_DAEMON=demo -e RGW_NAME=localhost -e MON_IP=127.0.0.1 \
  -e CEPH_PUBLIC_NETWORK=0.0.0.0/0 -e CEPH_DEMO_UID=demo \
  -e CEPH_DEMO_ACCESS_KEY=demoaccesskey123456 \
  -e CEPH_DEMO_SECRET_KEY=demosecretkey1234567890abcdefghijklmnop \
  quay.io/ceph/demo:latest-reef
# add --platform linux/amd64 on Apple Silicon; ~40s to come up

import re, requests
from requests_aws4auth import AWS4Auth

auth = AWS4Auth('demoaccesskey123456',
                'demosecretkey1234567890abcdefghijklmnop', 'us-east-1', 's3')
r = requests.get('http://127.0.0.1:7480/', auth=auth)
print(r.status_code, re.search(r'<Code>(.*?)</Code>', r.text) or 'OK')

MinIO is a lighter substitute if Ceph is too heavy:

docker run -d --rm -p 9111:9000 \
  -e MINIO_ROOT_USER=demoaccesskey123456 \
  -e MINIO_ROOT_PASSWORD=demosecretkey1234567890abcdefghijklmnop \
  minio/minio server /data

Generated by Claude (Opus 5). Edited to correct an error in the first version: it said HEAD was wrong on "5 of 11" where the count is 7 — the table was right, the summary sentence was not. Also corrected the malformed-port note, which the first version overstated as a live defect in 8e1417c; both HTTP clients reject those URLs before auth runs. The behavioral claims and every table are reproducible with the scripts above; the #65 analysis is reasoning plus indirect evidence, not a direct measurement of that setup, and remains the part most worth a second pair of eyes.

fix: sign the Host header that is actually sent

Feel free to test the PR branch of #81 for your use case.

Works for me
Tested on custom port and on default https with 443 specified

fix: stop pinning the Host header on the request

@dreznichuk

Update on #81: I removed one part of the fix, so it needs a retest.

The first version did two things: computed the correct host to sign (port kept unless it is the scheme's default), and also wrote that value into the request's Host header. The second part caused a regression. When requests follows a redirect to a different host, it drops Authorization but copies every other header, so the redirect target received the original host instead of its own. The current release doesn't have this problem.

f0623a7 removes the header pinning and adds a regression test for the redirect case. The host computation that fixes this issue is unchanged, and the RadosGW integration job on port 7480 still passes.

@dreznichuk, since the code you tested has changed, could you retest the current ted/issue-79-fix branch on your custom-port and explicit-:443 setups?

Hi

fresh tests on branch ted/issue-79-fix

custom port still works
explicit 443 with https - works also

tested on 1.3.2 to ensure that my venv is clean.
still dont work, as expected

great- I'll release it.

@dreznichuk

Released in v1.4.0 (PyPI, release notes). Thanks for testing, @dreznichuk.

I checked the package installed from PyPI, against 1.3.2:

check	1.4.0	1.3.2
RadosGW on 7480 (this issue's server)	pass	`SignatureDoesNotMatch`
MinIO on 9111	pass	`SignatureDoesNotMatch`
IPv6 `http://[::1]:<port>` on both	200	403 `SignatureDoesNotMatch`
real S3, with and without `:443`	200	200

The #65 regression that got this fix reverted in 2023 is written up on #65. Reproduced through a tunnel to S3, 1.4.0 works and 1.3.2 fails, the opposite of that report.

fix: sign the Host header that is actually sent

fix: stop pinning the Host header on the request

Navigation Menu

Repository navigation

SignatureDoesNotMatch when endpoint URL contains a port #79

Short version

Reproduced against real servers

Where HEAD is wrong

Why only `requests` is affected

Two bugs in here that aren't in any issue

Fixing it automatically, without a flag

The open question: was #65 real?

Test approach: a wire-truth oracle

Integration testing in CI

Repro

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

SignatureDoesNotMatch when endpoint URL contains a port #79

Description

Activity

tedder commented on Jul 27, 2026

tedder commented on Jul 27, 2026

Short version

Reproduced against real servers

Where HEAD is wrong

Why only requests is affected

Two bugs in here that aren't in any issue

Fixing it automatically, without a flag

The open question: was #65 real?

Test approach: a wire-truth oracle

Integration testing in CI

Repro

tedder commented on Jul 27, 2026

dreznichuk commented on Jul 28, 2026

tedder commented on Oct 4, 2026

dreznichuk commented on Oct 9, 2026

tedder commented on Oct 9, 2026

tedder commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Why only `requests` is affected