Every Outbound HTTP Call Needs a Timeout

One Monday morning, a scheduled job that sends stock price notifications hit a 5.8% error rate in production. Every single error was the same: a call to our market-data provider dying with a socket hang-up or a dropped TLS connection.

The same week, I found a request path where a user editing their phone number or address could be held open by a call to another internal API with no limit at all.

A few weeks before that, I'd found a job whose configured 15-second timeout was actually 5 seconds.

Different services, different symptoms, same root: nobody had decided how long an outbound call was allowed to take. The library had decided for us.

The cron that opened 500 connections at once

The notification job needed quotes for every symbol it was about to push. It fired one request per symbol, up to 500 at once, each on a fresh TLS connection, with no timeout.

Most mornings that worked. Then one morning it didn't. In one burst, 46 requests started within 100ms of each other and hung for about 21 seconds together before failing.

Nothing about that was a decision. The per-symbol fan-out, the fresh connection per call and the missing timeout were all just what the code did by default. For many Node HTTP clients, the default timeout is "wait forever."

The fix had four parts:

• Batch the symbols. The provider accepts many symbols per request, so 500 calls became at most 5.
• Keep-alive. Reuse connections instead of a TLS handshake per call.
• A 10-second timeout. Explicit, in the client config, where the next person will see it.
• Host failover. The provider exposes a second host, so a failed call retries there.

Any one of those would have helped. Together, the job went from hundreds of concurrent connections to a handful of calls with a ceiling on how long each can hang.

The user request with no ceiling

The second one was quieter. When a member updated their profile, phone or address, our service made an inline call to another internal API to keep it in sync. That call had no timeout at all, so the member's request waited for as long as the other service did.

I didn't just add a timeout there. The sync didn't need to block the user in the first place. The change now publishes a message to the service's own RabbitMQ queue and returns. The delivery side has a 5-second cap. None of the six callers had to change.

That's worth calling out: sometimes the right timeout on a request path is "don't make this call on the request path."

The 15-second timeout that was really 5

This was the one that surprised me most.

A daily job made HTTP calls to an internal API with axios, configured with a 15-second timeout. It kept failing with socket hang up, at around 5 seconds rather than 15.

The config set an httpsAgent. The call used http://. So axios ignored the custom agent and every request fell through to Node's global HTTP agent, which (since Node 19) carries its own 5-second socket timeout. The global agent destroyed the socket before axios's timer ever fired, and reported it as a hang-up, not a timeout.

The configured timeout was dead code on that path. It looked right in review. It just never ran.

The first fix removed that ceiling and exposed a second fault. The upstream service never set server.keepAliveTimeout, so it closed idle sockets at Node's 5-second default, and now we were holding pooled sockets longer than that and reusing ones the server had already dropped. Since the job runs once a day over an in-cluster hop, I turned pooling off for it. A handshake per request is noise at that volume.

The next scheduled run had zero connection errors. The same workload had produced 88 on one day and 48 the next.

Measure the shape, not the count

That job had actually been through three fixes, each hiding the next. Before the agent issue, the pod's CPU limit was low enough that throttling stalled the Node event loop for up to 78 seconds, so a 15-second timer could never fire on time.

What made each layer findable was looking at the duration of failures, not how many there were. Failure durations went from about 479,000ms, to 5,600ms, to 2,600ms, to none. The error count said "still broken" all three times. The shape of the failure changed every time, and each change was the next clue.

A fix that changes how something fails without removing the failure isn't a failed fix. It's evidence.

What I do now

When I add or review an outbound client, I check these explicitly:

import http from 'node:http';
import https from 'node:https';
import axios from 'axios';

export const client = axios.create({
  baseURL: process.env.UPSTREAM_URL,
  timeout: 10_000, // a decision, written down
  // set the agent that matches the URL scheme, or set both
  httpAgent: new http.Agent({ keepAlive: true }),
  httpsAgent: new https.Agent({ keepAlive: true }),
});

And a few questions that go beyond the config:

1. Is there an explicit timeout? If the answer is "whatever the library does," that's not an answer.
2. Is the configured timeout the one that actually applies? Agents, proxies and the platform can each impose their own. Test that a slow upstream fails at the number you wrote.
3. How many of these calls can be in flight at once? Unbounded fan-out turns one slow dependency into hundreds of hung sockets.
4. Does this call need to be on the request path at all? If the user doesn't need the result, put it on a queue.
5. Do keep-alive settings agree on both sides? A client holding sockets longer than the server keeps them open will reuse dead connections.

Takeaways

• No timeout is still a choice. It's just one nobody made on purpose.
• A configured timeout isn't proof. Verify which layer actually enforces it.
• Batch before you parallelize. 500 calls that could be 5 is a design bug, not a tuning problem.
• When a fix changes the failure's shape, read the new shape. It's pointing at the next layer.
• If two services each learn the same lesson about the same provider, that's a sign the client should be shared. We ended up with host failover written separately in two places, which is exactly how the third service learns it again.