Skip to content

Commit cb54b3c

Browse files
joyeecheungjoyeecheung
authored and
aduh95
committed
http: add http.setGlobalProxyFromEnv()
This adds an API to dynamically enable built-in proxy support for all of fetch() and http.request()/https.request(), so that users do not have to be aware of them all and configure them one by one. PR-URL: #60953 Reviewed-By: Matteo Collina <matteo.collina@gmail.com> Reviewed-By: Tim Perry <pimterry@gmail.com>
1 parent 407807b commit cb54b3c

File tree

25 files changed

+807
-44
lines changed

25 files changed

+807
-44
lines changed

‎doc/api/http.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4338,6 +4338,32 @@ added:
43384338
43394339
Set the maximum number of idle HTTP parsers.
43404340
4341+
## `http.setGlobalProxyFromEnv([proxyEnv])`
4342+
4343+
<!-- YAML
4344+
added:
4345+
- REPLACEME
4346+
-->
4347+
4348+
* `proxyEnv` {Object} An object containing proxy configuration. This accepts the
4349+
same options as the `proxyEnv` option accepted by [`Agent`][]. **Default:**
4350+
`process.env`.
4351+
* Returns: {Function} A function that restores the original agent and dispatcher
4352+
settings to the state before this `http.setGlobalProxyFromEnv()` is invoked.
4353+
4354+
Dynamically resets the global configurations to enable built-in proxy support for
4355+
`fetch()` and `http.request()`/`https.request()` at runtime, as an alternative
4356+
to using the `--use-env-proxy` flag or `NODE_USE_ENV_PROXY` environment variable.
4357+
It can also be used to override settings configured from the environment variables.
4358+
4359+
As this function resets the global configurations, any previously configured
4360+
`http.globalAgent`, `https.globalAgent` or undici global dispatcher would be
4361+
overridden after this function is invoked. It's recommended to invoke it before any
4362+
requests are made and avoid invoking it in the middle of any requests.
4363+
4364+
See [Built-in Proxy Support][] for details on proxy URL formats and `NO_PROXY`
4365+
syntax.
4366+
43414367
## Class: `WebSocket`
43424368
43434369
<!-- YAML
@@ -4358,6 +4384,9 @@ added: v24.5.0
43584384
When Node.js creates the global agent, if the `NODE_USE_ENV_PROXY` environment variable is
43594385
set to `1` or `--use-env-proxy` is enabled, the global agent will be constructed
43604386
with `proxyEnv: process.env`, enabling proxy support based on the environment variables.
4387+
4388+
To enable proxy support dynamically and globally, use [`http.setGlobalProxyFromEnv()`][].
4389+
43614390
Custom agents can also be created with proxy support by passing a
43624391
`proxyEnv` option when constructing the agent. The value can be `process.env`
43634392
if they just want to inherit the configuration from the environment variables,
@@ -4413,6 +4442,86 @@ Or the `--use-env-proxy` flag.
44134442
HTTP_PROXY=http://proxy.example.com:8080 NO_PROXY=localhost,127.0.0.1 node --use-env-proxy client.js
44144443
```
44154444
4445+
To enable proxy support dynamically and globally with `process.env` (the default option of `http.setGlobalProxyFromEnv()`):
4446+
4447+
```cjs
4448+
const http = require('node:http');
4449+
4450+
// Reads proxy-related environment variables from process.env
4451+
const restore = http.setGlobalProxyFromEnv();
4452+
4453+
// Subsequent requests will use the configured proxies from environment variables
4454+
http.get('http://www.example.com', (res) => {
4455+
// This request will be proxied if HTTP_PROXY or http_proxy is set
4456+
});
4457+
4458+
fetch('https://www.example.com', (res) => {
4459+
// This request will be proxied if HTTPS_PROXY or https_proxy is set
4460+
});
4461+
4462+
// To restore the original global agent and dispatcher settings, call the returned function.
4463+
// restore();
4464+
```
4465+
4466+
```mjs
4467+
import http from 'node:http';
4468+
4469+
// Reads proxy-related environment variables from process.env
4470+
http.setGlobalProxyFromEnv();
4471+
4472+
// Subsequent requests will use the configured proxies from environment variables
4473+
http.get('http://www.example.com', (res) => {
4474+
// This request will be proxied if HTTP_PROXY or http_proxy is set
4475+
});
4476+
4477+
fetch('https://www.example.com', (res) => {
4478+
// This request will be proxied if HTTPS_PROXY or https_proxy is set
4479+
});
4480+
4481+
// To restore the original global agent and dispatcher settings, call the returned function.
4482+
// restore();
4483+
```
4484+
4485+
To enable proxy support dynamically and globally with custom settings:
4486+
4487+
```cjs
4488+
const http = require('node:http');
4489+
4490+
const restore = http.setGlobalProxyFromEnv({
4491+
http_proxy: 'http://proxy.example.com:8080',
4492+
https_proxy: 'https://proxy.example.com:8443',
4493+
no_proxy: 'localhost,127.0.0.1,.internal.example.com',
4494+
});
4495+
4496+
// Subsequent requests will use the configured proxies
4497+
http.get('http://www.example.com', (res) => {
4498+
// This request will be proxied through proxy.example.com:8080
4499+
});
4500+
4501+
fetch('https://www.example.com', (res) => {
4502+
// This request will be proxied through proxy.example.com:8443
4503+
});
4504+
```
4505+
4506+
```mjs
4507+
import http from 'node:http';
4508+
4509+
http.setGlobalProxyFromEnv({
4510+
http_proxy: 'http://proxy.example.com:8080',
4511+
https_proxy: 'https://proxy.example.com:8443',
4512+
no_proxy: 'localhost,127.0.0.1,.internal.example.com',
4513+
});
4514+
4515+
// Subsequent requests will use the configured proxies
4516+
http.get('http://www.example.com', (res) => {
4517+
// This request will be proxied through proxy.example.com:8080
4518+
});
4519+
4520+
fetch('https://www.example.com', (res) => {
4521+
// This request will be proxied through proxy.example.com:8443
4522+
});
4523+
```
4524+
44164525
To create a custom agent with built-in proxy support:
44174526
44184527
```cjs
@@ -4476,6 +4585,7 @@ const agent2 = new http.Agent({ proxyEnv: process.env });
44764585
[`http.get()`]: #httpgetoptions-callback
44774586
[`http.globalAgent`]: #httpglobalagent
44784587
[`http.request()`]: #httprequestoptions-callback
4588+
[`http.setGlobalProxyFromEnv()`]: #httpsetglobalproxyfromenvproxyenv
44794589
[`message.headers`]: #messageheaders
44804590
[`message.rawHeaders`]: #messagerawheaders
44814591
[`message.socket`]: #messagesocket

‎lib/_http_agent.js‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ const {
4040
kProxyConfig,
4141
checkShouldUseProxy,
4242
kWaitForProxyTunnel,
43-
filterEnvForProxies,
43+
getGlobalAgent,
4444
} = require('internal/http');
4545
const { AsyncResource } = require('async_hooks');
4646
const { async_id_symbol } = require('internal/async_hooks').symbols;
@@ -627,9 +627,5 @@ function asyncResetHandle(socket) {
627627

628628
module.exports = {
629629
Agent,
630-
globalAgent: new Agent({
631-
keepAlive: true, scheduling: 'lifo', timeout: 5000,
632-
// This normalized from both --use-env-proxy and NODE_USE_ENV_PROXY settings.
633-
proxyEnv: getOptionValue('--use-env-proxy') ? filterEnvForProxies(process.env) : undefined,
634-
}),
630+
globalAgent: getGlobalAgent(getOptionValue('--use-env-proxy') ? process.env : undefined, Agent),
635631
};

‎lib/http.js‎

Lines changed: 60 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,11 +25,12 @@ const {
2525
ObjectDefineProperty,
2626
} = primordials;
2727

28-
const { validateInteger } = require('internal/validators');
28+
const { validateInteger, validateObject } = require('internal/validators');
2929
const httpAgent = require('_http_agent');
3030
const { ClientRequest } = require('_http_client');
3131
const { methods, parsers } = require('_http_common');
3232
const { IncomingMessage } = require('_http_incoming');
33+
const { ERR_PROXY_INVALID_CONFIG } = require('internal/errors').codes;
3334
const {
3435
validateHeaderName,
3536
validateHeaderValue,
@@ -41,6 +42,11 @@ const {
4142
Server,
4243
ServerResponse,
4344
} = require('_http_server');
45+
const {
46+
parseProxyUrl,
47+
getGlobalAgent,
48+
} = require('internal/http');
49+
const { URL } = require('internal/url');
4450
let maxHeaderSize;
4551
let undici;
4652

@@ -123,6 +129,58 @@ function lazyUndici() {
123129
return undici ??= require('internal/deps/undici/undici');
124130
}
125131

132+
function setGlobalProxyFromEnv(env = process.env) {
133+
validateObject(env, 'proxyEnv');
134+
const httpProxy = parseProxyUrl(env, 'http:');
135+
const httpsProxy = parseProxyUrl(env, 'https:');
136+
const noProxy = env.no_proxy || env.NO_PROXY;
137+
138+
if (!httpProxy && !httpsProxy) {
139+
return () => {};
140+
}
141+
142+
if (httpProxy && !URL.canParse(httpProxy)) {
143+
throw new ERR_PROXY_INVALID_CONFIG(httpProxy);
144+
}
145+
if (httpsProxy && !URL.canParse(httpsProxy)) {
146+
throw new ERR_PROXY_INVALID_CONFIG(httpsProxy);
147+
}
148+
149+
let originalDispatcher, originalHttpsAgent, originalHttpAgent;
150+
if (httpProxy || httpsProxy) {
151+
// Set it for fetch.
152+
const { setGlobalDispatcher, getGlobalDispatcher, EnvHttpProxyAgent } = lazyUndici();
153+
const envHttpProxyAgent = new EnvHttpProxyAgent({
154+
__proto__: null, httpProxy, httpsProxy, noProxy,
155+
});
156+
originalDispatcher = getGlobalDispatcher();
157+
setGlobalDispatcher(envHttpProxyAgent);
158+
}
159+
160+
if (httpProxy) {
161+
originalHttpAgent = module.exports.globalAgent;
162+
module.exports.globalAgent = getGlobalAgent(env, httpAgent.Agent);
163+
}
164+
if (httpsProxy && !!process.versions.openssl) {
165+
const https = require('https');
166+
originalHttpsAgent = https.globalAgent;
167+
https.globalAgent = getGlobalAgent(env, https.Agent);
168+
}
169+
170+
return function restore() {
171+
if (originalDispatcher) {
172+
const { setGlobalDispatcher } = lazyUndici();
173+
setGlobalDispatcher(originalDispatcher);
174+
}
175+
if (originalHttpAgent) {
176+
module.exports.globalAgent = originalHttpAgent;
177+
}
178+
if (originalHttpsAgent) {
179+
require('https').globalAgent = originalHttpsAgent;
180+
}
181+
};
182+
}
183+
126184
module.exports = {
127185
_connectionListener,
128186
METHODS: methods.toSorted(),
@@ -142,6 +200,7 @@ module.exports = {
142200
validateInteger(max, 'max', 1);
143201
parsers.max = max;
144202
},
203+
setGlobalProxyFromEnv,
145204
};
146205

147206
ObjectDefineProperty(module.exports, 'maxHeaderSize', {

‎lib/https.js‎

Lines changed: 2 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -50,8 +50,8 @@ const tls = require('tls');
5050
const {
5151
kProxyConfig,
5252
checkShouldUseProxy,
53-
filterEnvForProxies,
5453
kWaitForProxyTunnel,
54+
getGlobalAgent,
5555
} = require('internal/http');
5656
const { Agent: HttpAgent } = require('_http_agent');
5757
const {
@@ -602,11 +602,7 @@ Agent.prototype._evictSession = function _evictSession(key) {
602602
delete this._sessionCache.map[key];
603603
};
604604

605-
const globalAgent = new Agent({
606-
keepAlive: true, scheduling: 'lifo', timeout: 5000,
607-
// This normalized from both --use-env-proxy and NODE_USE_ENV_PROXY settings.
608-
proxyEnv: getOptionValue('--use-env-proxy') ? filterEnvForProxies(process.env) : undefined,
609-
});
605+
const globalAgent = getGlobalAgent(getOptionValue('--use-env-proxy') ? process.env : undefined, Agent);
610606

611607
/**
612608
* Makes a request to a secure web server.

‎lib/internal/http.js‎

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -186,11 +186,7 @@ class ProxyConfig {
186186
}
187187
}
188188

189-
function parseProxyConfigFromEnv(env, protocol, keepAlive) {
190-
// We only support proxying for HTTP and HTTPS requests.
191-
if (protocol !== 'http:' && protocol !== 'https:') {
192-
return null;
193-
}
189+
function parseProxyUrl(env, protocol) {
194190
// Get the proxy url - following the most popular convention, lower case takes precedence.
195191
// See https://about.gitlab.com/blog/we-need-to-talk-no-proxy/#http_proxy-and-https_proxy
196192
const proxyUrl = (protocol === 'https:') ?
@@ -204,6 +200,20 @@ function parseProxyConfigFromEnv(env, protocol, keepAlive) {
204200
throw new ERR_PROXY_INVALID_CONFIG(`Invalid proxy URL: ${proxyUrl}`);
205201
}
206202

203+
return proxyUrl;
204+
}
205+
206+
function parseProxyConfigFromEnv(env, protocol, keepAlive) {
207+
// We only support proxying for HTTP and HTTPS requests.
208+
if (protocol !== 'http:' && protocol !== 'https:') {
209+
return null;
210+
}
211+
212+
const proxyUrl = parseProxyUrl(env, protocol);
213+
if (proxyUrl === null) {
214+
return null;
215+
}
216+
207217
// Only http:// and https:// proxies are supported.
208218
// Ignore instead of throw, in case other protocols are supposed to be
209219
// handled by the user land.
@@ -244,6 +254,13 @@ function filterEnvForProxies(env) {
244254
};
245255
}
246256

257+
function getGlobalAgent(proxyEnv, Agent) {
258+
return new Agent({
259+
keepAlive: true, scheduling: 'lifo', timeout: 5000,
260+
proxyEnv,
261+
});
262+
}
263+
247264
module.exports = {
248265
kOutHeaders: Symbol('kOutHeaders'),
249266
kNeedDrain: Symbol('kNeedDrain'),
@@ -257,4 +274,6 @@ module.exports = {
257274
getNextTraceEventId,
258275
isTraceHTTPEnabled,
259276
filterEnvForProxies,
277+
getGlobalAgent,
278+
parseProxyUrl,
260279
};

‎test/client-proxy/test-http-proxy-fetch.mjs‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,7 @@ await once(proxy, 'listening');
2020
const serverHost = `localhost:${server.address().port}`;
2121

2222
// FIXME(undici:4083): undici currently always tunnels the request over
23-
// CONNECT if proxyTunnel is not explicitly set to false, but what we
24-
// need is for it to be automatically false for HTTP requests to be
25-
// consistent with curl.
23+
// CONNECT if proxyTunnel is not explicitly set to false.
2624
const expectedLogs = [{
2725
method: 'CONNECT',
2826
url: serverHost,

‎test/client-proxy/test-http-set-global-proxy-from-env-fetch-default.mjs‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
// Tests that http.setGlobalProxyFromEnv() without arguments uses process.env.
2+
3+
import '../common/index.mjs';
4+
import assert from 'node:assert';
5+
import { startTestServers, checkProxiedFetch } from '../common/proxy-server.js';
6+
7+
const { proxyLogs, proxyUrl, shutdown, httpEndpoint: { serverHost, requestUrl } } = await startTestServers({
8+
httpEndpoint: true,
9+
});
10+
11+
// Test that calling setGlobalProxyFromEnv() without arguments uses process.env
12+
await checkProxiedFetch({
13+
FETCH_URL: requestUrl,
14+
// Set the proxy in the environment instead of passing it to SET_GLOBAL_PROXY
15+
http_proxy: proxyUrl,
16+
SET_GLOBAL_PROXY_DEFAULT: '1', // Signal to call without arguments
17+
}, {
18+
stdout: 'Hello world',
19+
});
20+
21+
shutdown();
22+
23+
// FIXME(undici:4083): undici currently always tunnels the request over
24+
// CONNECT if proxyTunnel is not explicitly set to false.
25+
const expectedLogs = [{
26+
method: 'CONNECT',
27+
url: serverHost,
28+
headers: {
29+
'connection': 'close',
30+
'host': serverHost,
31+
'proxy-connection': 'keep-alive',
32+
},
33+
}];
34+
35+
assert.deepStrictEqual(proxyLogs, expectedLogs);

‎test/client-proxy/test-http-set-global-proxy-from-env-fetch-empty.mjs‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
// Tests that http.setGlobalProxyFromEnv() is a no-op when no proxy is configured.
2+
3+
import '../common/index.mjs';
4+
import assert from 'node:assert';
5+
import { startTestServers, checkProxiedFetch } from '../common/proxy-server.js';
6+
7+
const { proxyLogs, shutdown, httpEndpoint: { requestUrl } } = await startTestServers({
8+
httpEndpoint: true,
9+
});
10+
11+
await checkProxiedFetch({
12+
FETCH_URL: requestUrl,
13+
SET_GLOBAL_PROXY: JSON.stringify({}),
14+
}, {
15+
stdout: 'Hello world',
16+
});
17+
18+
shutdown();
19+
20+
// Verify request did NOT go through proxy.
21+
assert.deepStrictEqual(proxyLogs, []);

0 commit comments

Comments
 (0)