proxy-agents

repository·main·Indexed 22 days ago

https://github.com/tootallnate/proxy-agents

A Node.js monorepo containing multiple HTTP Agent implementations for interacting with various proxy protocols including HTTP, HTTPS, SOCKS, and PAC. The collection includes packages such as http-proxy-agent, https-proxy-agent, agent-base for creating custom agents, get-uri for obtaining readable streams from various URI protocols, data-uri-to-buffer for converting Data URIs to ArrayBuffers, and degenerator for transpiling synchronous-looking code into async functions.

Tokens
20.4K
Snippets
74
Records
93
Agent score
78%

What's inside proxy-agents

  1. Overview of the proxy-agents monorepo

    main

    The proxy-agents monorepo provides a collection of various Node.js HTTP Agent implementations designed to operate over proxies using different protocols. It includes specialized agents for HTTP, HTTPS, SOCKS, and PAC (Proxy Auto-Config) protocols.

    Recommendation: For most standard use cases, you should use the proxy-agent module. This high-level module acts as a wrapper that utilizes the more specialized, low-level agent implementations within the monorepo based on the provided proxy URL.

  2. What is agent-base and how to use it

    main

    The agent-base module provides an abstract base class for creating custom http.Agent instances. It is designed to wrap a connect() function that is responsible for creating the underlying socket (a Duplex stream) used by HTTP client requests.

    To use agent-base, you must extend the Agent class and implement the connect(req, opts) method. This method can be asynchronous and can return either a socket or another http.Agent instance to delegate the request to.

    To support both http and https modules, you should check the secureEndpoint property within the opts parameter of the connect() function. If secureEndpoint is true, the request is targeting an HTTPS endpoint; otherwise, it is targeting an HTTP endpoint.

    import { Agent } from 'agent-base';
    
    class MyAgent extends Agent {
      connect(req, opts) {
        // Implementation logic here
      }
    }
  3. Use pac-resolver to resolve proxies from a PAC file

    main

    The pac-resolver module generates an asynchronous FindProxyForURL() function from the contents of a Proxy Auto-Config (PAC) file. This function can then be used to determine the appropriate proxy for a given URL.

    To use it, you must provide the PAC file contents (as a string or Buffer) and a QuickJS module instance (obtained from quickjs-emscripten).

    import { readFileSync } from 'fs';
    import { createPacResolver } from 'pac-resolver';
    
    // Assuming createPacResolver is the high-level entry point provided by the package
    const FindProxyForURL = createPacResolver(readFileSync('proxy.pac'));
    
    const res = await FindProxyForURL('http://foo.com/');
    console.log(res);
    // "DIRECT"
  4. Implement a custom Agent by extending Agent

    main

    You can create a custom agent by extending the Agent class from agent-base. This is useful for implementing custom socket logic, such as choosing between net.connect and tls.connect based on the endpoint security.

    In the connect(req, opts) method:

    • Use opts.secureEndpoint to determine if the connection should be encrypted (HTTPS).
    • Return a net.Socket, tls.Socket, or any Duplex stream.
    • You can also return another http.Agent to delegate the connection.

    Once instantiated, pass the agent to the agent option in Node.js http.get, https.get, or other request methods.

    import * as net from 'net';
    import * as tls from 'tls';
    import * as http from 'http';
    import { Agent } from 'agent-base';
    
    class MyAgent extends Agent {
      connect(req, opts) {
        // `secureEndpoint` is true when using the "https" module
        if (opts.secureEndpoint) {
          return tls.connect(opts);
        } else {
          return net.connect(opts);
        }
      }
    }
    
    // Keep alive enabled means that `connect()` will only be
    // invoked when a new connection needs to be created
    const agent = new MyAgent({ keepAlive: true });
    
    // Pass the `agent` option when creating the HTTP request
    http.get('http://nodejs.org/api/', { agent }, (res) => {
      console.log('"response" event!', res.headers);
      res.pipe(process.stdout);
    });
  5. Install and use http-proxy-agent

    main

    The http-proxy-agent package provides an http.Agent implementation that allows you to route HTTP requests through a specified HTTP or HTTPS proxy server using Node.js's built-in http module.

    Important Note: If you need to use an HTTP proxy with the https module, you should use https-proxy-agent instead of this package.

    ```ts
    import * as http from 'http';
    import { HttpProxyAgent } from 'http-proxy-agent';
    
    const agent = new HttpProxyAgent('http://168.63.76.32:3128');
    
    http.get('http://nodejs.org/api/', { agent }, (res) => {
      console.log('
  6. Use proxy-agent to automatically route HTTP requests through proxies

    main

    The proxy-agent module provides an http.Agent implementation that automatically selects and uses the appropriate proxy server based on environment variables such as HTTP_PROXY, HTTPS_PROXY, and NO_PROXY.

    It uses an LRU cache to transparently re-use http.Agent instances for subsequent requests to the same proxy server, improving performance. The specific proxy used for a request is determined by the proxy-from-env module logic.

    import * as https from 'https';
    import { ProxyAgent } from 'proxy-agent';
    
    // The correct proxy `Agent` implementation to use will be determined
    // via the `http_proxy` / `https_proxy` / `no_proxy` / etc. env vars
    const agent = new ProxyAgent();
    
    // The rest works just like any other normal HTTP request
    https.get('https://jsonip.com', { agent }, (res) => {
      console.log(res.statusCode, res.headers);
      res.pipe(process.stdout);
    });
  7. Use socks-proxy-agent for HTTP and HTTPS requests

    main

    The socks-proxy-agent package provides a SocksProxyAgent class, which is an http.Agent implementation. It allows you to route HTTP and HTTPS requests through a SOCKS proxy server. You can pass an instance of SocksProxyAgent to the agent option in Node.js built-in http.get, https.get, or other request methods.

    import https from 'https';
    import { SocksProxyAgent } from 'socks-proxy-agent';
    
    const agent = new SocksProxyAgent(
    	'socks://your-name%40gmail.com:abcdef12345124@br41.nordvpn.com'
    );
    
    https.get('https://ipinfo.io', { agent }, (res) => {
    	console.log(res.headers);
    	res.pipe(process.stdout);
    });
  8. Use pac-proxy-agent for PAC file proxy resolution

    main

    The pac-proxy-agent module provides an http.Agent implementation for HTTP and HTTPS requests. It retrieves a specified Proxy Auto-Config (PAC) file and uses it to resolve whether to use an HTTP, HTTPS, SOCKS proxy, or a direct connection for a given endpoint. It is designed to work seamlessly with Node.js's built-in http and https modules by passing the agent into the request options.

    import * as http from 'http';
    import { PacProxyAgent } from 'pac-proxy-agent';
    
    // Initialize the agent with the URL of the PAC file
    const agent = new PacProxyAgent('pac+https://cloudup.com/ceGH2yZ0Bjp+');
    
    // Use the agent in a standard http.get call
    http.get('http://nodejs.org/api/', { agent }, (res) => {
      console.log('"response" event!', res.headers);
      res.pipe(process.stdout);
    });
  9. Use degenerator to compile sync code into async functions

    main

    The degenerator module allows you to write synchronous-looking JavaScript code that is transpiled into async functions. This is useful when you want to provide a user-facing API that looks synchronous but actually performs asynchronous operations (like HTTP requests) under the hood using await.

    To use it, you provide a string containing one or more synchronous functions and an array of function names (as strings or RegExps) that should be treated as asynchronous. The module will transform calls to those specific names into await expressions and prefix the functions with the async keyword.

    import vm from 'vm';
    import { degenerator } from 'degenerator';
    
    // 1. Define the async implementation of the target function
    function get(endpoint: string) {
      return new Promise((resolve) => {
        // ... implementation ...
      });
    }
    
    // 2. The user-provided sync-looking code (as a string)
    const str = `function myFn() {
      const one = get('https://google.com');
      return one;
    }`;
    
    // 3. Transpile the string to include async/await
    const asyncStr = degenerator(str, ['get']);
    
    // 4. Evaluate the transpiled string into a real function using Node.js `vm`
    const asyncFn = vm.runInNewContext(`(${asyncStr})`, { get });
    
    // 5. Invoke the resulting async function
    asyncFn().then((res) => {
      console.log(res);
    });
  10. Convert a Data URI to an ArrayBuffer using dataUriToBuffer

    main

    The data-uri-to-buffer module converts a Data URI string into an ArrayBuffer instance containing the decoded data. It supports both plain-text and base64-encoded data and is designed to work across multiple JavaScript runtimes, including Node.js and web browsers.

    import { dataUriToBuffer } from 'data-uri-to-buffer';
    
    // plain-text data is supported
    let uri = 'data:,Hello%2C%20World!';
    let parsed = dataUriToBuffer(uri);
    console.log(new TextDecoder().decode(parsed.buffer));
    // 'Hello, World!'
    
    // base64-encoded data is supported
    uri = 'data:text/plain;base64,SGVsbG8sIFdvcmxkIQ%3D%3D';
    parsed = dataUriToBuffer(uri);
    console.log(new TextDecoder().decode(parsed.buffer));
    // 'Hello, World!'
  11. Implement cacheability using the cache option

    main

    To avoid re-downloading resources that haven't changed, you can use the cache option.

    1. Perform an initial call to getUri() and store the returned stream.Readable instance.
    2. On subsequent calls for the same URI, pass that previous stream instance in the options object as { cache: previousStream }.
    3. If the remote resource has not changed, getUri() will throw a NotModifiedError with the code "ENOTMODIFIED".
    4. When you catch "ENOTMODIFIED", you should re-use the results from your previous successful call.
    // First time fetches for real
    const stream = await getUri('http://example.com/resource.json');
    
    try {
      // Pass the previous stream as the cache option
      await getUri('http://example.com/resource.json', { cache: stream });
    } catch (err) {
      if (err.code === 'ENOTMODIFIED') {
        // source file has not been modified since last time it was requested,
        // so you are expected to re-use results from a previous call to `getUri()`
      } else {
        throw err;
      }
    }