repository·main·Indexed 41 days ago
https://github.com/projectdiscovery/rawhttpA specialized Go library for performing low-level, 'raw' HTTP requests. Optimized by adapting features from gorilla/http and fasthttp, it provides high-level methods like Get, Post, and Head, as well as granular control via DoRaw and a PipelineClient for pipelined HTTP requests to improve throughput.
gorilla/http and fasthttp, inspired by the work of tomnomnom/rawhttp. It is intended for low-level or highly customized HTTP interactions.rawhttp internally utilizes fastdialer for connection dialing, it may create temporary files on disk for DNS lookup caching. To ensure these temporary files are removed and to prevent resource leaks, you must call the .Close() method at the end of your program's lifecycle.To get started with rawhttp, you can refer to the simple implementation example provided in the repository. For full API documentation, including method signatures and types, consult the official GoDoc documentation.
Example implementation path: example/simple/main.go.
// Refer to example/simple/main.go for a complete working exampleWhen initializing a PipelineClient, you can tune several parameters to control connection pooling and pipelining behavior:
| Field | Type | Description |
|---|---|---|
Addr | string | The target address (e.g., example.com:80). |
MaxConns | int | Maximum number of connection clients to maintain for the host. |
MaxPendingRequests | int | Maximum number of requests that can be queued per connection. Defaults to DefaultMaxPendingRequests (1024). |
MaxBatchDelay | time.Duration | Delay used to batch multiple requests together before flushing to the wire. |
Dial | DialFunc | Custom dial function for establishing connections. |
DialDualStack | bool | Whether to use dual-stack dialing if Dial is nil. |
IsTLS | bool | Whether to use TLS. |
TLSConfig | *tls.Config | Custom TLS configuration. |
MaxIdleConnDuration | time.Duration | How long an idle connection stays open before being closed. |
ReadBufferSize | int | Size of the buffer for reading responses. |
WriteBufferSize | int | Size of the buffer for writing requests. |
ReadTimeout | time.Duration | Timeout for reading a response. |
WriteTimeout | time.Duration | Timeout for writing a request. |
If you need to perform a request by specifying components like the URI path and headers directly without building a full http.Request object, use the DoRaw family of functions.
DoRaw(method, url, uripath string, headers map[string][]string, body io.Reader): Executes a request with the specified method, base URL, URI path, headers, and body.DoRawWithOptions(method, url, uripath string, headers map[string][]string, body io.Reader, options *Options): Same as DoRaw, but allows passing a pointer to an Options struct to customize the request behavior.import (
"strings"
"github.com/projectdiscovery/rawhttp"
)
headers := map[string][]string{
"User-Agent": {"rawhttp-client"},
}
body := strings.NewReader("payload")
// Raw request with specific path and headers
resp, err := rawhttp.DoRaw("POST", "https://example.com", "/api/v1/resource", headers, body)
// Raw request with custom options
opts := &rawhttp.Options{ /* custom config */ }
resp, err := rawhttp.DoRawWithOptions("POST", "https://example.com", "/api/v1/resource", headers, body, opts)The Dialer interface is the primary entrypoint for establishing remote HTTP connections. It supports direct dialing, dialing with timeouts, and dialing through proxies (HTTP or SOCKS5/SOCKS5h).
When using Dial or DialTimeout, the implementation may reuse existing connections to the same address if they have been released.
Supported protocols include http and https. For https, the dialer automatically handles TLS handshakes, supporting custom SNI via the Options struct.
// Example of using a Dialer to establish a connection
var d rawhttp.Dialer
// Dial a standard HTTP connection
conn, err := d.Dial("http", "example.com:80", &rawhttp.Options{})
if err != nil {
return err
}
// Dial an HTTPS connection with a specific timeout
conn, err = d.DialTimeout("https", "example.com:443", 5*time.Second, &rawhttp.Options{})
if err != nil {
return err
}
// Dial through a proxy
conn, err = d.DialWithProxy("http", "example.com:80", "http://proxy.example.com:8080", 10*time.Second, &rawhttp.Options{})
if err != nil {
return err
}
defer conn.Close()The ReadResponse(forceReadAll bool) (*Response, error) method unmarshals an HTTP response from the connection.
It includes logic to skip duplicate status lines emitted by non-compliant servers (up to maxDuplicateStatusLines).
forceReadAll is false, the returned Response.Body is wrapped in an io.LimitReader based on the Content-Length header to prevent over-reading.Transfer-Encoding: chunked, the body is automatically wrapped in an httputil.NewChunkedReader.resp, err := c.ReadResponse(false)
if err != nil {
// handle error
}
// Use resp.Body (io.Reader)
body, _ := io.ReadAll(resp.Body)PendingRequests() method on a PipelineClient to get the total number of requests currently in flight or queued across all managed connection clients. This is useful for monitoring load and preventing overflows.To enable connection reuse within a Dialer, call the Release() method on a Conn object.
When Release() is called, the connection is not closed; instead, it is added to an internal map in the dialer associated with its remote address. The next time Dial is called for that same address, the Dialer will attempt to return this released connection instead of creating a new one.
// Instead of closing, release the connection to allow reuse
conn.Release()Use NewPipelineClient to create a high-performance client designed for pipelined HTTP requests. You must provide a PipelineOptions struct to configure connection pooling, timeouts, and host settings.
client := rawhttp.NewPipelineClient(rawhttp.PipelineOptions{
Host: "example.com",
MaxConnections: 10,
MaxPendingRequests: 100,
Timeout: 30 * time.Second,
})The DumpRequestRaw function generates a byte slice representing the raw HTTP request. It constructs the request line (method, path, query, and version), appends headers, handles Content-Length automatically if enabled, and includes the request body.
If options.CustomRawBytes is provided in the Options struct, this function will return those bytes directly instead of generating them.
Key behaviors:
Host header if it is missing from the provided headers map.uripath to override the path if a non-empty string is provided.client.NewLine to ensure consistent formatting.TeeReader to read the body, allowing the body to be captured without exhausting the original io.Reader if it is used elsewhere (though the returned slice contains the full body).// Example usage of DumpRequestRaw
// Note: Requires an initialized Options object
options := &rawhttp.Options{}
headers := map[string][]string{
"User-Agent": {"rawhttp-client"},
}
body := strings.NewReader("request body content")
rawBytes, err := rawhttp.DumpRequestRaw("POST", "http://example.com/api", "", headers, body, options)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s", rawBytes)The PipelineClient is the primary entrypoint for performing HTTP requests using a pipelining model. It manages a pool of connection clients (pipelineConnClient) to a specific address, allowing multiple requests to be sent over the same connection before waiting for responses, which improves throughput.
To use it, initialize a PipelineClient with your desired configuration and call the Do method with a Request and a Response object.
// Example usage pattern
client := &clientpipeline.PipelineClient{
Addr: "example.com:80",
MaxConns: 512,
IsTLS: false,
}
req := &Request{ /* ... */ }
resp := &Response{ /* ... */ }
err := client.Do(req, resp)
if err != nil {
// handle error
}