repository·master·Indexed 42 days ago
https://github.com/d-rickyy-b/certstream-server-goA high-performance Go-based replacement for the original Certstream server that aggregates, parses, and streams real-time Certificate Transparency (CT) log data to clients via WebSockets. It includes a Prometheus metrics endpoint, a configuration validation tool, and the certpicker CLI for fetching specific certificate entries from CT logs.
Use the prebuilt Docker image 0rickyy0/certstream-server-go. To use a custom configuration, you must mount your config.yaml file into the container. If you do not mount a config file, the default config.sample.yaml will be used.
docker run -d -v /path/to/config.yaml:/app/config.yaml -p 8080:8080 0rickyy0/certstream-server-goYou can set up an instance of the certstream server by downloading and compiling the source code or by using precompiled binaries from the releases page.
By default, the server monitors all logs listed in the Google Log list. You can define additional logs or customize behavior using a configuration file (see the sample config file).
The IPWhitelist middleware restricts access to specific IP addresses or CIDR ranges.
Crucially, it does not check the potentially rewritten r.RemoteAddr (which might be a proxy IP). Instead, it retrieves the original TCP connection IP from the request context (stored by realIPMiddleware under origConnAddrKey). This ensures that even if a client spoofs headers, the whitelist check is performed against the actual source of the TCP connection.
The realIPMiddleware is used to handle requests coming through reverse proxies.
r.RemoteAddr) and stores it in the request context under the origConnAddrKey to prevent loss during rewriting.realIP is enabled, it attempts to extract the client IP from X-Forwarded-For or X-Real-IP headers.trustedProxies are configured, the middleware only performs the rewrite if the actual connection IP is within the provided list of IPs or CIDR ranges. If the connection is not trusted, it logs a warning and leaves r.RemoteAddr unchanged.Starting with version v1.9.0, the Docker container expects the configuration file to be located at /app/config/config.yml. It is recommended to mount a local directory containing your configuration to this path using a volume.
volumes:
- ./certstream/config:/app/configTo see a live example of the data format for any endpoint, append /example.json to the endpoint URL (e.g., /full-stream/example.json). This returns a single certificate update in the 'lite' format.
{
"data": {
"cert_index": 712420366,
"cert_link": "https://yeti2022-2.ct.digicert.com/log/ct/v1/get-entries?start=712420366&end=712420366",
"leaf_cert": {
"all_domains": [
"cmslieferhit.e06.k-k.de"
],
"extensions": {
"authorityInfoAccess": "URI:http://r3.i.lencr.org/, URI:http://r3.o.lencr.org",
"authorityKeyIdentifier": "keyid:14:2e:b3:17:b7:58:56:cb:ae:50:09:40:e6:1f:af:9d:8b:14:c2:c6",
"basicConstraints": "CA:FALSE",
"keyUsage": "Digital Signature, Key Encipherment",
"subjectAltName": "DNS:cmslieferhit.e06.k-k.de",
"subjectKeyIdentifier": "keyid:4e:cb:ae:47:84:a8:92:f7:e7:de:78:d1:00:9e:d9:cc:80:ac:0b:ce"
},
"fingerprint": "27:58:3D:01:3D:71:B8:D3:A6:6E:2C:7A:86:3A:E9:1F:DB:F0:1B:5D",
"sha1": "27:58:3D:01:3D:71:B8:D3:A6:6E:2C:7A:86:3A:E9:1F:DB:F0:1B:5D",
"sha256": "57:61:38:C0:3C:03:A3:34:6A:0B:32:89:11:1B:74:AB:8A:DF:A5:02:9F:06:43:E6:F3:0E:69:F3:0E:4E:4E:FC",
"not_after": 1667028404,
"not_before": 1659252405,
"serial_number": "0498BDF812FAF923FEBD5EF7B374899FC61A",
"signature_algorithm": "sha256, rsa",
"subject": {
"C": null,
"CN": "cmslieferhit.e06.k-k.de",
"L": null,
"O": null,
"OU": null,
"ST": null,
"aggregated": "/CN=cmslieferhit.e06.k-k.de",
"email_address": null
},
"issuer": {
"C": "US",
"CN": "R3",
"L": null,
"O": "Let's Encrypt",
"OU": null,
"ST": null,
"aggregated": "/C=US/CN=R3/O=Let's Encrypt",
"email_address": null
},
"is_ca": false
},
"seen": 1659301203.904,
"source": {
"name": "DigiCert Yeti2022-2 Log",
"url": "https://yeti2022-2.ct.digicert.com/log"
},
"update_type": "PrecertLogEntry"
},
"message_type": "certificate_update"
}The server provides several WebSocket endpoints for streaming certificate data. You must open a websocket connection to one of the following endpoints:
| Config | Default | Function |
|---|---|---|
full_url | /full-stream | Constant stream of new certificates with all details available |
lite_url | / | Constant stream of new certificates with reduced details (no as_der and chain fields) |
domains_only_url | /domains-only | Constant stream of domains found in new certificates |
Important: Heartbeat Requirement To prevent disconnection, your client must send a ping message at least every 60 seconds. It is recommended to use a 30-second interval. The server will not send ping messages to your client.
You can override default server settings by providing environment variables in your Docker configuration. The following variables are available:
CERTSTREAM_LISTEN_ADDR: Sets the address the server listens on (e.g., 0.0.0.0).CERTSTREAM_LISTEN_PORT: Sets the port the server listens on (e.g., 8080).environment:
- CERTSTREAM_LISTEN_ADDR="0.0.0.0"
- CERTSTREAM_LISTEN_PORT=8080/metrics. You can use this to collect and visualize server performance data using Prometheus and Grafana.Use NewMetricsServer to create a web server dedicated to exposing Prometheus metrics. This server automatically applies realIPMiddleware and IPWhitelist based on the config.AppConfig.Prometheus configuration. It is designed to listen on a specific network interface and port, with optional TLS support via certPath and keyPath.
// Example initialization (conceptual usage)
metricsServer := web.NewMetricsServer("0.0.0.0", 9090, "/path/to/cert", "/path/to/key")
metricsServer.Start()The RegisterPrometheus method allows you to attach a custom metrics endpoint to a Server instance. You provide a URL path and a callback function. The callback is executed when the endpoint is hit, receiving the http.ResponseWriter and a boolean indicating whether system metrics should be exposed (determined by config.AppConfig.Prometheus.ExposeSystemMetrics).
ws.RegisterPrometheus("/metrics", func(w io.Writer, exposeProcessMetrics bool) {
// Your custom metrics logic here
})Use NewWebsocketServer to create a web server that handles Certstream websocket connections. It configures the websocket.Upgrader using settings from config.AppConfig.Webserver (such as CompressionEnabled) and sets up routes for different subscription types: Full, Lite, and Domains Only. It also supports realIPMiddleware and IPWhitelist based on the config.AppConfig.Webserver configuration.
// Example initialization (conceptual usage)
wsServer := web.NewWebsocketServer("0.0.0.0", 8080, "", "")
wsServer.Start()