docconv

To add image support (OCR) to the docconv library, follow these steps:

1. Install and build gosseract.
2. If on macOS, install tesseract via brew: brew install tesseract.
3. Fetch/build docconv using the ocr build tag:

$ go get -tags ocr code.sajari.com/docconv/v2/...

The docd tool can be used in three different modes:

1. As an HTTP service: Runs on port 8888 by default. You can send documents via multipart POST requests to receive plain text and metadata as a JSON object.
2. As a Docker container: Official images are available at https://hub.docker.com/r/sajari/docd. You can build it locally:

   $ cd docd
   $ docker build -t docd .

3. Via Command Line: Pass a document as an argument:

   $ docd -input document.pdf

To deploy the docd service to Google App Engine, navigate to the docd/appengine directory within the repository and use the Google Cloud CLI to perform the deployment.

gcloud app deploy

The docconv library requires several system-level dependencies to handle different file formats. Install them based on your operating system:

Debian-based Linux

$ sudo apt-get install poppler-utils wv unrtf tidy
$ go get github.com/JalfResi/justext

macOS

$ brew install poppler-qt5 wv unrtf tidy-html5
$ go get github.com/JalfResi/justext

To install the docd executable, use the go install command. Ensure that the resulting executable's location is included in your PATH environment variable.

$ go install code.sajari.com/docconv/v2/docd@latest

By default, docd starts an HTTP server that listens for conversion requests. The service exposes a /convert endpoint.

To start the server on a specific address, use the -addr flag:

docd -addr :8080

Once running, you can request a conversion via HTTP (e.g., using curl):

curl http://localhost:8080/convert?url=https://example.com

To install this specific version of the Snappy library for Go, use the go get command.

WARNING: This version is a modified 'hack' specifically designed to handle Apple files that fail to set CRC checks and stream identifiers. If you are working on any other project that does not require this specific Apple-file compatibility, DO NOT USE THIS VERSION. Use the standard code.google.com/p/snappy-go/snappy instead.

$ go get code.google.com/p/snappy-go/snappy

If you have the system dependencies installed, you can use the docconv package directly in your Go code to convert a file path to text.

package main

import (
	"fmt"

	"code.sajari.com/docconv/v2"
)

func main() {
	res, err := docconv.ConvertPath("your-file.pdf")
	if err != nil {
		// TODO: handle
	}
	fmt.Println(res)
}

You can use the client package to interact with a running docd service (defaulting to localhost:8888).

package main

import (
	"fmt"

	"code.sajari.com/docconv/v2/client"
)

func main() {
	// Create a new client, using the default endpoint (localhost:8888)
	c := client.New()

	res, err := client.ConvertPath(c, "your-file.pdf")
	if err != nil {
		// TODO: handle
	}
	fmt.Println(res)
}

Alternatively, you can use curl to send a multipart POST request:

$ curl -s -F input=@your-file.pdf http://localhost:8888/convert

The docd service allows fine-tuning of the HTML readability extraction logic via CLI flags. These flags map to docconv.HTMLReadabilityOptions:

• readability-length-low / readability-length-high: Controls the length constraints for readability.
• readability-stopwords-low / readability-stopwords-high: Controls the stopwords density thresholds.
• readability-max-link-density: Sets the maximum allowed link density.
• readability-max-heading-distance: Sets the maximum distance between headings.
• readability-use-classes: A comma-separated list of CSS classes used to identify readable content (e.g., "good,neargood").

The HTMLReadability function is configured via the global variable HTMLReadabilityOptionsValues of type HTMLReadabilityOptions.

Note: These settings are global and affect all subsequent calls to HTMLReadability or ConvertHTML (when readability is true).

type HTMLReadabilityOptions struct {
	LengthLow             int
	LengthHigh            int
	StopwordsLow          float64
	StopwordsHigh         float64
	MaxLinkDensity        float64
	MaxHeadingDistance    int
	ReadabilityUseClasses string
}

// Set global options
docconv.HTMLReadabilityOptionsValues = docconv.HTMLReadabilityOptions{
	LengthLow:      20,
	LengthHigh:     100,
	MaxLinkDensity: 0.5,
}