DOCSEARCHbyAlgolia
Submit Docs

gse

repository·master·Indexed 25 days ago

https://github.com/go-ego/gse

An efficient, multilingual NLP and text segmentation library for Go. It implements a Jieba-like segmentation algorithm and supports English, Chinese, Japanese, and other languages. Key features include POS tagging, HMM-based segmentation, TF-IDF dictionary support, and multiple segmentation modes such as Cut, CutSearch, CutAll, and CutDAG.

Tokens
7.5K
Snippets
8
Records
82
Agent score
83%

What's inside gse

  3. Install the gse package

    master

    To install gse using Go modules (Go 1.11+), import the package directly into your project:

    import "github.com/go-ego/gse"

    Alternatively, you can install it using the go get command:

    go get -u github.com/go-ego/gse
    go get -u github.com/go-ego/gse

  5. Run the gse segmentation server

    master

    The gse server provides a web interface for demonstration, an endpoint to add words to the dictionary, and a JSON-RPC service for remote text segmentation.

    CLI Flags

    • -host: HTTP server hostname (default: empty string).
    • -port: HTTP server port (default: 8080).
    • -hmm: Use HMM (Hidden Markov Model) for segmentation (default: false).
    • -dict: Path to the dictionary file (default: ../data/dict/dictionary.txt).
    • -static_folder: Directory containing static web pages (default: static).

  6. Basic text segmentation with Segmenter

    master

    You can use the gse.Segmenter struct to perform text segmentation. You can load dictionaries from files using LoadDict or from embedded strings using LoadDictEmbed. The Cut method performs the segmentation.

    var seg1 gse.Segmenter
seg1.DictSep = ","
err := seg1.LoadDict("./testdata/test_en.txt")
if err != nil {
	fmt.Println("Load dictionary error: ", err)
}

s1 := seg1.Cut(text)
fmt.Println("seg1 Cut: ", s1)

  7. Use gse for basic text segmentation

    master

    You can perform text segmentation by initializing a gse.Segmenter and loading a dictionary. The Cut method performs the segmentation.

    Key configuration options for Segmenter:

    • DictSep: Sets the separator for the dictionary file.
    • AlphaNum: If true, treats alphanumeric characters as separate segments.
    • ToLower: (Global) If set to false, preserves original capitalization.
    package main

import (
	"fmt"
	"github.com/go-ego/gse"
)

func main() {
	var seg1 gse.Segmenter
	seg1.DictSep = ","
	err := seg1.LoadDict("./testdata/test_en.txt")
	if err != nil {
		fmt.Println("Load dictionary error: ", err)
	}

	text := "To be or not to be, that's the question!"
	s1 := seg1.Cut(text)
	fmt.Println("seg1 Cut: ", s1)
}

  8. Use gse for text segmentation

    master

    Gse provides several segmentation modes including normal, search engine, full mode, precise mode, and HMM mode. It supports multiple languages such as English, Chinese, and Japanese.

    package main

import (
	"fmt"
	"regexp"

	"github.com/go-ego/gse"
	"github.com/go-ego/gse/hmm/pos"
)

var (
	seg gse.Segmenter
	posSeg pos.Segmenter

	new, _ = gse.New("zh,testdata/test_en_dict3.txt", "alpha")

	text = "你好世界, Hello world, Helloworld."
)

func main() {
	// Load default dictionary
	seg.LoadDict()

	cut()
	segCut()
}

func cut() {
	hmm := new.Cut(text, true)
	fmt.Println("cut use hmm: ", hmm)

	hmm = new.CutSearch(text, true)
	fmt.Println("cut search use hmm: ", hmm)
	fmt.Println("analyze: ", new.Analyze(hmm, text))

	hmm = new.CutAll(text)
	fmt.Println("cut all: ", hmm)

	reg := regexp.MustCompile(`(\d+年|\d+月|\d+日|[\p{Latin}]+|[\p{Hangul}]+|\d+\.\d+|[a-zA-Z0-9]+)`)
	text1 := `헬로월드 헬로 서울, 2021年09月10日, 3.14`
	hmm = seg.CutDAG(text1, reg)
	fmt.Println("Cut with hmm and regexp: ", hmm, hmm[0], hmm[6])
}

func segCut() {
	// Segment text
	tb := []byte("山达尔星联邦共和国联邦政府")

	// Output segmentation results as string (search mode)
	fmt.Println("Output string, search mode: ", seg.String(tb, true))
	// Output segmentation results as slice
	fmt.Println("Output slice: ", seg.Slice(tb))

	segments := seg.Segment(tb)
	// Process segments (normal mode)
	fmt.Println(gse.ToString(segments))

	segments1 := seg.Segment([]byte(text))
	// Process segments (search mode)
	fmt.Println(gse.ToString(segments1, true))
}

  9. Use embedded dictionaries with gse.NewEmbed

    master

    You can use Go's embed feature to include custom dictionaries directly in your binary using gse.NewEmbed.

    package main

import (
	"fmt"
	_ "embed"

	"github.com/go-ego/gse"
)

//go:embed test_en_dict3.txt
var testDict string

func main() {
	// Initialize with embedded dictionary
	seg, err := gse.NewEmbed("zh, word 20 n"+testDict, "en")
	if err != nil {
		panic(err)
	}
	seg.LoadStopEmbed()

	text1 := "所以, 你好, 再见"
	fmt.Println(seg.Cut(text1, true))
	fmt.Println(seg.String(text1, true))

	segments := seg.Segment([]byte(text1))
	fmt.Println(gse.ToString(segments))
}

  10. Initialize Segmenter with gse.New()

    master
    Use gse.New() to quickly initialize a segmenter with a specific language and dictionary. Supported languages include Chinese (zh), Japanese (jp), etc. The second argument can specify modes like alpha for alphanumeric handling.

  11. Perform HMM and Search segmentation

    master
    The library supports advanced segmentation modes including HMM (Hidden Markov Model) and Search modes. Use Cut with the HMM flag enabled, or use CutSearch for search-engine optimized segmentation. You can also use CutAll to get all possible segments.

  12. Initialize a Segmenter with gse.New()

    master

    The gse.New() function provides a quick way to create a segmenter with specific language support and dictionary paths.

    Signature pattern: gse.New(langAndDict string, mode string) (Segmenter, error)

    Example usage for Chinese with an alpha-numeric mode: