repository·master·Indexed 26 days ago
https://github.com/chai2010/go-ast-bookA technical guide on using Go's standard library (go/ast, go/token) to analyze and manipulate Go source code via its Abstract Syntax Tree. The documentation covers lexical analysis with scanner.Scanner, managing source files with FileSet, and understanding AST structures like BlockStmt and ExprStmt. It also includes appendices on generating parsers and lexers using goyacc, flex, and ANTLR4.
go/types package. The go/types package was developed by Robert Griesemer and is used for performing type checking within the Go ecosystem.ssa package.Chapter 15 provides a high-level introduction to LLVM. Rather than focusing on deep compiler theory or complex internal implementations, the chapter focuses on two practical aspects:
Note: Due to copyright restrictions, the full content of this chapter is not available in this repository. To access the complete material, you must refer to the physical/digital book.
The Go AST Book (formerly Go Syntax Tree Introduction) is a technical guide focused on the Go Abstract Syntax Tree (AST). It explores the semantic representation of Go source files using the standard library's go/ast and go/token packages. The book provides practical knowledge for building tools similar to go fmt and go doc by analyzing Go programs through their syntax tree structures.
Key Resources:
To perform lexical analysis on source code, use token.FileSet and token.File objects to manage file positions and collections.
token.FileSet: Represents a collection of files. It manages a global position space using token.Pos (an integer type representing an index in the underlying array).token.File: Represents a single file within a FileSet. It contains the filename, a base position, and a size.To create a File object, you must use the AddFile method of a FileSet instance. You cannot construct a File directly.
In the Go AST, functions and methods are represented by the *ast.FuncDecl type. A method is essentially a function with a Receiver.
Grammar Rules:
FunctionDecl: func + MethodName + Signature + [FunctionBody]MethodDecl: func + Receiver + MethodName + Signature + [FunctionBody]Key distinction: The function signature (type) includes only the input parameters and return values. The function name and the method receiver are not part of the function signature.
Composite expressions in Go are formed by combining operands with high-order operators. The syntax for PrimaryExpr includes:
. identifier (e.g., x.y)[ Expression ] (e.g., x[y])[ [ Expression ] : [ Expression ] ] or [ [ Expression ] : Expression : Expression ] (e.g., x[1:2:3]). ( Type ) (e.g., x.(y))( [ ( ExpressionList | Type [ "," ExpressionList ] ) [ "..." ] [ "," ] ] ) (e.g., int(x) or f(x, y))In the AST, many of these are parsed based on structure before the semantic meaning (like whether a name is a type or a function) is fully resolved.
In Go, Basic Literals are values directly represented in code (e.g., 2 in x+2*y). The Go specification defines basic literals as integers, floating-point numbers, complex numbers, characters, and strings.
Note that while true and false are treated as predefined literal types by users, they are technically built-in boolean identifiers in the Go specification. In the go/token package, basic literals are represented by specific tokens between literal_beg and literal_end.
When performing type checking on code that imports other packages, types.Config needs an Importer to resolve dependencies. Without an importer, Check will fail with an error like could not import <package_name> (Config.Importer not installed).
For standard library packages, use go/importer to provide the default implementation:
import "go/importer"
import "go/types"
// ...
conf := types.Config{Importer: importer.Default()}
pkg, err := conf.Check("hello.go", fset, []*ast.File{f}, nil)To handle custom package structures or manual loading, implement the types.Importer interface:
type Importer interface {
Import(path string) (*Package, error)
}The go/scanner package provides the Scanner type to perform lexical analysis.
To use it:
token.FileSet using token.NewFileSet().fset.AddFile(filename, base, len(src)).s.Init(file, src, err, mode).s.Scan() in a loop until it returns token.EOF.Use fset.Position(pos) to convert the returned token.Pos into a human-readable token.Position (filename, line, and column).
package main
import (
"fmt"
"go/scanner"
"go/token"
)
func main() {
var src = []byte(`println("你好，世界")`)
var fset = token.NewFileSet()
var file = fset.AddFile("hello.go", fset.Base(), len(src))
var s scanner.Scanner
s.Init(file, src, nil, scanner.ScanComments)
for {
pos, tok, lit := s.Scan()
if tok == token.EOF {
break
}
fmt.Printf("%s\t%s\t%q\n", fset.Position(pos), tok, lit)
}
}