csvutil

repository·master·Indexed 19 days ago

https://github.com/jszwec/csvutil

A fast, idiomatic, and dependency-free Go package for mapping between CSV data and Go values. Built as a mapping layer on top of standard Go CSV reader/writer interfaces, it provides functionality to marshal and unmarshal Go structs, handle custom struct tags, support nested/inline structs, and implement custom type encoding and decoding via Marshaler and Unmarshaler interfaces.

Tokens
7.1K
Snippets
38
Records
40
Agent score
75%

What's inside csvutil

  1. Use Nested and Inline structs

    master

    csvutil supports nested or embedded structs.

    • Embedded structs: Fields are flattened into the CSV.
    • Inline tags: Use the csv:",inline" tag to flatten a struct. You can also provide a prefix (e.g., csv:"prefix_,inline") to namespace the underlying fields in the CSV columns.
    type Address struct {
    	Street string `csv:"street"`
    	City   string `csv:"city"`
    }
    
    type User struct {
    	Name        string  `csv:"name"`
    	Address     Address `csv:",inline"` // Flattened
    	HomeAddress Address `csv:"home_,inline"` // Prefixed: home_street, home_city
    	WorkAddress Address `csv:"work_,inline"` // Prefixed: work_street, work_city
    	Age         int     `csv:"age,omitempty"`
    }
  2. Handle extra metadata with Decoder.Unused

    master

    If your CSV contains headers that are not mapped to your struct fields, you can capture them as metadata. Use csvutil.NewDecoder and call dec.Unused() after each dec.Decode() call to get the indexes of the unused columns. You can then use dec.Header() to map those indexes back to their column names.

    type User struct {
    	Name      string            `csv:"name"`
    	City      string            `csv:"city"`
    	Age       int               `csv:"age"`
    	OtherData map[string]string `csv:"-"`
    }
    
    csvReader := csv.NewReader(strings.NewReader(`
    name,age,city,zip
    alice,25,la,90005
    bob,30,ny,10005`))
    
    dec, err := csvutil.NewDecoder(csvReader)
    if err != nil {
    	log.Fatal(err)
    }
    
    header := dec.Header()
    var users []User
    for {
    	u := User{OtherData: make(map[string]string)}
    
    	if err := dec.Decode(&u); err == io.EOF {
    		break
    	} else if err != nil {
    		log.Fatal(err)
    	}
    
    	for _, i := range dec.Unused() {
    		u.OtherData[header[i]] = dec.Record()[i]
    	}
    	users = append(users, u)
    }
  3. Customize type encoding and decoding

    master

    You can override how specific types are handled using four methods, listed in order of precedence:

    1. Registered functions: Use Encoder.WithMarshalers or Decoder.WithUnmarshalers to register custom functions for specific types.
    2. Interface implementation (Registered): Register an interface (e.g., fmt.Stringer) and implement it on your types.
    3. csvutil interfaces: Implement csvutil.Marshaler and csvutil.Unmarshaler on your types.
    4. Standard library interfaces: Implement encoding.TextMarshaler and encoding.TextUnmarshaler on your types.
    // Example: Implementing csvutil.Marshaler
    type Foo int64
    
    func (f Foo) MarshalCSV() ([]byte, error) {
    	return strconv.AppendInt(nil, int64(f), 16), nil
    }
    
    func (f *Foo) UnmarshalCSV(data []byte) error {
    	i, err := strconv.ParseInt(string(data), 16, 64)
    	if err != nil {
    		return err
    	}
    	*f = Foo(i)
    	return nil
    }
  4. Decode CSV files without headers

    master

    To decode a CSV file that lacks a header row, first generate a header slice using csvutil.Header based on your target struct. Then, pass this header slice to csvutil.NewDecoder as additional arguments.

    type User struct {
    	ID   int
    	Name string
    	Age  int `csv:",omitempty"`
    	City string
    }
    
    csvReader := csv.NewReader(strings.NewReader(`
    1,John,27,la
    2,Bob,,ny`))
    
    // Generate header from struct
    userHeader, err := csvutil.Header(User{}, "csv")
    if err != nil {
    	log.Fatal(err)
    }
    
    // Pass header to NewDecoder
    dec, err := csvutil.NewDecoder(csvReader, userHeader...)
    if err != nil {
    	log.Fatal(err)
    }
    
    var users []User
    for {
    	var u User
    	if err := dec.Decode(&u); err == io.EOF {
    		break
    	} else if err != nil {
    		log.Fatal(err)
    	}
    	users = append(users, u)
    }
  5. Use different separators (e.g., TSV)

    master

    To use a different delimiter like a tab (\t), configure the underlying csv.Reader or csv.Writer before passing them to csvutil.

    // Decoder for TSV
    csvReader := csv.NewReader(r)
    csvReader.Comma = '\t'
    dec, err := csvutil.NewDecoder(csvReader)
    
    // Encoder for TSV
    var buf bytes.Buffer
    w := csv.NewWriter(&buf)
    w.Comma = '\t'
    enc := csvutil.NewEncoder(w)
  6. Handle Slice and Map fields

    master

    Since the CSV specification does not define how to encode slices or maps, csvutil does not support them by default. To use them, create a type alias for the slice or map and implement the csvutil.Marshaler and csvutil.Unmarshaler interfaces.

    type Strings []string
    
    func (s Strings) MarshalCSV() ([]byte, error) {
    	return []byte(strings.Join(s, ",")), nil
    }
    
    type StringMap map[string]string
    
    func (sm StringMap) MarshalCSV() ([]byte, error) {
    	return []byte(fmt.Sprint(sm)), nil
    }
  7. How Decoder and Unmarshalers work together

    master

    The Decoder uses a mapping system to connect CSV columns to struct fields. When Decode is called, it looks up the appropriate decoding function for each field.

    Custom decoding logic is provided via Unmarshalers. When a field's type matches a registered concrete type, that function is used. If no concrete match is found, the Decoder checks if the field implements any of the registered interfaces in the order they were registered. This allows for flexible, polymorphic decoding of CSV data.

  8. Register custom encoding functions with Marshalers

    master

    To provide custom encoding logic for specific types, use the Marshalers type and the MarshalFunc[T] helper. This is the recommended, type-safe way to extend encoding behavior.

    1. Create a Marshalers object using MarshalFunc[T](f) where f is func(T) ([]byte, error).
    2. Apply them to an encoder using WithMarshalers(m).

    When encoding, the encoder matches the concrete type first. If no match is found, it checks registered interfaces in the order they were registered.

    Note: The Encoder.Register(f any) method is deprecated. Use MarshalFunc instead for better performance and type safety.

    // 1. Define a custom marshaler for a specific type
    myMarshaler := csvutil.MarshalFunc(func(t MyType) ([]byte, error) {
    	return []byte(fmt.Sprintf("custom:%v", t)), nil
    })
    
    // 2. Create encoder and apply marshalers
    enc := csvutil.NewEncoder(writer)
    enc.WithMarshalers(myMarshaler)
    
    // 3. Encode
    enc.Encode(myStruct)
  9. Override custom struct tags

    master

    By default, csvutil looks for the csv tag. You can change this by setting the Tag field on the Encoder or Decoder.

    type Foo struct {
    	Bar int `custom:"bar"`
    }
    
    // Use 'custom' tag instead of 'csv'
    dec, err := csvutil.NewDecoder(r)
    dec.Tag = "custom"
    
    enc := csvutil.NewEncoder(w)
    enc.Tag = "custom"
  10. Normalize data using Decoder.Map

    master

    The Decoder.Map function allows you to intercept and modify raw field values before they are decoded into Go types. This is useful for data normalization, such as converting custom strings (e.g., 'n/a') into standard formats (e.g., 'NaN') that strconv can handle.

    dec, err := csvutil.NewDecoder(r)
    if err != nil {
    	log.Fatal(err)
    }
    
    // Normalize 'n/a' to 'NaN' for float64 fields
    dec.Map = func(field, column string, v any) string {
    	if _, ok := v.(float64); ok && field == "n/a" {
    		return "NaN"
    	}
    	return field
    }
  11. Unmarshal CSV data into Go structs

    master

    Use csvutil.Unmarshal for simple, one-off decoding of CSV bytes into a slice of structs. It uses the standard Go csv.Reader with default options. For streaming or more advanced use cases, use csvutil.NewDecoder.

    var csvInput = []byte(`
    name,age,CreatedAt
    jacek,26,2012-04-01T15:00:00Z
    john,,0001-01-01T00:00:00Z`,
    )
    
    type User struct {
    	Name      string `csv:"name"`
    	Age       int    `csv:"age,omitempty"`
    	CreatedAt time.Time
    }
    
    var users []User
    if err := csvutil.Unmarshal(csvInput, &users); err != nil {
    	fmt.Println("error:", err)
    }
    
    for _, u := range users {
    	fmt.Printf("%+v\n", u)
    }