DOCSEARCHbyAlgolia
Submit Docs

Scio Documentation

repository·main·Indexed 25 days ago

https://github.com/spotify/scio

A Scala API for Apache Beam and Google Cloud Dataflow, inspired by Apache Spark and Scalding. Scio provides a unified batch and streaming programming model with strong integration for Google Cloud products and various data IOs. It includes a modular set of artifacts for Avro, Cassandra, Elasticsearch, BigQuery, Pub/Sub, JDBC, and more, along with macro annotations for Avro case class generation and utilities for command line argument parsing.

Tokens
60.6K
Snippets
168
Records
285
Agent score
82%

What's inside Scio

  1. Explore Scio IO integrations

    main

    Scio provides specialized IO modules for interacting with various data formats and storage systems. Key integrations include:

    • Avro: Using Scio with Avro files.
    • BigQuery: Type-safe interaction with BigQuery.
    • Bigtable: Using Scio with Google Cloud Bigtable.
    • Parquet: Using Scio with Parquet files.
    • Protobuf: Using Scio with Protocol Buffers.

  2. Core Scio Concepts and Execution

    main

    Scio is a Scala wrapper around Apache Beam. The core types are:

    • ScioContext: Wraps Beam's Pipeline. It is the entry point for reads and pipeline execution.
    • SCollection[T]: Wraps Beam's PCollection[T]. Represents data in the pipeline.
    • ScioResult: Wraps Beam's PipelineResult. Used to access results and metrics after execution.

    To execute a pipeline, call run() on your ScioContext and use waitUntilDone() to await completion.

    import com.spotify.scio._
val sc: ScioContext = ???
sc.run().waitUntilDone()

  3. Understand Scio Coders and their role in Apache Beam

    main

    In Scio, Coders are used to describe how elements of an SCollection[T] are encoded to and decoded from byte strings. This is essential for Apache Beam to transfer data between workers during:

    • Shuffling data: Any *byKey transform (e.g., groupByKey, reduceByKey) triggers a shuffle.
    • Cluster scaling: Redistributing data when workers are added or removed.
    • GroupByKey: Both for serialization during shuffles and for testing key equality (Beam compares serialized forms to determine equality).

    Important: When used for equality testing (like in groupByKey), Coders must be deterministic. Using a non-deterministic Coder for equality will throw an exception.

  4. Explore end-to-end complete pipeline examples

    main

    The complete examples directory provides end-to-end pipelines for complex data processing tasks. These examples demonstrate advanced patterns such as streaming vs. batch processing, windowing, and complex joins. Key examples include:

    • AutoComplete: Computes popular hashtags for prefixes. Demonstrates using the same pipeline for both streaming and batch, as well as combiners and composite transforms.
    • StreamingWordExtract: A streaming pipeline that ingests text from Cloud Pub/Sub, splits/capitalizes words, and writes to BigQuery.
    • TfIdf: Computes a TF-IDF search table for a directory or Cloud Storage prefix. Demonstrates joining data, side inputs, and logging.
    • TopWikipediaSessions: Reads Wikipedia edit data from Cloud Storage to find users with the longest edit strings (within one hour of each other) per month. Demonstrates Cloud Dataflow Windowing for time-based aggregations.
    • TrafficMaxLaneFlow: A streaming pipeline in the traffic sensor domain. Demonstrates the Cloud Dataflow streaming runner, sliding windows, Cloud Pub/Sub ingestion, AvroCoder for custom classes, and custom Combine transforms.
    • TrafficRoutes: A streaming pipeline in the traffic sensor domain. Demonstrates the Cloud Dataflow streaming runner, GroupByKey, keyed state, sliding windows, and Cloud Pub/Sub ingestion.

  5. Use Scio additional features

    main

    Scio includes several advanced features for data processing:

    • DistCache: Allows workers to pull files from Google Cloud Storage to be used locally within transforms, similar to Hadoop's distributed cache.
    • Type-safe BigQuery IO: Uses Scala macros to generate case classes and converters at compile time based on BigQuery schemas, avoiding generic JSON handling.
    • Pipeline Orchestration with ClosedTap[T]: Sinks (methods like saveAs*) return a ClosedTap[T]. This tap can be opened in a subsequent pipeline as an SCollection[T] or as an Iterator[T] after the current pipeline completes, enabling complex orchestration.

  8. Configure Dataflow options for Scio REPL

    main

    To run pipelines on the Google Cloud Dataflow service instead of locally, pass Dataflow pipeline options when starting the REPL JAR. These options will also be used by the :newScio command.

    java -jar scio-repl-<version>.jar \
  --project=<project-id> \
  --stagingLocation=<staging-dir> \
  --tempLocation=<temp-dir> \
  --runner=DataflowRunner

  9. Tune Scio pipeline execution parameters

    main

    When tuning pipeline execution on Dataflow, follow these steps to manage costs and performance:

    • Start small: Begin with smaller workerMachineType instances (e.g., moving from n1-standard-1 to n1-standard-4).
    • Scale workers appropriately: Set a maxNumWorkers value that reflects your input size.
    • Consider costs: Be aware that increasing the number of workers increases shuffle costs and that large GCE instances may have limited availability.

  10. Configure OverrideTypeProvider via JVM System Property

    main

    To use a custom OverrideTypeProvider, you must specify it as a JVM System property named override.type.provider. This provider is loaded via Reflection at both macro expansion time and runtime.

    Because this feature relies on Scala macros, the property must be set at initialization time. If the property is not specified, Scio falls back to its default type mapping behavior. Note that only one OverrideTypeProvider is allowed per sbt project.

    System.setProperty(
  "override.type.provider",
  "com.spotify.scio.bigquery.validation.SampleOverrideTypeProvider")

  11. Use Taps and Materialization to access pipeline results

    main

    Scio provides ClosedTap and materialize to capture and access data at specific points in a pipeline after it completes.

    • Taps: Writing to a sink (like saveAsTextFile) returns a ClosedTap. You can use a ScioResult to extract a Tap from a ClosedTap, allowing you to access the values as an Iterator or open them as a new SCollection in a subsequent Scio context.
    • Materialize: The materialize method saves the contents of an SCollection to a temporary location, making them available as a ClosedTap once the pipeline finishes.
    import com.spotify.scio._
import com.spotify.scio.io.{Tap, ClosedTap}
import com.spotify.scio.values.SCollection

val sc: ScioContext = ???
val elements: SCollection[String] = ???
val writeTap: ClosedTap[String] = elements.saveAsTextFile("gs://output-path")

val sr: ScioResult = sc.run().waitUntilDone()

val textTap: Tap[String] = sr.tap(writeTap)
val textContexts: Iterator[String] = textTap.value

val sc2: ScioContext = ???
val results: SCollection[String] = textTap.open(sc)