Overview of Apache SeaTunnel for distributed data integration
# Source code available at: https://github.com/apache/seatunnelwebsite·Indexed 29 days ago
https://raymondlam1.github.ioA personal technical blog focusing on Java runtime, garbage collection, and high-performance Java persistence. The site includes reading notes on the Go programming language, Vim guides, and discussions on Domain-Driven Design (DDD), Apache Calcite, and monitoring tools like Grafana and Prometheus.
# Source code available at: https://github.com/apache/seatunnel# Source code available at: https://github.com/apache/calcite/Vim commands follow a simple grammatical structure: verb + noun.
Combining these allows you to 'speak' to the editor. For example, to yank (verb) to the end of the line (noun), you use y$. To delete (verb) the next word (noun), you use dw.
A DDD architecture replaces the traditional MVC flow with a more decoupled structure consisting of the following layers:
JPA (Java Persistence API) is a specification that defines interfaces and metadata for object-relational mapping. Hibernate is a full implementation of the JPA specification that also provides its own native API for non-standard features. ORM tools like Hibernate solve several JDBC limitations, including:
The Java Transaction API (JTA) supports distributed transactions across multiple heterogeneous resource managers (e.g., multiple DataSources or JMS queues) using the X/Open XA architecture. It employs a Two-Phase Commit (2PC) protocol:
Optimization (1PC): When a transaction involves only a single resource manager, the prepare phase can be skipped (One-Phase Commit) to reduce network overhead. This is triggered by setting the onePhase flag to true in XAResource.commit(Xid xid, boolean onePhase).
Durability ensures that all committed transaction changes are permanent and recoverable even after a system crash. This is achieved through two primary logging mechanisms:
JDBC provides options for result set updatability and persistence (holdability) across transactions.
Updatability: Updatable result sets allow modifying records while iterating. However, for Web applications, they are generally discouraged because keeping a result set (and its underlying connection) open across multiple HTTP requests severely limits scalability. Read-only, forward-only cursors typically offer better performance.
Holdability (Persistence): Determines if a result set remains open after a transaction is committed.
HOLD_CURSORS_OVER_COMMIT.CLOSE_CURSORS_AT_COMMIT.CLOSE_CURSORS_AT_COMMIT but supports HOLD_CURSORS_OVER_COMMIT.Best Practice: To ensure scalability and release system resources, explicitly close all ResultSet and Statement objects as soon as they are no longer needed, regardless of the holdability setting.
fzf and ripgrep (rg) command-line tools. To ensure fzf uses ripgrep for file listing, define the FZF_DEFAULT_COMMAND environment variable in your shell configuration file (.zshrc or .bashrc).# In .zshrc or .bashrc
if type rg &> /dev/null; then
export FZF_DEFAULT_COMMAND='rg --files'
export FZF_DEFAULT_OPTS='-m' # Allows multiple selections with <Tab>/<Shift-Tab>
fiVariables are declared using the var keyword. Go supports type inference and ensures every variable is initialized to a 'zero value' if no explicit value is provided, preventing uninitialized variable errors.
Zero Values:
0false"" (empty string)nil// General syntax: var name type = expression
var s string // Initialized to "" (zero value)
var i, j, k int // Multiple variables of the same type
var b, f, s = true, 2.3, "four" // Type inference from expressions
// Initialize from function returning multiple values
var f, err = os.Open(name)Short variable declarations use the := operator to declare and initialize local variables inside a function. The type is automatically inferred from the expression on the right.
Key rules:
a, b := 1, 2), at least one variable on the left side must be new to the current lexical scope. If some variables were already declared in the same scope, they are simply assigned new values.:= is for declaration and initialization; = is for assignment to an existing variable.// Single variable
t := 0.0
// Multiple variables
i, j := 0, 1
// Using function return values
f, err := os.Open(name)
// Partial redeclaration: 'out' is new, 'err' is reused
in, err := os.Open(infile)
out, err := os.Create(outfile)
// This would cause a compile error: "no new variables"
f, err := os.Open(infile)
f, err := os.Create(outfile)