repository·main·Indexed 21 days ago
https://github.com/facebook/starlark-rustA deterministic, Python-inspired configuration language implementation written in Rust, used by build systems like Buck2. The project includes the main starlark library for embedding, starlark_lsp for IDE integration, and supporting crates such as starlark_syntax and starlark_map. It also provides utility libraries like Allocative for memory profiling and gazebo for common Rust type extensions.
Allocative is a lightweight memory profiler for Rust that enables object traversal and memory size introspection. Unlike traditional malloc profilers that focus on call stacks, Allocative provides an object-by-object tree view.
Key capabilities include:
bumpalo bumps).To use it, you must implement the Allocative trait for the types you wish to measure. This is typically done using a procedural macro provided by the crate.
This directory contains a mirrored set of evaluation test cases from the original Go Starlark project. These test cases are used to verify the correctness of the Starlark implementation.
Note on selection: Not all files from the original source were included. Some files were excluded because they are considered unsuitable tests for Starlark, as determined by the test_go function in the original repository.
The gazebo::dupe module provides the Dupe trait with a dupe method. This is conceptually similar to Clone, but intended for types where cloning is extremely inexpensive (e.g., Arc or usize).
Mental Model:
Use dupe instead of clone to distinguish between cheap operations and expensive reallocations (like String or Vec). This makes it easier to audit code for performance by making expensive clone calls stand out.
When deciding between Allocative and a tool like the jemalloc heap profiler, consider these differences:
| Feature | Allocative | Call-stack Malloc Profiler |
|---|---|---|
| Primary View | Object-by-object tree | Call stack |
| Setup | Requires Allocative trait implementation | Usually requires no code changes |
| Memory Gaps | Shows spare capacity and padding | Focuses on allocation sites |
| Allocation Types | Supports non-malloc (e.g., bump allocators) | Primarily tracks malloc |
| Granularity | Can profile specific subsets of data | Profiles the entire heap |
To optimize evaluation, Starlark accesses variables via integer 'slots' rather than string lookups.
Variables are stored in a Slots structure. A slot can be:
Frozen: Used for variables from imported modules (stored behind an Arc).Slots: Mutable variables for the current scope (stored behind Rc<RefCell<Vec<Option<Value>>>>).The mapping from a string name to a slot index is managed by the Names enum:
Frozen(FrozenNames): Uses Arc<HashMap<String, usize>> for immutable mappings.Names(Rc<RefCell<HashMap<String, usize>>>): Uses Rc<RefCell<...>> for mutable mappings.When a scope is frozen (e.g., when a module is imported), the mutable Names and Slots are consumed and replaced with their Frozen variants.
Warning: It is unsafe to access the original mutable slots after a freeze operation has been called.
enum Slots {
Frozen(FrozenSlots),
Slots(Rc<RefCell<Vec<Option<Value>>>>),
}
struct FrozenSlots(Arc<Vec<Option<FrozenValue>>>);
enum Names {
Frozen(FrozenNames),
Names(Rc<RefCell<HashMap<String, usize>>>),
}
struct FrozenNames(Arc<HashMap<String, usize>>);Starlark uses a hierarchy of environments (scopes) that allow nested functions to access and mutate variables from outer scopes, but not assign to them.
list.append(x, 1) where x is defined in an outer scope).=) inside a scope creates a new local variable that shadows the outer variable. You cannot assign to a variable defined in an outer scope from within a nested scope.x = [] # Module environment
def foo():
y = True # Environment of foo
def bar():
z = 1 # Environment of bar
list.append(x, 1) # Accesses x from module, list.append from globalx = []
def foo():
y = True
def bar():
z = 1
list.append(x, 1)Starlark manages memory through three distinct heap types, each with different ownership and sharing rules:
Heap: Allocates Values. It cannot be cloned or shared.FrozenHeap: Allocates FrozenValues. It cannot be cloned or shared.FrozenHeapRef: A read-only version of a FrozenHeap. It can be cloned and shared, and it is responsible for keeping the underlying FrozenHeap alive.When a value from one heap is used to construct a value in another heap (e.g., putting a FrozenValue from h1 into a list in h2), you must ensure the source heap is added as a reference to the destination heap to prevent the source from being deallocated while in use.
Starlark distinguishes between frozen and unfrozen values to allow for safe, parallel sharing of imported modules without expensive copying.
Once a module finishes executing, its local values are frozen and can be reused freely by other parts of the program.
To optimize performance when functions return list literals (which are technically unfrozen and mutable), Starlark uses a thaw-on-write strategy.
Instead of returning a fully mutable list, the interpreter can return a shared reference to a frozen list. If a caller attempts to mutate this list, the interpreter 'thaws' the value by copying it into a mutable variant. This allows most callers to treat the returned value as immutable without the overhead of constant copying, only paying the cost if mutation actually occurs.
List comprehensions (e.g., [x for x in [1,2,3]]) create a specialized, temporary environment.
x) is immediately initialized and shadows any existing x in the outer scope.Slots data type without adding them to the Names mapping.[x for x in [1,2,3]]Certain data types in Starlark are considered Pseudo values. These are types that are themselves immutable but contain references to mutable data.
Key characteristics:
lambda, def, or method calls like a.b()) fall into this category.