Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ Inspired by the [*Starlight*](https://github.com/starlight-go/starlight) and [*S
| [`regex`](/lib/regex) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/regex.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/regex) | Python-style regex (RE2) with Match objects, flags, named groups |
| [`runtime`](/lib/runtime) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/runtime.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/runtime) | Provides Go and app runtime information |
| [`serial`](/lib/serial) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/serial.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/serial) | Lossless data-value serialization (bytes/set/tuple/bigint/time) |
| [`starlark`](/lib/starlark) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/starlark.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/starlark) | Read-only data snapshots for Starlark expressions and structured try errors |
| [`stats`](/lib/stats) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/stats.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/stats) | Statistics: mean/median/stddev, correlation, percentiles, etc. |
| [`string`](/lib/string) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/string.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/string) | Constants and functions to manipulate strings |

Expand Down Expand Up @@ -106,7 +107,7 @@ This may output:
```
Starlark: Hello, Starlet!
Go: Hello, Starlet!
Modules: [atom base64 csv file go_idiomatic hashlib http json log math net path random re regex runtime serial stats string struct time]
Modules: [atom base64 csv file go_idiomatic hashlib http json log math net path random re regex runtime serial starlark stats string struct time]
```

Use CLI to interact with the read-eval-print loop (REPL):
Expand Down
3 changes: 3 additions & 0 deletions config.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import (
libregex "github.com/1set/starlet/lib/regex"
librt "github.com/1set/starlet/lib/runtime"
libserial "github.com/1set/starlet/lib/serial"
libsl "github.com/1set/starlet/lib/starlark"
libstat "github.com/1set/starlet/lib/stats"
libstr "github.com/1set/starlet/lib/string"
stdmath "go.starlark.net/lib/math"
Expand Down Expand Up @@ -65,6 +66,7 @@ var allBuiltinModules = ModuleLoaderMap{
librt.ModuleName: librt.LoadModule,
libserial.ModuleName: libserial.LoadModule,
libstr.ModuleName: libstr.LoadModule,
libsl.ModuleName: libsl.LoadModule,
libstat.ModuleName: libstat.LoadModule,
}

Expand Down Expand Up @@ -190,6 +192,7 @@ var builtinModuleCapabilities = map[string]ModuleCapability{
"regex": CapPure,
"runtime": CapProcess, // exposes host identity, directories, process info
"serial": CapPure,
"starlark": CapPure, // explicit data-only evaluation; no caller capabilities inherited
"stats": CapPure,
"string": CapPure,
"struct": CapPure,
Expand Down
145 changes: 145 additions & 0 deletions lib/starlark/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
# starlark

`starlark` evaluates a single **Starlark expression** against an explicit,
read-only data snapshot. It provides normal and tuple-returning error handling,
without inheriting caller functions, modules or host objects.

The name identifies the language. It does not evaluate
[Expr-language](https://github.com/expr-lang/expr) syntax, and does not guess or
retry another language. `expr` is left available for a separate future adapter.

## Functions

| Function | Result |
| --- | --- |
| `eval(source, vars={})` | A frozen native data value; raises an error on failure. |
| `try_eval(source, vars={})` | `(value, None)` on success, or `(None, error_info)` on an ordinary evaluation failure. |

`source` must be a string. `vars` must be a native dict whose keys are valid
Starlark identifiers. Omitting `vars` supplies an empty environment. The module
does not inspect caller locals or copy a Machine's implicit globals.

```python
load("starlark", "eval", "try_eval")

items = [2, 3]
result = eval(
'{"title": title.upper(), "values": [x * 2 for x in items]}',
vars={"title": "example", "items": items},
)
# {"title": "EXAMPLE", "values": [4, 6]}

value, error = try_eval("items.append(9)", vars={"items": items})
# value == None; error["phase"] == "runtime"
items.append(4) # The original input was neither changed nor frozen.

value, error = try_eval("None")
# (None, None): a valid None result is distinct from failure.
```

A preloaded module is also available as `starlark.eval(...)` and
`starlark.try_eval(...)`. When importing both this module and `json.eval`, use
explicit load aliases to distinguish the engines.

## Data and method boundaries

Inputs and results preserve native `None`, bool, int (including big integers),
float, string, bytes, list, dict, tuple and set values. Nested data is supported;
aliases are preserved within a copied graph. Reference cycles, nil values,
functions, bound methods, modules, iterators and custom or Go-backed values are
rejected with a data path. Convert a host object into an explicit native data
projection first; wrapping it with Starlight does not create a snapshot.

Only private input copies are frozen. Native read methods such as `upper`,
`lower`, `get` and `index`, comprehensions and approved computational builtins
remain usable. Mutating a frozen input fails. Expressions may create and modify
their own temporary containers; results are checked, copied and frozen before
publication. Even a frozen lambda or bound method remains callable, so callable
results are rejected recursively.

Caller functions and external modules are not inherited. Unapproved Universe
builtins, including `print`, are blocked without changing the global Universe or
calling the parent's Print handler. The standard computational builtin instances
are captured during package initialization; the embedding application's startup
must retain the standard definitions then. As upstream requires, do not modify
Universe concurrently with execution. This policy is independent of Starlet's
CapPure classification: random and clock sources are not automatically exposed.

The source graph must be exclusively owned/synchronized during evaluation, or
already deeply frozen native data. A lack of writers alone is insufficient:
iterating an unfrozen native set can change its iterator bookkeeping. The helper
does not acquire a Machine or application data lock for the caller.

## Errors and execution state

`error_info` is a frozen dict with these fields:

- `phase` and `kind`: the failing stage or policy category;
- `message`: a diagnostic string;
- `position`: `None` or `{file, line, column}`;
- `frames`: a tuple of `{file, line, column, name}` dicts;
- `truncated`: whether diagnostic text or frames were shortened.

Stages distinguish arguments, input, source admission, parsing, name resolution,
runtime, policy and output checks. Messages are limited to 4096 bytes, the most
recent 32 frames are retained, and frame file/name text is bounded.

Both functions execute on the caller's existing Thread. Its cancellation,
OnMaxSteps behavior and cumulative VM step count remain in force; no new step
allowance is installed or reset. Execution checkpoints also consume VM steps.
`try_eval` does **not** turn caller cancellation, exhausted total steps or a caller
panic into a successful tuple. It only handles errors occurring inside the call;
errors while evaluating the call's arguments still belong to the outer script.

There is no rollback of host callback effects. Such callbacks are excluded from
the data boundary. Errors discard the private evaluation data.

## Admission limits

| Limit | Value |
| --- | --- |
| Source bytes | 4096 |
| Non-whitespace source bytes | 512 |
| AST depth / nodes | 64 / 512 |
| Each copied data graph's depth / visited edges | 64 / 4096 |
| Each copied graph's scalar/name payload | 1 MiB |
| Integer magnitude | At most 4096 bits |

The pre-parser complexity count includes non-whitespace bytes in strings and
comments; only ASCII space, tab, CR and LF are excluded. Put large data in `vars`.
Input and output copying have separate graph quotas. Every visited edge counts,
including memo hits; payload accounting follows the copied graph rather than an
expanded JSON serialization.

These are admission and result limits, **not hard RSS or CPU limits**. The pinned
parser's general recursion issue is not repaired here. VM steps do not account for
every operation inside a native builtin; an intermediate allocation can occur
before output admission. Configure the caller's execution limits, and use a
separately validated process boundary when stronger isolation is required.

## Go usage

```go
import (
evalmod "github.com/1set/starlet/lib/starlark"
"go.starlark.net/starlark"
)

thread := &starlark.Thread{Name: "expression"}
thread.SetMaxExecutionSteps(10000)
result, err := evalmod.Eval(thread, "qty * 2", starlark.StringDict{
"qty": starlark.MakeInt(3),
})
```

The caller owns the Thread and source graph. Direct Go callers must wire
cancellation to `Thread.Cancel` when cancellation during VM execution is needed;
a context stored in Thread.Local is checked at helper checkpoints, not a new
cancellation goroutine. Machine calls retain their existing cancellation bridge.
The Go helper returns `starlark.Value` and an error preserving parser, resolver and
EvalError causes through `errors.As`/`errors.Is`. It does not implicitly convert
the result to Go or JSON.

The module is registered as CapPure because it grants no filesystem, network,
process or logging access. Downstream module sets still decide whether to expose
it; CapPure does not itself establish deterministic execution or a sandbox.
207 changes: 207 additions & 0 deletions lib/starlark/snapshot.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
package starlark

import (
"fmt"
"math/big"
"reflect"

std "go.starlark.net/starlark"
)

const (
maxDataDepth = 64
maxDataNodes = 4096
maxDataBytes = 1 << 20
maxIntBits = 4096
)

var integerBound = std.MakeBigInt(new(big.Int).Lsh(big.NewInt(1), maxIntBits))
var negativeIntegerBound = std.MakeBigInt(new(big.Int).Neg(new(big.Int).Lsh(big.NewInt(1), maxIntBits)))

type objectKey struct {
kind byte
ptr uintptr
len int
}

type dataCopy struct {
thread *std.Thread
nodes int
bytes int
active map[objectKey]bool
memo map[objectKey]std.Value
height map[objectKey]int
}

func newDataCopy(thread *std.Thread) *dataCopy {
return &dataCopy{thread: thread, active: make(map[objectKey]bool), memo: make(map[objectKey]std.Value), height: make(map[objectKey]int)}
}

func (c *dataCopy) chargeBytes(n int, path string) error {
if n > maxDataBytes-c.bytes {
return fmt.Errorf("%s: data exceeds %d bytes", path, maxDataBytes)
}
c.bytes += n
return nil
}

func (c *dataCopy) value(value std.Value, path string, depth int) (std.Value, error) {
c.nodes++ // Every edge counts, including an alias that hits the memo.
if c.nodes > maxDataNodes {
return nil, fmt.Errorf("%s: data exceeds %d nodes", path, maxDataNodes)
}
if depth > maxDataDepth {
return nil, fmt.Errorf("%s: data exceeds depth %d", path, maxDataDepth)
}
if c.nodes%64 == 0 {
if err := checkThread(c.thread); err != nil {
return nil, err
}
}
switch x := value.(type) {
case std.NoneType, std.Bool, std.Float:
return x, c.chargeBytes(8, path)
case std.Int:
upper, _ := x.Cmp(integerBound, 0)
lower, _ := x.Cmp(negativeIntegerBound, 0)
if upper >= 0 || lower <= 0 {
return nil, fmt.Errorf("%s: integer exceeds %d bits", path, maxIntBits)
}
return x, c.chargeBytes((x.BigInt().BitLen()+7)/8, path)
case std.String:
return x, c.chargeBytes(len(x), path)
case std.Bytes:
return x, c.chargeBytes(len(x), path)
case *std.List:
if x == nil {
return nil, fmt.Errorf("%s: nil list", path)
}
key := objectKey{kind: 'l', ptr: reflect.ValueOf(x).Pointer()}
if known, found, err := c.lookup(key, path, depth); found || err != nil {
return known, err
}
if x.Len() > maxDataNodes-c.nodes {
return nil, fmt.Errorf("%s: list exceeds node limit", path)
}
out := std.NewList(nil)
c.memo[key], c.active[key] = out, true
defer delete(c.active, key)
for i := 0; i < x.Len(); i++ {
item, err := c.value(x.Index(i), fmt.Sprintf("%s[%d]", path, i), depth+1)
if err != nil {
return nil, err
}
_ = out.Append(item)
c.childHeight(key, x.Index(i))
}
return out, nil
case *std.Dict:
if x == nil {
return nil, fmt.Errorf("%s: nil dict", path)
}
key := objectKey{kind: 'd', ptr: reflect.ValueOf(x).Pointer()}
if known, found, err := c.lookup(key, path, depth); found || err != nil {
return known, err
}
if x.Len() > (maxDataNodes-c.nodes)/2 {
return nil, fmt.Errorf("%s: dict exceeds node limit", path)
}
out := std.NewDict(x.Len())
c.memo[key], c.active[key] = out, true
defer delete(c.active, key)
for i, pair := range x.Items() {
k, err := c.value(pair[0], fmt.Sprintf("%s.key[%d]", path, i), depth+1)
if err != nil {
return nil, err
}
v, err := c.value(pair[1], fmt.Sprintf("%s.value[%d]", path, i), depth+1)
if err != nil {
return nil, err
}
if err := out.SetKey(k, v); err != nil {
return nil, fmt.Errorf("%s: %w", path, err)
}
c.childHeight(key, pair[0])
c.childHeight(key, pair[1])
}
return out, nil
case std.Tuple:
key := objectKey{kind: 't', ptr: reflect.ValueOf(x).Pointer(), len: len(x)}
if known, found, err := c.lookup(key, path, depth); found || err != nil {
return known, err
}
if len(x) > maxDataNodes-c.nodes {
return nil, fmt.Errorf("%s: tuple exceeds node limit", path)
}
out := make(std.Tuple, len(x))
c.memo[key], c.active[key] = out, true
defer delete(c.active, key)
for i, item := range x {
v, err := c.value(item, fmt.Sprintf("%s[%d]", path, i), depth+1)
if err != nil {
return nil, err
}
out[i] = v
c.childHeight(key, item)
}
return out, nil
case *std.Set:
if x == nil {
return nil, fmt.Errorf("%s: nil set", path)
}
key := objectKey{kind: 's', ptr: reflect.ValueOf(x).Pointer()}
if known, found, err := c.lookup(key, path, depth); found || err != nil {
return known, err
}
if x.Len() > maxDataNodes-c.nodes {
return nil, fmt.Errorf("%s: set exceeds node limit", path)
}
out := std.NewSet(x.Len())
c.memo[key], c.active[key] = out, true
defer delete(c.active, key)
it := x.Iterate()
defer it.Done()
var item std.Value
for i := 0; it.Next(&item); i++ {
v, err := c.value(item, fmt.Sprintf("%s[%d]", path, i), depth+1)
if err != nil {
return nil, err
}
if err := out.Insert(v); err != nil {
return nil, fmt.Errorf("%s: %w", path, err)
}
c.childHeight(key, item)
}
return out, nil
default:
return nil, fmt.Errorf("%s: unsupported data value %T", path, value)
}
}

func (c *dataCopy) lookup(key objectKey, path string, depth int) (std.Value, bool, error) {
if c.active[key] {
return nil, false, fmt.Errorf("%s: reference cycle is unsupported", path)
}
v, found := c.memo[key]
if found && depth+c.height[key] > maxDataDepth {
return nil, false, fmt.Errorf("%s: aliased data exceeds depth %d", path, maxDataDepth)
}
return v, found, nil
}

func (c *dataCopy) childHeight(parent objectKey, value std.Value) {
height := 1
switch x := value.(type) {
case *std.List:
height += c.height[objectKey{kind: 'l', ptr: reflect.ValueOf(x).Pointer()}]
case *std.Dict:
height += c.height[objectKey{kind: 'd', ptr: reflect.ValueOf(x).Pointer()}]
case *std.Set:
height += c.height[objectKey{kind: 's', ptr: reflect.ValueOf(x).Pointer()}]
case std.Tuple:
height += c.height[objectKey{kind: 't', ptr: reflect.ValueOf(x).Pointer(), len: len(x)}]
}
if height > c.height[parent] {
c.height[parent] = height
}
}
Loading
Loading