From b12f37e95113ed75eae22e6c0f86873c730afe97 Mon Sep 17 00:00:00 2001 From: Kevin Tang <73975146+vt128@users.noreply.github.com> Date: Sun, 8 Nov 2026 19:20:35 +0800 Subject: [PATCH 1/5] [feat] add read-only eval builtins (LET-36) --- README.md | 2 +- lib/goidiomatic/README.md | 155 +++++++++- lib/goidiomatic/eval.go | 341 +++++++++++++++++++++ lib/goidiomatic/eval_test.go | 484 ++++++++++++++++++++++++++++++ lib/goidiomatic/idiomatic.go | 2 + lib/goidiomatic/idiomatic_test.go | 32 ++ lib/goidiomatic/snapshot.go | 207 +++++++++++++ lib/goidiomatic/snapshot_test.go | 203 +++++++++++++ lib/goidiomatic/source.go | 73 +++++ lib/goidiomatic/source_test.go | 107 +++++++ module_test.go | 85 ++++++ 11 files changed, 1689 insertions(+), 2 deletions(-) create mode 100644 lib/goidiomatic/eval.go create mode 100644 lib/goidiomatic/eval_test.go create mode 100644 lib/goidiomatic/snapshot.go create mode 100644 lib/goidiomatic/snapshot_test.go create mode 100644 lib/goidiomatic/source.go create mode 100644 lib/goidiomatic/source_test.go diff --git a/README.md b/README.md index 35e693a7..5d6d75bc 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ Inspired by the [*Starlight*](https://github.com/starlight-go/starlight) and [*S | [`base64`](/lib/base64) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/base64.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/base64) | Base64 encoding & decoding functions | | [`csv`](/lib/csv) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/csv.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/csv) | Parses and writes comma-separated values (csv) contents | | [`file`](/lib/file) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/file.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/file) | Functions to interact with the file system | -| [`goidiomatic`](/lib/goidiomatic) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/goidiomatic.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/goidiomatic) | Go idiomatic functions and values for Starlark | +| [`goidiomatic`](/lib/goidiomatic) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/goidiomatic.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/goidiomatic) | Extended builtins, data-only expression evaluation and constructors | | [`hashlib`](/lib/hashlib) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/hashlib.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/hashlib) | Hash primitives for Starlark | | [`http`](/lib/http) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/http.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/http) | HTTP client and server handler implementation for Starlark | | [`json`](/lib/json) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/json.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/json) | Utilities for converting Starlark values to/from JSON strings | diff --git a/lib/goidiomatic/README.md b/lib/goidiomatic/README.md index cd16a9c2..e450e843 100644 --- a/lib/goidiomatic/README.md +++ b/lib/goidiomatic/README.md @@ -1,6 +1,6 @@ # go_idiomatic -`go_idiomatic` provides Go-flavored helpers, constants, and constructors for Starlark scripts — base conversions, `length`/`sum`/`distinct` utilities, struct and shared-dict factories, and `print`-style output. It is **not** a mirror of any single Python module; it borrows the spirit of several Python builtins (`hex`/`oct`/`bin`/`sum`/`bytes.hex`) while adding Star\* idioms. +`go_idiomatic` provides Go-flavored helpers, constants, and constructors for Starlark scripts — base conversions, `length`/`sum`/`distinct` utilities, struct and shared-dict factories, data-only expression evaluation, and `print`-style output. It is **not** a mirror of any single Python module; it borrows the spirit of several Python builtins (`hex`/`oct`/`bin`/`sum`/`bytes.hex`) while adding Star\* idioms. Capability profile: **Log + Process**. `eprint`/`pprint` write to stderr or the thread's `Print` handler (Log); `sleep` blocks the thread and `exit`/`quit` halt the program (Process). Everything else is pure. @@ -10,6 +10,8 @@ Load name for `load()`: `go_idiomatic`. | function | description | |----------|-------------| +| `eval(source, vars={}) -> value` | Evaluate one Starlark expression against copied, read-only native data. | +| `try_eval(source, vars={}) -> (value, error_info)` | Return ordinary evaluation errors as frozen data; preserve caller cancellation and total step limits. | | `length(obj) -> int` | Length of a string (in Unicode code points), bytes, or any sequence. | | `sum(iterable, start=0) -> number` | Sum of `start` plus the items of `iterable`, left to right. | | `distinct(iterable) -> iterable` | Iterable with duplicates removed; shape follows the input type. | @@ -283,3 +285,154 @@ pprint({"key": "value", "list": [1, 2, 3]}) - **Process effects.** `sleep` blocks (and respects context cancellation); `exit`/`quit` do not stop the goroutine themselves — they return `ErrSystemExit` for the host runner to act on, and set the thread-local `exit_code`. - **Determinism.** `struct`/`make_struct`/`module` print fields in sorted order; `to_dict` of a `GoStruct` keeps the Go field names verbatim. - **`shared_dict` scope.** Locking protects a single instance across threads; it does not make the values it holds deeply immutable or persist anything beyond the process. + +## Expression evaluation + +`eval` and `try_eval` evaluate one **Starlark expression**, using only explicit +variables and approved computational builtins. They do not execute Go or +[Expr-language](https://github.com/expr-lang/expr) syntax, guess the language, or +retry failures with another engine. The `expr` module name remains available +for a separate future Expr-language adapter. + +`source` must be a string. `vars` must be a native dict whose keys are valid +Starlark identifiers. Omitting `vars` supplies an empty environment. The module +does not inspect caller locals or copy a Machine's implicit globals. + +```python +load("go_idiomatic", "eval", "try_eval") + +items = [2, 3] +result = eval( + '{"title": title.upper(), "values": [x * 2 for x in items]}', + vars={"title": "example", "items": items}, +) +# {"title": "EXAMPLE", "values": [4, 6]} + +value, error = try_eval("items.append(9)", vars={"items": items}) +# value == None; error["phase"] == "runtime" +items.append(4) # The original input was neither changed nor frozen. + +value, error = try_eval("None") +# (None, None): a valid None result is distinct from failure. +``` + +Preloading `go_idiomatic` makes `eval(...)` and `try_eval(...)` available as +bare functions, like its existing `sum` and `hex` helpers. A `NewDefault()` +Machine does not preload modules; configure this module explicitly. Neither a +`starlark` module nor a `go_idiomatic.eval` attribute wrapper is registered. +When importing `json.eval` or a future Expr-language evaluator too, use explicit +load aliases to distinguish the engines. + +The existing Machine precedence also applies to these two new names: preload +functions override same-named host globals; per-run extras override preloads; +explicit `load` bindings take precedence in the script. Use lazy loading with +an alias, such as `load("go_idiomatic", evaluate="eval")`, when keeping a host's +existing `eval` name matters. + +### Data and method boundaries + +Inputs and results preserve native `None`, bool, int (including big integers), +float, string, bytes, list, dict, tuple and set values. Nested data is supported; +aliases are preserved within a copied graph. Reference cycles, nil values, +functions, bound methods, modules, iterators and custom or Go-backed values are +rejected with a data path. Convert a host object into an explicit native data +projection first; wrapping it with Starlight does not create a snapshot. + +Only private input copies are frozen. Native read methods such as `upper`, +`lower`, `get` and `index`, comprehensions and approved computational builtins +remain usable. Mutating a frozen input fails. Expressions may create and modify +their own temporary containers; results are checked, copied and frozen before +publication. Even a frozen lambda or bound method remains callable, so callable +results are rejected recursively. + +Caller functions and external modules are not inherited. Unapproved Universe +builtins, including `print`, are blocked without changing the global Universe or +calling the parent's Print handler. The standard computational builtin instances +are captured during package initialization; the embedding application's startup +must retain the standard definitions then. As upstream requires, do not modify +Universe concurrently with execution. Other `go_idiomatic` helpers retain their existing Log/Process capabilities; +they are not inherited by an expression. Random and clock sources are not +automatically exposed either. + +The source graph must be exclusively owned/synchronized during evaluation, or +already deeply frozen native data. A lack of writers alone is insufficient: +iterating an unfrozen native set can change its iterator bookkeeping. The helper +does not acquire a Machine or application data lock for the caller. + +### Errors and execution state + +`error_info` is a frozen dict with these fields: + +- `phase` and `kind`: the failing stage or policy category; +- `message`: a diagnostic string; +- `position`: `None` or `{file, line, column}`; +- `frames`: a tuple of `{file, line, column, name}` dicts; +- `truncated`: whether diagnostic text or frames were shortened. + +Stages distinguish arguments, input, source admission, parsing, name resolution, +runtime, policy and output checks. Messages are limited to 4096 bytes, the most +recent 32 frames are retained, and frame file/name text is bounded. + +Both functions execute on the caller's existing Thread. Its cancellation, +OnMaxSteps behavior and cumulative VM step count remain in force; no new step +allowance is installed or reset. Execution checkpoints also consume VM steps. +`try_eval` does **not** turn caller cancellation, exhausted total steps or a caller +panic into a successful tuple. It only handles errors occurring inside the call; +errors while evaluating the call's arguments still belong to the outer script. + +There is no rollback of host callback effects. Such callbacks are excluded from +the data boundary. Errors discard the private evaluation data. + +### Admission limits + +| Limit | Value | +| --- | --- | +| Source bytes | 4096 | +| Non-whitespace source bytes | 512 | +| AST depth / nodes | 64 / 512 | +| Each copied data graph's depth / visited edges | 64 / 4096 | +| Each copied graph's scalar/name payload | 1 MiB | +| Integer magnitude | At most 4096 bits | + +The pre-parser complexity count includes non-whitespace bytes in strings and +comments; only ASCII space, tab, CR and LF are excluded. Put large data in `vars`. +Input and output copying have separate graph quotas. Every visited edge counts, +including memo hits; payload accounting follows the copied graph rather than an +expanded JSON serialization. + +These are admission and result limits, **not hard RSS or CPU limits**. The pinned +parser's general recursion issue is not repaired here. VM steps do not account for +every operation inside a native builtin; an intermediate allocation can occur +before output admission. Configure the caller's execution limits, and use a +separately validated process boundary when stronger isolation is required. + +### Go usage + +```go +import ( + "github.com/1set/starlet/lib/goidiomatic" + "go.starlark.net/starlark" +) + +thread := &starlark.Thread{Name: "expression"} +thread.SetMaxExecutionSteps(10000) +result, err := goidiomatic.Eval(thread, "qty * 2", starlark.StringDict{ + "qty": starlark.MakeInt(3), +}) +``` + +The caller owns the Thread and source graph. Direct Go callers must wire +cancellation to `Thread.Cancel` when cancellation during VM execution is needed; +a context stored in Thread.Local is checked at helper checkpoints, not a new +cancellation goroutine. Machine calls retain their existing cancellation bridge. +The Go helper returns `starlark.Value` and an error preserving parser, resolver and +EvalError causes through `errors.As`/`errors.Is`. It does not implicitly convert +the result to Go or JSON. + +The existing `go_idiomatic` capability classification remains Log + Process. +Adding these data-only helpers does not reduce that classification or expose the +module's other helpers inside an expression. Consumers that already allow or +preload `go_idiomatic` receive the new names with it; in particular, Starbox's +Safe and Network module sets already contain this module. Hosts needing a +separate explicit opt-in can instead pass `goidiomatic.Eval` through their own +loader or builtin policy. None of these policies establishes a hard sandbox. diff --git a/lib/goidiomatic/eval.go b/lib/goidiomatic/eval.go new file mode 100644 index 00000000..428d28b5 --- /dev/null +++ b/lib/goidiomatic/eval.go @@ -0,0 +1,341 @@ +// The expression helpers provide data-only, read-only Starlark evaluation. +// They use explicit variables, preserve their native types, and do not expose +// caller functions, modules, host wrappers, or resource handles. This is not a +// process sandbox or a hard allocation limit; the caller owns execution limits. +package goidiomatic + +import ( + "context" + "errors" + "fmt" + "sync" + "unicode" + "unicode/utf8" + + "go.starlark.net/resolve" + std "go.starlark.net/starlark" + "go.starlark.net/syntax" +) + +// Error identifies the stage of a failed expression evaluation. Cause retains +// parser/resolver errors and EvalError backtraces for errors.As/errors.Is. +type Error struct { + Phase string + Kind string + Cause error +} + +func (e *Error) Error() string { + message, _ := diagnosticText(e.Cause.Error(), maxErrorMessage) + return "eval: " + e.Phase + ": " + message +} +func (e *Error) Unwrap() error { return e.Cause } + +func evalError(phase string, cause error) *Error { + return &Error{Phase: phase, Kind: phase, Cause: cause} +} + +var ( + // Builtins are immutable and capture only their fixed error-handling mode. + // Reuse them across fresh module dictionaries; no Thread or vars are cached. + evalBuiltin = std.NewBuiltin("eval", evaluationBuiltin(false)) + tryEvalBuiltin = std.NewBuiltin("try_eval", evaluationBuiltin(true)) + // Capture the standard, immutable builtin instances before evaluation. + // Like upstream, this package requires Universe not be mutated concurrently + // with execution. A later replacement of an allowed name is not trusted. + pureBuiltins = captureBuiltins() + probeOnce sync.Once + threadProbe *std.Function +) + +func captureBuiltins() std.StringDict { + names := []string{"None", "True", "False", "abs", "any", "all", "bool", "bytes", "chr", "dict", "dir", "enumerate", "fail", "float", "getattr", "hasattr", "hash", "int", "len", "list", "max", "min", "ord", "range", "repr", "reversed", "set", "sorted", "str", "tuple", "type", "zip"} + result := make(std.StringDict, len(names)) + for _, name := range names { + result[name] = std.Universe[name] + } + return result +} + +func makeThreadProbe() *std.Function { + f, err := std.ExprFuncOptions(&syntax.FileOptions{Set: true}, "", "0", nil) + if err != nil { + panic(err) // A fixed integer literal cannot fail to compile. + } + f.Freeze() + return f +} + +// checkThread runs a constant, closed expression on the caller's existing VM. +// This checks its private cancellation/step state without inspecting internals, +// resetting limits, or relying on the wording of an EvalError. In particular, +// an OnMaxSteps panic propagates unchanged to the caller's execution boundary. +func checkThread(thread *std.Thread) error { + if thread == nil { + return evalError("input", fmt.Errorf("thread is nil")) + } + if ctx, ok := thread.Local("context").(context.Context); ok && ctx != nil { + if err := ctx.Err(); err != nil { + return evalError("execution", err) + } + } + probeOnce.Do(func() { threadProbe = makeThreadProbe() }) + if _, err := std.Call(thread, threadProbe, nil, nil); err != nil { + return evalError("execution", err) + } + return nil +} + +// Eval evaluates one Starlark expression against an owned, frozen copy of vars. +// The caller must provide an active or exclusively owned Thread and either an +// exclusively owned/synchronized variables graph or already-frozen native +// data. Merely avoiding writes does not make an unfrozen set safe to share: +// native iteration updates its bookkeeping. Values and names are checked +// before interpretation; results are copied, frozen native data only. +// +// Eval preserves the caller's cancellation and cumulative step accounting. +// It does not set a new step limit, recover caller panics, or capture locals. +func Eval(thread *std.Thread, source string, vars std.StringDict) (std.Value, error) { + if err := checkThread(thread); err != nil { + return std.None, err + } + expr, phase, err := parseSource(source) + if err != nil { + return std.None, evalError(phase, err) + } + if len(vars) > maxDataNodes { + return std.None, evalError("input", fmt.Errorf("too many variables")) + } + nameBytes := 0 + for name := range vars { + if len(name) > maxDataBytes-nameBytes { + return std.None, evalError("input", fmt.Errorf("variable names exceed data byte limit")) + } + nameBytes += len(name) + } + copy := newDataCopy(thread) + env := make(std.StringDict, len(vars)+len(std.Universe)) + names := vars.Keys() + for _, name := range names { + if err := copy.chargeBytes(len(name), "vars"); err != nil { + return std.None, evalError("input", err) + } + if !validName(name) { + preview, _ := diagnosticText(name, 64) + return std.None, evalError("input", fmt.Errorf("invalid variable name %q (%d bytes)", preview, len(name))) + } + value, err := copy.value(vars[name], "vars."+name, 0) + if err != nil { + return std.None, wrapInputError(err) + } + env[name] = value + } + // Only private copies are frozen, after the full input graph was accepted. + env.Freeze() + for name := range std.Universe { + if _, exists := env[name]; exists { + continue + } + if value, approved := pureBuiltins[name]; approved { + env[name] = value + } else { + env[name] = deniedBuiltin(name) + } + } + // Approved instances remain usable even if another host replaced or removed + // a Universe entry before execution. Data variables keep normal shadowing. + for name, value := range pureBuiltins { + if _, exists := env[name]; !exists { + env[name] = value + } + } + if err := checkThread(thread); err != nil { + return std.None, err + } + value, err := std.EvalExprOptions(&syntax.FileOptions{Set: true}, thread, expr, env) + if err != nil { + if terminal := checkThread(thread); terminal != nil { + return std.None, terminal + } + var own *Error + if errors.As(err, &own) { + return std.None, &Error{Phase: own.Phase, Kind: own.Kind, Cause: err} + } + var unresolved resolve.ErrorList + if errors.As(err, &unresolved) { + return std.None, evalError("resolve", err) + } + return std.None, evalError("runtime", err) + } + result, err := newDataCopy(thread).value(value, "result", 0) + if err != nil { + var terminal *Error + if errors.As(err, &terminal) { + return std.None, err + } + return std.None, evalError("output", err) + } + if err := checkThread(thread); err != nil { + return std.None, err + } + result.Freeze() + return result, nil +} + +func wrapInputError(err error) error { + var execution *Error + if errors.As(err, &execution) { + return err + } + return evalError("input", err) +} + +func validName(name string) bool { + if len(name) == 0 || len(name) > 512 || !utf8.ValidString(name) { + return false + } + for i, r := range name { + if r != '_' && !unicode.IsLetter(r) && (i == 0 || !unicode.IsDigit(r)) { + return false + } + } + expr, err := (&syntax.FileOptions{}).ParseExpr("", name, 0) + if err != nil { + return false + } + ident, ok := expr.(*syntax.Ident) + return ok && ident.Name == name +} + +func deniedBuiltin(name string) *std.Builtin { + return std.NewBuiltin(name, func(*std.Thread, *std.Builtin, std.Tuple, []std.Tuple) (std.Value, error) { + return std.None, &Error{Phase: "policy", Kind: "capability", Cause: fmt.Errorf("builtin %q is unavailable", name)} + }) +} + +func evaluationBuiltin(try bool) func(*std.Thread, *std.Builtin, std.Tuple, []std.Tuple) (std.Value, error) { + return func(thread *std.Thread, fn *std.Builtin, args std.Tuple, kwargs []std.Tuple) (std.Value, error) { + if err := checkThread(thread); err != nil { + return std.None, err + } + var sourceValue std.Value + var varsValue std.Value + err := std.UnpackArgs(fn.Name(), args, kwargs, "source", &sourceValue, "vars?", &varsValue) + vars := make(std.StringDict) + source, sourceOK := sourceValue.(std.String) + if err != nil { + err = evalError("arguments", err) + } else if !sourceOK { + err = evalError("input", fmt.Errorf("source must be a native string, got %T", sourceValue)) + } else if varsValue != nil { + dict, ok := varsValue.(*std.Dict) + if !ok || dict == nil { + err = evalError("input", fmt.Errorf("vars must be a native dict")) + } else if dict.Len() > maxDataNodes { + err = evalError("input", fmt.Errorf("too many variables")) + } else { + for _, pair := range dict.Items() { + key, ok := pair[0].(std.String) + if !ok { + err = evalError("input", fmt.Errorf("vars keys must be strings")) + break + } + vars[string(key)] = pair[1] + } + } + } + value := std.Value(std.None) + if err == nil { + value, err = Eval(thread, string(source), vars) + } + if terminal := checkThread(thread); terminal != nil { + return std.None, terminal + } + if err != nil { + var execution *Error + if errors.As(err, &execution) && execution.Phase == "execution" { + // Preserve a terminal failure even if the owner concurrently + // cleared the Thread cancellation before the final probe. + return std.None, err + } + if try { + return std.Tuple{std.None, errorInfo(err)}, nil + } + return std.None, err + } + if try { + return std.Tuple{value, std.None}, nil + } + return value, nil + } +} + +const ( + maxErrorMessage = 4096 + maxErrorFrames = 32 +) + +func diagnosticText(text string, limit int) (string, bool) { + if len(text) <= limit { + return text, false + } + const suffix = "... (truncated)" + end := limit - len(suffix) + for end > 0 && !utf8.RuneStart(text[end]) { + end-- + } + return text[:end] + suffix, true +} + +func errorInfo(err error) *std.Dict { + phase, kind := "runtime", "runtime" + var own *Error + if errors.As(err, &own) { + phase, kind = own.Phase, own.Kind + } + position := std.Value(std.None) + frames := std.Tuple{} + message, truncated := diagnosticText(err.Error(), maxErrorMessage) + var parse syntax.Error + var unresolved resolve.ErrorList + var runtime *std.EvalError + if errors.As(err, &parse) { + position = positionInfo(parse.Pos) + truncated = truncated || len(parse.Pos.Filename()) > 512 + } else if errors.As(err, &unresolved) && len(unresolved) > 0 { + position = positionInfo(unresolved[0].Pos) + truncated = truncated || len(unresolved[0].Pos.Filename()) > 512 + } else if errors.As(err, &runtime) { + start := 0 + if len(runtime.CallStack) > maxErrorFrames { + start = len(runtime.CallStack) - maxErrorFrames + truncated = true + } + for _, frame := range runtime.CallStack[start:] { + f := positionInfo(frame.Pos) + truncated = truncated || len(frame.Pos.Filename()) > 512 + name, shortened := diagnosticText(frame.Name, 256) + truncated = truncated || shortened + _ = f.SetKey(std.String("name"), std.String(name)) + frames = append(frames, f) + if frame.Pos.IsValid() && frame.Pos.Filename() != "" { + position = positionInfo(frame.Pos) + } + } + } + info := std.NewDict(6) + for _, pair := range []std.Tuple{{std.String("phase"), std.String(phase)}, {std.String("kind"), std.String(kind)}, {std.String("message"), std.String(message)}, {std.String("position"), position}, {std.String("frames"), frames}, {std.String("truncated"), std.Bool(truncated)}} { + _ = info.SetKey(pair[0], pair[1]) + } + info.Freeze() + return info +} + +func positionInfo(pos syntax.Position) *std.Dict { + d := std.NewDict(3) + file, _ := diagnosticText(pos.Filename(), 512) + _ = d.SetKey(std.String("file"), std.String(file)) + _ = d.SetKey(std.String("line"), std.MakeInt(int(pos.Line))) + _ = d.SetKey(std.String("column"), std.MakeInt(int(pos.Col))) + return d +} diff --git a/lib/goidiomatic/eval_test.go b/lib/goidiomatic/eval_test.go new file mode 100644 index 00000000..e35f58fe --- /dev/null +++ b/lib/goidiomatic/eval_test.go @@ -0,0 +1,484 @@ +package goidiomatic_test + +// Sections: data isolation; result boundaries; errors; caller execution state. + +import ( + "context" + "errors" + "strings" + "sync" + "testing" + + mod "github.com/1set/starlet/lib/goidiomatic" + std "go.starlark.net/starlark" + "go.starlark.net/syntax" +) + +func builtin(t *testing.T, name string) std.Value { + t.Helper() + m, err := mod.LoadModule() + if err != nil { + t.Fatal(err) + } + v := m[name] + if v == nil { + t.Fatalf("missing flat builtin %q", name) + } + return v +} + +func call(t *testing.T, name, source string, vars std.Value) (std.Value, error) { + t.Helper() + return std.Call(&std.Thread{}, builtin(t, name), std.Tuple{std.String(source), vars}, nil) +} + +func TestEvalReadOnlyData(t *testing.T) { + items := std.NewList([]std.Value{std.MakeInt(1), std.MakeInt(2)}) + vars := std.StringDict{"items": items, "name": std.String("Ada")} + v, err := mod.Eval(&std.Thread{}, `{"name": name.upper(), "values": [n * 2 for n in items]}`, vars) + if err != nil || v.String() != `{"name": "ADA", "values": [2, 4]}` { + t.Fatalf("%v, %v", v, err) + } + if err := items.Append(std.MakeInt(3)); err != nil { + t.Fatalf("parent was frozen: %v", err) + } + if v.String() != `{"name": "ADA", "values": [2, 4]}` { + t.Fatalf("result shares parent: %v", v) + } + if err := v.(*std.Dict).SetKey(std.String("bad"), std.None); err == nil { + t.Fatal("result must be frozen") + } + for _, src := range []string{`items.append(9)`, `items.clear()`} { + if _, err := mod.Eval(&std.Thread{}, src, vars); err == nil { + t.Fatal("accepted input mutation", src) + } + } + if items.String() != "[1, 2, 3]" { + t.Fatal(items) + } + v, err = mod.Eval(&std.Thread{}, `(lambda xs: (xs.append(8), xs))([7])`, nil) + if err != nil || v.String() != "(None, [7, 8])" { + t.Fatalf("private intermediate: %v, %v", v, err) + } +} + +func TestEvalResultBoundary(t *testing.T) { + for _, src := range []string{`lambda: 1`, `"abc".upper`, `{"nested": [len]}`} { + _, err := mod.Eval(&std.Thread{}, src, nil) + var e *mod.Error + if !errors.As(err, &e) || e.Phase != "output" { + t.Fatalf("%q: %v", src, err) + } + } + v, err := mod.Eval(&std.Thread{}, `(123456789012345678901234567890, b"abc", set([1, 2]), None)`, nil) + if err != nil { + t.Fatal(err) + } + tuple, ok := v.(std.Tuple) + if !ok || len(tuple) != 4 || tuple[0].Type() != "int" || tuple[1].Type() != "bytes" || tuple[2].Type() != "set" || tuple[3] != std.None { + t.Fatal(v) + } +} + +func TestEvalErrors(t *testing.T) { + for _, tc := range []struct{ source, phase string }{{"1 +", "parse"}, {"missing + 1", "resolve"}, {"1 // 0", "runtime"}} { + _, err := mod.Eval(&std.Thread{}, tc.source, nil) + var e *mod.Error + if !errors.As(err, &e) || e.Phase != tc.phase { + t.Fatalf("%q: %v", tc.source, err) + } + v, err := call(t, "try_eval", tc.source, std.NewDict(0)) + if err != nil { + t.Fatal(err) + } + pair, ok := v.(std.Tuple) + if !ok || len(pair) != 2 || pair[0] != std.None { + t.Fatalf("%v", v) + } + info, ok := pair[1].(*std.Dict) + if !ok { + t.Fatalf("not error data: %v", v) + } + phase, found, err := info.Get(std.String("phase")) + if err != nil || !found || phase != std.String(tc.phase) { + t.Fatal(info, err) + } + if err := info.SetKey(std.String("x"), std.None); err == nil { + t.Fatal("mutable error") + } + } + v, err := call(t, "try_eval", "None", std.NewDict(0)) + if err != nil || v.String() != "(None, None)" { + t.Fatalf("%v, %v", v, err) + } + if _, err := call(t, "eval", "1 // 0", std.NewDict(0)); err == nil { + t.Fatal("eval swallowed error") + } + if _, err := mod.Eval(nil, "1", nil); err == nil { + t.Fatal("accepted nil thread") + } + if _, err := call(t, "eval", "1", std.NewList(nil)); err == nil { + t.Fatal("accepted non-dict vars") + } +} + +func TestEvalBuiltinPolicy(t *testing.T) { + prints, touches := 0, 0 + thread := &std.Thread{Print: func(*std.Thread, string) { prints++ }} + for _, src := range []string{`print("bad")`, `getattr("abc", "upper")() if False else print("bad")`} { + if _, err := mod.Eval(thread, src, nil); err == nil { + t.Fatal("allowed print") + } + } + if prints != 0 { + t.Fatal("called parent Print") + } + std.Universe["dangerous_eval_test"] = std.NewBuiltin("dangerous_eval_test", func(*std.Thread, *std.Builtin, std.Tuple, []std.Tuple) (std.Value, error) { + touches++ + return std.None, nil + }) + defer delete(std.Universe, "dangerous_eval_test") + if _, err := mod.Eval(&std.Thread{}, "dangerous_eval_test()", nil); err == nil || touches != 0 { + t.Fatalf("dangerous builtin: %v/%d", err, touches) + } + v, err := mod.Eval(&std.Thread{}, `getattr("abc", "upper")()`, nil) + if err != nil || v != std.String("ABC") { + t.Fatalf("native getattr: %v, %v", v, err) + } + v, err = mod.Eval(&std.Thread{}, "len", std.StringDict{"len": std.MakeInt(7)}) + if err != nil || v.String() != "7" { + t.Fatalf("data shadow: %v, %v", v, err) + } + original := std.Universe["len"] + std.Universe["len"] = std.NewBuiltin("len", func(*std.Thread, *std.Builtin, std.Tuple, []std.Tuple) (std.Value, error) { + touches++ + return std.None, nil + }) + v, err = mod.Eval(&std.Thread{}, "len([1])", nil) + std.Universe["len"] = original + if err != nil || v.String() != "1" || touches != 0 { + t.Fatalf("replaced builtin: %v, %v, %d", v, err, touches) + } + delete(std.Universe, "len") + v, err = mod.Eval(&std.Thread{}, "len([1])", nil) + std.Universe["len"] = original + if err != nil || v.String() != "1" { + t.Fatalf("removed builtin: %v, %v", v, err) + } +} + +func TestEvalOuterHelpersAndDataShadow(t *testing.T) { + vars := std.StringDict{"eval": std.MakeInt(7), "try_eval": std.NewList([]std.Value{std.MakeInt(8)})} + result, err := mod.Eval(&std.Thread{}, "eval + try_eval[0]", vars) + if err != nil || result != std.MakeInt(15) { + t.Fatalf("data shadows builtin names: %v, %v", result, err) + } + for _, source := range []string{ + "eval('1')", "try_eval('1')", "sleep(0)", "exit(0)", "quit(0)", + "eprint('hidden')", "pprint('hidden')", "getenv('PATH')", "runtime.getenv('PATH')", + "outer_value", "callback()", + } { + if _, err := mod.Eval(&std.Thread{}, source, nil); err == nil { + t.Fatalf("inherited outer helper: %s", source) + } + } + first, err := mod.LoadModule() + if err != nil { + t.Fatal(err) + } + delete(first, "eval") + second, err := mod.LoadModule() + if err != nil || second["eval"] == nil { + t.Fatalf("module dictionaries share mutations: %v", err) + } + second["try_eval"].Freeze() + value, err := std.Call(&std.Thread{}, second["try_eval"], std.Tuple{std.String("1 + 1")}, nil) + if err != nil || value.String() != "(2, None)" { + t.Fatalf("shared immutable builtin: %v, %v", value, err) + } +} + +func TestEvalPreservesCallerState(t *testing.T) { + thread := &std.Thread{} + if _, err := mod.Eval(thread, "1 + 2", nil); err != nil { + t.Fatal(err) + } + steps := thread.ExecutionSteps() + if _, err := mod.Eval(thread, "3 + 4", nil); err != nil { + t.Fatal(err) + } + if thread.ExecutionSteps() <= steps { + t.Fatal("steps reset") + } + thread.Cancel("caller cancelled") + _, err := std.Call(thread, builtin(t, "try_eval"), std.Tuple{std.String("1 +")}, nil) + if err == nil || !strings.Contains(err.Error(), "caller cancelled") { + t.Fatalf("try swallowed cancellation: %v", err) + } + ctx, cancel := context.WithCancel(context.Background()) + cancel() + ctxThread := &std.Thread{} + ctxThread.SetLocal("context", ctx) + _, err = std.Call(ctxThread, builtin(t, "try_eval"), std.Tuple{std.String("1")}, nil) + if !errors.Is(err, context.Canceled) { + t.Fatalf("context cancellation: %v", err) + } + limitThread := &std.Thread{} + limitThread.SetMaxExecutionSteps(1) + _, err = std.Call(limitThread, builtin(t, "try_eval"), std.Tuple{std.String("1")}, nil) + if err == nil || !strings.Contains(err.Error(), "too many steps") { + t.Fatalf("try swallowed limit: %v", err) + } + panics := &std.Thread{} + panics.SetMaxExecutionSteps(1) + panics.OnMaxSteps = func(*std.Thread) { panic("parent limit") } + func() { + defer func() { + if recover() != "parent limit" { + t.Error("parent panic swallowed") + } + }() + _, _ = mod.Eval(panics, "1", nil) + }() +} + +func TestEvalArgumentsAndDiagnostics(t *testing.T) { + called := false + var list *std.List + for _, name := range []string{"eval", "try_eval"} { + for _, source := range []std.Value{nil, list, hostileValue{&called}, std.MakeInt(1)} { + v, err := std.Call(&std.Thread{}, builtin(t, name), std.Tuple{source}, nil) + if name == "eval" && err == nil || name == "try_eval" && (err != nil || v.(std.Tuple)[0] != std.None) { + t.Fatalf("%s(%T): %v, %v", name, source, v, err) + } + } + } + if called { + t.Fatal("source rejection executed host behavior") + } + for _, args := range []std.Tuple{nil, {std.String("1"), std.NewDict(0), std.None}} { + v, err := std.Call(&std.Thread{}, builtin(t, "try_eval"), args, nil) + if err != nil || v.(std.Tuple)[0] != std.None { + t.Fatalf("argument error: %v, %v", v, err) + } + } + v, err := std.Call(&std.Thread{}, builtin(t, "eval"), std.Tuple{std.String("2 + 3")}, nil) + if err != nil || v.String() != "5" { + t.Fatalf("omitted vars: %v, %v", v, err) + } + vars := std.NewDict(1) + _ = vars.SetKey(std.String("n"), std.MakeInt(5)) + v, err = std.Call(&std.Thread{}, builtin(t, "eval"), std.Tuple{std.String("n * 2")}, []std.Tuple{{std.String("vars"), vars}}) + if err != nil || v.String() != "10" { + t.Fatalf("keyword vars: %v, %v", v, err) + } + tooMany := std.NewDict(4097) + for i := 0; i < 4097; i++ { + _ = tooMany.SetKey(std.String(strings.Repeat("x", i+1)), std.None) + } + if _, err := call(t, "eval", "1", tooMany); err == nil { + t.Fatal("accepted too many dict variables") + } + v, err = call(t, "try_eval", `fail("é" * 10000)`, std.NewDict(0)) + if err != nil { + t.Fatal(err) + } + info := v.(std.Tuple)[1].(*std.Dict) + message, _, _ := info.Get(std.String("message")) + truncated, _, _ := info.Get(std.String("truncated")) + if len(string(message.(std.String))) > 4096 || truncated != std.True || !strings.Contains(string(message.(std.String)), "truncated") { + t.Fatal("unbounded diagnostic") + } + depth := 0 + var outer *std.Builtin + try := builtin(t, "try_eval") + outer = std.NewBuiltin("outer", func(th *std.Thread, _ *std.Builtin, _ std.Tuple, _ []std.Tuple) (std.Value, error) { + if depth < 64 { + depth++ + defer func() { depth-- }() + return std.Call(th, outer, nil, nil) + } + return std.Call(th, try, std.Tuple{std.String("1 // 0")}, nil) + }) + v, err = std.Call(&std.Thread{}, outer, nil, nil) + if err != nil { + t.Fatal(err) + } + info = v.(std.Tuple)[1].(*std.Dict) + frames, _, _ := info.Get(std.String("frames")) + truncated, _, _ = info.Get(std.String("truncated")) + if len(frames.(std.Tuple)) != 32 || truncated != std.True { + t.Fatal("unbounded call frames") + } + _, err = std.ExecFileOptions(&syntax.FileOptions{Set: true}, &std.Thread{}, strings.Repeat("p", 1024), `result = evaluator("1 // 0")`, std.StringDict{"evaluator": try}) + if err != nil { + t.Fatal(err) + } + // Inspect through a capturing wrapper so the outer filename is part of + // the diagnostic while the error itself remains inside try_eval. + var captured std.Value + capture := std.NewBuiltin("capture", func(_ *std.Thread, _ *std.Builtin, a std.Tuple, _ []std.Tuple) (std.Value, error) { + captured = a[0] + return std.None, nil + }) + _, err = std.ExecFileOptions(&syntax.FileOptions{Set: true}, &std.Thread{}, strings.Repeat("p", 1024), `capture(evaluator("1 // 0"))`, std.StringDict{"evaluator": try, "capture": capture}) + if err != nil { + t.Fatal(err) + } + info = captured.(std.Tuple)[1].(*std.Dict) + truncated, _, _ = info.Get(std.String("truncated")) + if truncated != std.True { + t.Fatal("long filename truncation not marked") + } +} + +func TestEvalConcurrentFrozenData(t *testing.T) { + set := std.NewSet(2) + _ = set.Insert(std.MakeInt(1)) + _ = set.Insert(std.MakeInt(2)) + vars := std.StringDict{"items": std.NewList([]std.Value{std.MakeInt(2), std.MakeInt(3)}), "tags": set} + vars.Freeze() + exports, err := mod.LoadModule() + if err != nil { + t.Fatal(err) + } + arguments := std.NewDict(len(vars)) + for name, value := range vars { + if err := arguments.SetKey(std.String(name), value); err != nil { + t.Fatal(err) + } + } + arguments.Freeze() + cancelled := &std.Thread{} + cancelled.Cancel("one caller only") + if _, err := std.Call(cancelled, exports["try_eval"], std.Tuple{std.String("1")}, nil); err == nil { + t.Fatal("cancelled caller succeeded") + } + var group sync.WaitGroup + for i := 0; i < 8; i++ { + group.Add(1) + go func(index int) { + defer group.Done() + v, err := mod.Eval(&std.Thread{}, `(sorted(tags), [n * 2 for n in items])`, vars) + if err != nil || v.String() != "([1, 2], [4, 6])" { + t.Errorf("%v, %v", v, err) + } + v, err = std.Call(&std.Thread{}, exports["eval"], std.Tuple{std.String(`(sorted(tags), [n * 2 for n in items])`), arguments}, nil) + if err != nil || v.String() != "([1, 2], [4, 6])" { + t.Errorf("shared builtin: %v, %v", v, err) + } + local := std.NewDict(1) + if err := local.SetKey(std.String("n"), std.MakeInt(index)); err != nil { + t.Error(err) + return + } + v, err = std.Call(&std.Thread{}, exports["try_eval"], std.Tuple{std.String("n + 1"), local}, nil) + if err != nil { + t.Errorf("independent caller: %v", err) + return + } + pair, ok := v.(std.Tuple) + if !ok || len(pair) != 2 || pair[0] != std.MakeInt(index+1) || pair[1] != std.None { + t.Errorf("crossed caller data/state: %v", v) + } + }(i) + } + group.Wait() +} + +func TestEvalBudgetBoundaries(t *testing.T) { + // Across entry, copy, execution and result publication checkpoints, a + // depleted caller budget must never turn into a successful try tuple. + for limit := uint64(1); limit <= 40; limit++ { + for _, source := range []string{"1 // 0", "list(range(100))"} { + thread := &std.Thread{} + thread.SetMaxExecutionSteps(limit) + value, err := std.Call(thread, builtin(t, "try_eval"), std.Tuple{std.String(source)}, nil) + if err != nil && !strings.Contains(err.Error(), "too many steps") { + t.Fatal(err) + } + if err == nil && value == nil { + t.Fatal("nil success") + } + } + } + thread := &std.Thread{} + thread.SetMaxExecutionSteps(3) + items := make([]std.Value, 100) + for i := range items { + items[i] = std.None + } + _, err := mod.Eval(thread, "x", std.StringDict{"x": std.NewList(items)}) + if err == nil || !strings.Contains(err.Error(), "too many steps") { + t.Fatalf("copy budget: %v", err) + } + t.Run("owner_clears_cancellation_after_failure", func(t *testing.T) { + caller := &std.Thread{} + caller.SetMaxExecutionSteps(1) + checkpoints := 0 + cleared := false + caller.OnMaxSteps = func(thread *std.Thread) { + checkpoints++ + // The fixed VM's closed integer probe has two instructions. + // Let the entry probe finish, cancel during Eval's next probe, + // then let the owner clear that state at the final checkpoint. + // try_eval must retain the error that has already occurred. + if checkpoints == 3 { + thread.Cancel("owner checkpoint cancellation") + } else if checkpoints > 3 { + thread.Uncancel() + cleared = true + } + } + value, err := std.Call(caller, builtin(t, "try_eval"), std.Tuple{std.String("1")}, nil) + var execution *mod.Error + if !cleared || value != std.None || !errors.As(err, &execution) || execution.Phase != "execution" || !strings.Contains(err.Error(), "owner checkpoint cancellation") { + t.Fatalf("lost prior execution error after owner reset: value=%v error=%v cleared=%v checkpoints=%d", value, err, cleared, checkpoints) + } + // The owner's reset is real: a later evaluation may succeed. It + // does not retroactively turn the failed try_eval into a tuple. + value, err = mod.Eval(caller, "7", nil) + if err != nil || value.String() != "7" { + t.Fatalf("owner did not clear cancellation: %v, %v", value, err) + } + }) +} + +func BenchmarkEval(b *testing.B) { + for _, name := range []string{"Scalar", "Nested", "TryError"} { + b.Run(name, func(b *testing.B) { + thread := &std.Thread{} + vars := std.StringDict{"qty": std.MakeInt(3), "items": std.NewList([]std.Value{std.MakeInt(1), std.MakeInt(2), std.MakeInt(3)})} + if name == "Scalar" { + vars = std.StringDict{"qty": std.MakeInt(3)} + } + module, err := mod.LoadModule() + if err != nil { + b.Fatal(err) + } + try := module["try_eval"] + if try == nil { + b.Fatal("missing try_eval builtin") + } + if _, err := mod.Eval(thread, "0", nil); err != nil { + b.Fatal(err) + } + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + var err error + switch name { + case "Scalar": + _, err = mod.Eval(thread, "qty * 2", vars) + case "Nested": + _, err = mod.Eval(thread, `{"values": [x * qty for x in items]}`, vars) + case "TryError": + _, err = std.Call(thread, try, std.Tuple{std.String("1 // 0")}, nil) + } + if err != nil { + b.Fatal(err) + } + } + }) + } +} diff --git a/lib/goidiomatic/idiomatic.go b/lib/goidiomatic/idiomatic.go index 06e269df..ff4b3b7e 100644 --- a/lib/goidiomatic/idiomatic.go +++ b/lib/goidiomatic/idiomatic.go @@ -23,6 +23,8 @@ const ModuleName = "go_idiomatic" // LoadModule loads the Go idiomatic module. func LoadModule() (starlark.StringDict, error) { return starlark.StringDict{ + "eval": evalBuiltin, + "try_eval": tryEvalBuiltin, "true": starlark.True, "false": starlark.False, "nil": starlark.None, diff --git a/lib/goidiomatic/idiomatic_test.go b/lib/goidiomatic/idiomatic_test.go index 17c0b71d..f0382f47 100644 --- a/lib/goidiomatic/idiomatic_test.go +++ b/lib/goidiomatic/idiomatic_test.go @@ -1,5 +1,7 @@ package goidiomatic_test +// Sections: helper behavior; data constructors; flat expression builtins. + import ( "fmt" "testing" @@ -1119,3 +1121,33 @@ func TestLoadModule_GoIdiomatic(t *testing.T) { }) } } + +// TestLoadModuleEvaluationFunctions pins the existing module's flat surface. +func TestLoadModuleEvaluationFunctions(t *testing.T) { + exports, err := goidiomatic.LoadModule() + if err != nil { + t.Fatal(err) + } + if _, exists := exports["starlark"]; exists { + t.Fatal("unexpected language-named wrapper") + } + for _, name := range []string{"eval", "try_eval"} { + function, ok := exports[name].(*starlark.Builtin) + if !ok { + t.Fatalf("missing flat builtin %q: %T", name, exports[name]) + } + if function.Name() != name { + t.Fatalf("builtin name = %q; want %q", function.Name(), name) + } + value, err := starlark.Call(&starlark.Thread{}, function, starlark.Tuple{starlark.String("2 + 3")}, nil) + if err != nil { + t.Fatal(err) + } + if name == "eval" && value != starlark.MakeInt(5) { + t.Fatalf("eval = %v", value) + } + if name == "try_eval" && value.String() != "(5, None)" { + t.Fatalf("try_eval = %v", value) + } + } +} diff --git a/lib/goidiomatic/snapshot.go b/lib/goidiomatic/snapshot.go new file mode 100644 index 00000000..169a6543 --- /dev/null +++ b/lib/goidiomatic/snapshot.go @@ -0,0 +1,207 @@ +package goidiomatic + +import ( + "fmt" + "math/big" + "reflect" + + std "go.starlark.net/starlark" +) + +const ( + maxDataDepth = 64 + maxDataNodes = 4096 + maxDataBytes = 1 << 20 + maxIntBits = 4096 +) + +var integerBound = std.MakeBigInt(new(big.Int).Lsh(big.NewInt(1), maxIntBits)) +var negativeIntegerBound = std.MakeBigInt(new(big.Int).Neg(new(big.Int).Lsh(big.NewInt(1), maxIntBits))) + +type objectKey struct { + kind byte + ptr uintptr + len int +} + +type dataCopy struct { + thread *std.Thread + nodes int + bytes int + active map[objectKey]bool + memo map[objectKey]std.Value + height map[objectKey]int +} + +func newDataCopy(thread *std.Thread) *dataCopy { + return &dataCopy{thread: thread, active: make(map[objectKey]bool), memo: make(map[objectKey]std.Value), height: make(map[objectKey]int)} +} + +func (c *dataCopy) chargeBytes(n int, path string) error { + if n > maxDataBytes-c.bytes { + return fmt.Errorf("%s: data exceeds %d bytes", path, maxDataBytes) + } + c.bytes += n + return nil +} + +func (c *dataCopy) value(value std.Value, path string, depth int) (std.Value, error) { + c.nodes++ // Every edge counts, including an alias that hits the memo. + if c.nodes > maxDataNodes { + return nil, fmt.Errorf("%s: data exceeds %d nodes", path, maxDataNodes) + } + if depth > maxDataDepth { + return nil, fmt.Errorf("%s: data exceeds depth %d", path, maxDataDepth) + } + if c.nodes%64 == 0 { + if err := checkThread(c.thread); err != nil { + return nil, err + } + } + switch x := value.(type) { + case std.NoneType, std.Bool, std.Float: + return x, c.chargeBytes(8, path) + case std.Int: + upper, _ := x.Cmp(integerBound, 0) + lower, _ := x.Cmp(negativeIntegerBound, 0) + if upper >= 0 || lower <= 0 { + return nil, fmt.Errorf("%s: integer exceeds %d bits", path, maxIntBits) + } + return x, c.chargeBytes((x.BigInt().BitLen()+7)/8, path) + case std.String: + return x, c.chargeBytes(len(x), path) + case std.Bytes: + return x, c.chargeBytes(len(x), path) + case *std.List: + if x == nil { + return nil, fmt.Errorf("%s: nil list", path) + } + key := objectKey{kind: 'l', ptr: reflect.ValueOf(x).Pointer()} + if known, found, err := c.lookup(key, path, depth); found || err != nil { + return known, err + } + if x.Len() > maxDataNodes-c.nodes { + return nil, fmt.Errorf("%s: list exceeds node limit", path) + } + out := std.NewList(nil) + c.memo[key], c.active[key] = out, true + defer delete(c.active, key) + for i := 0; i < x.Len(); i++ { + item, err := c.value(x.Index(i), fmt.Sprintf("%s[%d]", path, i), depth+1) + if err != nil { + return nil, err + } + _ = out.Append(item) + c.childHeight(key, x.Index(i)) + } + return out, nil + case *std.Dict: + if x == nil { + return nil, fmt.Errorf("%s: nil dict", path) + } + key := objectKey{kind: 'd', ptr: reflect.ValueOf(x).Pointer()} + if known, found, err := c.lookup(key, path, depth); found || err != nil { + return known, err + } + if x.Len() > (maxDataNodes-c.nodes)/2 { + return nil, fmt.Errorf("%s: dict exceeds node limit", path) + } + out := std.NewDict(x.Len()) + c.memo[key], c.active[key] = out, true + defer delete(c.active, key) + for i, pair := range x.Items() { + k, err := c.value(pair[0], fmt.Sprintf("%s.key[%d]", path, i), depth+1) + if err != nil { + return nil, err + } + v, err := c.value(pair[1], fmt.Sprintf("%s.value[%d]", path, i), depth+1) + if err != nil { + return nil, err + } + if err := out.SetKey(k, v); err != nil { + return nil, fmt.Errorf("%s: %w", path, err) + } + c.childHeight(key, pair[0]) + c.childHeight(key, pair[1]) + } + return out, nil + case std.Tuple: + key := objectKey{kind: 't', ptr: reflect.ValueOf(x).Pointer(), len: len(x)} + if known, found, err := c.lookup(key, path, depth); found || err != nil { + return known, err + } + if len(x) > maxDataNodes-c.nodes { + return nil, fmt.Errorf("%s: tuple exceeds node limit", path) + } + out := make(std.Tuple, len(x)) + c.memo[key], c.active[key] = out, true + defer delete(c.active, key) + for i, item := range x { + v, err := c.value(item, fmt.Sprintf("%s[%d]", path, i), depth+1) + if err != nil { + return nil, err + } + out[i] = v + c.childHeight(key, item) + } + return out, nil + case *std.Set: + if x == nil { + return nil, fmt.Errorf("%s: nil set", path) + } + key := objectKey{kind: 's', ptr: reflect.ValueOf(x).Pointer()} + if known, found, err := c.lookup(key, path, depth); found || err != nil { + return known, err + } + if x.Len() > maxDataNodes-c.nodes { + return nil, fmt.Errorf("%s: set exceeds node limit", path) + } + out := std.NewSet(x.Len()) + c.memo[key], c.active[key] = out, true + defer delete(c.active, key) + it := x.Iterate() + defer it.Done() + var item std.Value + for i := 0; it.Next(&item); i++ { + v, err := c.value(item, fmt.Sprintf("%s[%d]", path, i), depth+1) + if err != nil { + return nil, err + } + if err := out.Insert(v); err != nil { + return nil, fmt.Errorf("%s: %w", path, err) + } + c.childHeight(key, item) + } + return out, nil + default: + return nil, fmt.Errorf("%s: unsupported data value %T", path, value) + } +} + +func (c *dataCopy) lookup(key objectKey, path string, depth int) (std.Value, bool, error) { + if c.active[key] { + return nil, false, fmt.Errorf("%s: reference cycle is unsupported", path) + } + v, found := c.memo[key] + if found && depth+c.height[key] > maxDataDepth { + return nil, false, fmt.Errorf("%s: aliased data exceeds depth %d", path, maxDataDepth) + } + return v, found, nil +} + +func (c *dataCopy) childHeight(parent objectKey, value std.Value) { + height := 1 + switch x := value.(type) { + case *std.List: + height += c.height[objectKey{kind: 'l', ptr: reflect.ValueOf(x).Pointer()}] + case *std.Dict: + height += c.height[objectKey{kind: 'd', ptr: reflect.ValueOf(x).Pointer()}] + case *std.Set: + height += c.height[objectKey{kind: 's', ptr: reflect.ValueOf(x).Pointer()}] + case std.Tuple: + height += c.height[objectKey{kind: 't', ptr: reflect.ValueOf(x).Pointer(), len: len(x)}] + } + if height > c.height[parent] { + c.height[parent] = height + } +} diff --git a/lib/goidiomatic/snapshot_test.go b/lib/goidiomatic/snapshot_test.go new file mode 100644 index 00000000..36d43be5 --- /dev/null +++ b/lib/goidiomatic/snapshot_test.go @@ -0,0 +1,203 @@ +package goidiomatic_test + +// Sections: unsupported host values; cycles and aliases; data quotas. + +import ( + "errors" + "math/big" + "strings" + "testing" + + mod "github.com/1set/starlet/lib/goidiomatic" + std "go.starlark.net/starlark" +) + +type hostileValue struct{ called *bool } + +func (v hostileValue) String() string { *v.called = true; panic("String called") } +func (v hostileValue) Type() string { *v.called = true; panic("Type called") } +func (v hostileValue) Freeze() { *v.called = true; panic("Freeze called") } +func (v hostileValue) Truth() std.Bool { *v.called = true; panic("Truth called") } +func (v hostileValue) Hash() (uint32, error) { *v.called = true; panic("Hash called") } + +func TestEvalRejectsUnsupportedInput(t *testing.T) { + var list *std.List + var dict *std.Dict + var set *std.Set + called := false + values := []std.Value{nil, list, dict, set, hostileValue{&called}, std.NewBuiltin("callback", nil)} + for _, v := range values { + _, err := mod.Eval(&std.Thread{}, "x", std.StringDict{"x": std.NewList([]std.Value{v})}) + var e *mod.Error + if !errors.As(err, &e) || e.Phase != "input" || !strings.Contains(err.Error(), "x") { + t.Fatalf("%T: %v", v, err) + } + } + if called { + t.Fatal("input inspection invoked host behavior") + } + for _, key := range []string{"", "1x", "try", "x.y"} { + if _, err := mod.Eval(&std.Thread{}, "1", std.StringDict{key: std.None}); err == nil { + t.Fatal("accepted bad name", key) + } + } + vars := std.NewDict(1) + _ = vars.SetKey(std.MakeInt(1), std.None) + if _, err := call(t, "eval", "1", vars); err == nil { + t.Fatal("accepted nonstring variable name") + } +} + +func TestEvalCyclesAliasesAndQuota(t *testing.T) { + list := std.NewList(nil) + _ = list.Append(list) + if _, err := mod.Eval(&std.Thread{}, "x", std.StringDict{"x": list}); err == nil { + t.Fatal("accepted cycle") + } + if err := list.Append(std.None); err != nil { + t.Fatal("rejected input was frozen") + } + shared := std.NewList([]std.Value{std.MakeInt(1)}) + v, err := mod.Eval(&std.Thread{}, "(a, b)", std.StringDict{"a": shared, "b": shared}) + if err != nil { + t.Fatal(err) + } + items := v.(std.Tuple) + if items[0] != items[1] || items[0] == shared { + t.Fatal("aliases not privately preserved") + } + bigValue := std.MakeBigInt(new(big.Int).Lsh(big.NewInt(1), 4096)) + for _, value := range []std.Value{std.String(strings.Repeat("x", (1<<20)+1)), bigValue, std.NewList(make([]std.Value, 4097))} { + if _, err := mod.Eval(&std.Thread{}, "x", std.StringDict{"x": value}); err == nil { + t.Fatalf("accepted oversized %T", value) + } + } + var deep std.Value = std.None + for i := 0; i < 65; i++ { + deep = std.NewList([]std.Value{deep}) + } + if _, err := mod.Eval(&std.Thread{}, "x", std.StringDict{"x": deep}); err == nil { + t.Fatal("accepted deep graph") + } + if _, err := mod.Eval(&std.Thread{}, `"x" * 1048577`, nil); err == nil { + t.Fatal("accepted oversized output") + } +} + +func TestEvalContainerGraphBoundaries(t *testing.T) { + dict := std.NewDict(1) + _ = dict.SetKey(std.String("self"), dict) + tuple := make(std.Tuple, 1) + tuple[0] = tuple + for _, value := range []std.Value{dict, tuple} { + if _, err := mod.Eval(&std.Thread{}, "x", std.StringDict{"x": value}); err == nil { + t.Fatal("accepted cycle") + } + } + var shared std.Value = std.None + for i := 0; i < 40; i++ { + shared = std.NewList([]std.Value{shared}) + } + longer := shared + for i := 0; i < 40; i++ { + longer = std.NewList([]std.Value{longer}) + } + if _, err := mod.Eval(&std.Thread{}, "1", std.StringDict{"a": shared, "b": longer}); err == nil { + t.Fatal("memo bypassed path depth") + } + containers := []std.Value{std.Tuple(make([]std.Value, 4097)), std.Bytes(strings.Repeat("x", (1<<20)+1))} + largeDict := std.NewDict(2050) + largeSet := std.NewSet(4097) + for i := 0; i < 4097; i++ { + _ = largeSet.Insert(std.MakeInt(i)) + if i < 2050 { + _ = largeDict.SetKey(std.MakeInt(i), std.None) + } + } + containers = append(containers, largeDict, largeSet) + for _, value := range containers { + if _, err := mod.Eval(&std.Thread{}, "x", std.StringDict{"x": value}); err == nil { + t.Fatalf("accepted oversized %T", value) + } + } + for _, src := range []string{`[0] * 5000`, `int("f" * 1025, 16)`, `(lambda x: (x.append(x), x)[1])([])`} { + _, err := mod.Eval(&std.Thread{}, src, nil) + var e *mod.Error + if !errors.As(err, &e) || e.Phase != "output" { + t.Fatalf("%s: %v", src, err) + } + } + variables := make(std.StringDict, 4097) + for i := 0; i < 4097; i++ { + variables[strings.Repeat("a", i+1)] = std.None + } + if _, err := mod.Eval(&std.Thread{}, "1", variables); err == nil { + t.Fatal("accepted too many variables") + } + if _, err := mod.Eval(&std.Thread{}, "1", std.StringDict{strings.Repeat("a", (1<<20)+1): std.None}); err == nil || len(err.Error()) > 5000 { + t.Fatal("unbounded variable name rejection") + } + if _, err := mod.Eval(&std.Thread{}, "1", std.StringDict{strings.Repeat("a", 513): std.None}); err == nil || len(err.Error()) > 5000 { + t.Fatal("unbounded invalid identifier") + } + // Host mutation of a tuple after insertion can invalidate a native key. + // Reject the malformed graph rather than panic or silently lose the item. + for _, kind := range []string{"dict", "set"} { + key := std.Tuple{std.MakeInt(1)} + var value std.Value + if kind == "dict" { + d := std.NewDict(1) + _ = d.SetKey(key, std.None) + value = d + } else { + s := std.NewSet(1) + _ = s.Insert(key) + value = s + } + key[0] = std.NewList(nil) + if _, err := mod.Eval(&std.Thread{}, "x", std.StringDict{"x": value}); err == nil { + t.Fatal("accepted malformed", kind) + } + } + called := false + for _, kind := range []string{"dict", "set"} { + key := std.Tuple{std.MakeInt(1)} + var value std.Value + if kind == "dict" { + d := std.NewDict(1) + _ = d.SetKey(key, std.None) + value = d + } else { + s := std.NewSet(1) + _ = s.Insert(key) + value = s + } + key[0] = hostileValue{&called} + if _, err := mod.Eval(&std.Thread{}, "x", std.StringDict{"x": value}); err == nil { + t.Fatal("accepted invalid native key", kind) + } + } + if called { + t.Fatal("key validation executed host behavior") + } + inside := std.NewDict(1) + _ = inside.SetKey(std.String("tuple"), std.Tuple{std.MakeInt(1)}) + if _, err := mod.Eval(&std.Thread{}, "x", std.StringDict{"x": std.NewList([]std.Value{inside})}); err != nil { + t.Fatal(err) + } + set := std.NewSet(1) + _ = set.Insert(std.MakeInt(1)) + if _, err := mod.Eval(&std.Thread{}, "(a,b)", std.StringDict{"a": set, "b": set}); err != nil { + t.Fatal(err) + } + if _, err := mod.Eval(&std.Thread{}, "1", std.StringDict{"a": std.String(strings.Repeat("x", (1<<20)-1)), "b": std.None}); err == nil { + t.Fatal("ignored names in total bytes") + } + full := make([]std.Value, 4094) + for i := range full { + full[i] = std.None + } + if _, err := mod.Eval(&std.Thread{}, "x", std.StringDict{"x": std.Tuple{std.NewList(full), std.None}}); err == nil { + t.Fatal("ignored later graph edge") + } +} diff --git a/lib/goidiomatic/source.go b/lib/goidiomatic/source.go new file mode 100644 index 00000000..45db8ed3 --- /dev/null +++ b/lib/goidiomatic/source.go @@ -0,0 +1,73 @@ +package goidiomatic + +import ( + "fmt" + + "go.starlark.net/syntax" +) + +const ( + maxSourceBytes = 4096 + maxSourceCodeBytes = 512 + maxSourceASTDepth = 64 + maxSourceASTNodes = 512 +) + +// parseSource admits a small source before invoking the pinned recursive +// parser, then checks the resulting tree before resolution and compilation. +// Counting literal and comment bytes too is deliberately conservative: large +// data belongs in vars. These limits are not a repair of the upstream parser +// or a hard process memory/CPU boundary. +func parseSource(source string) (syntax.Expr, string, error) { + if len(source) > maxSourceBytes { + return nil, "source", fmt.Errorf("source exceeds %d bytes", maxSourceBytes) + } + codeBytes := 0 + for i := 0; i < len(source); i++ { + switch source[i] { + case ' ', '\t', '\r', '\n': + default: + codeBytes++ + } + } + if codeBytes > maxSourceCodeBytes { + return nil, "source", fmt.Errorf("source exceeds %d non-whitespace bytes", maxSourceCodeBytes) + } + + opts := syntax.FileOptions{Set: true} + expr, err := opts.ParseExpr("", source, 0) + if err != nil { + return nil, "parse", err + } + if err := checkSourceAST(expr); err != nil { + return nil, "source", err + } + return expr, "", nil +} + +// checkSourceAST prunes traversal before descending past its own depth bound. +func checkSourceAST(expr syntax.Expr) error { + var guardErr error + depth, nodes := 0, 0 + syntax.Walk(expr, func(node syntax.Node) bool { + if node == nil { + depth-- + return false + } + if guardErr != nil { + return false + } + nodes++ + if nodes > maxSourceASTNodes { + guardErr = fmt.Errorf("source AST exceeds %d nodes", maxSourceASTNodes) + return false + } + if depth == maxSourceASTDepth { + guardErr = fmt.Errorf("source AST exceeds depth %d", maxSourceASTDepth) + return false + } + depth++ + return true + }) + return guardErr +} diff --git a/lib/goidiomatic/source_test.go b/lib/goidiomatic/source_test.go new file mode 100644 index 00000000..2a22adb2 --- /dev/null +++ b/lib/goidiomatic/source_test.go @@ -0,0 +1,107 @@ +package goidiomatic + +// Sections: accepted expressions; source admission limits; parse errors; +// independently bounded AST traversal. + +import ( + "errors" + "strings" + "testing" + + "go.starlark.net/syntax" +) + +func TestParseSourceAccepted(t *testing.T) { + tests := []struct { + name string + source string + }{ + {"number", "1"}, + {"total_bytes_boundary", "1" + strings.Repeat(" ", maxSourceBytes-1)}, + {"code_bytes_boundary", `"` + strings.Repeat("a", maxSourceCodeBytes-2) + `"`}, + {"unicode_bytes_boundary", `"` + strings.Repeat("é", (maxSourceCodeBytes-2)/2) + `"`}, + {"json_whitespace", " \t\r\n1 \t\r\n"}, + {"unicode_string", `"你好"`}, + {"unicode_identifier", "名字"}, + {"comments", "# comment\n1 # tail\n"}, + {"raw_string", `r"\n"`}, + {"bytes", `b"\x00"`}, + {"triple_string", "\"\"\"first\nsecond\"\"\""}, + {"lambda", "(lambda x: x + 1)(2)"}, + {"comprehension", "[x for x in items if x]"}, + {"compound_data", `({"x": [1, 2]}, items[1:], set([1]))`}, + {"depth_boundary", strings.Repeat("(", maxSourceASTDepth-1) + "1" + strings.Repeat(")", maxSourceASTDepth-1)}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + expr, phase, err := parseSource(tt.source) + if err != nil || phase != "" || expr == nil { + t.Fatalf("parseSource = (%T, %q, %v); want expression and no error", expr, phase, err) + } + }) + } +} + +func TestParseSourceAdmissionLimits(t *testing.T) { + tests := []struct { + name string + source string + want string + }{ + {"total_bytes", "1" + strings.Repeat(" ", maxSourceBytes), "4096 bytes"}, + {"code_bytes", strings.Repeat("a", maxSourceCodeBytes+1), "512 non-whitespace bytes"}, + {"literal_bytes", `"` + strings.Repeat("a", maxSourceCodeBytes-1) + `"`, "512 non-whitespace bytes"}, + {"comment_bytes", "#" + strings.Repeat("a", maxSourceCodeBytes) + "\n1", "512 non-whitespace bytes"}, + {"unicode_counts_bytes", `"` + strings.Repeat("é", maxSourceCodeBytes/2) + `"`, "512 non-whitespace bytes"}, + {"vertical_tab_is_not_exempt", strings.Repeat("\v", maxSourceCodeBytes+1), "512 non-whitespace bytes"}, + {"form_feed_is_not_exempt", strings.Repeat("\f", maxSourceCodeBytes+1), "512 non-whitespace bytes"}, + {"nbsp_is_not_exempt", strings.Repeat("\u00a0", maxSourceCodeBytes), "512 non-whitespace bytes"}, + // These are deliberately rejected before invoking the recursive parser. + {"not_chain", strings.Repeat("not ", maxSourceCodeBytes/3+1) + "1", "512 non-whitespace bytes"}, + {"minus_chain", strings.Repeat("-", maxSourceCodeBytes+1) + "1", "512 non-whitespace bytes"}, + {"plus_chain", strings.Repeat("+", maxSourceCodeBytes+1) + "1", "512 non-whitespace bytes"}, + {"tilde_chain", strings.Repeat("~", maxSourceCodeBytes+1) + "1", "512 non-whitespace bytes"}, + {"lambda_chain", strings.Repeat("lambda: ", maxSourceCodeBytes/7+1) + "1", "512 non-whitespace bytes"}, + {"ternary_chain", strings.Repeat("1 if 1 else ", maxSourceCodeBytes/8+1) + "0", "512 non-whitespace bytes"}, + {"paren_chain", strings.Repeat("(", maxSourceCodeBytes/2+1) + "1" + strings.Repeat(")", maxSourceCodeBytes/2+1), "512 non-whitespace bytes"}, + {"ast_depth", strings.Repeat("(", maxSourceASTDepth) + "1" + strings.Repeat(")", maxSourceASTDepth), "depth 64"}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + expr, phase, err := parseSource(tt.source) + if err == nil || phase != "source" || expr != nil || !strings.Contains(err.Error(), tt.want) { + t.Fatalf("parseSource = (%T, %q, %v); want source error containing %q", expr, phase, err, tt.want) + } + }) + } +} + +func TestParseSourceParseErrors(t *testing.T) { + for _, source := range []string{"", " \t\r\n", "1 +", "x = 1", "1; 2", "'unterminated", "[(1]", "load(\"x\", \"a\")", string([]byte{0xff})} { + t.Run(source, func(t *testing.T) { + expr, phase, err := parseSource(source) + var syntaxErr syntax.Error + if expr != nil || phase != "parse" || !errors.As(err, &syntaxErr) { + t.Fatalf("parseSource = (%T, %q, %T %v); want original syntax.Error", expr, phase, err, err) + } + }) + } +} + +func TestCheckSourceASTNodeLimit(t *testing.T) { + // Construct a flat tree directly: the admission byte limit is tighter + // than most source spellings of this defensive AST node boundary. + for _, count := range []int{maxSourceASTNodes - 1, maxSourceASTNodes, maxSourceASTNodes + 1} { + expr := &syntax.ListExpr{List: make([]syntax.Expr, count)} + for i := range expr.List { + expr.List[i] = &syntax.Ident{Name: "item"} + } + err := checkSourceAST(expr) + if count == maxSourceASTNodes-1 && err != nil { + t.Fatalf("tree with exactly %d nodes rejected: %v", maxSourceASTNodes, err) + } + if count >= maxSourceASTNodes && (err == nil || !strings.Contains(err.Error(), "512 nodes")) { + t.Fatalf("tree over node bound = %v, want node-limit error", err) + } + } +} diff --git a/module_test.go b/module_test.go index c7bbff47..17bcb48f 100644 --- a/module_test.go +++ b/module_test.go @@ -1446,3 +1446,88 @@ func TestDocCoverage(t *testing.T) { t.Errorf("%d module member(s) are not documented in their README — see the report above", len(missing)) } } + +// Evaluation helpers extend the existing flat go_idiomatic module. +func TestBuiltinEvaluationLoading(t *testing.T) { + capability, ok := starlet.GetBuiltinModuleCapability("go_idiomatic") + if !ok || capability != starlet.CapLog|starlet.CapProcess { + t.Fatalf("changed module capabilities: %v, %v", capability, ok) + } + for _, tc := range []struct { + name string + globals starlet.StringAnyMap + preload, lazy []string + extras starlet.StringAnyMap + source string + want int64 + }{ + {name: "explicit_load", lazy: []string{"go_idiomatic"}, source: `load("go_idiomatic", "eval", "try_eval") +result = eval("2 + 3") +_, problem = try_eval("1 // 0") +check = fail("missing ordinary error") if problem == None else None`, want: 5}, + {name: "preload_overrides_host_globals", globals: starlet.StringAnyMap{"eval": "host value", "try_eval": "host value"}, preload: []string{"go_idiomatic"}, source: `result = eval("2 + 3") +value, problem = try_eval("None") +check = fail("valid None failed") if value != None or problem != None else None`, want: 5}, + {name: "extras_override_preload", preload: []string{"go_idiomatic"}, extras: starlet.StringAnyMap{"eval": int64(7), "try_eval": int64(8)}, source: "result = eval + try_eval", want: 15}, + {name: "load_overrides_extras", preload: []string{"go_idiomatic"}, lazy: []string{"go_idiomatic"}, extras: starlet.StringAnyMap{"eval": int64(7), "try_eval": int64(8)}, source: `load("go_idiomatic", "eval", "try_eval") +result = eval("2 + 3")`, want: 5}, + {name: "load_alias_preserves_existing_name", lazy: []string{"go_idiomatic"}, extras: starlet.StringAnyMap{"eval": int64(7)}, source: `load("go_idiomatic", run_expression="eval") +result = run_expression("2 + 3") + eval`, want: 12}, + } { + t.Run(tc.name, func(t *testing.T) { + mac := starlet.NewWithNames(tc.globals, tc.preload, tc.lazy) + mac.SetMaxExecutionSteps(10000) + out, err := mac.RunScript([]byte(tc.source), tc.extras) + if err != nil || out["result"] != tc.want { + t.Fatalf("output=%#v error=%v", out, err) + } + }) + } + empty := starlet.NewDefault() + if _, err := empty.RunScript([]byte(`result = eval("1")`), nil); err == nil || !strings.Contains(err.Error(), "undefined: eval") { + t.Fatalf("unconfigured Machine exposed eval: %v", err) + } +} + +func TestBuiltinEvaluationOuterCapabilities(t *testing.T) { + touches, prints := 0, 0 + mac := starlet.NewWithNames(starlet.StringAnyMap{ + "getenv": func(string) string { touches++; return "host value" }, + "callback": func() int { touches++; return 9 }, + "outer_value": 17, + }, []string{"go_idiomatic", "runtime"}, nil) + mac.SetMaxExecutionSteps(100000) + mac.SetPrintFunc(func(*starlark.Thread, string) { prints++ }) + out, err := mac.RunScript([]byte(` +def check_capabilities(): + # Helpers exist outside, while expressions receive only selected data. + sleep(0) + if exit == None or quit == None or pprint == None or eprint == None: fail("missing outer helper") + if runtime.getenv == None or getenv == None or callback == None: fail("missing host capability") + for source in ["sleep(0)", "exit(0)", "quit(0)", "pprint('hidden')", "eprint('hidden')", "print('hidden')", "runtime.getenv('PATH')", "getenv('PATH')", "callback()", "outer_value", "eval('1')", "try_eval('1')"]: + value, problem = try_eval(source) + if value != None or problem == None: fail("inherited outer capability: " + source) + _, callback_problem = try_eval("fn()", vars={"fn": callback}) + if callback_problem == None or callback_problem["phase"] != "input": fail("callable crossed data boundary") + items = [1, 2] + result = eval('{"n": len(items), "text": text.upper()}', vars={"items": items, "text": "ok"}) + _, mutation_problem = try_eval("items.append(3)", vars={"items": items}) + if mutation_problem == None: fail("input mutation accepted") + items.append(4) + return result, items +result, items = check_capabilities() +`), nil) + if err != nil { + t.Fatal(err) + } + if touches != 0 || prints != 0 { + t.Fatalf("host side effects: touches=%d prints=%d", touches, prints) + } + result, ok := out["result"].(map[interface{}]interface{}) + if !ok || result["n"] != int64(2) || result["text"] != "OK" { + t.Fatalf("unexpected result: %#v", out["result"]) + } + if !reflect.DeepEqual(out["items"], []interface{}{int64(1), int64(2), int64(4)}) { + t.Fatalf("parent items = %#v", out["items"]) + } +} From 47b5562511a8122092583f6b87df34cf286d0e0d Mon Sep 17 00:00:00 2001 From: Kevin Tang <73975146+vt128@users.noreply.github.com> Date: Sun, 8 Nov 2026 21:16:02 +0800 Subject: [PATCH 2/5] [doc] clarify eval usage and error_info (LET-36) --- README.md | 2 +- lib/goidiomatic/README.md | 319 ++++++++++++++++++++------------------ 2 files changed, 165 insertions(+), 156 deletions(-) diff --git a/README.md b/README.md index 5d6d75bc..4080a2d3 100644 --- a/README.md +++ b/README.md @@ -38,7 +38,7 @@ Inspired by the [*Starlight*](https://github.com/starlight-go/starlight) and [*S | [`base64`](/lib/base64) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/base64.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/base64) | Base64 encoding & decoding functions | | [`csv`](/lib/csv) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/csv.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/csv) | Parses and writes comma-separated values (csv) contents | | [`file`](/lib/file) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/file.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/file) | Functions to interact with the file system | -| [`goidiomatic`](/lib/goidiomatic) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/goidiomatic.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/goidiomatic) | Extended builtins, data-only expression evaluation and constructors | +| [`goidiomatic`](/lib/goidiomatic) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/goidiomatic.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/goidiomatic) | Go-style helpers, values and constructors; read-only expression evaluation | | [`hashlib`](/lib/hashlib) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/hashlib.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/hashlib) | Hash primitives for Starlark | | [`http`](/lib/http) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/http.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/http) | HTTP client and server handler implementation for Starlark | | [`json`](/lib/json) | [![godoc](https://pkg.go.dev/badge/github.com/1set/starlet/lib/json.svg)](https://pkg.go.dev/github.com/1set/starlet/lib/json) | Utilities for converting Starlark values to/from JSON strings | diff --git a/lib/goidiomatic/README.md b/lib/goidiomatic/README.md index e450e843..ef8fcfd7 100644 --- a/lib/goidiomatic/README.md +++ b/lib/goidiomatic/README.md @@ -1,6 +1,6 @@ # go_idiomatic -`go_idiomatic` provides Go-flavored helpers, constants, and constructors for Starlark scripts — base conversions, `length`/`sum`/`distinct` utilities, struct and shared-dict factories, data-only expression evaluation, and `print`-style output. It is **not** a mirror of any single Python module; it borrows the spirit of several Python builtins (`hex`/`oct`/`bin`/`sum`/`bytes.hex`) while adding Star\* idioms. +`go_idiomatic` provides Go-flavored helpers, values and constructors for Starlark, including `true`/`false`/`nil`, structs and shared dictionaries. It also adds collection, output and read-only expression-evaluation helpers. Capability profile: **Log + Process**. `eprint`/`pprint` write to stderr or the thread's `Print` handler (Log); `sleep` blocks the thread and `exit`/`quit` halt the program (Process). Everything else is pure. @@ -10,8 +10,8 @@ Load name for `load()`: `go_idiomatic`. | function | description | |----------|-------------| -| `eval(source, vars={}) -> value` | Evaluate one Starlark expression against copied, read-only native data. | -| `try_eval(source, vars={}) -> (value, error_info)` | Return ordinary evaluation errors as frozen data; preserve caller cancellation and total step limits. | +| `eval(source, vars={}) -> value` | Evaluate a Starlark expression. [Parameters and examples](#expression-evaluation). | +| `try_eval(source, vars={}) -> (value, error_info)` | Go-style value/error results. [Return values and errors](#returns-and-errors). | | `length(obj) -> int` | Length of a string (in Unicode code points), bytes, or any sequence. | | `sum(iterable, start=0) -> number` | Sum of `start` plus the items of `iterable`, left to right. | | `distinct(iterable) -> iterable` | Iterable with duplicates removed; shape follows the input type. | @@ -45,6 +45,167 @@ print(true, false, nil) # Output: True False None ``` +## Expression evaluation + +Evaluate one **Starlark expression** with `eval`. Use `try_eval` for Go-style +`(value, error)` results: `error_info == None` means success. + +```python +load("go_idiomatic", "eval", "try_eval") + +total = eval("price * qty", vars={"price": 12, "qty": 3}) +print(total) # 36 + +value, error = try_eval("price") # No implicit caller variables. +print(value, error["phase"]) # None resolve +``` + +### Parameters + +```text +eval(source, vars={}) -> value +try_eval(source, vars={}) -> (value, error_info) +``` + +Both parameters accept positional or keyword arguments. + +| Parameter | Type | Required / default | Meaning | +| --- | --- | --- | --- | +| `source` | Native `string` | Required | A single expression, such as `"price * qty"`. Statements are not accepted. | +| `vars` | Native `dict` with string keys | Optional; `{}` | Variables visible to the expression. Keys must be valid Starlark identifiers, at most 512 bytes each. `None` is not accepted. | + +`vars` supports nested native `None`, bool, int, float, string, bytes, list, +tuple, dict and set values. Functions, methods, modules, iterators, cycles and +Go-backed/custom values (including `shared_dict`) are rejected. + +Input snapshots and returned containers are frozen; originals are untouched. +Native reads such as `.upper()`, `.get()` and `.index()` work; input writes fail. +New temporary containers inside an expression may be mutated. + +```python +items = [1, 2] +value, error = try_eval("items.append(9)", vars={"items": items}) +print(error["phase"]) # runtime +items.append(3) # Original list remains writable: [1, 2, 3]. +``` + +Expressions see only `vars` and these standard computation names: + +```text +None True False abs any all bool bytes chr dict dir enumerate fail float +getattr hasattr hash int len list max min ord range repr reversed set +sorted str tuple type zip +``` + +Caller locals, other helpers (`sum`, `sleep`, `exit`) and modules such as +`runtime` are not inherited. `print()` is disabled. + +### Returns and errors + +| Outcome | `eval` | `try_eval` | +| --- | --- | --- | +| Success | Frozen native value | `(value, None)` | +| Successful expression `None` | `None` | `(None, None)` | +| Argument, input or evaluation failure | Raises an error | `(None, error_info)` | +| Caller cancellation or total step limit reached | Raises an execution error | Raises an execution error | + +Test `error != None` for failure; valid results include `None`, `False`, `0` and +empty containers. Cancellation and cumulative steps remain in force. +`try_eval` does not catch caller panics or errors computing its arguments: +`try_eval("1 // 0")` catches division by zero; `try_eval(1 // 0)` does not. + +### `error_info` structure + +`error_info` is a **frozen dict** with six fields. Use `error["phase"]` to read +fields; nested dicts and frames are frozen too. + +| Field | Type | Meaning | +| --- | --- | --- | +| `phase` | `string` | Failing stage, listed below. | +| `kind` | `string` | Same as `phase`, except `policy` uses `"capability"`. | +| `message` | `string` | Human-readable diagnostic, at most 4096 bytes. Do not match its exact wording in program logic. | +| `position` | `None` or `dict` | Best available source location: `{"file": string, "line": int, "column": int}`. Lines and Unicode character (rune) columns are 1-based; 0 means unknown. | +| `frames` | `tuple` of `dict` | Frames from outermost to innermost, retaining the most recent 32. Each has the location fields above plus `name: string`. Builtin frames can use `""`, 0, 0. | +| `truncated` | `bool` | Whether message/location/frame text or the frame count was truncated. File names are capped at 512 bytes, frame names at 256. | + +| `phase` | Typical cause | +| --- | --- | +| `arguments` | Missing/extra arguments, duplicate or unknown keywords. | +| `input` | Wrong parameter type, invalid variable name or unsupported/oversized input data. | +| `source` | Source length or AST complexity limit exceeded. | +| `parse` | Invalid expression syntax. | +| `resolve` | Undefined name, such as an omitted variable or caller-only helper. | +| `policy` | Denied builtin, such as `print`; `kind` is `"capability"`. | +| `runtime` | Division by zero, invalid operation or attempted input mutation. | +| `output` | Result contains a callable/unsupported value or exceeds data limits. | + +For `try_eval("price")` with no variables: + +```python +{ + "phase": "resolve", + "kind": "resolve", + "message": "eval: resolve: :1:1: undefined: price", + "position": {"file": "", "line": 1, "column": 1}, + "frames": (), + "truncated": False, +} +``` + +Argument/input/source errors usually have `position=None` and `frames=()`. +Runtime/policy errors can include caller frames; `position` uses the innermost +available non-builtin location. Cancellation/budget errors are raised, never +returned with `phase="execution"`. + +### Loading and limits + +Preload `go_idiomatic` for bare calls, or enable lazy loading for `load(...)`. +`NewDefault()` enables neither. Name precedence is host globals < preloads < +per-run extras < explicit `load`. Avoid clashes with an alias: +`load("go_idiomatic", evaluate="eval")`. + +| Limit | Value | +| --- | --- | +| Source | 4096 bytes; 512 non-whitespace bytes, including literals/comments | +| AST | Depth 64; 512 nodes | +| Each copied input/output graph | Depth 64; 4096 visited edges, including repeated references | +| Each graph's scalar/name payload | 1 MiB | +| Integer magnitude | 4096 bits | + +Only ASCII space, tab, CR and LF are excluded from the source count. Put large +data in `vars`. These quotas do not cap native intermediate allocations, total +CPU or process memory; configure caller execution limits for your workload. + +### Go callers + +The Go helper returns `(starlark.Value, error)`, not an `error_info` dict: + +```go +import ( + "github.com/1set/starlet/lib/goidiomatic" + "go.starlark.net/starlark" +) + +func evaluateQuantity(qty int) (starlark.Value, error) { + thread := &starlark.Thread{Name: "expression"} + thread.SetMaxExecutionSteps(10000) + return goidiomatic.Eval(thread, "qty * 2", starlark.StringDict{ + "qty": starlark.MakeInt(qty), + }) +} +``` + +Own the Thread and own/synchronize input data, or deeply freeze it first. +Bridge direct Go cancellation to `Thread.Cancel`; Machine does this already. +Use `errors.As` with `*goidiomatic.Error` to inspect `Phase` and `Kind`; error +causes remain available through `errors.As`/`errors.Is`. Results are not converted +to Go/JSON. Keep upstream Universe standard at startup and never mutate it +concurrently. See [`Eval`](eval.go) for the host contract. + +The module remains Log + Process and is already in Starbox Safe/Network. +Use host module policy for opt-in; hard CPU/memory isolation requires a separately +validated process boundary. + ## Types ### `shared_dict` @@ -279,160 +440,8 @@ pprint({"key": "value", "list": [1, 2, 3]}) ## Notes / boundaries -- **Not a Python module clone.** Names echo Python builtins, but this is a Star\* utility grab-bag, not a 1:1 API mirror — there is no module-level namespace; members load individually. - **`length` vs `len`.** `length` counts Unicode code points for strings; the builtin `len` counts UTF-8 bytes. They agree for sequences. - **`distinct` ordering.** First-seen order is preserved for list/tuple/custom inputs; dict-key order from `.keys()` is **not** guaranteed (sort it if you need stability). Sets are returned unchanged. - **Process effects.** `sleep` blocks (and respects context cancellation); `exit`/`quit` do not stop the goroutine themselves — they return `ErrSystemExit` for the host runner to act on, and set the thread-local `exit_code`. - **Determinism.** `struct`/`make_struct`/`module` print fields in sorted order; `to_dict` of a `GoStruct` keeps the Go field names verbatim. - **`shared_dict` scope.** Locking protects a single instance across threads; it does not make the values it holds deeply immutable or persist anything beyond the process. - -## Expression evaluation - -`eval` and `try_eval` evaluate one **Starlark expression**, using only explicit -variables and approved computational builtins. They do not execute Go or -[Expr-language](https://github.com/expr-lang/expr) syntax, guess the language, or -retry failures with another engine. The `expr` module name remains available -for a separate future Expr-language adapter. - -`source` must be a string. `vars` must be a native dict whose keys are valid -Starlark identifiers. Omitting `vars` supplies an empty environment. The module -does not inspect caller locals or copy a Machine's implicit globals. - -```python -load("go_idiomatic", "eval", "try_eval") - -items = [2, 3] -result = eval( - '{"title": title.upper(), "values": [x * 2 for x in items]}', - vars={"title": "example", "items": items}, -) -# {"title": "EXAMPLE", "values": [4, 6]} - -value, error = try_eval("items.append(9)", vars={"items": items}) -# value == None; error["phase"] == "runtime" -items.append(4) # The original input was neither changed nor frozen. - -value, error = try_eval("None") -# (None, None): a valid None result is distinct from failure. -``` - -Preloading `go_idiomatic` makes `eval(...)` and `try_eval(...)` available as -bare functions, like its existing `sum` and `hex` helpers. A `NewDefault()` -Machine does not preload modules; configure this module explicitly. Neither a -`starlark` module nor a `go_idiomatic.eval` attribute wrapper is registered. -When importing `json.eval` or a future Expr-language evaluator too, use explicit -load aliases to distinguish the engines. - -The existing Machine precedence also applies to these two new names: preload -functions override same-named host globals; per-run extras override preloads; -explicit `load` bindings take precedence in the script. Use lazy loading with -an alias, such as `load("go_idiomatic", evaluate="eval")`, when keeping a host's -existing `eval` name matters. - -### Data and method boundaries - -Inputs and results preserve native `None`, bool, int (including big integers), -float, string, bytes, list, dict, tuple and set values. Nested data is supported; -aliases are preserved within a copied graph. Reference cycles, nil values, -functions, bound methods, modules, iterators and custom or Go-backed values are -rejected with a data path. Convert a host object into an explicit native data -projection first; wrapping it with Starlight does not create a snapshot. - -Only private input copies are frozen. Native read methods such as `upper`, -`lower`, `get` and `index`, comprehensions and approved computational builtins -remain usable. Mutating a frozen input fails. Expressions may create and modify -their own temporary containers; results are checked, copied and frozen before -publication. Even a frozen lambda or bound method remains callable, so callable -results are rejected recursively. - -Caller functions and external modules are not inherited. Unapproved Universe -builtins, including `print`, are blocked without changing the global Universe or -calling the parent's Print handler. The standard computational builtin instances -are captured during package initialization; the embedding application's startup -must retain the standard definitions then. As upstream requires, do not modify -Universe concurrently with execution. Other `go_idiomatic` helpers retain their existing Log/Process capabilities; -they are not inherited by an expression. Random and clock sources are not -automatically exposed either. - -The source graph must be exclusively owned/synchronized during evaluation, or -already deeply frozen native data. A lack of writers alone is insufficient: -iterating an unfrozen native set can change its iterator bookkeeping. The helper -does not acquire a Machine or application data lock for the caller. - -### Errors and execution state - -`error_info` is a frozen dict with these fields: - -- `phase` and `kind`: the failing stage or policy category; -- `message`: a diagnostic string; -- `position`: `None` or `{file, line, column}`; -- `frames`: a tuple of `{file, line, column, name}` dicts; -- `truncated`: whether diagnostic text or frames were shortened. - -Stages distinguish arguments, input, source admission, parsing, name resolution, -runtime, policy and output checks. Messages are limited to 4096 bytes, the most -recent 32 frames are retained, and frame file/name text is bounded. - -Both functions execute on the caller's existing Thread. Its cancellation, -OnMaxSteps behavior and cumulative VM step count remain in force; no new step -allowance is installed or reset. Execution checkpoints also consume VM steps. -`try_eval` does **not** turn caller cancellation, exhausted total steps or a caller -panic into a successful tuple. It only handles errors occurring inside the call; -errors while evaluating the call's arguments still belong to the outer script. - -There is no rollback of host callback effects. Such callbacks are excluded from -the data boundary. Errors discard the private evaluation data. - -### Admission limits - -| Limit | Value | -| --- | --- | -| Source bytes | 4096 | -| Non-whitespace source bytes | 512 | -| AST depth / nodes | 64 / 512 | -| Each copied data graph's depth / visited edges | 64 / 4096 | -| Each copied graph's scalar/name payload | 1 MiB | -| Integer magnitude | At most 4096 bits | - -The pre-parser complexity count includes non-whitespace bytes in strings and -comments; only ASCII space, tab, CR and LF are excluded. Put large data in `vars`. -Input and output copying have separate graph quotas. Every visited edge counts, -including memo hits; payload accounting follows the copied graph rather than an -expanded JSON serialization. - -These are admission and result limits, **not hard RSS or CPU limits**. The pinned -parser's general recursion issue is not repaired here. VM steps do not account for -every operation inside a native builtin; an intermediate allocation can occur -before output admission. Configure the caller's execution limits, and use a -separately validated process boundary when stronger isolation is required. - -### Go usage - -```go -import ( - "github.com/1set/starlet/lib/goidiomatic" - "go.starlark.net/starlark" -) - -thread := &starlark.Thread{Name: "expression"} -thread.SetMaxExecutionSteps(10000) -result, err := goidiomatic.Eval(thread, "qty * 2", starlark.StringDict{ - "qty": starlark.MakeInt(3), -}) -``` - -The caller owns the Thread and source graph. Direct Go callers must wire -cancellation to `Thread.Cancel` when cancellation during VM execution is needed; -a context stored in Thread.Local is checked at helper checkpoints, not a new -cancellation goroutine. Machine calls retain their existing cancellation bridge. -The Go helper returns `starlark.Value` and an error preserving parser, resolver and -EvalError causes through `errors.As`/`errors.Is`. It does not implicitly convert -the result to Go or JSON. - -The existing `go_idiomatic` capability classification remains Log + Process. -Adding these data-only helpers does not reduce that classification or expose the -module's other helpers inside an expression. Consumers that already allow or -preload `go_idiomatic` receive the new names with it; in particular, Starbox's -Safe and Network module sets already contain this module. Hosts needing a -separate explicit opt-in can instead pass `goidiomatic.Eval` through their own -loader or builtin policy. None of these policies establishes a hard sandbox. From 27f7e1e300c999e3a539a0785e4466e04e6da40f Mon Sep 17 00:00:00 2001 From: Kevin Tang <73975146+vt128@users.noreply.github.com> Date: Sun, 8 Nov 2026 22:00:46 +0800 Subject: [PATCH 3/5] [test] compare direct script and eval performance (LET-36) --- lib/goidiomatic/README.md | 31 ++++++++++ lib/goidiomatic/eval_test.go | 113 ++++++++++++++++++++++++++++++++++- 2 files changed, 143 insertions(+), 1 deletion(-) diff --git a/lib/goidiomatic/README.md b/lib/goidiomatic/README.md index ef8fcfd7..a78b0a54 100644 --- a/lib/goidiomatic/README.md +++ b/lib/goidiomatic/README.md @@ -206,6 +206,37 @@ The module remains Log + Process and is already in Starbox Safe/Network. Use host module policy for opt-in; hard CPU/memory isolation requires a separately validated process boundary. +### Performance + +Local measurements on an Apple M4 Pro, Go 1.27.3, darwin/arm64, `-cpu=1`: +medians of six 500 ms runs, in **microseconds per call** (1000 µs = 1 ms). +Both cases use `qty=3`; the scalar case uses `price=12`, and the nested case +evaluates `{"values": [x * qty for x in items]}` with `items=list(range(32))`. + +| Path | `price * qty` | 32-item nested result | +| --- | ---: | ---: | +| Normal compiled Starlark function | 0.062 µs | 1.821 µs | +| `eval`, reused `vars` dict | 4.504 µs (72.4×) | 19.709 µs (10.8×) | +| `eval`, new `vars` dict per call | 4.657 µs (75.2×) | 20.022 µs (11.0×) | +| `try_eval`, reused `vars`, success | 4.630 µs (74.6×) | 19.896 µs (11.0×) | + +The normal/reused-`eval` allocation costs were 80/6976 B and 1/111 allocations +for the scalar case, and 1760/25025 B and 11/682 allocations for the nested case. +Every path calls a compiled Starlark function with identical inputs. Outer +compilation, input preparation and warmup are excluded; per-call `eval` parsing, +compilation, copying, validation and freezing are included. Normal execution +does not provide this isolation. Ratios are medians of the six run-wise ratios; +these are local examples, not application-wide latency or throughput guarantees. + +Use direct script code for fixed calculations and hot loops. `eval` is intended +for dynamic expressions: it does not cache compiled expressions, even when +`vars` is reused. Run the same workloads with: + +```sh +go test ./lib/goidiomatic -run '^$' -bench '^BenchmarkEvalComparison$' \ + -benchmem -benchtime=500ms -cpu=1 -count=6 +``` + ## Types ### `shared_dict` diff --git a/lib/goidiomatic/eval_test.go b/lib/goidiomatic/eval_test.go index e35f58fe..ca02bc09 100644 --- a/lib/goidiomatic/eval_test.go +++ b/lib/goidiomatic/eval_test.go @@ -1,6 +1,7 @@ package goidiomatic_test -// Sections: data isolation; result boundaries; errors; caller execution state. +// Sections: data isolation; result boundaries; errors; caller execution state; +// evaluation benchmarks. import ( "context" @@ -482,3 +483,113 @@ func BenchmarkEval(b *testing.B) { }) } } + +// BenchmarkEvalComparison compares ordinary compiled Starlark with the same +// expression passed to eval. Each path reuses one caller Thread with identical +// argument values and a three-argument Starlark call shape. Normal uses its first +// two parameters; EvalReusedVars and TryEvalReusedVars use the equivalent prebuilt +// vars dict; EvalNewVars builds that dict from the parameters on each call. +// +// Input creation, wrapper compilation, initialization and result checks are not +// timed. The eval paths include source parsing/compilation and data isolation on +// every call; there is no expression cache. This is a feature-cost comparison, +// not a before/after regression benchmark. +func BenchmarkEvalComparison(b *testing.B) { + items := make([]std.Value, 32) + products := make([]std.Value, len(items)) + for i := range items { + items[i] = std.MakeInt(i) + products[i] = std.MakeInt(i * 3) + } + nestedResult := std.NewDict(1) + if err := nestedResult.SetKey(std.String("values"), std.NewList(products)); err != nil { + b.Fatal(err) + } + cases := []struct { + name, source, newVars string + names []string + values std.Tuple + want std.Value + }{ + { + name: "Scalar", source: "price * qty", + newVars: `{"price": price, "qty": qty}`, + names: []string{"price", "qty"}, + values: std.Tuple{std.MakeInt(12), std.MakeInt(3)}, + want: std.MakeInt(36), + }, + { + name: "Nested32", source: `{"values": [x * qty for x in items]}`, + newVars: `{"items": items, "qty": qty}`, + names: []string{"items", "qty"}, + values: std.Tuple{std.NewList(items), std.MakeInt(3)}, + want: nestedResult, + }, + } + for _, tc := range cases { + b.Run(tc.name, func(b *testing.B) { + vars := std.NewDict(len(tc.names)) + for i, name := range tc.names { + if err := vars.SetKey(std.String(name), tc.values[i]); err != nil { + b.Fatal(err) + } + } + vars.Freeze() + args := append(append(std.Tuple(nil), tc.values...), vars) + module, err := mod.LoadModule() + if err != nil { + b.Fatal(err) + } + parameters := strings.Join(tc.names, ", ") + ", vars" + quotedSource := std.String(tc.source).String() + script := "def normal(" + parameters + "):\n return " + tc.source + "\n\n" + + "def eval_reused(" + parameters + "):\n return eval(" + quotedSource + ", vars=vars)\n\n" + + "def eval_new(" + parameters + "):\n return eval(" + quotedSource + ", vars=" + tc.newVars + ")\n\n" + + "def try_reused(" + parameters + "):\n value, error = try_eval(" + quotedSource + ", vars=vars)\n return value\n" + functions, err := std.ExecFileOptions(&syntax.FileOptions{Set: true}, &std.Thread{}, "eval_comparison.star", script, module) + if err != nil { + b.Fatal(err) + } + for _, path := range []struct{ name, function string }{ + {"Normal", "normal"}, + {"EvalReusedVars", "eval_reused"}, + {"EvalNewVars", "eval_new"}, + {"TryEvalReusedVars", "try_reused"}, + } { + b.Run(path.name, func(b *testing.B) { + thread := &std.Thread{Name: "eval-comparison"} + function := functions[path.function] + if path.function == "try_reused" { + value, err := std.Call(thread, module["try_eval"], std.Tuple{std.String(tc.source), vars}, nil) + if err != nil { + b.Fatal(err) + } + pair, ok := value.(std.Tuple) + if !ok || len(pair) != 2 || pair[1] != std.None { + b.Fatalf("try_eval did not succeed: %v", value) + } + } + result, err := std.Call(thread, function, args, nil) + if err != nil { + b.Fatal(err) + } + if equal, err := std.Equal(result, tc.want); err != nil || !equal { + b.Fatalf("wrong result: %v, want %v (error %v)", result, tc.want, err) + } + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + result, err = std.Call(thread, function, args, nil) + if err != nil { + b.Fatal(err) + } + } + b.StopTimer() + if equal, err := std.Equal(result, tc.want); err != nil || !equal { + b.Fatalf("wrong final result: %v, want %v (error %v)", result, tc.want, err) + } + }) + } + }) + } +} From 53bce9644ca8cdffe256b439fb2b1b27eea51b0a Mon Sep 17 00:00:00 2001 From: Kevin Tang <73975146+vt128@users.noreply.github.com> Date: Sun, 8 Nov 2026 22:25:42 +0800 Subject: [PATCH 4/5] [test] measure eval overhead across workload sizes (LET-36) --- lib/goidiomatic/README.md | 33 +++++++++++++++++ lib/goidiomatic/eval_test.go | 69 ++++++++++++++++++++++++++++++++++++ 2 files changed, 102 insertions(+) diff --git a/lib/goidiomatic/README.md b/lib/goidiomatic/README.md index a78b0a54..eec563f5 100644 --- a/lib/goidiomatic/README.md +++ b/lib/goidiomatic/README.md @@ -237,6 +237,39 @@ go test ./lib/goidiomatic -run '^$' -bench '^BenchmarkEvalComparison$' \ -benchmem -benchtime=500ms -cpu=1 -count=6 ``` +#### Increasing the work per call + +`eval` reuses the caller's Thread. Each call builds a private variable environment +and compiles an expression; it does not fork the whole runtime. To measure how +this preparation cost compares with longer computation, keep the source fixed: +`len([(i * 3 + 7) % 97 for i in range(n)])`. Only integer `n` is passed in and +returned, so larger input/output graphs do not add copying work. + +On the same host, six alternating normal/eval pairs at 300 ms per workload gave: + +| Iterations | Normal (µs) | eval (µs) | Paired extra time (µs) | Paired extra (%) | +| --- | ---: | ---: | ---: | ---: | +| 0 | 0.238 | 7.394 | 7.155 | 3013.1% | +| 10 | 1.336 | 8.603 | 7.255 | 544.4% | +| 100 | 9.514 | 17.097 | 7.619 | 80.9% | +| 1,000 | 91.241 | 99.590 | 8.878 | 9.7% | +| 10,000 | 984.503 | 1035.482 | 54.321 | 5.5% | +| 100,000 | 13076.609 | 12915.692 | -156.653* | -1.3%* | + +Times are per-path medians; extra time/percentages are medians of paired +differences/ratios, so subtracting the displayed medians need not give the delta. +At zero iterations, about **7.2 µs** estimates the extra cost for this source and +small data graph. The extra allocation was roughly 9.4 KB per call. Relative +overhead becomes smaller as computation grows, but the time gap is not constant. +*At 100,000 iterations, individual paired differences ranged from -1.21 to ++1.38 ms: the negative median does not establish a speedup or absence of cost.* + +These timings depend on CPU, OS, architecture, Go/compiler, GC and system load. +Larger source or input/output graphs also increase preparation work. Re-run +`BenchmarkEvalWorkload` with `-benchtime=300ms -cpu=1 -count=6` on your target +system; these results are estimates for the measured workloads, not a universal +per-call constant. + ## Types ### `shared_dict` diff --git a/lib/goidiomatic/eval_test.go b/lib/goidiomatic/eval_test.go index ca02bc09..cf407e8f 100644 --- a/lib/goidiomatic/eval_test.go +++ b/lib/goidiomatic/eval_test.go @@ -593,3 +593,72 @@ func BenchmarkEvalComparison(b *testing.B) { }) } } + +// BenchmarkEvalWorkload keeps source, input shape and scalar output shape fixed +// while increasing work inside the expression. Normal and EvalReusedVars use +// identical arguments and the same two-argument Starlark call shape. Every eval +// includes parsing, compilation and data isolation; setup and result checks are +// outside timing. Differences estimate the incremental cost of this expression +// on the measured system, not a universal constant or a before/after regression. +func BenchmarkEvalWorkload(b *testing.B) { + const source = "len([(i * 3 + 7) % 97 for i in range(n)])" + module, err := mod.LoadModule() + if err != nil { + b.Fatal(err) + } + script := "def normal(n, vars):\n return " + source + "\n\n" + + "def eval_reused(n, vars):\n return eval(" + std.String(source).String() + ", vars=vars)\n" + functions, err := std.ExecFileOptions(&syntax.FileOptions{Set: true}, &std.Thread{}, "eval_workload.star", script, module) + if err != nil { + b.Fatal(err) + } + for _, workload := range []struct { + name string + n int + }{ + {"N0", 0}, + {"N10", 10}, + {"N100", 100}, + {"N1000", 1000}, + {"N10000", 10000}, + {"N100000", 100000}, + } { + b.Run(workload.name, func(b *testing.B) { + want := std.MakeInt(workload.n) + vars := std.NewDict(1) + if err := vars.SetKey(std.String("n"), want); err != nil { + b.Fatal(err) + } + vars.Freeze() + args := std.Tuple{want, vars} + for _, path := range []struct{ name, function string }{ + {"Normal", "normal"}, + {"EvalReusedVars", "eval_reused"}, + } { + b.Run(path.name, func(b *testing.B) { + thread := &std.Thread{Name: "eval-workload"} + function := functions[path.function] + result, err := std.Call(thread, function, args, nil) + if err != nil { + b.Fatal(err) + } + if equal, err := std.Equal(result, want); err != nil || !equal { + b.Fatalf("wrong result: %v, want %v (error %v)", result, want, err) + } + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + result, err = std.Call(thread, function, args, nil) + if err != nil { + b.Fatal(err) + } + } + b.StopTimer() + if equal, err := std.Equal(result, want); err != nil || !equal { + b.Fatalf("wrong final result: %v, want %v (error %v)", result, want, err) + } + }) + } + }) + } +} From f8a2c347ef6ca15b226267afccce5cc8c2638628 Mon Sep 17 00:00:00 2001 From: Kevin Tang <73975146+vt128@users.noreply.github.com> Date: Sun, 8 Nov 2026 22:37:36 +0800 Subject: [PATCH 5/5] [doc] summarize expression evaluation tradeoffs (LET-36) --- lib/goidiomatic/BENCHMARKS.md | 69 +++++++++++++++++++++++++++++++ lib/goidiomatic/README.md | 78 ++++++++--------------------------- 2 files changed, 86 insertions(+), 61 deletions(-) create mode 100644 lib/goidiomatic/BENCHMARKS.md diff --git a/lib/goidiomatic/BENCHMARKS.md b/lib/goidiomatic/BENCHMARKS.md new file mode 100644 index 00000000..57a4cca7 --- /dev/null +++ b/lib/goidiomatic/BENCHMARKS.md @@ -0,0 +1,69 @@ +# Expression evaluation benchmark notes + +These local measurements compare direct compiled Starlark with the public +`eval` and `try_eval` builtins. See [usage and qualitative guidance](README.md#performance). +They describe the measured workloads and systems; run the benchmarks on your +deployment target before making latency or throughput assumptions. + +Local measurements on an Apple M4 Pro, Go 1.27.3, darwin/arm64, `-cpu=1`: +medians of six 500 ms runs, in **microseconds per call** (1000 µs = 1 ms). +Both cases use `qty=3`; the scalar case uses `price=12`, and the nested case +evaluates `{"values": [x * qty for x in items]}` with `items=list(range(32))`. + +| Path | `price * qty` | 32-item nested result | +| --- | ---: | ---: | +| Normal compiled Starlark function | 0.062 µs | 1.821 µs | +| `eval`, reused `vars` dict | 4.504 µs (72.4×) | 19.709 µs (10.8×) | +| `eval`, new `vars` dict per call | 4.657 µs (75.2×) | 20.022 µs (11.0×) | +| `try_eval`, reused `vars`, success | 4.630 µs (74.6×) | 19.896 µs (11.0×) | + +The normal/reused-`eval` allocation costs were 80/6976 B and 1/111 allocations +for the scalar case, and 1760/25025 B and 11/682 allocations for the nested case. +Every path calls a compiled Starlark function with identical inputs. Outer +compilation, input preparation and warmup are excluded; per-call `eval` parsing, +compilation, copying, validation and freezing are included. Normal execution +does not provide this isolation. Ratios are medians of the six run-wise ratios; +these are local examples, not application-wide latency or throughput guarantees. + +Use direct script code for fixed calculations and hot loops. `eval` is intended +for dynamic expressions: it does not cache compiled expressions, even when +`vars` is reused. Run the same workloads with: + +```sh +go test ./lib/goidiomatic -run '^$' -bench '^BenchmarkEvalComparison$' \ + -benchmem -benchtime=500ms -cpu=1 -count=6 +``` + +## Increasing the work per call + +`eval` reuses the caller's Thread. Each call builds a private variable environment +and compiles an expression; it does not fork the whole runtime. To measure how +this preparation cost compares with longer computation, keep the source fixed: +`len([(i * 3 + 7) % 97 for i in range(n)])`. Only integer `n` is passed in and +returned, so larger input/output graphs do not add copying work. + +On the same host, six alternating normal/eval pairs at 300 ms per workload gave: + +| Iterations | Normal (µs) | eval (µs) | Paired extra time (µs) | Paired extra (%) | +| --- | ---: | ---: | ---: | ---: | +| 0 | 0.238 | 7.394 | 7.155 | 3013.1% | +| 10 | 1.336 | 8.603 | 7.255 | 544.4% | +| 100 | 9.514 | 17.097 | 7.619 | 80.9% | +| 1,000 | 91.241 | 99.590 | 8.878 | 9.7% | +| 10,000 | 984.503 | 1035.482 | 54.321 | 5.5% | +| 100,000 | 13076.609 | 12915.692 | -156.653* | -1.3%* | + +Times are per-path medians; extra time/percentages are medians of paired +differences/ratios, so subtracting the displayed medians need not give the delta. +At zero iterations, about **7.2 µs** estimates the extra cost for this source and +small data graph. The extra allocation was roughly 9.4 KB per call. Relative +overhead becomes smaller as computation grows, but the time gap is not constant. +*At 100,000 iterations, individual paired differences ranged from -1.21 to ++1.38 ms: the negative median does not establish a speedup or absence of cost.* + +These timings depend on CPU, OS, architecture, Go/compiler, GC and system load. +Larger source or input/output graphs also increase preparation work. Re-run +`BenchmarkEvalWorkload` with `-benchtime=300ms -cpu=1 -count=6` on your target +system; these results are estimates for the measured workloads, not a universal +per-call constant. + diff --git a/lib/goidiomatic/README.md b/lib/goidiomatic/README.md index eec563f5..9178b033 100644 --- a/lib/goidiomatic/README.md +++ b/lib/goidiomatic/README.md @@ -208,67 +208,23 @@ validated process boundary. ### Performance -Local measurements on an Apple M4 Pro, Go 1.27.3, darwin/arm64, `-cpu=1`: -medians of six 500 ms runs, in **microseconds per call** (1000 µs = 1 ms). -Both cases use `qty=3`; the scalar case uses `price=12`, and the nested case -evaluates `{"values": [x * qty for x in items]}` with `items=list(range(32))`. - -| Path | `price * qty` | 32-item nested result | -| --- | ---: | ---: | -| Normal compiled Starlark function | 0.062 µs | 1.821 µs | -| `eval`, reused `vars` dict | 4.504 µs (72.4×) | 19.709 µs (10.8×) | -| `eval`, new `vars` dict per call | 4.657 µs (75.2×) | 20.022 µs (11.0×) | -| `try_eval`, reused `vars`, success | 4.630 µs (74.6×) | 19.896 µs (11.0×) | - -The normal/reused-`eval` allocation costs were 80/6976 B and 1/111 allocations -for the scalar case, and 1760/25025 B and 11/682 allocations for the nested case. -Every path calls a compiled Starlark function with identical inputs. Outer -compilation, input preparation and warmup are excluded; per-call `eval` parsing, -compilation, copying, validation and freezing are included. Normal execution -does not provide this isolation. Ratios are medians of the six run-wise ratios; -these are local examples, not application-wide latency or throughput guarantees. - -Use direct script code for fixed calculations and hot loops. `eval` is intended -for dynamic expressions: it does not cache compiled expressions, even when -`vars` is reused. Run the same workloads with: - -```sh -go test ./lib/goidiomatic -run '^$' -bench '^BenchmarkEvalComparison$' \ - -benchmem -benchtime=500ms -cpu=1 -count=6 -``` - -#### Increasing the work per call - -`eval` reuses the caller's Thread. Each call builds a private variable environment -and compiles an expression; it does not fork the whole runtime. To measure how -this preparation cost compares with longer computation, keep the source fixed: -`len([(i * 3 + 7) % 97 for i in range(n)])`. Only integer `n` is passed in and -returned, so larger input/output graphs do not add copying work. - -On the same host, six alternating normal/eval pairs at 300 ms per workload gave: - -| Iterations | Normal (µs) | eval (µs) | Paired extra time (µs) | Paired extra (%) | -| --- | ---: | ---: | ---: | ---: | -| 0 | 0.238 | 7.394 | 7.155 | 3013.1% | -| 10 | 1.336 | 8.603 | 7.255 | 544.4% | -| 100 | 9.514 | 17.097 | 7.619 | 80.9% | -| 1,000 | 91.241 | 99.590 | 8.878 | 9.7% | -| 10,000 | 984.503 | 1035.482 | 54.321 | 5.5% | -| 100,000 | 13076.609 | 12915.692 | -156.653* | -1.3%* | - -Times are per-path medians; extra time/percentages are medians of paired -differences/ratios, so subtracting the displayed medians need not give the delta. -At zero iterations, about **7.2 µs** estimates the extra cost for this source and -small data graph. The extra allocation was roughly 9.4 KB per call. Relative -overhead becomes smaller as computation grows, but the time gap is not constant. -*At 100,000 iterations, individual paired differences ranged from -1.21 to -+1.38 ms: the negative median does not establish a speedup or absence of cost.* - -These timings depend on CPU, OS, architecture, Go/compiler, GC and system load. -Larger source or input/output graphs also increase preparation work. Re-run -`BenchmarkEvalWorkload` with `-benchtime=300ms -cpu=1 -count=6` on your target -system; these results are estimates for the measured workloads, not a universal -per-call constant. +`eval` reuses the caller's Thread and builds a private environment from explicit +variables. Every call parses and compiles the expression, copies and validates +input/output data, and checks the caller's execution limits. Reusing `vars` +does not cache the compiled expression or remove this preparation work. + +Benchmarks against the same expression in a compiled Starlark function found +that this preparation has a noticeable cost for small calculations. With fixed +source and small input/output graphs, more computation can reduce the relative +cost. The total time difference is not a single constant: longer source, larger +data graphs, allocation and GC also affect it. Large-workload variation does not +establish that dynamic evaluation is faster or has no overhead. + +Use direct script code for fixed calculations and hot loops; use `eval` when the +expression itself is supplied at runtime. Timings depend on CPU, OS, +architecture, Go/compiler and system load. Measure on your target system with +`BenchmarkEvalComparison` and `BenchmarkEvalWorkload`; [benchmark notes](BENCHMARKS.md) +provide the workloads, sample results and reproduction commands. ## Types