diff --git a/USAGE.md b/USAGE.md index e5bf715..3cc743c 100644 --- a/USAGE.md +++ b/USAGE.md @@ -214,39 +214,32 @@ print("new" in original, original["items"]) # Output: False [1, 2] ``` -Prefer one owner goroutine for a shared value, with other goroutines sending it requests. Serialize both updates and output at that owner. Default Starlark formatting/printing and the Go `String`/`ToJSON` methods do not lock the mapping themselves. +`String()` (including Starlark `print`/`str`/`repr`) and `SnapshotString()` now capture mapping entries under an internal lock, then release that lock before formatting callbacks. Concurrent output operations are serialized without waiting: if an output or mapping lock is already held, they return the fixed marker `shared_dict()`. This also applies to nested SharedDict values and reentrant calls. Normal output retains insertion order and configured type names. `SetTypeName` is safe to call concurrently. -If multiple goroutines need access, use one application mutex for **every access to the shared mutable graph**, including output and reads/writes through nested aliases. Hold it for an entire Machine `Run`/`Call` that can read or change the same value. The dictionary's internal locks do not make other accesses participate in this application mutex. This minimal Go example keeps the dictionary private to the synchronized operations: +Go `ToJSON() (string, error)` and Starlark `to_json()` use the same nonblocking serialization path. Go callers can check `errors.Is(err, dataconv.ErrSharedDictBusy)`; a failed call returns no JSON. Retry after leaving the current lock or callback, rather than retrying in a loop that still holds it. ```go package main import ( + "errors" "fmt" "log" - "sync" "github.com/1set/starlet/dataconv" "go.starlark.net/starlark" ) func main() { - var gate sync.Mutex shared := dataconv.NewSharedDict() - store := func(n int) error { - gate.Lock() - defer gate.Unlock() - return shared.SetKey(starlark.String("count"), starlark.MakeInt(n)) - } - encode := func() (string, error) { - gate.Lock() - defer gate.Unlock() - return shared.ToJSON() - } - if err := store(1); err != nil { + if err := shared.SetKey(starlark.String("count"), starlark.MakeInt(1)); err != nil { log.Fatal(err) } - text, err := encode() + text, err := shared.ToJSON() + if errors.Is(err, dataconv.ErrSharedDictBusy) { + log.Print("shared output is busy; retry outside the current critical section") + return + } if err != nil { log.Fatal(err) } @@ -255,34 +248,16 @@ func main() { } ``` -The example contains only a native integer. Formatting and JSON encoding of Go-backed values can invoke host callbacks: while holding the application mutex, do not reenter it or wait for work that needs it. A callback at the owner goroutine must likewise not wait for that owner to handle another request. For complex host values, have the owner construct independent plain data, then encode that data after leaving the ownership or lock boundary. `to_dict()`/`CloneDict()` are shallow copies; freezing the original shared value changes that original rather than creating an isolated copy. Neither substitutes for ownership of nested mutable values. - -For on-demand debugging and diagnostics in Go, use -`text := shared.SnapshotString()` to capture and format the mapping's entries -without racing with mapping updates. Formatting callbacks run after the mapping -lock is released; mutable nested values remain shared. -Call `SnapshotString` without already holding that dictionary's mutex, and -outside callbacks from operations that hold it, such as key `Hash`/`Equal` or -value `Freeze`. This Go method does not add a Starlark builtin or change default -printing. See [shared_dict methods and callback rules](lib/goidiomatic/README.md#shared_dict). - -Cycle detection follows list/dict/set references through tuples, nested -`SharedDict` values and structs with unique field names. Nested `SharedDict` -values are not locked; synchronize them separately. Sets capture their elements -before formatting callbacks run. Custom values and Go-created structs with -duplicate field names keep their own `String` behavior. Go-created cycles -consisting only of tuples are not detected. There is no general depth or output -limit. - -Snapshot formatting copies entries and tracks container paths, adding temporary -memory and work. Large sets and nested containers can take longer and use more -bytes than direct formatting. Call it only when diagnostic output is needed; -avoid high-frequency business loops. Normal execution and default printing do -not call it automatically. Costs depend on the values, Go version and machine. -Run the included comparisons with: +SharedDict JSON accepts native JSON scalars and containers, native structs/modules, duration, and the interpreter's immutable range/string/bytes iterator sources. Dictionary keys must be strings. Unsupported values and native container cycles return errors. **Other Go-backed values, including `GoStruct`, `GoMap`, `GoSlice` and custom JSON/attribute implementations, are rejected by SharedDict JSON.** Convert such values to independently owned plain data before storing or encoding them. The general `EncodeStarlarkJSON` helper retains its existing behavior, including host callbacks, and requires the caller to own or synchronize its input. + +Internal output locks do not take ownership of raw nested List/Dict/Set aliases or native data shared by independent SharedDict owners. Keep those values under one owner or a common application mutex for every access. Use that same boundary for whole Machine `Run`/`Call` operations and compound updates that need an atomic application snapshot. Formatting callbacks must not reenter an application mutex or wait for its owner while that boundary is held. `to_dict()`/`CloneDict()` are shallow copies; freezing the original changes that original rather than creating an isolated copy. + +`SnapshotString` remains a Go-only name for the same formatter used by default output. Cycle detection follows list/dict/set references through tuples, nested SharedDict values and structs with unique field names. Formatting callbacks run after mapping locks are released; nested references reuse captured owner views, while reentrant output calls return a busy marker. Custom values and Go-created structs with duplicate field names retain their own `String` behavior. Go-created tuple-only cycles and general depth/output limits remain outside this formatter's guarantees. Other dictionary operations retain their ordinary mutex discipline. + +Output locking, entry snapshots, cycle paths and JSON validation add work and temporary memory. Paired benchmarks observed higher costs for set formatting and JSON encoding; ordinary `Get`/`SetKey` added no allocations. Setting a nonempty type name also allocates. Results depend on data, compiler, machine and load; avoid unnecessary formatting in business hot loops. Run the included read and output benchmarks with: ```bash -go test ./dataconv -run '^$' -bench '^BenchmarkSharedDictReadPaths$' -benchmem +go test ./dataconv -run '^$' -bench '^BenchmarkSharedDict(ReadPaths|OutputSafety)$' -benchmem ``` ## Terminal REPL diff --git a/dataconv/share.go b/dataconv/share.go index 6c8d5a8..febc99d 100644 --- a/dataconv/share.go +++ b/dataconv/share.go @@ -1,14 +1,16 @@ package dataconv import ( + "errors" "fmt" - "sort" + "reflect" "strings" "sync" + "sync/atomic" tps "github.com/1set/starlet/dataconv/types" itn "github.com/1set/starlet/internal" - stdjson "go.starlark.net/lib/json" + starlarktime "go.starlark.net/lib/time" "go.starlark.net/starlark" "go.starlark.net/starlarkstruct" "go.starlark.net/syntax" @@ -18,25 +20,30 @@ import ( // methods, Len, CloneDict, LoadJSON, Freeze and comparison. // Returned keys/values and shallow clones can retain mutable nested aliases; // the mapping's lock does not synchronize access through those aliases. -// String and the Go ToJSON method do not lock: stop or externally synchronize -// writers before formatting or serializing the dictionary. Configure its type -// name before sharing it between goroutines. -// SnapshotString provides explicit snapshot formatting from Go without holding -// the mapping lock while formatting values. +// String and SnapshotString capture entries before formatting, and ToJSON +// serializes native Starlark data under the mapping lock. Output methods do not +// wait for a busy dictionary: formatting returns shared_dict(), while +// ToJSON returns ErrSharedDictBusy. Type names can be changed concurrently. // A frozen SharedDict rejects mapping updates and freezes its contained values. type SharedDict struct { _ itn.DoNotCompare sync.RWMutex - dict *starlark.Dict - frozen bool - name string + outputMu sync.Mutex + dict *starlark.Dict + frozen bool + name atomic.Pointer[string] } const ( defaultSharedDictSize = 8 defaultSharedDictName = "shared_dict" + sharedDictBusyString = "shared_dict()" ) +// ErrSharedDictBusy reports output that could not safely acquire its locks. +// Retry outside callbacks or critical sections; a busy result contains no JSON. +var ErrSharedDictBusy = errors.New("shared dictionary is busy") + var ( _ starlark.Value = (*SharedDict)(nil) _ starlark.Comparable = (*SharedDict)(nil) @@ -54,10 +61,9 @@ func NewSharedDict() *SharedDict { // NewNamedSharedDict creates a new SharedDict instance with the given name. func NewNamedSharedDict(name string) *SharedDict { - return &SharedDict{ - dict: starlark.NewDict(defaultSharedDictSize), - name: name, - } + s := NewSharedDict() + s.SetTypeName(name) + return s } // NewSharedDictFromDict creates a new SharedDict instance from the given starlark.Dict. @@ -74,46 +80,22 @@ func NewSharedDictFromDict(d *starlark.Dict) *SharedDict { } func (s *SharedDict) String() string { - var v string - if s != nil && s.dict != nil { - v = s.dict.String() - } - return fmt.Sprintf("%s(%s)", s.getTypeName(), v) + return s.SnapshotString() } // SnapshotString formats a shallow snapshot of the mapping in insertion order. -// It releases the mapping lock before invoking formatting callbacks. Mutable -// nested values remain shared and require their own synchronization. -// Nested SharedDict values are not locked by this method. Cycle detection spans +// It releases mapping locks before invoking formatting callbacks. Output from +// each visited SharedDict is serialized without waiting for a busy lock; a busy +// dictionary is rendered as shared_dict(), including reentrant calls. +// Mutable native aliases still require their own ownership or synchronization. +// Cycle detection spans // native lists/dictionaries/sets, SharedDict and structs with unique field names, // following references via tuples. Other values retain their own String behavior. // Set elements are captured before formatting callbacks run; the values remain shared. -// Call it without holding this dictionary's mutex, and outside callbacks from -// operations that hold it (for example a key's Hash/Equal or a value's Freeze). func (s *SharedDict) SnapshotString() string { - s.RLock() - dict, name := s.dict, s.getTypeName() - var items []starlark.Tuple - if dict != nil { - items = dict.Items() - } - s.RUnlock() - - if dict == nil { - return name + "()" - } - if len(items) == 0 { - return name + "({})" - } var f sharedDictFormatter - f.shared = s - f.name = name - f.root = dict - f.items = items - f.WriteString(name) - f.WriteByte('(') - f.write(dict, nil) - f.WriteByte(')') + defer f.release() + f.write(s, nil) return f.String() } @@ -122,12 +104,51 @@ func (s *SharedDict) SnapshotString() string { // keys or invoking host callbacks under the mapping lock. type sharedDictFormatter struct { strings.Builder + snapshots []sharedDictSnapshot +} + +type sharedDictSnapshot struct { shared *SharedDict + dict *starlark.Dict name string - root *starlark.Dict items []starlark.Tuple } +func (f *sharedDictFormatter) release() { + for i := len(f.snapshots) - 1; i >= 0; i-- { + f.snapshots[i].shared.outputMu.Unlock() + } +} + +func (f *sharedDictFormatter) capture(shared *SharedDict) (*sharedDictSnapshot, bool) { + for i := range f.snapshots { + if f.snapshots[i].shared == shared { + return &f.snapshots[i], true + } + } + if !shared.outputMu.TryLock() { + return nil, false + } + if !shared.TryLock() { + shared.outputMu.Unlock() + return nil, false + } + captured := false + defer func() { + shared.Unlock() + if !captured { + shared.outputMu.Unlock() + } + }() + snapshot := sharedDictSnapshot{shared: shared, dict: shared.dict, name: shared.getTypeName()} + if snapshot.dict != nil { + snapshot.items = snapshot.dict.Items() + } + f.snapshots = append(f.snapshots, snapshot) + captured = true + return &f.snapshots[len(f.snapshots)-1], true +} + func (f *sharedDictFormatter) write(value starlark.Value, path []starlark.Value) { switch value := value.(type) { case nil: @@ -159,14 +180,19 @@ func (f *sharedDictFormatter) write(value starlark.Value, path []starlark.Value) } f.WriteByte(')') case *SharedDict: - dict, name := f.root, f.name - if value != f.shared { - dict, name = value.dict, value.getTypeName() + if value == nil { + f.WriteString(defaultSharedDictName + "()") + return } - f.WriteString(name) + snapshot, ok := f.capture(value) + if !ok { + f.WriteString(sharedDictBusyString) + return + } + f.WriteString(snapshot.name) f.WriteByte('(') - if dict != nil { - f.write(dict, path) + if snapshot.dict != nil { + f.write(snapshot.dict, path) } f.WriteByte(')') case *starlark.Set: @@ -178,8 +204,15 @@ func (f *sharedDictFormatter) write(value starlark.Value, path []starlark.Value) if sharedDictPathContains(path, value) { f.WriteString("...") } else { - items := f.items - if value != f.root { + var items []starlark.Tuple + captured := false + for _, snapshot := range f.snapshots { + if snapshot.dict == value { + items, captured = snapshot.items, true + break + } + } + if !captured { items = value.Items() } valuePath := append(path, value) @@ -287,15 +320,20 @@ func sharedDictPathContains(path []starlark.Value, value starlark.Value) bool { // SetTypeName sets the type name of the SharedDict. func (s *SharedDict) SetTypeName(name string) { - s.name = name + if name == "" { + s.name.Store(nil) + return + } + s.name.Store(&name) } // getTypeName returns the type name of the SharedDict. func (s *SharedDict) getTypeName() string { - if s.name == "" { + name := s.name.Load() + if name == nil { return defaultSharedDictName } - return s.name + return *name } // Type returns the type name of the SharedDict. @@ -365,58 +403,55 @@ func (s *SharedDict) SetKey(k, v starlark.Value) error { // Attr returns the value of the specified attribute, or (nil, nil) if the attribute is not found. // It implements the starlark.HasAttrs interface. func (s *SharedDict) Attr(name string) (starlark.Value, error) { - s.Lock() - defer s.Unlock() - - // basic check - if s.dict == nil { - return nil, nil - } - - var ( - attr starlark.Value - err error - ) - // try to get the new custom builtin - if b, ok := customSharedDictMethods[name]; ok { - attr = b.BindReceiver(s.dict) - } else { - // get the builtin from the original dict - attr, err = s.dict.Attr(name) - } - - // convert to builtin - if attr == nil || err != nil { - return attr, err + if name == "to_json" { + return starlark.NewBuiltin(name, s.sharedDictToJSON), nil } - btl, ok := attr.(*starlark.Builtin) + btl, ok := customSharedDictMethods[name] if !ok { - return nil, fmt.Errorf("unsupported attribute: %s", name) + attr, err := sharedDictMethodPrototype.Attr(name) + if attr == nil || err != nil { + return attr, err + } + btl, ok = attr.(*starlark.Builtin) + if !ok { + return nil, fmt.Errorf("unsupported attribute: %s", name) + } } - - // wrap the builtin - return starlark.NewBuiltin(name, func(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) { - // lock the shared dict + // Metadata lookup does not touch the live mapping. Acquire its lock only + // when calling the method, preserving the existing method-call contract. + return starlark.NewBuiltin(name, func(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) { s.Lock() defer s.Unlock() - - // call the original builtin - return starlark.Call(thread, btl, args, kwargs) + s.ensureDict() + return starlark.Call(thread, btl.BindReceiver(s.dict), args, kwargs) }), nil } // AttrNames returns a new slice containing the names of all the attributes of the SharedDict. // It implements the starlark.HasAttrs interface. func (s *SharedDict) AttrNames() []string { - if s.dict != nil { - names := s.dict.AttrNames() - for cn := range customSharedDictMethods { - names = append(names, cn) + if s == nil { + return nil + } + return append([]string(nil), sharedDictAttrNames...) +} + +var sharedDictMethodPrototype = starlark.NewDict(0) + +var sharedDictAttrNames = []string{ + "clear", "from_json", "get", "items", "keys", "len", "perform", "pop", + "popitem", "setdefault", "to_dict", "to_json", "update", "values", +} + +// The caller holds the mapping's write lock. A zero-value dictionary is empty; +// initialization must preserve a Freeze that happened before its first use. +func (s *SharedDict) ensureDict() { + if s.dict == nil { + s.dict = starlark.NewDict(defaultSharedDictSize) + if s.frozen { + s.dict.Freeze() } - sort.Strings(names) - return names } - return nil } // CompareSameType compares the SharedDict with another value of the same type. @@ -481,15 +516,166 @@ func (s *SharedDict) CloneDict() (*starlark.Dict, error) { return CloneDict(s.dict) } -// ToJSON serializes the SharedDict with the Starlark JSON encoder. -// Direct Go callers must synchronize it with mapping writes; the Starlark -// to_json method acquires the mapping lock through its attribute wrapper. -// Mutable nested values require their own ownership or synchronization. -// The encoder rejects unsupported values and cycles in native Starlark containers. -// This does not guarantee cycle detection or termination inside arbitrary host -// MarshalJSON methods, attribute callbacks or Go-backed values. +// ToJSON serializes native Starlark data with the official JSON encoder. +// It returns ErrSharedDictBusy rather than waiting for another output operation +// or an already-held mapping lock. Supported JSON data includes native scalars, +// containers, structs/modules, duration and the interpreter's immutable range +// and string/bytes iterator sources. Dictionary keys must be strings. +// Other Go-backed values and custom JSON/attribute implementations are rejected +// before encoding, so no host code runs while the mapping lock is held. +// Mutable native aliases still require ownership or external synchronization. +// The Starlark to_json method uses the same rules. Errors return an empty string. func (s *SharedDict) ToJSON() (string, error) { - return EncodeStarlarkJSON(s.dict) + return s.toJSON(nil) +} + +func (s *SharedDict) toJSON(thread *starlark.Thread) (string, error) { + if s == nil { + return "", fmt.Errorf("shared_dict.to_json: nil shared dictionary") + } + if !s.outputMu.TryLock() { + return "", ErrSharedDictBusy + } + defer s.outputMu.Unlock() + if !s.TryLock() { + return "", ErrSharedDictBusy + } + defer s.Unlock() + s.ensureDict() + if err := sharedDictCheckJSON(s.dict); err != nil { + return "", err + } + if thread == nil { + return EncodeStarlarkJSON(s.dict) + } + value, err := starlark.Call(thread, starJSONEncode, starlark.Tuple{s.dict}, nil) + if err != nil { + return "", err + } + return string(value.(starlark.String)), nil +} + +// Validate only the exact native types whose JSON traversal invokes no host +// callbacks. A work list and visited identities also terminate on Go-created +// tuple cycles; the encoder itself reports supported-container cycle errors. +func sharedDictCheckJSON(root starlark.Value) error { + type tupleIdentity struct { + first *starlark.Value + len int + } + seen := make(map[interface{}]bool) + visit := func(key interface{}) bool { + if seen[key] { + return false + } + seen[key] = true + return true + } + invalid := func(value starlark.Value) error { + return fmt.Errorf("shared_dict.to_json: unsupported value %T; want native Starlark JSON data", value) + } + pending := []starlark.Value{root} + for len(pending) > 0 { + value := pending[len(pending)-1] + pending = pending[:len(pending)-1] + switch value := value.(type) { + case starlark.NoneType, starlark.Bool, starlark.Int, starlark.Float, starlark.String, + starlarktime.Duration, starlarktime.Time: + continue + case *starlark.Dict: + if value == nil { + return invalid(value) + } + if !visit(value) { + continue + } + for _, item := range value.Items() { + if _, ok := item[0].(starlark.String); !ok { + return fmt.Errorf("shared_dict.to_json: dictionary key must be string (got %T)", item[0]) + } + pending = append(pending, item[1]) + } + case *starlark.List: + if value == nil { + return invalid(value) + } + if !visit(value) { + continue + } + for i := 0; i < value.Len(); i++ { + pending = append(pending, value.Index(i)) + } + case starlark.Tuple: + if len(value) == 0 || !visit(tupleIdentity{&value[0], len(value)}) { + continue + } + pending = append(pending, value...) + case *starlark.Set: + if value == nil { + return invalid(value) + } + if !visit(value) { + continue + } + func() { + iter := value.Iterate() + defer iter.Done() + var element starlark.Value + for iter.Next(&element) { + pending = append(pending, element) + } + }() + case *starlarkstruct.Struct: + if value == nil { + return invalid(value) + } + if !visit(value) { + continue + } + for _, name := range value.AttrNames() { + field, err := value.Attr(name) + if err != nil || field == nil { + return fmt.Errorf("shared_dict.to_json: invalid native struct field") + } + pending = append(pending, field) + } + case *starlarkstruct.Module: + if value == nil { + return invalid(value) + } + if !visit(value) { + continue + } + for _, name := range value.AttrNames() { + field, err := value.Attr(name) + if err != nil || field == nil { + return fmt.Errorf("shared_dict.to_json: invalid native module member") + } + pending = append(pending, field) + } + default: + if sharedDictSafeJSONIterable(value) { + continue + } + return invalid(value) + } + } + return nil +} + +// These four immutable, private iterator sources are closed over scalar data in +// the pinned interpreter. Do not trust arbitrary Iterable implementations or +// pre-traverse a potentially large range merely to validate its element types. +func sharedDictSafeJSONIterable(value starlark.Value) bool { + typ := reflect.TypeOf(value) + if typ == nil || typ.Kind() != reflect.Struct || typ.PkgPath() != "go.starlark.net/starlark" { + return false + } + switch typ.Name() { + case "rangeValue", "stringElems", "stringCodepoints", "bytesIterable": + return true + } + return false } // LoadJSON updates the SharedDict instance with key-value pairs decoded from a given JSON string. @@ -540,7 +726,6 @@ var ( "len": starlark.NewBuiltin("len", sharedDictLen), "perform": starlark.NewBuiltin("perform", sharedDictPerform), "to_dict": starlark.NewBuiltin("to_dict", sharedDictToDict), - "to_json": starlark.NewBuiltin("to_json", sharedDictToJSON), "from_json": starlark.NewBuiltin("from_json", sharedDictFromJSON), } ) @@ -588,24 +773,17 @@ func sharedDictToDict(thread *starlark.Thread, b *starlark.Builtin, args starlar } // sharedDictToJSON converts the underlying dictionary to a JSON string. -func sharedDictToJSON(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) { +func (s *SharedDict) sharedDictToJSON(thread *starlark.Thread, b *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error) { // check the arguments: no arguments if err := starlark.UnpackPositionalArgs(b.Name(), args, kwargs, 0); err != nil { return nil, err } - // get the receiver - d := b.Receiver().(*starlark.Dict) - - // get the JSON encoder - jm, ok := stdjson.Module.Members["encode"] - if !ok { - return nil, fmt.Errorf("json.encode not found") + text, err := s.toJSON(thread) + if err != nil { + return nil, fmt.Errorf("%s: %w", b.Name(), err) } - enc := jm.(*starlark.Builtin) - - // convert to JSON - return starlark.Call(thread, enc, starlark.Tuple{d}, nil) + return starlark.String(text), nil } // sharedDictFromJSON converts a starlark.Value to a starlark.Dict, and wraps it with a SharedDict. diff --git a/dataconv/share_test.go b/dataconv/share_test.go index ac25555..131febf 100644 --- a/dataconv/share_test.go +++ b/dataconv/share_test.go @@ -1,16 +1,19 @@ package dataconv // SharedDict tests cover constructors/names, script methods, frozen/nil values, -// concurrent access, caller-synchronized output, snapshots, and JSON conversion. +// concurrent access, nonblocking output, snapshots, and native JSON conversion. import ( "context" + "encoding/json" + "errors" "fmt" "math" "os" "os/exec" "reflect" "runtime/debug" + "sort" "strings" "sync" "testing" @@ -126,10 +129,12 @@ func TestSharedDict_SnapshotString(t *testing.T) { } }) t.Run("empty_root", func(t *testing.T) { + zero := new(SharedDict) + zero.SetTypeName("zero") for _, test := range []struct { dict *SharedDict want string - }{{NewSharedDict(), "shared_dict({})"}, {NewNamedSharedDict("named"), "named({})"}, {&SharedDict{name: "zero"}, "zero()"}} { + }{{NewSharedDict(), "shared_dict({})"}, {NewNamedSharedDict("named"), "named({})"}, {zero, "zero()"}} { if got := test.dict.SnapshotString(); got != test.want { t.Fatalf("SnapshotString() = %q; want %q", got, test.want) } @@ -261,6 +266,15 @@ func TestSharedDict_SnapshotStringCallbackPanic(t *testing.T) { t.Fatal("mapping lock remained held after a formatting panic") } sd.Unlock() + if err := sd.SetKey(starlark.String("value"), starlark.None); err != nil { + t.Fatal(err) + } + if sd.String() == sharedDictBusyString || sd.SnapshotString() == sharedDictBusyString { + t.Fatal("output lock leaked after formatting panic") + } + if _, err := sd.ToJSON(); err != nil { + t.Fatalf("ToJSON after formatting panic: %v", err) + } } func TestSharedDict_SnapshotStringConcurrent(t *testing.T) { @@ -287,7 +301,7 @@ func TestSharedDict_SnapshotStringConcurrent(t *testing.T) { <-start for i := 0; i < 1000; i++ { got := sd.SnapshotString() - if !strings.HasPrefix(got, `shared_dict({"counter": `) || !strings.HasSuffix(got, "})") { + if got != sharedDictBusyString && (!strings.HasPrefix(got, `shared_dict({"counter": `) || !strings.HasSuffix(got, "})")) { t.Errorf("malformed concurrent String: %q", got) return } @@ -297,8 +311,423 @@ func TestSharedDict_SnapshotStringConcurrent(t *testing.T) { workers.Wait() } -// This verifies the caller's synchronization discipline; String and ToJSON do -// not acquire the application mutex or become safe for unsynchronized callers. +func TestSharedDict_OutputLocks(t *testing.T) { + sd := NewNamedSharedDict("named") + if err := sd.SetKey(starlark.String("n"), starlark.MakeInt(1)); err != nil { + t.Fatal(err) + } + encode, err := sd.Attr("to_json") + if err != nil { + t.Fatal(err) + } + for _, lock := range []struct { + name string + lock func() + unlock func() + }{{"write", sd.Lock, sd.Unlock}, {"read", sd.RLock, sd.RUnlock}, {"output", sd.outputMu.Lock, sd.outputMu.Unlock}} { + t.Run(lock.name, func(t *testing.T) { + lock.lock() + defer lock.unlock() + if got := sd.String(); got != sharedDictBusyString { + t.Errorf("String while busy = %q", got) + } + if got := sd.SnapshotString(); got != sharedDictBusyString { + t.Errorf("SnapshotString while busy = %q", got) + } + if text, err := sd.ToJSON(); text != "" || !errors.Is(err, ErrSharedDictBusy) { + t.Errorf("ToJSON while busy = (%q, %v)", text, err) + } + if _, err := starlark.Call(&starlark.Thread{}, encode, nil, nil); !errors.Is(err, ErrSharedDictBusy) { + t.Errorf("script to_json while busy = %v", err) + } + if lock.name == "write" { + if method, err := sd.Attr("to_json"); method == nil || err != nil { + t.Errorf("to_json lookup while locked = (%v,%v)", method, err) + } + if names := sd.AttrNames(); !reflect.DeepEqual(names, sharedDictAttrNames) { + t.Errorf("attribute names while locked = %v", names) + } + } + }) + } + if got := sd.String(); got != `named({"n": 1})` { + t.Errorf("String after busy calls = %q", got) + } + if text, err := sd.ToJSON(); text != `{"n":1}` || err != nil { + t.Errorf("ToJSON after busy calls = (%q, %v)", text, err) + } + var nilDict *SharedDict + if names := nilDict.AttrNames(); names != nil { + t.Errorf("nil AttrNames = %v", names) + } + if got := nilDict.String(); got != "shared_dict()" { + t.Errorf("nil String = %q", got) + } + if text, err := nilDict.ToJSON(); text != "" || err == nil { + t.Errorf("nil ToJSON = (%q, %v)", text, err) + } + if text, err := new(SharedDict).ToJSON(); text != "{}" || err != nil { + t.Errorf("zero ToJSON = (%q, %v)", text, err) + } +} + +func TestSharedDict_OutputConcurrent(t *testing.T) { + for _, withWriter := range []bool{false, true} { + t.Run(fmt.Sprintf("writer=%v", withWriter), func(t *testing.T) { + sd := NewSharedDict() + if err := sd.SetKey(starlark.String("counter"), starlark.MakeInt(0)); err != nil { + t.Fatal(err) + } + set := starlark.NewSet(2) + for _, n := range []int{1, 2} { + if err := set.Insert(starlark.MakeInt(n)); err != nil { + t.Fatal(err) + } + } + if err := sd.SetKey(starlark.String("list"), starlark.NewList([]starlark.Value{starlark.MakeInt(1)})); err != nil { + t.Fatal(err) + } + if err := sd.SetKey(starlark.String("set"), set); err != nil { + t.Fatal(err) + } + start := make(chan struct{}) + var workers sync.WaitGroup + for _, format := range []func() string{sd.String, sd.SnapshotString} { + format := format + workers.Add(1) + go func() { + defer workers.Done() + <-start + for i := 0; i < 128; i++ { + got := format() + if got != sharedDictBusyString && (!strings.HasPrefix(got, `shared_dict({"counter": `) || !strings.HasSuffix(got, `, "list": [1], "set": set([1, 2])})`)) { + t.Errorf("concurrent format = %q", got) + return + } + } + }() + } + workers.Add(1) + go func() { + defer workers.Done() + <-start + for i := 0; i < 128; i++ { + text, err := sd.ToJSON() + if errors.Is(err, ErrSharedDictBusy) && text == "" { + continue + } + var got struct { + Counter int `json:"counter"` + List []int `json:"list"` + Set []int `json:"set"` + } + if err != nil || json.Unmarshal([]byte(text), &got) != nil || got.Counter < 0 || got.Counter >= 128 || !reflect.DeepEqual(got.List, []int{1}) || !reflect.DeepEqual(got.Set, []int{1, 2}) { + t.Errorf("concurrent ToJSON = (%q, %v)", text, err) + return + } + } + }() + if withWriter { + workers.Add(1) + go func() { + defer workers.Done() + <-start + for i := 0; i < 128; i++ { + if err := sd.SetKey(starlark.String("counter"), starlark.MakeInt(i)); err != nil { + t.Error(err) + return + } + } + }() + } + close(start) + workers.Wait() + counter := 0 + if withWriter { + counter = 127 + } + wantText := fmt.Sprintf(`shared_dict({"counter": %d, "list": [1], "set": set([1, 2])})`, counter) + if got := sd.String(); got != wantText { + t.Errorf("uncontended String = %q, want %q", got, wantText) + } + wantJSON := fmt.Sprintf(`{"counter":%d,"list":[1],"set":[1,2]}`, counter) + if got, err := sd.ToJSON(); got != wantJSON || err != nil { + t.Errorf("uncontended ToJSON = (%q,%v), want %q", got, err, wantJSON) + } + }) + } +} + +func TestSharedDict_ConcurrentTypeName(t *testing.T) { + sd := NewSharedDict() + start := make(chan struct{}) + var workers sync.WaitGroup + workers.Add(2) + go func() { + defer workers.Done() + <-start + for i := 0; i < 128; i++ { + sd.SetTypeName([]string{"left", "right", ""}[i%3]) + } + }() + go func() { + defer workers.Done() + <-start + for i := 0; i < 128; i++ { + if name := sd.Type(); name != "left" && name != "right" && name != "shared_dict" { + t.Errorf("concurrent Type = %q", name) + } + if text := sd.String(); text != "left({})" && text != "right({})" && text != "shared_dict({})" && text != sharedDictBusyString { + t.Errorf("concurrent named String = %q", text) + } + } + }() + close(start) + workers.Wait() +} + +func TestSharedDict_ConcurrentNestedOutput(t *testing.T) { + a, b := NewNamedSharedDict("a"), NewNamedSharedDict("b") + for _, pair := range []struct{ owner, child *SharedDict }{{a, b}, {b, a}} { + if err := pair.owner.SetKey(starlark.String("items"), starlark.NewList([]starlark.Value{pair.child})); err != nil { + t.Fatal(err) + } + } + start := make(chan struct{}) + var workers sync.WaitGroup + for _, dict := range []*SharedDict{a, b} { + dict := dict + workers.Add(1) + go func() { + defer workers.Done() + <-start + for i := 0; i < 128; i++ { + if text := dict.String(); text == "" { + t.Error("empty opposite-root output") + return + } + } + }() + } + workers.Add(1) + go func() { + defer workers.Done() + <-start + for i := 0; i < 128; i++ { + if err := b.SetKey(starlark.String("counter"), starlark.MakeInt(i)); err != nil { + t.Error(err) + return + } + } + }() + close(start) + workers.Wait() + if got, found, err := b.Get(starlark.String("counter")); err != nil || !found || got != starlark.MakeInt(127) { + t.Fatalf("child writer result = (%v,%v,%v)", got, found, err) + } + if got := a.String(); got != `a({"items": [b({"items": [a({...})], "counter": 127})]})` { + t.Errorf("cycle after concurrent output = %q", got) + } +} + +func TestSharedDict_NestedOutputLocks(t *testing.T) { + root, child := NewNamedSharedDict("root"), NewNamedSharedDict("child") + if err := child.SetKey(starlark.String("n"), starlark.MakeInt(1)); err != nil { + t.Fatal(err) + } + if err := root.SetKey(starlark.String("items"), starlark.NewList([]starlark.Value{child, child})); err != nil { + t.Fatal(err) + } + child.Lock() + got := root.String() + child.Unlock() + if got != `root({"items": [shared_dict(), shared_dict()]})` { + t.Errorf("locked child = %q", got) + } + panicValue := &sharedDictFormatValue{stringFunc: func() string { panic("format panic") }} + if err := child.SetKey(starlark.String("panic"), panicValue); err != nil { + t.Fatal(err) + } + func() { + defer func() { + if recover() == nil { + t.Error("expected nested formatting panic") + } + }() + root.SnapshotString() + }() + if err := child.SetKey(starlark.String("panic"), starlark.None); err != nil { + t.Fatal(err) + } + for _, dict := range []*SharedDict{root, child} { + if got := dict.String(); got == sharedDictBusyString || strings.Contains(got, "") { + t.Errorf("output lock leaked after panic: %q", got) + } + } +} + +type sharedDictRejectedJSONValue struct{ calls *int } + +func (v *sharedDictRejectedJSONValue) called() { (*v.calls)++; panic("JSON invoked host code") } +func (v *sharedDictRejectedJSONValue) String() string { v.called(); return "host" } +func (v *sharedDictRejectedJSONValue) Type() string { v.called(); return "host" } +func (v *sharedDictRejectedJSONValue) Freeze() { v.called() } +func (*sharedDictRejectedJSONValue) Truth() starlark.Bool { return true } +func (*sharedDictRejectedJSONValue) Hash() (uint32, error) { return 1, nil } +func (v *sharedDictRejectedJSONValue) MarshalJSON() ([]byte, error) { v.called(); return nil, nil } +func (v *sharedDictRejectedJSONValue) AttrNames() []string { v.called(); return nil } +func (v *sharedDictRejectedJSONValue) Attr(string) (starlark.Value, error) { + v.called() + return nil, nil +} + +func TestSharedDict_JSONMethodThread(t *testing.T) { + sd := NewSharedDict() + method, err := sd.Attr("to_json") + if err != nil { + t.Fatal(err) + } + method.Freeze() + if err := sd.SetKey(starlark.String("n"), starlark.MakeInt(1)); err != nil { + t.Fatalf("freezing method froze receiver: %v", err) + } + thread := &starlark.Thread{Name: "script-caller"} + marker := new(int) + thread.SetLocal("json-marker", marker) + old := starJSONEncode + defer func() { starJSONEncode = old }() + called := false + starJSONEncode = starlark.NewBuiltin("encode", func(got *starlark.Thread, _ *starlark.Builtin, _ starlark.Tuple, _ []starlark.Tuple) (starlark.Value, error) { + called = got == thread && got.Local("json-marker") == marker + return starlark.String(`{"n":1}`), nil + }) + if _, err := starlark.Call(thread, method, nil, nil); err != nil || !called { + t.Fatalf("JSON lost caller thread: called=%v, err=%v", called, err) + } + starJSONEncode = old + if err := sd.SetKey(starlark.String("bad"), starlark.Float(math.NaN())); err != nil { + t.Fatal(err) + } + _, err = starlark.ExecFile(thread, "json-source.star", "def encode_here():\n return sd.to_json()\nencode_here()\n", starlark.StringDict{"sd": sd}) + if eval, ok := err.(*starlark.EvalError); !ok || !strings.Contains(eval.Backtrace(), "json-source.star") || !strings.Contains(eval.Backtrace(), "encode_here") { + t.Fatalf("JSON error source lost: %v", err) + } +} + +func TestSharedDict_AttributeNames(t *testing.T) { + sd := new(SharedDict) + want := starlark.NewDict(0).AttrNames() + for name := range customSharedDictMethods { + want = append(want, name) + } + want = append(want, "to_json") + sort.Strings(want) + if got := sd.AttrNames(); !reflect.DeepEqual(got, want) { + t.Fatalf("zero AttrNames = %v, want %v", got, want) + } + names := sd.AttrNames() + names[0] = "changed" + if reflect.DeepEqual(names, sd.AttrNames()) { + t.Fatal("AttrNames returned shared mutable list") + } + for _, frozen := range []bool{false, true} { + sd := new(SharedDict) + if frozen { + sd.Freeze() + } + method, err := sd.Attr("to_json") + if err != nil { + t.Fatal(err) + } + if got, err := starlark.Call(&starlark.Thread{}, method, nil, nil); err != nil || got != starlark.String("{}") { + t.Fatalf("zero JSON = (%v,%v)", got, err) + } + update, err := sd.Attr("update") + if err != nil { + t.Fatal(err) + } + arg := starlark.NewDict(1) + if err := arg.SetKey(starlark.String("n"), starlark.MakeInt(1)); err != nil { + t.Fatal(err) + } + _, err = starlark.Call(&starlark.Thread{}, update, starlark.Tuple{arg}, nil) + if (err != nil) != frozen { + t.Fatalf("zero initialization lost frozen=%v: %v", frozen, err) + } + } +} + +func TestSharedDict_ToJSONAdmission(t *testing.T) { + calls := 0 + host := &sharedDictRejectedJSONValue{calls: &calls} + var list *starlark.List + var dict *starlark.Dict + var set *starlark.Set + var record *starlarkstruct.Struct + var module *starlarkstruct.Module + longTuple := starlark.Tuple{starlark.MakeInt(1), host} + badKey := starlark.NewDict(1) + if err := badKey.SetKey(host, starlark.MakeInt(1)); err != nil { + t.Fatal(err) + } + for _, value := range []starlark.Value{host, list, dict, set, record, module, nil, badKey, starlarkstruct.FromStringDict(starlarkstruct.Default, starlark.StringDict{"nil": nil}), &starlarkstruct.Module{Name: "nil_member", Members: starlark.StringDict{"nil": nil}}, starlarkstruct.FromStringDict(starlarkstruct.Default, starlark.StringDict{"bad": host}), &starlarkstruct.Module{Name: "host_member", Members: starlark.StringDict{"bad": host}}, starlark.Bytes("x"), starlark.Universe["len"], starlark.NewList([]starlark.Value{longTuple, longTuple[:1]})} { + sd := NewSharedDict() + if err := sd.SetKey(starlark.String("value"), value); err != nil { + t.Fatal(err) + } + if text, err := sd.ToJSON(); text != "" || err == nil || errors.Is(err, ErrSharedDictBusy) { + t.Errorf("unsupported %T produced (%q, %v)", value, text, err) + } + if calls != 0 { + t.Fatal("JSON invoked a host callback") + } + if err := sd.SetKey(starlark.String("value"), starlark.MakeInt(1)); err != nil { + t.Fatal(err) + } + if text, err := sd.ToJSON(); text != `{"value":1}` || err != nil { + t.Errorf("JSON locks leaked after rejection: (%q, %v)", text, err) + } + } + globals, err := starlark.ExecFile(&starlark.Thread{}, "json-values.star", "values = [range(3), 'abc'.elems(), 'abc'.elem_ords(), '雪'.codepoints(), '雪'.codepoint_ords(), b'abc'.elems()]", nil) + if err != nil { + t.Fatal(err) + } + aliasSet := starlark.NewSet(1) + if err := aliasSet.Insert(starlark.MakeInt(1)); err != nil { + t.Fatal(err) + } + aliasList := starlark.NewList([]starlark.Value{starlark.MakeInt(2)}) + aliasRecord := starlarkstruct.FromStringDict(starlarkstruct.Default, starlark.StringDict{"n": starlark.MakeInt(1)}) + aliasModule := &starlarkstruct.Module{Name: "alias", Members: starlark.StringDict{"n": starlark.MakeInt(1)}} + aliases := starlark.Tuple{aliasList, aliasList, aliasSet, aliasSet, aliasRecord, aliasRecord, aliasModule, aliasModule} + values := []starlark.Value{aliases, globals["values"], starlarktime.Duration(time.Second), starlarktime.Time(time.Date(2024, 1, 2, 3, 4, 5, 0, time.UTC)), &starlarkstruct.Struct{}, starlarkstruct.FromStringDict(host, starlark.StringDict{"n": starlark.MakeInt(1)}), &starlarkstruct.Module{Name: "module", Members: starlark.StringDict{"n": starlark.MakeInt(1)}}} + for _, value := range values { + sd := NewSharedDict() + if err := sd.SetKey(starlark.String("value"), value); err != nil { + t.Fatal(err) + } + want, wantErr := EncodeStarlarkJSON(sd.dict) + got, gotErr := sd.ToJSON() + if got != want || fmt.Sprint(gotErr) != fmt.Sprint(wantErr) { + t.Errorf("native %T: got (%q,%v), want (%q,%v)", value, got, gotErr, want, wantErr) + } + } + if calls != 0 { + t.Fatal("native struct JSON invoked its constructor") + } + cycle := make(starlark.Tuple, 1) + cycle[0] = cycle + sd := NewSharedDict() + if err := sd.SetKey(starlark.String("cycle"), cycle); err != nil { + t.Fatal(err) + } + if text, err := sd.ToJSON(); text != "" || err == nil || !strings.Contains(err.Error(), "cycle in JSON structure") { + t.Errorf("tuple cycle = (%q, %v)", text, err) + } +} + +// An application mutex is still needed for compound updates to native aliases; +// output locks do not make the two assignments one atomic application operation. func TestSharedDict_ExternalSynchronization(t *testing.T) { sd := NewSharedDict() key := starlark.String("generation") @@ -384,6 +813,29 @@ func TestSharedDict_SnapshotStringFormattingGraph(t *testing.T) { } else { debug.SetMaxStack(128 << 10) } + t.Run("json_metadata_held_lock", func(t *testing.T) { + sd := NewSharedDict() + sd.Lock() + text, err := EncodeStarlarkJSON(sd) + sd.Unlock() + if text != "" || err == nil { + t.Fatalf("generic JSON while locked = (%q,%v)", text, err) + } + }) + t.Run("json_method_freeze", func(t *testing.T) { + sd := NewSharedDict() + method, err := sd.Attr("to_json") + if err != nil { + t.Fatal(err) + } + if err := sd.SetKey(starlark.String("method"), method); err != nil { + t.Fatal(err) + } + sd.Freeze() + if err := sd.SetKey(starlark.String("n"), starlark.MakeInt(1)); err == nil { + t.Fatal("Freeze did not freeze mapping") + } + }) put := func(t *testing.T, dict starlark.HasSetKey, key string, value starlark.Value) { t.Helper() if err := dict.SetKey(starlark.String(key), value); err != nil { @@ -429,9 +881,11 @@ func TestSharedDict_SnapshotStringFormattingGraph(t *testing.T) { check(t, a, `a({"items": [b({"items": (c({"items": {"back": a({...})}}),)})]})`) }) t.Run("aliases_and_empty", func(t *testing.T) { + zero := new(SharedDict) + zero.SetTypeName("zero") root, child := NewNamedSharedDict("root"), NewNamedSharedDict("child") put(t, child, "n", starlark.MakeInt(1)) - put(t, root, "items", starlark.NewList([]starlark.Value{child, child, NewNamedSharedDict("empty"), &SharedDict{name: "zero"}})) + put(t, root, "items", starlark.NewList([]starlark.Value{child, child, NewNamedSharedDict("empty"), zero})) check(t, root, `root({"items": [child({"n": 1}), child({"n": 1}), empty({}), zero()]})`) }) }) @@ -463,7 +917,11 @@ func TestSharedDict_SnapshotStringFormattingGraph(t *testing.T) { put(t, root, "next", starlark.NewList([]starlark.Value{child})) put(t, child, "next", starlark.NewList([]starlark.Value{root})) child.Lock() - defer child.Unlock() + got := root.SnapshotString() + child.Unlock() + if want := `root({"next": [shared_dict()]})`; got != want { + t.Fatalf("locked child snapshot = %q; want %q", got, want) + } check(t, root, `root({"next": [child({"next": [root({...})]})]})`) }) t.Run("callbacks", func(t *testing.T) { @@ -525,7 +983,7 @@ func TestSharedDict_SnapshotStringFormattingGraph(t *testing.T) { go func() { defer workers.Done() for i := 0; i < 128; i++ { - if got := root.SnapshotString(); !strings.HasPrefix(got, `root({"items": [child({"n": 1})], "counter": `) || !strings.HasSuffix(got, "})") { + if got := root.SnapshotString(); got != sharedDictBusyString && (!strings.HasPrefix(got, `root({"items": [child({"n": 1})], "counter": `) || !strings.HasSuffix(got, "})")) { t.Errorf("malformed snapshot: %q", got) return } @@ -541,19 +999,22 @@ func TestSharedDict_SnapshotStringFormattingGraph(t *testing.T) { for _, test := range []struct { dict *SharedDict want string - }{{a, `a({"next": [b({"next": [a({...})]})]})`}, {b, `b({"next": [a({"next": [b({...})]})]})`}} { + busy string + }{{a, `a({"next": [b({"next": [a({...})]})]})`, `a({"next": [shared_dict()]})`}, {b, `b({"next": [a({"next": [b({...})]})]})`, `b({"next": [shared_dict()]})`}} { workers.Add(1) - go func(dict *SharedDict, want string) { + go func(dict *SharedDict, want, busy string) { defer workers.Done() for i := 0; i < 128; i++ { - if got := dict.SnapshotString(); got != want { + if got := dict.SnapshotString(); got != want && got != busy && got != sharedDictBusyString { t.Errorf("SnapshotString() = %q; want %q", got, want) return } } - }(test.dict, test.want) + }(test.dict, test.want, test.busy) } workers.Wait() + check(t, a, `a({"next": [b({"next": [a({...})]})]})`) + check(t, b, `b({"next": [a({"next": [b({...})]})]})`) }) t.Run("struct", func(t *testing.T) { t.Run("list_cycle", func(t *testing.T) { @@ -773,36 +1234,62 @@ func (v *sharedDictFormatValue) CompareSameType(op syntax.Token, other starlark. return equal == (op == syntax.EQL), nil } -func TestSharedDict_StringLegacyReentry(t *testing.T) { - for _, test := range []string{"external_lock", "Hash", "Equal", "Freeze"} { +func TestSharedDict_StringBusyReentry(t *testing.T) { + for _, test := range []string{"external_lock", "external_read_lock", "Hash", "Equal", "Freeze"} { t.Run(test, func(t *testing.T) { sd := NewNamedSharedDict("named") - var pending []chan string + type result struct { + text string + err error + } + var pending []chan result checks := 0 - checkString := func(want string) { + checkOutput := func() { checks++ - done := make(chan string, 1) - go func() { done <- sd.String() }() - select { - case got := <-done: - if got != want { - t.Errorf("String while locked = %q; want %q", got, want) + for _, output := range []struct { + name string + call func() (string, error) + json bool + }{{"String", func() (string, error) { return sd.String(), nil }, false}, {"SnapshotString", func() (string, error) { return sd.SnapshotString(), nil }, false}, {"ToJSON", sd.ToJSON, true}} { + done := make(chan result, 1) + call := output.call + go func() { text, err := call(); done <- result{text, err} }() + select { + case got := <-done: + if output.json { + if got.text != "" || !errors.Is(got.err, ErrSharedDictBusy) { + t.Errorf("locked %s = (%q,%v)", output.name, got.text, got.err) + } + } else if got.text != sharedDictBusyString || got.err != nil { + t.Errorf("locked %s = (%q,%v)", output.name, got.text, got.err) + } + case <-time.After(2 * time.Second): + t.Errorf("%s waited for the mapping lock", output.name) + pending = append(pending, done) } - case <-time.After(2 * time.Second): - t.Error("legacy String unexpectedly waited for the mapping lock") - // Return from the callback so the operation unlocks, then - // drain the worker below even if a regression blocks it. - pending = append(pending, done) } } key := &sharedDictFormatValue{stringFunc: func() string { return "key" }} switch test { case "external_lock": sd.Lock() - checkString("named({})") + checkOutput() sd.Unlock() + case "external_read_lock": + key.stringFunc = func() string { + if err := sd.SetKey(starlark.String("new"), starlark.None); err != nil { + t.Error(err) + } + return "key" + } + if err := sd.SetKey(starlark.String("value"), key); err != nil { + t.Fatal(err) + } + sd.RLock() + checkOutput() + sd.RUnlock() case "Hash": - key.hashFunc = func() (uint32, error) { checkString("named({})"); return 1, nil } + key.hashFunc = func() (uint32, error) { checkOutput(); return 1, nil } if err := sd.SetKey(key, starlark.True); err != nil { t.Fatal(err) } @@ -810,13 +1297,12 @@ func TestSharedDict_StringLegacyReentry(t *testing.T) { if err := sd.SetKey(key, starlark.True); err != nil { t.Fatal(err) } - other := &sharedDictFormatValue{stringFunc: func() string { return "other" }} - other.compareFunc = func() { checkString("named({key: True})") } + other := &sharedDictFormatValue{stringFunc: func() string { return "other" }, compareFunc: checkOutput} if err := sd.SetKey(other, starlark.False); err != nil { t.Fatal(err) } case "Freeze": - key.freezeFunc = func() { checkString(`named({"value": key})`) } + key.freezeFunc = checkOutput if err := sd.SetKey(starlark.String("value"), key); err != nil { t.Fatal(err) } @@ -826,11 +1312,11 @@ func TestSharedDict_StringLegacyReentry(t *testing.T) { select { case <-done: case <-time.After(2 * time.Second): - t.Fatal("formatting worker remained blocked after the operation unlocked") + t.Fatal("output worker remained blocked after unlocking") } } if checks != 1 { - t.Fatalf("String compatibility callback count = %d; want 1", checks) + t.Fatalf("callback count = %d; want 1", checks) } }) } @@ -1763,9 +2249,8 @@ func TestSharedDict_NilDict(t *testing.T) { name: `attr: len`, script: itn.HereDoc(` load('share', 'sd') - sd.len() + assert.eq(sd.len(), 0) `), - wantErr: `shared_dict has no .len field or method`, }, { name: `to json`, @@ -1773,13 +2258,12 @@ func TestSharedDict_NilDict(t *testing.T) { load('share', 'sd') assert.eq(sd.to_json(), '{}') `), - wantErr: `shared_dict has no .to_json field or method`, }, { name: `attr names`, script: itn.HereDoc(` load('share', 'sd') - assert.eq(dir(sd), []) + assert.eq(dir(sd), ['clear', 'from_json', 'get', 'items', 'keys', 'len', 'perform', 'pop', 'popitem', 'setdefault', 'to_dict', 'to_json', 'update', 'values']) `), }, { @@ -2166,3 +2650,168 @@ func TestSharedDict_LoadJSON(t *testing.T) { }) } } + +// Shared by the baseline validation copy and candidate. Native children remain +// mutable and owner-exclusive during these single-threaded success-path runs. +var ( + sharedDictOutputSafetyStringSink string + sharedDictOutputSafetyDictSink *SharedDict + sharedDictOutputSafetyValueSink starlark.Value +) + +func BenchmarkSharedDictOutputSafety(b *testing.B) { + put := func(dict *SharedDict, key string, value starlark.Value) { + if err := dict.SetKey(starlark.String(key), value); err != nil { + b.Fatal(err) + } + } + flat := func(size int) (*SharedDict, string, string) { + dict := NewSharedDict() + text, encoded := make([]string, size), make([]string, size) + for i := 0; i < size; i++ { + key := fmt.Sprintf("k%d", i) + put(dict, key, starlark.MakeInt(i)) + text[i] = fmt.Sprintf("%q: %d", key, i) + encoded[i] = fmt.Sprintf("%q:%d", key, i) + } + return dict, "shared_dict({" + strings.Join(text, ", ") + "})", "{" + strings.Join(encoded, ",") + "}" + } + flat0, text0, _ := flat(0) + flat8, text8, json8 := flat(8) + flat64, text64, _ := flat(64) + values, numbers := make([]starlark.Value, 32), make([]string, 32) + set := starlark.NewSet(32) + for i := range values { + values[i] = starlark.MakeInt(i) + numbers[i] = fmt.Sprint(i) + if err := set.Insert(values[i]); err != nil { + b.Fatal(err) + } + } + list := starlark.NewList(values) + setDict, listDict := NewSharedDict(), NewSharedDict() + put(setDict, "items", set) + put(listDict, "items", list) + setText := "shared_dict({\"items\": set([" + strings.Join(numbers, ", ") + "])})" + sequenceJSON := "{\"items\":[" + strings.Join(numbers, ",") + "]}" + for _, test := range []struct { + name string + dict *SharedDict + want string + snapshot bool + json bool + }{ + {name: "StringFlat0", dict: flat0, want: text0}, + {name: "StringFlat8", dict: flat8, want: text8}, + {name: "StringFlat64", dict: flat64, want: text64}, + {name: "StringSet32", dict: setDict, want: setText}, + {name: "SnapshotSet32", dict: setDict, want: setText, snapshot: true}, + {name: "JSONFlat8", dict: flat8, want: json8, json: true}, + {name: "JSONNestedList32", dict: listDict, want: sequenceJSON, json: true}, + {name: "JSONSet32", dict: setDict, want: sequenceJSON, json: true}, + } { + test := test + b.Run(test.name, func(b *testing.B) { + b.ReportAllocs() + switch { + case test.json: + if got, err := test.dict.ToJSON(); err != nil || got != test.want { + b.Fatalf("ToJSON() = %q, %v; want %q, nil", got, err, test.want) + } + b.ResetTimer() + for i := 0; i < b.N; i++ { + got, err := test.dict.ToJSON() + if err != nil { + b.Fatal(err) + } + sharedDictOutputSafetyStringSink = got + } + case test.snapshot: + if got := test.dict.SnapshotString(); got != test.want { + b.Fatalf("SnapshotString() = %q; want %q", got, test.want) + } + b.ResetTimer() + for i := 0; i < b.N; i++ { + sharedDictOutputSafetyStringSink = test.dict.SnapshotString() + } + default: + if got := test.dict.String(); got != test.want { + b.Fatalf("String() = %q; want %q", got, test.want) + } + b.ResetTimer() + for i := 0; i < b.N; i++ { + sharedDictOutputSafetyStringSink = test.dict.String() + } + } + }) + } + b.Run("NewShared", func(b *testing.B) { + if dict := NewSharedDict(); dict.Len() != 0 || dict.Type() != "shared_dict" { + b.Fatal("unexpected default constructor result") + } + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + sharedDictOutputSafetyDictSink = NewSharedDict() + } + }) + b.Run("NewNamed", func(b *testing.B) { + if dict := NewNamedSharedDict("bench"); dict.Len() != 0 || dict.Type() != "bench" { + b.Fatal("unexpected named constructor result") + } + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + sharedDictOutputSafetyDictSink = NewNamedSharedDict("bench") + } + }) + b.Run("SetTypeName", func(b *testing.B) { + dict := NewSharedDict() + names := [2]string{"bench_a", "bench_b"} + sharedDictOutputSafetyDictSink = dict + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + dict.SetTypeName(names[i&1]) + } + b.StopTimer() + if b.N > 0 && dict.Type() != names[(b.N-1)&1] { + b.Fatal("SetTypeName did not store the final name") + } + }) + b.Run("SetKeyExisting", func(b *testing.B) { + dict := NewSharedDict() + key := starlark.String("existing") + values := [2]starlark.Value{starlark.MakeInt(0), starlark.MakeInt(1)} + if err := dict.SetKey(key, values[0]); err != nil { + b.Fatal(err) + } + sharedDictOutputSafetyDictSink = dict + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + if err := dict.SetKey(key, values[i&1]); err != nil { + b.Fatal(err) + } + } + }) + b.Run("GetExisting", func(b *testing.B) { + dict := NewSharedDict() + key, want := starlark.String("existing"), starlark.MakeInt(1) + if err := dict.SetKey(key, want); err != nil { + b.Fatal(err) + } + if got, found, err := dict.Get(key); err != nil || !found || got != want { + b.Fatalf("Get() = %v, %v, %v; want %v, true, nil", got, found, err, want) + } + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + got, found, err := dict.Get(key) + if err != nil || !found { + b.Fatalf("Get() found=%v, err=%v", found, err) + } + sharedDictOutputSafetyValueSink = got + } + }) +} diff --git a/lib/goidiomatic/README.md b/lib/goidiomatic/README.md index 82b0cc2..2936ea2 100644 --- a/lib/goidiomatic/README.md +++ b/lib/goidiomatic/README.md @@ -25,7 +25,7 @@ Load name for `load()`: `go_idiomatic`. | `module(name, **kv) -> module` | Build a `starlarkstruct` module named `name`. | | `struct(**kv) -> struct` | Build an anonymous comparable struct. | | `make_struct(name, **kv) -> struct` | Build a struct whose constructor name is `name`. | -| `shared_dict() -> shared_dict` | Create an empty thread-safe shared dictionary. | +| `shared_dict() -> shared_dict` | Create an empty dictionary with synchronized top-level access. | | `make_shared_dict(name="", data=None) -> shared_dict` | Create a shared dictionary with an optional type name and initial data. | | `to_dict(v) -> value` | Convert a dict, `module`, `struct`, `GoStruct`, or `shared_dict`; usually returns a dict. | | `eprint(*args, sep=" ")` | `print`-style output to stderr. | @@ -230,16 +230,16 @@ provide the workloads, sample results and reproduction commands. ### `shared_dict` -A mutable dictionary returned by `shared_dict()` and `make_shared_dict()`. Indexing and dictionary methods lock the top-level mapping; this does not protect mutable nested values or make every operation safe during concurrent writes. In particular, format/print the dictionary only while writers are stopped or externally synchronized. Its `type()` is `shared_dict` by default, or the custom name passed to `make_shared_dict`. It supports indexing (`sd["k"] = v`, `sd["k"]`), membership (`in`), the `.len()` method, and `==`/`!=` comparison with another `shared_dict`. It is **not** iterable, and the builtin `len()` does not apply — use the `.len()` method. +A mutable dictionary returned by `shared_dict()` and `make_shared_dict()`. Indexing and dictionary methods lock the top-level mapping. Default output takes an internal snapshot or returns `shared_dict()`; JSON output reports an error when it cannot safely proceed. Mutable native aliases still require their own ownership or synchronization. Its `type()` is `shared_dict` by default, or the custom name passed to `make_shared_dict`. It supports indexing (`sd["k"] = v`, `sd["k"]`), membership (`in`), the `.len()` method, and `==`/`!=` comparison with another `shared_dict`. It is **not** iterable, and the builtin `len()` does not apply — use the `.len()` method. Beyond the standard dict methods it inherits (`clear`, `get`, `items`, `keys`, `pop`, `popitem`, `setdefault`, `update`, `values`), it adds the methods below. | method | description | |--------|-------------| | `len() -> int` | Number of items in the dictionary. | -| `perform(fn)` | Call `fn(dict)` with the underlying plain dict while holding the lock, for atomic compound updates. Do not retain that dict or reenter this `shared_dict` from the callback. | +| `perform(fn)` | Call `fn(dict)` with the underlying plain dict while holding the lock, for atomic compound updates. Do not retain that dict or call locking methods on the owning `shared_dict` from the callback; output calls report busy rather than waiting. | | `to_dict() -> dict` | Copy top-level entries into a plain dict. Inserting/removing clone entries is independent; nested values remain shared. | -| `to_json() -> str` | Serialize the contents to a JSON string. | +| `to_json() -> str` | Serialize supported native data; a busy lock or unsupported host value raises an error. | | `from_json(json_str)` | Decode a JSON object string and merge its pairs into the dictionary. | ```python @@ -283,7 +283,7 @@ print(sd.to_json()) # Output: {"new_key":"new_value"} ``` -For Go `String`/`ToJSON` and shared nested values, use one owner or a common application mutex for every access, including all writers. See the [shared-value synchronization example and callback limits](../../USAGE.md#shared-values). +Default output captures entries and returns `shared_dict()` instead of waiting for an occupied output or mapping lock. `to_json()` uses the native-data-only Go `ToJSON` path and reports a busy or unsupported-value error without returning partial JSON. Raw nested aliases still require one owner or common synchronization. See the [shared-value output and ownership rules](../../USAGE.md#shared-values). ## Details & examples