Repository navigation
Conversation
|
✅ No conflicts with other open PRs targeting |
|
I see you updated files related to
|
|
7ff9593 to
01b7262
Compare
01b7262 to
8e88824
Compare
|
|
||
| return cmp.Or(cmp.Compare(a.Deadline.Seconds, b.Deadline.Seconds), | ||
| cmp.Compare(a.Deadline.Nanos, b.Deadline.Nanos), | ||
| cmp.Compare(a.TriggerEventID, b.TriggerEventID)) < 0 |
There was a problem hiding this comment.
How are trigger event IDs derived? Perhaps we should use blobHash here instead?
| WorkflowCapacities: make(map[string]uint32), | ||
| } | ||
| for wf, wfCap := range capacity.PerWorkflow { | ||
| //TODO only for events proposed and queued in the KV store? |
There was a problem hiding this comment.
@MStreet3 Was the intent to filter this capacity map down to only what we need?
| obsEvent.ObservedAts = append(obsEvent.ObservedAts, eventToAdd.ObservedAts[i]) | ||
| obsEvent.Deadlines = append(obsEvent.Deadlines, eventToAdd.Deadlines[i]) | ||
| obsEvent.Blobs = append(obsEvent.Blobs, eventToAdd.Blobs[i]) | ||
| } //TODO what if a node reports twice with different deadline or nodeID? do we need to overwrite? |
There was a problem hiding this comment.
@MStreet3 @cedric-cordenier WDYT about this? Should we allow nodes to update their observations? I think the only valid change is to the deadline field, and we don't currently do that (though we could in order to navigate split reconfiguration more smoothly).
| cfg: cfg, | ||
| triggerCoordinator: triggerCoordinator, | ||
| expirationDur: func(ctx context.Context) (time.Duration, error) { | ||
| //TODO also allow this grace period in the trigger coordinator |
There was a problem hiding this comment.
I noticed the deadline is being enforced as a hard cutoff in other places. I think we will have to relax that to respect this grace period too. The grace period is important to allow jumping the line with a very near deadline (as soon as "now"), without increasing the risk of being dropped.
| const roundDur = 10 * time.Second //TODO dynamic calibration? | ||
| expSeqNr := seqNr + uint64(expirationGraceDur/roundDur) //TODO pad extra time and/or rounds |
There was a problem hiding this comment.
@cedric-cordenier Is there precedent for how to calculate safe expiration sequence numbers?
CORA - Pending Reviewers
Legend: ✅ Approved | ❌ Changes Requested | 💬 Commented | 🚫 Dismissed | ⏳ Pending | ❓ Unknown For more details, see the full review summary. |
6663160 to
b0d2926
Compare
nolag
left a comment
There was a problem hiding this comment.
Reviewed one pass up to and including queue.go. I need to drop for a bit so I'll publish what I have so far.
| cache map[string]func() ([]byte, error) // blob handle -> blob | ||
| } | ||
|
|
||
| func (c *blobCache) fetchBlob(ctx context.Context, handle []byte) ([]byte, error) { |
There was a problem hiding this comment.
Doesn't the BlobFetcher already keep everything locally once it's fetched?
There was a problem hiding this comment.
I tried to look for an implementation to reassure me that it would be cached efficiently, but IIUC we would at least be paying extra synchronization and allocation overhead.
There was a problem hiding this comment.
libocr caches on their end but you'd still be paying the network cost to fetch the blob from cache so the cache here is helpful
| return coordEvent, b, nil | ||
| } | ||
|
|
||
| const coordinatedEventsFieldNumber = 1 // pb.CoordinatedEventList.Events |
There was a problem hiding this comment.
Is it better to use something like this to get the field number?
ageFieldNumber := (&person.Person{}).
ProtoReflect().
Descriptor().
Fields().
ByName("age").
Number()
There was a problem hiding this comment.
I have a test set up to ensure it doesn't drift. I suppose we could derive it directly, though we have to give up having a const
| num, typ, n := protowire.ConsumeTag(b) | ||
| if n < 0 { | ||
| return nil, protowire.ParseError(n) | ||
| } |
There was a problem hiding this comment.
Whitespace after the closing } (when it's not followed by another }) would make this a lot easier to read.
This issues is in many functions in the PR
| // Assemble an observation up to maxObservationBytes | ||
| var coordEvents []*pb.CoordinatedEvent | ||
| for coord := range toProtoSeq(c.triggerQueue.Sorted()) { | ||
| observedAt, deadline := coord.ObservedAt, coord.Deadline |
There was a problem hiding this comment.
is this considered best-practice in go? I've been reverting when Claude puts more than one assignment on a line unless it's a variable swap.
There was a problem hiding this comment.
I'm not sure that I've ever seen a formal rule about it, but when it is short (and symmetric like this one), it doesn't seem any harder to read than a multi-value return to me 🤷
| var addedCap bool | ||
| if _, ok := obs.WorkflowCapacities[coord.WorkflowID]; !ok { | ||
| addedCap = true | ||
| obs.WorkflowCapacities[coord.WorkflowID] = uint32(min(capacity.PerWorkflow[coord.WorkflowID], math.MaxUint32)) |
There was a problem hiding this comment.
Is there a realistic world where we need to worry about capacity being higher than max uint 32?
There was a problem hiding this comment.
No, just being defensive since it is an int. If we could align types instead, that would be ideal.
| @@ -58,45 +113,448 @@ func (c *consensusQueuePlugin) Query(_ context.Context, _ uint64, _ ocr3_1types. | |||
| } | |||
|
|
|||
| func (c *consensusQueuePlugin) Observation( | |||
There was a problem hiding this comment.
This would be a lot easier to read if you split it to smaller functions. I think I groked it all, but it was a lot to hold in my head.
| } | ||
|
|
||
| // update state | ||
| state.pruneExpiredObservations(obs.medianExpiration) |
There was a problem hiding this comment.
Note for me for later: It sucks that we need to do this in observations, it kills capacity when it wouldn't necessarily need to if the protocol handled it.
| // Medianize capacities, and when there is disagreement among honest nodes, scale back to 90%. | ||
| var medianCount int | ||
| res.medianCap, medianCount = median(capacities) | ||
| if medianCount < 2*f+1 { | ||
| res.medianCap = (res.medianCap * 9) / 10 | ||
| } | ||
| for wf, caps := range wfCapacities { | ||
| res.medianCapPerWF[wf], medianCount = median(caps) | ||
| if medianCount < 2*f+1 { | ||
| res.medianCapPerWF[wf] = (res.medianCapPerWF[wf] * 9) / 10 | ||
| } | ||
| } |
There was a problem hiding this comment.
I don’t think we can expect the median to have any agreement. It’s a snapshot of time, and WASMs won’t always take the same time to run on each box. Node mode is one example, you can retry a fetch and take a lot longer to run. Even just jitter on the box can cause a WASM to take a bit longer to start or run, then if you're busy that can compound.
Even a simple case of everyone running exactly the same with a snapshot being made in a race gives us 50% for each. For easy math, say we had 80 by half the nodes and 90 by the other half because we can run 10 workflows. With an even number of nodes reporting, it returns 85 and 0 agree, you take 72 as the value.
For capacity, I would take the minimum known honest value at F+1th lowest. Validate observation can ensure you have a capacity, as observation fails if you get an error from the capacity function so you know you have at QuorumTwoFPlusOne here. Even if you don't validate it, at most F nodes would be missing it and defaulting to zero so the first honest node would be used.
There was a problem hiding this comment.
Do we have a term for the F+1th lowest?
There was a problem hiding this comment.
I'm tempted to call it leastHonest(capacities)? 😆
There was a problem hiding this comment.
Updated but lmk if there is a preferred name. I just did an inline Sorted call with an f index 🤷
| } | ||
|
|
||
| // Check size | ||
| if protoOpts.Size(obs) > maxObservationBytes { |
There was a problem hiding this comment.
This checks if adding the event goes over maxObservationBytes but we have blobHandle placeholders in the obs now. If broadcast as blobs breaks it up into more than the placeholders we'll go over the maxObservation size and the observation will fail
There was a problem hiding this comment.
It looks like we have a maxObservedBlobHandles variable already, but we're not checking it in broadcastAsBlob
| maxObservedBlobHandles = 10 // sanity check on blobs in an observation | ||
| maxObservedEvents = 1000 // sanity check events in an observation | ||
|
|
||
| //TODO reduce these? connect to cre settings? |
There was a problem hiding this comment.
These should be connected to cre settings instead of using libocr's maxmax values. I don't think we've tested those values except for the observation bytes size. Since cre settings is local changing these limits isn't atomic (including any feature flags we add). There's a PR to change how that works in the vault which we may want to copy over here: #23650
| if err != nil { | ||
| return nil, -1, fmt.Errorf("failed to marshal blob: %w", err) | ||
| } | ||
| bh, err := broadcaster.BroadcastBlob(ctx, data, ocr3_1types.BlobExpirationHintSequenceNumber{SeqNr: expSeq}) |
There was a problem hiding this comment.
Broadcasting blobs is somewhat slow (600-800ms on Vault prod). We can do this in parallel to speed it up but make sure you adhere to libocr's concurrency limit. Vault reference: https://github.com/smartcontractkit/chainlink/blob/develop/core/services/ocr2/plugins/vault/plugin.go#L909
| var obs observations | ||
|
|
||
| go func() { | ||
| defer close(done) |
There was a problem hiding this comment.
done is nil here so close will panic
There was a problem hiding this comment.
Why is this a go routine? To save time on loading the state?
There was a problem hiding this comment.
Yeah, though I inherited that decision from an earlier design, so I don't have evidence that is it necessarily important to be doing.
fe77b85 to
d4f5a8d
Compare
d4f5a8d to
60e3b4d
Compare
https://smartcontract-it.atlassian.net/browse/CRE-6002
Requires:
sequenceDiagram participant trigger@{ "type" : "boundary"} as Trigger participant local@{ "type" : "queue"} as Local<br>Queue actor Observation as Observation() participant observation@{ "type" : "collections"} as Observation actor StateTransition as StateTransition() box KVState participant observed@{ "type" : "collections"} as Observed participant queued@{ "type" : "queue"} as Consensus<br>Queue participant consensus@{ "type" : "collections"} as Consensus<br>Events end box BlobStore participant blobserved@{ "type" : "database" } as Observed end actor Commit as Commit() participant coord@{ "type" : "entity" } as Trigger<br>Coordinator alt trigger->>+local: Cron else trigger->>local: HTTP else trigger->>local: Log end rect Note right of local: Observation() par TryStartExecute consensus->>+Commit: Read() and blobserved->>Commit: FetchBlob() end Commit->>-coord: Deliver() local->>+Observation: Sorted() par Observation->>+observation: observe and Observation->>+blobserved: BroadcastBlob() deactivate Observation end end %trigger->coord: rect Note right of observation: StateTransition() observation->>+StateTransition: arg observed->>StateTransition: Read() activate observed queued->>StateTransition: Read() activate queued alt <f+1 StateTransition->>observed: Write() deactivate observed else f+1 StateTransition->>queued: Write() deactivate queued StateTransition->>local: Remove() deactivate local else capacity limited batch StateTransition->>-consensus: Write() activate consensus end end rect Note right of consensus: Commit() par TryStartExecute consensus->>-Commit: Read() activate Commit and blobserved->>-Commit: FetchBlob() end Commit->>-coord: Deliver() end