Skip to content

core/services/ocr3_1/plugins/queue: ReportingPlugin implementation - #24102

Draft
jmank88 wants to merge 14 commits into
developfrom
CRE-6002-reporting-plugin-impl
Draft

jmank88 wants to merge 14 commits into
developfrom
CRE-6002-reporting-plugin-impl

Conversation

@jmank88

@jmank88 jmank88 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

https://smartcontract-it.atlassian.net/browse/CRE-6002

Requires:

sequenceDiagram
    participant trigger@{ "type" : "boundary"} as Trigger
    participant local@{ "type" : "queue"} as Local<br>Queue

    actor Observation as Observation()

    participant observation@{ "type" : "collections"} as Observation

    actor StateTransition as StateTransition()

    box KVState 
        participant observed@{ "type" : "collections"} as Observed
        participant queued@{ "type" : "queue"} as Consensus<br>Queue
        participant consensus@{ "type" : "collections"} as Consensus<br>Events
    end

    box BlobStore
        participant blobserved@{ "type" : "database" } as Observed
    end

    actor Commit as Commit()

    participant coord@{ "type" : "entity" } as Trigger<br>Coordinator

    alt
        trigger->>+local: Cron
    else
        trigger->>local: HTTP
    else
        trigger->>local: Log
    end

    
    rect
        Note right of local: Observation()

        par TryStartExecute
            consensus->>+Commit: Read()
        and
            blobserved->>Commit: FetchBlob()
        end
        Commit->>-coord: Deliver()
        
        local->>+Observation: Sorted()
        par
            Observation->>+observation: observe
        and
            Observation->>+blobserved: BroadcastBlob()
            deactivate Observation
        end
    end

    %trigger->coord: 

    rect
        Note right of observation: StateTransition()
        
        observation->>+StateTransition: arg        
        observed->>StateTransition: Read()
        activate observed
        queued->>StateTransition: Read()
        activate queued

        alt <f+1
            StateTransition->>observed: Write()
            deactivate observed
        else f+1            
            StateTransition->>queued: Write()
            deactivate queued
			StateTransition->>local: Remove()
			deactivate local

        else capacity limited batch
            StateTransition->>-consensus: Write()
            activate consensus
        end
    end

    rect
        Note right of consensus: Commit()
        par TryStartExecute
            consensus->>-Commit: Read()
            activate Commit
        and 
            blobserved->>-Commit: FetchBlob()
        end
        Commit->>-coord: Deliver()
    end

Loading

@github-actions

Copy link
Copy Markdown
Contributor

✅ No conflicts with other open PRs targeting develop

@github-actions

Copy link
Copy Markdown
Contributor

I see you updated files related to core. Please run make gocs in the root directory to add a changeset as well as in the text include at least one of the following tags:

  • #added For any new functionality added.
  • #breaking_change For any functionality that requires manual action for the node to boot.
  • #bugfix For bug fixes.
  • #changed For any change to the existing functionality.
  • #db_update For any feature that introduces updates to database schema.
  • #deprecation_notice For any upcoming deprecation functionality.
  • #internal For changesets that need to be excluded from the final changelog.
  • #nops For any feature that is NOP facing and needs to be in the official Release Notes for the release.
  • #removed For any functionality/config that is removed.
  • #updated For any functionality that is updated.
  • #wip For any change that is not ready yet and external communication about it should be held off till it is feature complete.

@trunk-io

trunk-io Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Static Badge   Static Badge   Static Badge

Failed Test Failure Summary Logs
TestHttpTriggerHandler_CheckRateLimit_PerOrgOverride The per-organization rate-limit override test failed because the workflow was still rate-limited, preventing execution from starting. Logs ↗︎

View Full Report ↗︎ ⋅ Docs

@jmank88
jmank88 force-pushed the CRE-6002-reporting-plugin-impl branch from 7ff9593 to 01b7262 Compare October 30, 2026 16:54
@jmank88
jmank88 force-pushed the CRE-6002-reporting-plugin-impl branch from 01b7262 to 8e88824 Compare October 30, 2026 18:55

return cmp.Or(cmp.Compare(a.Deadline.Seconds, b.Deadline.Seconds),
cmp.Compare(a.Deadline.Nanos, b.Deadline.Nanos),
cmp.Compare(a.TriggerEventID, b.TriggerEventID)) < 0

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How are trigger event IDs derived? Perhaps we should use blobHash here instead?

WorkflowCapacities: make(map[string]uint32),
}
for wf, wfCap := range capacity.PerWorkflow {
//TODO only for events proposed and queued in the KV store?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MStreet3 Was the intent to filter this capacity map down to only what we need?

obsEvent.ObservedAts = append(obsEvent.ObservedAts, eventToAdd.ObservedAts[i])
obsEvent.Deadlines = append(obsEvent.Deadlines, eventToAdd.Deadlines[i])
obsEvent.Blobs = append(obsEvent.Blobs, eventToAdd.Blobs[i])
} //TODO what if a node reports twice with different deadline or nodeID? do we need to overwrite?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MStreet3 @cedric-cordenier WDYT about this? Should we allow nodes to update their observations? I think the only valid change is to the deadline field, and we don't currently do that (though we could in order to navigate split reconfiguration more smoothly).

cfg: cfg,
triggerCoordinator: triggerCoordinator,
expirationDur: func(ctx context.Context) (time.Duration, error) {
//TODO also allow this grace period in the trigger coordinator

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed the deadline is being enforced as a hard cutoff in other places. I think we will have to relax that to respect this grace period too. The grace period is important to allow jumping the line with a very near deadline (as soon as "now"), without increasing the risk of being dropped.

Comment on lines +128 to +129
const roundDur = 10 * time.Second //TODO dynamic calibration?
expSeqNr := seqNr + uint64(expirationGraceDur/roundDur) //TODO pad extra time and/or rounds

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cedric-cordenier Is there precedent for how to calculate safe expiration sequence numbers?

Base automatically changed from CRE-6002-consensus-queue-wiring to develop November 17, 2026 16:02
@github-actions

Copy link
Copy Markdown
Contributor

CORA - Pending Reviewers

Codeowners Entry Overall Num Files Owners
* 💬 3 @smartcontractkit/foundations, @smartcontractkit/core
/core/services/ocr*/ 💬 8 @smartcontractkit/foundations, @smartcontractkit/core
/core/services/workflows/ 💬 1 @smartcontractkit/keystone
/deployment/cre/ 💬 1 @smartcontractkit/keystone, @smartcontractkit/operations-platform
go.mod 💬 6 @smartcontractkit/core, @smartcontractkit/foundations
go.sum 💬 6 @smartcontractkit/core, @smartcontractkit/foundations
integration-tests/go.mod 💬 1 @smartcontractkit/core, @smartcontractkit/devex-tooling, @smartcontractkit/foundations
integration-tests/go.sum 💬 1 @smartcontractkit/core, @smartcontractkit/devex-tooling, @smartcontractkit/foundations

Legend: ✅ Approved | ❌ Changes Requested | 💬 Commented | 🚫 Dismissed | ⏳ Pending | ❓ Unknown

For more details, see the full review summary.

@jmank88
jmank88 force-pushed the CRE-6002-reporting-plugin-impl branch from 6663160 to b0d2926 Compare November 17, 2026 16:54

@nolag nolag left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed one pass up to and including queue.go. I need to drop for a bit so I'll publish what I have so far.

cache map[string]func() ([]byte, error) // blob handle -> blob
}

func (c *blobCache) fetchBlob(ctx context.Context, handle []byte) ([]byte, error) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't the BlobFetcher already keep everything locally once it's fetched?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to look for an implementation to reassure me that it would be cached efficiently, but IIUC we would at least be paying extra synchronization and allocation overhead.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

libocr caches on their end but you'd still be paying the network cost to fetch the blob from cache so the cache here is helpful

return coordEvent, b, nil
}

const coordinatedEventsFieldNumber = 1 // pb.CoordinatedEventList.Events

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it better to use something like this to get the field number?

ageFieldNumber := (&person.Person{}).
ProtoReflect().
Descriptor().
Fields().
ByName("age").
Number()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a test set up to ensure it doesn't drift. I suppose we could derive it directly, though we have to give up having a const

num, typ, n := protowire.ConsumeTag(b)
if n < 0 {
return nil, protowire.ParseError(n)
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whitespace after the closing } (when it's not followed by another }) would make this a lot easier to read.

This issues is in many functions in the PR

// Assemble an observation up to maxObservationBytes
var coordEvents []*pb.CoordinatedEvent
for coord := range toProtoSeq(c.triggerQueue.Sorted()) {
observedAt, deadline := coord.ObservedAt, coord.Deadline

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this considered best-practice in go? I've been reverting when Claude puts more than one assignment on a line unless it's a variable swap.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure that I've ever seen a formal rule about it, but when it is short (and symmetric like this one), it doesn't seem any harder to read than a multi-value return to me 🤷

var addedCap bool
if _, ok := obs.WorkflowCapacities[coord.WorkflowID]; !ok {
addedCap = true
obs.WorkflowCapacities[coord.WorkflowID] = uint32(min(capacity.PerWorkflow[coord.WorkflowID], math.MaxUint32))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a realistic world where we need to worry about capacity being higher than max uint 32?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, just being defensive since it is an int. If we could align types instead, that would be ideal.

@@ -58,45 +113,448 @@ func (c *consensusQueuePlugin) Query(_ context.Context, _ uint64, _ ocr3_1types.
}

func (c *consensusQueuePlugin) Observation(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be a lot easier to read if you split it to smaller functions. I think I groked it all, but it was a lot to hold in my head.

}

// update state
state.pruneExpiredObservations(obs.medianExpiration)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note for me for later: It sucks that we need to do this in observations, it kills capacity when it wouldn't necessarily need to if the protocol handled it.

Comment on lines +440 to +451
// Medianize capacities, and when there is disagreement among honest nodes, scale back to 90%.
var medianCount int
res.medianCap, medianCount = median(capacities)
if medianCount < 2*f+1 {
res.medianCap = (res.medianCap * 9) / 10
}
for wf, caps := range wfCapacities {
res.medianCapPerWF[wf], medianCount = median(caps)
if medianCount < 2*f+1 {
res.medianCapPerWF[wf] = (res.medianCapPerWF[wf] * 9) / 10
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t think we can expect the median to have any agreement. It’s a snapshot of time, and WASMs won’t always take the same time to run on each box. Node mode is one example, you can retry a fetch and take a lot longer to run. Even just jitter on the box can cause a WASM to take a bit longer to start or run, then if you're busy that can compound.

Even a simple case of everyone running exactly the same with a snapshot being made in a race gives us 50% for each. For easy math, say we had 80 by half the nodes and 90 by the other half because we can run 10 workflows. With an even number of nodes reporting, it returns 85 and 0 agree, you take 72 as the value.

For capacity, I would take the minimum known honest value at F+1th lowest. Validate observation can ensure you have a capacity, as observation fails if you get an error from the capacity function so you know you have at QuorumTwoFPlusOne here. Even if you don't validate it, at most F nodes would be missing it and defaulting to zero so the first honest node would be used.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have a term for the F+1th lowest?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm tempted to call it leastHonest(capacities)? 😆

@jmank88 jmank88 Nov 19, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated but lmk if there is a preferred name. I just did an inline Sorted call with an f index 🤷

}

// Check size
if protoOpts.Size(obs) > maxObservationBytes {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This checks if adding the event goes over maxObservationBytes but we have blobHandle placeholders in the obs now. If broadcast as blobs breaks it up into more than the placeholders we'll go over the maxObservation size and the observation will fail

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we have a maxObservedBlobHandles variable already, but we're not checking it in broadcastAsBlob

maxObservedBlobHandles = 10 // sanity check on blobs in an observation
maxObservedEvents = 1000 // sanity check events in an observation

//TODO reduce these? connect to cre settings?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should be connected to cre settings instead of using libocr's maxmax values. I don't think we've tested those values except for the observation bytes size. Since cre settings is local changing these limits isn't atomic (including any feature flags we add). There's a PR to change how that works in the vault which we may want to copy over here: #23650

if err != nil {
return nil, -1, fmt.Errorf("failed to marshal blob: %w", err)
}
bh, err := broadcaster.BroadcastBlob(ctx, data, ocr3_1types.BlobExpirationHintSequenceNumber{SeqNr: expSeq})

@russell-stern russell-stern Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Broadcasting blobs is somewhat slow (600-800ms on Vault prod). We can do this in parallel to speed it up but make sure you adhere to libocr's concurrency limit. Vault reference: https://github.com/smartcontractkit/chainlink/blob/develop/core/services/ocr2/plugins/vault/plugin.go#L909

var obs observations

go func() {
defer close(done)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done is nil here so close will panic

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this a go routine? To save time on loading the state?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, though I inherited that decision from an earlier design, so I don't have evidence that is it necessarily important to be doing.

@jmank88
jmank88 force-pushed the CRE-6002-reporting-plugin-impl branch from fe77b85 to d4f5a8d Compare November 19, 2026 21:44
@jmank88
jmank88 force-pushed the CRE-6002-reporting-plugin-impl branch from d4f5a8d to 60e3b4d Compare November 19, 2026 21:47

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants