Skip to content

Allow request cancellation for wrapped C++ methods - #342

Open
xyzconstant wants to merge 6 commits into
bitcoin-core:masterfrom
xyzconstant:add-proxy-cancel
Open

xyzconstant wants to merge 6 commits into
bitcoin-core:masterfrom
xyzconstant:add-proxy-cancel

Conversation

@xyzconstant

@xyzconstant xyzconstant commented Aug 12, 2026 •

Copy link
Copy Markdown
Contributor

Currently, libmultiprocess requests are blocking, with no way to cancel them from either the client or server side. Downstream code works around this by pairing each blocking method with a dedicated interruptX() method whose only purpose is to wake it up. For example, bitcoin/bitcoin#33676 introduced BlockTemplate::interruptWait() specifically to wake an in-progress BlockTemplate::waitNext() call.

Cap'n Proto provides a useful cancellation mechanism that libmultiprocess can use to support cancellation from both non-libmultiprocess and libmultiprocess clients. When a promise is dropped, it sends a cancellation request. The server may either cancel immediately or ignore the signal.

This PR implements approach 4 from bitcoin/bitcoin#33575, adding cancellation support on both sides:

  • Allow ProxyClient method calls to be canceled from another thread. ClientInvokeContext provides two hooks an application overload can set: set_canceler, which receives the kj::Canceler that clientInvoke wraps the request promise in, and handle_error, which receives the kj::Exception from a failed request and decides what the caller sees. The second hook is how a cancel parameter turns the DISCONNECTED that a kj::Canceler raises into InterruptException.
  • Allow wrapped server methods to register a callback that runs when the request is canceled, whether because the promise was dropped or the connection was interrupted.

The PR also adds a new capnp annotation, $Proxy.extraParam, which declares C++-only parameters that are not sent through RPC. These parameters are handled by CustomBuildExtraParam on the client side and CustomReadExtraParam on the server side. Paired with the cancellation mechanism, this feature lets applications define their own cancel parameter types. A reference implementation lives under test/mp/test/foo-types.h and test/mp/test/foo.h, used in tests.

For example, given the following capnp schema method:

waitNext @0 (context :Proxy.Context) -> (result :Template) $Proxy.extraParam("cancel") $Cxx.allowCancellation;

and this C++ method with an application-defined cancel parameter:

using CancelFn = std::function<void()>;
class CancelGuard;   // unregisters the callback when destroyed
using CancelArg = std::function<CancelGuard(CancelFn)>;

virtual std::unique_ptr<Template> waitNext(CancelArg cancel) = 0;

The caller (client) passes a callback that receives the function that cancels the call:

CancelFn cancel_fn;

// ... blocks until the result arrives or another thread runs cancel_fn()
auto tmpl = client->waitNext([&](CancelFn fn) {
    cancel_fn = std::move(fn);
    return CancelGuard{};
});

The callee (server) registers a callback that interrupts its wait and retains the returned guard until the wait ends:

std::unique_ptr<Template> waitNext(CancelArg cancel) override
{
    CancelGuard guard;
    if (cancel) guard = cancel([this] { m_cv.notify_all(); });

    // ... wait on m_cv, checking for cancellation
}

Detecting a dropped promise on the server requires the $Cxx.allowCancellation annotation on the method, file, or interface. Without it, Cap'n Proto runs the abandoned call to completion. The annotation requires Cap'n Proto 1.0 (see the "Breaking change" section in https://capnproto.org/news/2023-07-28-capnproto-1.0.html), which this PR also sets as the minimum supported version.


NOTE: bitcoin/bitcoin#36097 builds on this PR by adding cancel arguments to the blocking mining methods (waitTipChanged, createNewBlock, BlockTemplate::waitNext) and deprecating the interrupt()/interruptWait() workarounds.

@DrahtBot

DrahtBot commented Aug 12, 2026 •

Copy link
Copy Markdown

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Reviews

See the guideline and AI policy for information on the review process.

Type Reviewers
Concept ACK enochaz, ViniciusCestarii
Approach ACK ryanofsky

If your review is incorrectly listed, please copy-paste <!--meta-tag:bot-skip--> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #393 (ci: Add OpenIndiana job by hebasto)
  • #392 (test: Skip thread name tests where OS thread names do not work by hebasto)
  • #377 (mpgen: Allow wrapped interfaces to inherit from unwrapped interfaces by ryanofsky)
  • #337 (proxy-types: Remove requirement for return types to be default-constructible by ryanofsky)
  • #175 (Set cmake_minimum_required(VERSION 3.22) by maflcko)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@xyzconstant

Copy link
Copy Markdown
Contributor Author

CI failures seem unrelated

@ryanofsky

ryanofsky commented Aug 13, 2026 •

Copy link
Copy Markdown
Collaborator

Nice work! I've had a number of ideas about this feature over the years so it's really interesting to see it implemented. I will just give some quick thoughts for now so I don't get nerdsniped and wind up spending all day or more on this. Thoughts:

  • I don't think the proxy.capnp Cancel struct should exist. The idea of allowing a CancelToken / interfaces::Cancel argument from RFC: Cancelling waitNext calls in the IPC mining interface bitcoin/bitcoin#33575 is just to provide a way for libmultiprocess C++ clients to cancel capnp::Response promises, and for libmultiprocess C++ servers to detect whether the promises they are fulfilling have been cancelled. Rust and python and non-libmultiprocess C++ clients and servers already have a way to do these things, so they should not be affected by implemntation of this feature, and not see any differences in .capnp schema files if a C++ method supports it or doesn't support it.

  • Relatedly, the implementation should be orthogonal to the $Cxx.allowCancellation(true); annotation. The annotation controls how capnproto sends cancellations, while CancelToken is way of letting libmultiprocess C++ classes send and receive them. CancelToken will probably be most useful combined with $Cxx.allowCancellation(true); annotations, but it could be useful without them too, for example to interrupt calls on unclean disconnects, but not interrupt calls when clients drop capnp::Response promises because they don't need the results.

  • In order for this feature to be useful in Bitcoin Core, it shouldn't require C++ interfaces to directly use the mp::CancelToken type. Bitcoin Core C++ interfaces in src/interfaces/ are intended to be used by node/wallet/gui code and compile without any dependency on libmultiprocess. So libmultiprocess could provide CustomBuildCancel and CustomReadCancel overloads analagous to CustomBuildField and CustomReadField overloads that applications can override to work with custom cancellation arguments. Probably the simplest cancellation argument type would look like:

    using CancelFn = std::function<void()>; // Called when a request is cancelled
    using CancelArg = std::function<void(CancelFn)>; // Called to set a CancelFn that is called when a request is cancelled.

    and support overloads like

    template <typename LocalType, typename Value>
    void CustomBuildCancel(TypeList<CancelArg>, InvokeContext& invoke_context, Value&& value)
    {
        // If client provied a CancelArg argument, call it to give them a CancelFn
        // callback they can use to interrupt this request.
        if (value) value([&invoke_context] { invoke_context.cancel_request(); } };
    }
    
    template <typename LocalType, typename ReadDest>
    decltype(auto) CustomReadCancel(TypeList<CancelArg>, InvokeContext& invoke_context, ReadDest&& read_dest)
    {
        // Return a CancelArg for servers call to register a CancelFn and be
        // notified if the current request is cancelled.
        return read_dest.construct([&invoke_context](CancelFn cancel_fn) invoke_context.on_request_cancelled(std::move(cancel_fn)); });
    }
    ``
    
    Having CustomBuildCancel/CustomReadCancel hooks would let libmultiprocess be agnostic to whatever cancellation interfaces C++ applications want to use, and just give clients a way to send cancellations and servers a way to receive them.
    
  • Alternately instead of adding CustomBuildCancel/CustomReadCancel hooks, we could use existing CustomBuildField/CustomReadField hooks with empty input/output arguments. I think to do this would need to add a new hook called something like CustomFieldExists() that is constexpr and returns true by default but false if no capnproto field corresponding to the C++ type will exist. This would let clientInvoke/serverInvoke code handle C++ arguments that don't have corresponding capnproto fields, and be be similar to existing CustomHasField() / CustomHasValue() overloads but be constexpr and reflect whether the field exists at all, not whether it is has a value set.

  • It looks like this implementation as of 0986a13 only allows libmultiprocess servers to detect cancellations, but doesn't allow libmultiprocess clients to request cancellations. This is ok, but it probably makes sense to support both because a single argument can support both, and so the feature can be more naturally tested end-to-end.

  • Commit 3f6b324 is an interesting way to provide compatibility with older versions of capnproto that don't support $Cxx.allowCancellation annotations. I think in practice it's probably fine to drop support for these older versions though, and if we did want to support them we should probably just provide an $Cxx.allowCancellation annotation for them to use and call context.allowCancellation() when it's present.

  • The CI failures in https://github.com/bitcoin-core/libmultiprocess/actions/runs/31636595747/job/94248350765?pr=342 are caused by this PR but they should be easy to fix. The -Wc++23-lambda-attributes errors are garbage from IWYU output but if you look below you will see real IWYU errors that need to be fixed. Once the IWYU errors are fixed all the IWYU output should disappear and CI should be green.

EDIT: Added CustomFieldExists idea above. Thinking about this more I believe it would be preferable to just add a new CustomFieldExists hook instead of adding CustomBuildCancel/CustomReadCancel hooks to be more general and also probably make the implementation simpler (just adding an argument handling option, not cancellation-specific argument handling options).

@xyzconstant

Copy link
Copy Markdown
Contributor Author

Thanks for your feedback @ryanofsky. Sorry for the late response. I took some time to fill some C++ gaps (mostly to understand whether client-side detection of the CancelFn/CancelArg types could be solved through templating), since I'm still learning the language.

  • I don't think the proxy.capnp Cancel struct should exist. The idea of allowing a CancelToken / interfaces::Cancel argument from RFC: Cancelling waitNext calls in the IPC mining interface bitcoin/bitcoin#33575 is just to provide a way for libmultiprocess C++ clients to cancel capnp::Response promises, and for libmultiprocess C++ servers to detect whether the promises they are fulfilling have been cancelled. Rust and python and non-libmultiprocess C++ clients and servers already have a way to do these things, so they should not be affected by implemntation of this feature, and not see any differences in .capnp schema files if a C++ method supports it or doesn't support it.

This makes complete sense. I believe I now grasp the original idea. This is cleaner for consumers and non-libmultiprocess clients, which don't need a Cancel schema parameter (it was only a marker for mpgen indicating the C++ method takes a token arg) because they already cancel their requests when promises are dropped.

  • Relatedly, the implementation should be orthogonal to the $Cxx.allowCancellation(true); annotation. The annotation controls how capnproto sends cancellations, while CancelToken is way of letting libmultiprocess C++ classes send and receive them. CancelToken will probably be most useful combined with $Cxx.allowCancellation(true); annotations, but it could be useful without them too, for example to interrupt calls on unclean disconnects, but not interrupt calls when clients drop capnp::Response promises because they don't need the results.

Yes, you're correct here as well. $Cxx.allowCancellation only changes how a server responds to a cancel signal (whether the callee wants to cancel the request immediately or not). This means we can drop the dispatchCall override too.

  • In order for this feature to be useful in Bitcoin Core, it shouldn't require C++ interfaces to directly use the mp::CancelToken type. Bitcoin Core C++ interfaces in src/interfaces/ are intended to be used by node/wallet/gui code and compile without any dependency on libmultiprocess. So libmultiprocess could provide CustomBuildCancel and CustomReadCancel overloads analagous to CustomBuildField and CustomReadField overloads that applications can override to work with custom cancellation arguments. Probably the simplest cancellation argument type would look like:

    using CancelFn = std::function<void()>; // Called when a request is cancelled
    using CancelArg = std::function<void(CancelFn)>; // Called to set a CancelFn that is called when a request is cancelled.

    and support overloads like

    template <typename LocalType, typename Value>
    void CustomBuildCancel(TypeList<CancelArg>, InvokeContext& invoke_context, Value&& value)
    {
        // If client provied a CancelArg argument, call it to give them a CancelFn
        // callback they can use to interrupt this request.
        if (value) value([&invoke_context] { invoke_context.cancel_request(); } };
    }
    
    template <typename LocalType, typename ReadDest>
    decltype(auto) CustomReadCancel(TypeList<CancelArg>, InvokeContext& invoke_context, ReadDest&& read_dest)
    {
        // Return a CancelArg for servers call to register a CancelFn and be
        // notified if the current request is cancelled.
        return read_dest.construct([&invoke_context](CancelFn cancel_fn) invoke_context.on_request_cancelled(std::move(cancel_fn)); });
    }

    Having CustomBuildCancel/CustomReadCancel hooks would let libmultiprocess be agnostic to whatever cancellation interfaces C++ applications want to use, and just give clients a way to send cancellations and servers a way to receive them.

This is interesting. I think we could combine this with OnCancel so it registers and unregisters the callback at the right time.

  • Alternately instead of adding CustomBuildCancel/CustomReadCancel hooks, we could use existing CustomBuildField/CustomReadField hooks with empty input/output arguments. I think to do this would need to add a new hook called something like CustomFieldExists() that is constexpr and returns true by default but false if no capnproto field corresponding to the C++ type will exist. This would let clientInvoke/serverInvoke code handle C++ arguments that don't have corresponding capnproto fields, and be be similar to existing CustomHasField() / CustomHasValue() overloads but be constexpr and reflect whether the field exists at all, not whether it is has a value set.

I prefer this over the dedicated hooks. However, I don't fully understand how it would handle the client side. For clientInvoke to accept an argument without a field, the generated method has to take that argument in the first place (mpgen prints the parameter list based on the schema's field counts), and it must match the interface method exactly.

So I think the hook can tell clientInvoke what to do with the argument, but I don't see how it makes mpgen print a parameter the schema doesn't mention. Am I missing something here? If I'm not, I only see one way to handle this, and it's still making mpgen learn about the argument from the schema, this time not through a direct schema parameter like the Cancel struct but through a capnp method annotation.

  • It looks like this implementation as of 0986a13 only allows libmultiprocess servers to detect cancellations, but doesn't allow libmultiprocess clients to request cancellations. This is ok, but it probably makes sense to support both because a single argument can support both, and so the feature can be more naturally tested end-to-end.

Yes, definitely we can cover both sides in this PR. I need to think about this more though, together with the mpgen issue above.

  • Commit 3f6b324 is an interesting way to provide compatibility with older versions of capnproto that don't support $Cxx.allowCancellation annotations. I think in practice it's probably fine to drop support for these older versions though, and if we did want to support them we should probably just provide an $Cxx.allowCancellation annotation for them to use and call context.allowCancellation() when it's present.

Ok perfect. I'll drop it.

Thanks, I really thought it wasn't caused by this PR, but I guess I had it wrong. I have some WIP rework locally, so these errors may be different the next time I push.

@ryanofsky

Copy link
Copy Markdown
Collaborator

This is interesting. I think we could combine this with OnCancel so it registers and unregisters the callback at the right time.

Hmm yeah. Actually when I wrote this I wasn't thinking about the need to unregister, and the CancelState and OnCancel interface seemed unnecessarily complicated to me. But unregistering is needed to deal with cancellation being sent of the the method finishes executing, so pretty important. (I still do think the vector of callbacks in CancelState might be too complicated though and a single std::function<void>() that could be set to null to unregister would probably be enough.)

I prefer this over the dedicated hooks. However, I don't fully understand how it would handle the client side. For clientInvoke to accept an argument without a field, the generated method has to take that argument in the first place (mpgen prints the parameter list based on the schema's field counts), and it must match the interface method exactly.

So I think the hook can tell clientInvoke what to do with the argument, but I don't see how it makes mpgen print a parameter the schema doesn't mention. Am I missing something here? If I'm not, I only see one way to handle this, and it's still making mpgen learn about the argument from the schema, this time not through a direct schema parameter like the Cancel struct but through a capnp method annotation.

You're right, I wasn't thinking about the client side of this very clearly. I was thinking as the client code was processing parameters it could use ProxyMethodTraits to see if the next parameter had a field mapping with CustomFieldExists. But this doesn't work at all because the code generator also needs to know the field mapping and doesn't have access to anything but the capnproto schema file.

So I think a capnproto method annotation of some kind might be necessary. I was thinking of adding annotations anyway to support more flexible mapping of c++ parameters to capnproto parameters like:

open @3 (name :Text $param(pos=1, type="CxxType")) -> (node :Node) $extraParam(name="foo",pos=2)) $extraParam(name="bar", pos=0]);

in context of #282. But something less general could also work well here.

@xyzconstant

xyzconstant commented Aug 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Hmm yeah. Actually when I wrote this I wasn't thinking about the need to unregister, and the CancelState and OnCancel interface seemed unnecessarily complicated to me. But unregistering is needed to deal with cancellation being sent of the the method finishes executing, so pretty important. (I still do think the vector of callbacks in CancelState might be too complicated though and a single std::function<void>() that could be set to null to unregister would probably be enough.)

The vector of callbacks exists because another callback (pre-existing) is registered in type-context.h to:

  1. Log that the request was canceled mid-execution
  2. Lock request_mutex so the event loop thread can't delete the params/results while the worker thread is still using them.

So during a cancellable request, there are two registrations: one internal, and another the wrapped method itself registers. But I do agree it looks a bit awkward. The raw pointers are there so OnCancel can own the callback by value and CancelState doesn't have to allocate, but there might be nicer ways to do that.

You're right, I wasn't thinking about the client side of this very clearly. I was thinking as the client code was processing parameters it could use ProxyMethodTraits to see if the next parameter had a field mapping with CustomFieldExists. But this doesn't work at all because the code generator also needs to know the field mapping and doesn't have access to anything but the capnproto schema file.

So I think a capnproto method annotation of some kind might be necessary. I was thinking of adding annotations anyway to support more flexible mapping of c++ parameters to capnproto parameters like:

open @3 (name :Text $param(pos=1, type="CxxType")) -> (node :Node) $extraParam(name="foo",pos=2)) $extraParam(name="bar", pos=0]);

in context of #282. But something less general could also work well here.

Good, this settles the client side then. I'd like to start with something minimal that we can scale up later into something more general.

I was thinking of a method annotation (your extraParam annotation but without the position parameter) that just names the extra C++ parameters that have no field:

waitValue @7 (context :Proxy.Context, timeoutMs :Int32) -> (result :Int32) $Proxy.extraParam("cancel");

@xyzconstant
xyzconstant force-pushed the add-proxy-cancel branch 3 times, most recently from 4c8b3db to 4cfc251 Compare August 23, 2026 13:52
@xyzconstant
xyzconstant force-pushed the add-proxy-cancel branch 2 times, most recently from 4821d99 to 4731f5d Compare August 24, 2026 18:27
@xyzconstant xyzconstant changed the title Allow wrapped C++ methods to detect request cancellations Allow request cancellation for wrapped C++ methods Aug 24, 2026
@xyzconstant

Copy link
Copy Markdown
Contributor Author

@ryanofsky Just addressed your comments, rebased with master, and pushed.

Added 9422b97, which supports the $Proxy.extraParam annotation, and built the cancellation feature on top of it. Also, bumped the minimum capnp version to 1.0 (mostly to pass in green olddeps) in f98bd92. If one of them seems like it needs to be moved to a separate PR, just let me know.

Updated the description too. Thanks for your feedback :)

@ryanofsky

Copy link
Copy Markdown
Collaborator

Thanks for the updates! And approach ACK f6944b2. I think it's good to add client and server cancellation support together in the same PR.

Quickly skimming changes, the extraParam implementation looks good and later changes also seem right, although I'm unclear in the later changes on why shared_ptr needs to be used, and why CancelState and ClientCancelState classes instead of plain std::function variables. I wonder if it might be possible to simplify more. But overall this looks good and I'm planning to review it.

It would also be great to see this put to use in bitcoin core by dropping the BlockTemplate::interruptWait and Mining::interrupt methods and supporting native request cancellation instead.

@xyzconstant
xyzconstant force-pushed the add-proxy-cancel branch 4 times, most recently from d2014d9 to 22000b4 Compare August 26, 2026 15:46
xyzconstant added a commit to xyzconstant/bitcoin that referenced this pull request Sep 11, 2026
@xyzconstant

xyzconstant commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor Author

FWIW: I just pushed changes to bitcoin/bitcoin#36097 to sync with this PR's current state. I also updated the description here to reference it.

@xyzconstant

Copy link
Copy Markdown
Contributor Author

Rebased onto master and solved conflicts: 5deb80e -> d8bf70b (compare).

@xyzconstant

Copy link
Copy Markdown
Contributor Author

Fixed surfaced failures exposed by the last rebase (clang-tidy's modernize-use-scoped-lock and -Wunused-parameter). d8bf70b -> f1e28d0 (compare).

Add a `$Proxy.extraParam` method annotation that declares an extra C++-only
parameter in the generated method signature. The parameter has no
corresponding capnp parameter and is not serialized or sent over RPC. The
annotation value names the parameter in generated C++ code.

Client behavior:
- A matching `CustomBuildExtraParam(TypeList<T>, ClientInvokeContext&, T&&)`
  overload MUST be implemented. The parameter is passed to it before the RPC
  message is dispatched.

Server behavior:
- A matching `CustomReadExtraParam(TypeList<T>, ServerContext&)` overload MUST
  be implemented. No data arrives for this parameter so this overload
  reconstructs the parameter value on the server side.

The test checks the values the client passes reach the client overload and
the ones the server reconstructs arrive instead.
…`request_mutex`

The mutex guards the request's params and results structs, not the
cancellation itself. The old names would be confusing next to the CancelState
class added in the following commits. Pure rename, no behavior change.
Add two optional hooks to `ClientInvokeContext` that client overloads can
set:

- `set_canceler` receives the `kj::Canceler` object wrapping the request promise
  once it is sent, and null when the request finishes and the canceler is
  destroyed. `clientInvoke` only creates the canceler (heap-allocated and
  attached to the promise) if the hook is set.
- `handle_error` receives the `kj::Exception` of a failed request and lets
  clients throw a custom exception, which `clientInvoke` rethrows to the caller.
  For example, `kj::Canceler::cancel` raises `DISCONNECTED`, the same error a
  disconnect produces, so only the parameter that canceled can tell them apart.
Add two members to `ServerInvokeContext` that server overloads can use:

- `request_mutex` is the mutex `request_lock` refers to, set together with it
  by the `Proxy.Context` `PassField` overload, and null for methods running on
  the event loop thread. Overloads lock it to touch `cancel_fn`.
- `cancel_fn` holds a callback that the cancel monitor runs on the event loop
  thread, when the request is canceled while the method executes. `ServerCall`
  clears it after the method returns.
The `$Cxx.allowCancellation` annotation used by the cancellation tests
does not exist in older versions. Remove the configure-time checks that
only covered them, and move the olddeps CI config to 1.0.0.
- One test cancels an in-flight `ProxyClient` call from another thread.
- Another drops the response promise mid-execution, imitating
  non-libmultiprocess clients.

Additionally, this commit adds a reference implementation of a cancel
parameter and its corresponding `CustomBuildExtraParam`/`CustomReadExtraParam`
overloads that applications could replicate to fully support cancellations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants