Skip to content

feat(wirings): windowed tier DataStream wiring + demo (stacks on #7)#8

Closed
estebanzimanyi wants to merge 1 commit into
MobilityDB:feat/flink-bounded-state-tier-wiringsfrom
estebanzimanyi:feat/flink-windowed-tier-wirings
Closed

feat(wirings): windowed tier DataStream wiring + demo (stacks on #7)#8
estebanzimanyi wants to merge 1 commit into
MobilityDB:feat/flink-bounded-state-tier-wiringsfrom
estebanzimanyi:feat/flink-windowed-tier-wirings

Conversation

@estebanzimanyi

@estebanzimanyi estebanzimanyi commented May 21, 2026

Copy link
Copy Markdown
Member

Third of four planned tier-specific wiring follow-ups on PR #5's generated MEOS facade. Stacks on PR #7 (bounded-state). Adds the windowed tier.

What's in this PR

File Purpose
MeosWindowedAggregate<K, IN, OUT, W> Generic ProcessWindowFunction wrapping any windowed MeosOps call; window-close-only aggregation
demo/MeosWindowedDemoJob Runnable 30s-tumbling-window per-vehicle aggregate-tbox pipeline (8 events × 2 vehicles × 2 windows)
README.md windowed row marked ✅ shipped

Design — fresh-per-window aggregation

Unlike bounded-state, no MEOS handle persists across window boundaries. Each window's MEOS value is built fresh from the iterable on window close, used to compute the output, then discarded. The iterable's events are Flink-side data; MEOS handles are short-lived per-window.

The adopter-facing signature stays slim — the lambda receives the window metadata, the iterable of in-window events, and a ContextLike<K> exposing key + processing-time + watermark (free of Flink internals):

new MeosWindowedAggregate<Integer, VehiclePoint, VehicleLength, TimeWindow>(
    (window, inWindowEvents, ctx) -> {
        Pointer trajectory = buildTrajectoryFromPoints(inWindowEvents);  // adopter helper
        double length = MeosOpsTemporal.temporal_length(trajectory);
        return new VehicleLength(ctx.getCurrentKey(), window.getStart(), length);
    });

Tier coverage after this PR

Tier Method count (v4 baseline) Wirable through
stateless 804 MeosStatelessMap / MeosStatelessFilter (PR #6)
bounded-state 797 MeosBoundedStateMap (PR #7)
windowed 161 MeosWindowedAggregate (this PR)
cross-stream 140 next follow-up (MeosCrossStreamJoin)
io-meta 195 covered transitively by MeosStatelessMap
sequence-only 14 n/a — inherently non-streamable

Cumulative: 1,957 of 2,097 generated methods (93%) are now wirable through 4 generic classes (Map + Filter + BoundedStateMap + WindowedAggregate). Only the cross-stream tier (140 methods, 7%) remains for the final follow-up.

Demo highlights

MeosWindowedDemoJob is an 8-event × 2-vehicle × 2-window pipeline:

  • Event-time tumbling windows of 30 seconds
  • Per-window aggregate computed by repeatedly unioning event tboxes (via MeosOpsFreeCore.union_tbox_tbox)
  • Per-window output: (vehicleId, windowStart, eventCount, aggregateTboxWKT)
  • Proves: window-close timing, per-key isolation, fresh-per-window aggregation

Stacking

This PR stacks on feat/flink-bounded-state-tier-wirings (PR #7). Additive-only: 3 file changes (1 new wiring class + 1 new demo + 1 README row marked ✅).

Compile verification

Locally green: 140 .class files total (135 from PR #7 base + 5 new — 1 wiring class + 2 nested lambda interfaces + 1 anonymous ContextLike + 1 demo class).

This was referenced May 21, 2026
Closed
Closed
@estebanzimanyi estebanzimanyi force-pushed the feat/flink-windowed-tier-wirings branch from fd5eb10 to ef47708 Compare May 29, 2026 12:42
estebanzimanyi added a commit to estebanzimanyi/MobilityFlink that referenced this pull request May 29, 2026
@estebanzimanyi
feat(wirings): cross-stream tier DataStream wiring + runnable demo (c…
10a03b6 
…ompletes the 4-tier matrix)

Adds MeosCrossStreamJoin<L, R, OUT> — the fourth and final
tier-wiring class in the org.mobilitydb.flink.meos.wirings package,
stacked on PR MobilityDB#8 (windowed wirings).

Cross-stream is the smallest streamable tier (140 of 2,097 emitted
methods, ~7%) — pairwise across two pre-keyed streams, time-bounded
match window. Canonical examples: spatial-relations between two
trajectories (edwithin_tgeo_tgeo, eintersects_tgeo_tgeo), distance
on two temporals (nad_tgeo_tgeo, mindistance_tgeo_tgeo).

## Design

Wraps any cross-stream MeosOps call as a ProcessJoinFunction — the
operator backing KeyedStream.intervalJoin(other). Both streams must
be pre-keyed by the same K; only events sharing a key are considered
for pairing. The .between(lowerBound, upperBound) declaration bounds
the time window for match-eligibility, and matches are emitted
event-time-aware (watermark-driven).

The adopter-facing signature keeps the slim ContextLike-pattern used
in MeosWindowedAggregate: the lambda receives the matched (left,
right) pair and a slim Context exposing left/right timestamps (the
bits a MEOS cross-stream call typically needs), free of Flink
internals.

## Files

- MeosCrossStreamJoin.java — the generic wiring class
- demo/MeosCrossStreamDemoJob.java — runnable interval-join demo
  matching two streams of (regionId, vehicleId, tboxWKT, ts) on
  shared regionId key within ±1 minute; emits per-pair overlap
  events via MeosOpsFreeCore.overlaps_tbox_tbox
- README — cross-stream row marked ✅ shipped

## Completes the 4-tier wiring matrix

After this PR, every streamable tier in the v4 baseline has a
generic wiring class in this package:

  stateless        804 methods   →  MeosStatelessMap / MeosStatelessFilter (PR MobilityDB#6)
  bounded-state    797 methods   →  MeosBoundedStateMap (PR MobilityDB#7)
  windowed         161 methods   →  MeosWindowedAggregate (PR MobilityDB#8)
  cross-stream     140 methods   →  MeosCrossStreamJoin (THIS PR)
  io-meta          195 methods   →  covered by MeosStatelessMap
  sequence-only     14 methods   →  inherently non-streamable

Total: 2,097 of 2,097 = 100% of streamable + io-meta generated
MeosOps* methods are wirable through 4 (+ 1 filter sibling) generic
classes; no per-method registration; adopters provide a serializable
lambda per use site.

## Stacks on PR MobilityDB#8

Additive-only; touches no existing file beyond the README row.
Locally compile-verified: 145 .class files total (140 from PR MobilityDB#8
base + 5 new — 1 wiring class + 2 nested lambda interfaces + 1
anonymous ContextLike + 1 demo class).
estebanzimanyi added a commit to estebanzimanyi/MobilityFlink that referenced this pull request May 29, 2026
@estebanzimanyi
feat(wirings): capstone demo composing all 4 tier wirings into one pi…
d322b8c 
…peline

Adds MeosAllTiersCapstoneDemo — a single Flink DataStream job that
exercises all four tier-wiring classes from the PR MobilityDB#6→MobilityDB#7→MobilityDB#8→MobilityDB#9 stack
in a coherent end-to-end pipeline.

Pipeline (each stage uses one tier-wiring class from the stack):

  ① MeosStatelessFilter    — drop events outside regions of interest
  ② MeosBoundedStateMap    — per-vehicle running tbox union (byte[] state)
  ③ MeosWindowedAggregate  — per-vehicle 30s tumbling tbox summary
  ④ MeosCrossStreamJoin    — interval-join vehicle aggregates against
                             region queries (±1m bound, regionId key)

The pipeline answers: 'for each region, which vehicles had an aggregate
trajectory (running union) overlapping the region's query bbox during
the latest 30-second window?'

Proves the wirings compose into a realistic pipeline shape (not just
work in isolation), each tier delivering its specific contract:
stateless filter is per-event, bounded-state persists handle state
across events as bytes, windowed aggregates window-close-only, cross-
stream interval-joins on shared key.

Stacks on PR MobilityDB#9; additive-only (1 new demo file). Locally compile-
verified: 146 .class files total (145 from PR MobilityDB#9 base + 1 new demo).
@estebanzimanyi
feat(wirings): windowed tier DataStream wiring + runnable demo
ad3cafe 
Adds MeosWindowedAggregate<K, IN, OUT, W> — the third tier-wiring
class in the org.mobilitydb.flink.meos.wirings package, stacked on
PR MobilityDB#7 (bounded-state wirings).

Windowed is the third streaming tier (161 of 2,097 emitted methods,
~8%) — output cardinality changes; one MEOS aggregate per window.
Canonical examples: temporal_length(tgeo) for per-window trajectory
length, temporal_twavg(tnumber) for time-weighted average per
window, per-class _trajectory / _time / _timespan accessors.

## Design

Wraps any windowed MeosOps call as a ProcessWindowFunction with a
slim adopter-facing signature: the lambda receives the window
metadata, the iterable of in-window events, and a slim Context
exposing key + processing-time + watermark (free of Flink internals).

Unlike bounded-state, NO MEOS handle persists across window
boundaries — each window's MEOS value is built fresh from the
iterable on window close, used to compute the output, discarded.
The iterable's events are Flink-side data; MEOS handles are
short-lived per-window.

## Files

- MeosWindowedAggregate.java — the generic wiring class
- demo/MeosWindowedDemoJob.java — runnable 30s-tumbling-window
  per-vehicle aggregate-tbox demo (8 events × 2 vehicles × 2 windows;
  demonstrates window-close timing, per-key isolation, fresh-per-
  window aggregation)
- README — windowed row marked ✅ shipped

## Stacks on PR MobilityDB#7

Additive-only; touches no existing file beyond the README row.
Locally compile-verified: 140 .class files total (135 from PR MobilityDB#7
base + 5 new — 1 wiring class + 2 nested lambda interfaces + 1
anonymous ContextLike + 1 demo class).

(cherry picked from commit ef47708)
@estebanzimanyi estebanzimanyi force-pushed the feat/flink-windowed-tier-wirings branch from ef47708 to ad3cafe Compare May 31, 2026 07:49
estebanzimanyi added a commit to estebanzimanyi/MobilityFlink that referenced this pull request May 31, 2026
@estebanzimanyi
feat(wirings): cross-stream tier DataStream wiring + runnable demo (c…
d71d460 
…ompletes the 4-tier matrix)

Adds MeosCrossStreamJoin<L, R, OUT> — the fourth and final
tier-wiring class in the org.mobilitydb.flink.meos.wirings package,
stacked on PR MobilityDB#8 (windowed wirings).

Cross-stream is the smallest streamable tier (140 of 2,097 emitted
methods, ~7%) — pairwise across two pre-keyed streams, time-bounded
match window. Canonical examples: spatial-relations between two
trajectories (edwithin_tgeo_tgeo, eintersects_tgeo_tgeo), distance
on two temporals (nad_tgeo_tgeo, mindistance_tgeo_tgeo).

## Design

Wraps any cross-stream MeosOps call as a ProcessJoinFunction — the
operator backing KeyedStream.intervalJoin(other). Both streams must
be pre-keyed by the same K; only events sharing a key are considered
for pairing. The .between(lowerBound, upperBound) declaration bounds
the time window for match-eligibility, and matches are emitted
event-time-aware (watermark-driven).

The adopter-facing signature keeps the slim ContextLike-pattern used
in MeosWindowedAggregate: the lambda receives the matched (left,
right) pair and a slim Context exposing left/right timestamps (the
bits a MEOS cross-stream call typically needs), free of Flink
internals.

## Files

- MeosCrossStreamJoin.java — the generic wiring class
- demo/MeosCrossStreamDemoJob.java — runnable interval-join demo
  matching two streams of (regionId, vehicleId, tboxWKT, ts) on
  shared regionId key within ±1 minute; emits per-pair overlap
  events via MeosOpsFreeCore.overlaps_tbox_tbox
- README — cross-stream row marked ✅ shipped

## Completes the 4-tier wiring matrix

After this PR, every streamable tier in the v4 baseline has a
generic wiring class in this package:

  stateless        804 methods   →  MeosStatelessMap / MeosStatelessFilter (PR MobilityDB#6)
  bounded-state    797 methods   →  MeosBoundedStateMap (PR MobilityDB#7)
  windowed         161 methods   →  MeosWindowedAggregate (PR MobilityDB#8)
  cross-stream     140 methods   →  MeosCrossStreamJoin (THIS PR)
  io-meta          195 methods   →  covered by MeosStatelessMap
  sequence-only     14 methods   →  inherently non-streamable

Total: 2,097 of 2,097 = 100% of streamable + io-meta generated
MeosOps* methods are wirable through 4 (+ 1 filter sibling) generic
classes; no per-method registration; adopters provide a serializable
lambda per use site.

## Stacks on PR MobilityDB#8

Additive-only; touches no existing file beyond the README row.
Locally compile-verified: 145 .class files total (140 from PR MobilityDB#8
base + 5 new — 1 wiring class + 2 nested lambda interfaces + 1
anonymous ContextLike + 1 demo class).

(cherry picked from commit 10a03b6)
estebanzimanyi added a commit to estebanzimanyi/MobilityFlink that referenced this pull request May 31, 2026
@estebanzimanyi
feat(wirings): capstone demo composing all 4 tier wirings into one pi…
173429b 
…peline

Adds MeosAllTiersCapstoneDemo — a single Flink DataStream job that
exercises all four tier-wiring classes from the PR MobilityDB#6→MobilityDB#7→MobilityDB#8→MobilityDB#9 stack
in a coherent end-to-end pipeline.

Pipeline (each stage uses one tier-wiring class from the stack):

  ① MeosStatelessFilter    — drop events outside regions of interest
  ② MeosBoundedStateMap    — per-vehicle running tbox union (byte[] state)
  ③ MeosWindowedAggregate  — per-vehicle 30s tumbling tbox summary
  ④ MeosCrossStreamJoin    — interval-join vehicle aggregates against
                             region queries (±1m bound, regionId key)

The pipeline answers: 'for each region, which vehicles had an aggregate
trajectory (running union) overlapping the region's query bbox during
the latest 30-second window?'

Proves the wirings compose into a realistic pipeline shape (not just
work in isolation), each tier delivering its specific contract:
stateless filter is per-event, bounded-state persists handle state
across events as bytes, windowed aggregates window-close-only, cross-
stream interval-joins on shared key.

Stacks on PR MobilityDB#9; additive-only (1 new demo file). Locally compile-
verified: 146 .class files total (145 from PR MobilityDB#9 base + 1 new demo).

(cherry picked from commit d322b8c)
@estebanzimanyi estebanzimanyi changed the base branch from main to feat/flink-bounded-state-tier-wirings June 11, 2026 16:46
@estebanzimanyi

estebanzimanyi commented Jun 11, 2026

Copy link
Copy Markdown
Member Author

Superseded by the Path-B consolidation: the former 18-deep stack is collapsed into two reviewable topical PRs on top of the merged scaffold — MEOS integration #30 → benchmark #31 — each one clean squashed commit with the generated-facade bulk, dead family-flag profiles, committed target/ artifacts, and invented synthetic corpus removed. Closing as folded into #30/#31.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@estebanzimanyi