Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Changelog

## Unreleased
- Added `CompactVector::to_bytes` and `from_bytes` for zero-copy serialization.
- Made `DacsByte` generic over its flag index type with a default of `Rank9SelIndex`.
- `DacsByte::from_slice` now accepts a generic index type, removing `from_slice_with_index`.
- Added `BitVectorBuilder` and zero-copy `BitVectorData` backed by `anybytes::View`.
Expand Down
1 change: 1 addition & 0 deletions INVENTORY.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
- Investigate alternative dense-select index strategies to replace removed `DArrayIndex`.
- Explore additional index implementations leveraging the new generic `DacsByte<I>`.
- Demonstrate the generic `from_slice` usage in examples and docs.
- Show `CompactVector::to_bytes` and `from_bytes` in examples.

## Discovered Issues
- `katex.html` performs manual string replacements; consider DOM-based manipulation.
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@ is backed by `anybytes::View`. The data can be serialized with
`BitVectorData::to_bytes` and reconstructed using `BitVectorData::from_bytes`,
allowing zero-copy loading from an mmap or any other source by passing the
byte region to `Bytes::from_source`.
`CompactVector` offers similar helpers: `CompactVector::to_bytes` returns a
metadata struct along with the raw bytes, and `CompactVector::from_bytes`
reconstructs the vector from that information.

## Examples

Expand Down
45 changes: 44 additions & 1 deletion src/int_vectors/compact_vector.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,10 @@ use anyhow::{anyhow, Result};
use num_traits::ToPrimitive;

use crate::bit_vector::BitVectorBuilder;
use crate::bit_vector::{BitVector, NoIndex};
use crate::bit_vector::{BitVector, BitVectorData, NoIndex};
use crate::int_vectors::prelude::*;
use crate::utils;
use anybytes::Bytes;

/// Mutable builder for [`CompactVector`].
///
Expand Down Expand Up @@ -184,6 +185,16 @@ impl Default for CompactVector {
}
}

/// Metadata returned by [`CompactVector::to_bytes`] and required by
/// [`CompactVector::from_bytes`].
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct CompactVectorMeta {
/// Number of integers stored.
pub len: usize,
/// Bit width for each integer.
pub width: usize,
}

impl CompactVector {
/// Creates a new empty builder storing integers within `width` bits each.
///
Expand Down Expand Up @@ -411,6 +422,30 @@ impl CompactVector {
pub const fn width(&self) -> usize {
self.width
}

/// Serializes the vector into a [`Bytes`] buffer and accompanying metadata.
pub fn to_bytes(&self) -> (CompactVectorMeta, Bytes) {
let (_, bytes) = self.chunks.data.to_bytes();
(
CompactVectorMeta {
len: self.len,
width: self.width,
},
bytes,
)
}

/// Reconstructs the vector from zero-copy [`Bytes`] and its metadata.
pub fn from_bytes(meta: CompactVectorMeta, bytes: Bytes) -> Result<Self> {
let data_len = meta.len * meta.width;
let data = BitVectorData::from_bytes(data_len, bytes)?;
let chunks = BitVector::new(data, NoIndex);
Ok(Self {
chunks,
len: meta.len,
width: meta.width,
})
}
}

impl Build for CompactVector {
Expand Down Expand Up @@ -656,4 +691,12 @@ mod tests {
let cv = CompactVector::from_slice(&[1, 2, 3]).unwrap();
assert_eq!(cv.to_vec(), vec![1, 2, 3]);
}

#[test]
fn from_bytes_roundtrip() {
let cv = CompactVector::from_slice(&[4, 5, 6]).unwrap();
let (meta, bytes) = cv.to_bytes();
let other = CompactVector::from_bytes(meta, bytes).unwrap();
assert_eq!(cv, other);
}
}