You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* split all API into cleanly defined "block" api and "variable length"
api. The API forces the input to be blocks of constant size, at compile
time, and for the codec to actually match that.
* Made code shorter by a thousand lines
* Identical C++ and Rust usage
* Introduce a new Composite codec that requires a generic `BlockCodec`
and `AnyLenCodec`
```rust
pub trait BlockCodec {
/// The fixed-size block type. Must be plain-old-data (`Pod`).
/// In practice this will be `[u32; 128]` or `[u32; 256]`.
type Block: Pod;
/// Compress a slice of complete, fixed-size blocks.
///
/// No remainder is possible — the caller must split the input first using
/// [`slice_to_blocks`] and handle any remainder separately.
fn encode_blocks(
&mut self,
blocks: &[Self::Block],
out: &mut Vec<u32>,
) -> Result<(), FastPForError>;
/// Decompress exactly `n_blocks` blocks from `input`.
fn decode_blocks(
&mut self,
input: &[u32],
n_blocks: usize,
out: &mut Vec<u32>,
) -> Result<(), FastPForError>;
}
/// Compresses and decompresses an arbitrary-length `&[u32]` slice.
pub trait AnyLenCodec {
/// Compress an arbitrary-length slice of `u32` values.
fn encode(&mut self, input: &[u32], out: &mut Vec<u32>) -> Result<(), FastPForError>;
/// Decompress a previously compressed slice of `u32` values.
fn decode(&mut self, input: &[u32], out: &mut Vec<u32>) -> Result<(), FastPForError>;
}
/// Split a flat `&[u32]` into `(&[Blocks::Block], &[u32])` without copying.
///
/// # Example
///
/// ```rust,ignore
/// let data: Vec<u32> = (0..600).collect(); // 2 × 256 + 88 remainder
/// let (blocks, remainder) = slice_to_blocks::<FastPFor256>(&data);
/// assert_eq!(blocks.len(), 2); // 2 blocks of [u32; 256]
/// assert_eq!(remainder.len(), 88);
/// ```
#[must_use]
pub fn slice_to_blocks<Blocks: BlockCodec>(input: &[u32]) -> (&[Blocks::Block], &[u32]) { ... }
/// Combines a block-oriented codec with an arbitrary-length tail codec.
///
/// `CompositeCodec<Blocks, Tail>` implements [`AnyLenCodec`]: it accepts any
/// input length, encodes the aligned prefix with `Blocks`, and the
/// sub-block remainder with `Tail`.
///
/// # Wire format
///
/// ```text
/// [ n_blocks: u32 ] [ Blocks encoded data... ] [ Tail encoded data... ]
/// ```
///
/// # Example
///
/// ```rust,ignore
/// use fastpfor::{AnyLenCodec, FastPFor256WithVByte};
///
/// let data: Vec<u32> = (0..600).collect(); // 2 × 256 + 88 remainder
/// let codec = FastPFor256WithVByte::default();
///
/// let mut encoded = Vec::new();
/// codec.encode(&data, &mut encoded).unwrap();
///
/// let mut decoded = Vec::new();
/// codec.decode(&encoded, &mut decoded).unwrap();
/// assert_eq!(decoded, data);
/// ```
pub struct CompositeCodec<Blocks: BlockCodec, Tail: AnyLenCodec> {...}
```
This is a Rust wrapper for the [C++ FastPFor library](https://github.com/fast-pack/FastPFor), as well as a pure Rust re-implementation. Supports 32-bit and 64-bit integers, and SIMD-optimized codecs for 128-bit and 256-bit vectors. Based on the [Decoding billions of integers per second through vectorization, 2012](https://arxiv.org/abs/1209.2137) paper.
11
+
Fast integer compression for Rust — both a pure-Rust implementation and a wrapper around the [C++ FastPFor library](https://github.com/fast-pack/FastPFor).
12
+
Supports 32-bit (and for some codecs 64-bit) integers.
13
+
Based on the [Decoding billions of integers per second through vectorization, 2012](https://arxiv.org/abs/1209.2137) paper.
12
14
13
15
The Rust **decoder** is about 29% faster than the C++ version. The Rust implementation contains no `unsafe` code, and when built without the `cpp` feature this crate has `#![forbid(unsafe_code)]`.
14
16
15
-
### Supported algorithms
16
-
Unless otherwise specified, all codecs support `&[u32]` only.
17
-
18
-
```text
19
-
* BP32
20
-
* Copy
21
-
* FastBinaryPacking16
22
-
* FastBinaryPacking32
23
-
* FastBinaryPacking8
24
-
* FastPFor128 (both `&[u32]` and `&[u64]`)
25
-
* FastPFor256 (both `&[u32]` and `&[u64]`)
26
-
* MaskedVByte
27
-
* NewPFor
28
-
* OptPFor
29
-
* PFor
30
-
* PFor2008
31
-
* SimdBinaryPacking
32
-
* SimdFastPFor128
33
-
* SimdFastPFor256
34
-
* SimdGroupSimple
35
-
* SimdGroupSimpleRingBuf
36
-
* SimdNewPFor
37
-
* SimdOptPFor
38
-
* SimdPFor
39
-
* SimdSimplePFor
40
-
* Simple16
41
-
* Simple8b
42
-
* Simple8bRle
43
-
* Simple9
44
-
* Simple9Rle
45
-
* SimplePFor
46
-
* StreamVByte
47
-
* VByte
48
-
* VarInt (both `&[u32]` and `&[u64]`)
49
-
* VarIntGb
17
+
## Usage
18
+
19
+
### Rust Implementation (default)
20
+
21
+
The simplest way is `FastPFor256` — a composite codec that handles any input
22
+
length by compressing aligned 256-element blocks with `FastPForBlock256` and encoding any
|`rust`|**yes**| Pure-Rust implementation — no `unsafe`, no build dependencies |
81
+
|`cpp`| no | C++ wrapper via CXX — requires a C++14 compiler with SIMD support |
82
+
|`cpp_portable`| no | Enables `cpp`, compiles C++ with SSE4.2 baseline (runs on any x86-64 from ~2008+) |
83
+
|`cpp_native`| no | Enables `cpp`, compiles C++ with `-march=native` for maximum throughput on the build machine |
84
+
85
+
The `FASTPFOR_SIMD_MODE` environment variable (`portable` or `native`) can override the SIMD mode at build time.
86
+
87
+
**Recommendation:** Use `cpp_portable` (not `cpp_native`) for distributable binaries.
88
+
89
+
## Supported Algorithms
90
+
91
+
### Rust (`rust` feature)
92
+
93
+
Rust block codecs require block-aligned input. `CompositeCodec` chains a block codec with a tail codec (e.g. `VariableByte`) to handle arbitrary-length input. `FastPFor256` and `FastPFor128` are type aliases for such composites.
Rust Encoding has not yet been either optimized or even fully verified.
71
-
72
-
## Usage
73
-
74
-
### Crate Features
75
-
*`cpp` - C++ implementation (uses portable SIMD mode)
76
-
*`rust` - Rust implementation (safe Rust code, no `unsafe` blocks)
77
-
78
-
#### SIMD Mode Configuration
79
-
80
-
The C++ backend can be compiled with different SIMD instruction sets. Control this by enabling one of these features:
81
-
| Mode | Description |
82
-
|------|-------------|
83
-
|`cpp_portable`|**Default.** Uses SSE4.2 baseline only. Binaries run on any x86-64 CPU from ~2008+. Best for distributable libraries. |
84
-
|`cpp_native`| Uses `-march=native` to enable all SIMD instructions supported by the build machine (AVX, AVX2, etc.). Maximum performance but may crash on CPUs lacking those instructions. |
85
-
86
-
Feature selection can be overridden with the `FASTPFOR_SIMD_MODE` environment variable set to "portable" or "native".
87
-
88
-
**Recommendation:** Use `portable` (default) for libraries and distributed binaries. Use `native` only when building for a specific machine where you need maximum performance.
89
-
90
-
### Using C++ Wrapper
91
-
92
-
```rust
93
-
usefastpfor::AnyLenCodecas _;
94
-
usefastpfor::cpp::CppSimdFastPFor128;
95
-
96
-
fnmain() {
97
-
letmutcodec=CppSimdFastPFor128::new();
98
-
99
-
letinput=vec![1u32, 2, 3, 4, 5];
100
-
letmutcompressed=Vec::new();
101
-
codec.encode(&input, &mutcompressed).unwrap();
102
-
103
-
letmutdecoded=Vec::new();
104
-
codec
105
-
.decode(&compressed, &mutdecoded, None)
106
-
.unwrap();
107
-
108
-
assert_eq!(input, decoded);
109
-
}
110
-
```
162
+
Rust encoding has not yet been fully optimized or verified.
111
163
112
164
## Build Requirements
113
165
114
-
- When using the **Rust implementation**:
115
-
no additional dependencies are required.
116
-
- When using the **C++ implementation**:
117
-
you need to have a C++ compiler that supports C++14 and SIMD intrinsics.
166
+
-**Rust feature** (`rust`, the default): no additional dependencies.
167
+
-**C++ feature** (`cpp`): requires a C++14-capable compiler with SIMD intrinsics.
118
168
See [FastPFor C++ requirements](https://github.com/fast-pack/FastPFor?tab=readme-ov-file#software-requirements).
119
169
120
170
### Linux
121
171
122
-
The default GitHub action runner for Linux has all the needed dependencies.
172
+
The default GitHub Actions runner has all needed dependencies.
123
173
124
-
For local development, you may need to install the following packages:
174
+
For local development:
125
175
126
176
```bash
127
177
# This list may be incomplete
128
178
sudo apt-get install build-essential
129
179
```
130
180
131
-
`libsimde-dev` is optional. On ARM/aarch64, the C++ build fetches `SIMDe` via `CMake`,
132
-
and the Rust CXX bridge now reuses that fetched include path automatically.
133
-
Install `libsimde-dev` only if you prefer a system package fallback.
181
+
`libsimde-dev` is optional. On ARM/aarch64, the C++ build fetches `SIMDe` via `CMake`
182
+
and the CXX bridge reuses that include path automatically.
134
183
135
184
### macOS
136
-
On Apple Silicon, manual `SIMDe` installation is usually not required.
137
-
The C++ build fetches `SIMDe` via `CMake`, and the Rust CXX bridge reuses that path.
138
185
139
-
If you prefer a system package fallback, install `SIMDe` with Homebrew and set include flags.
186
+
On Apple Silicon, `SIMDe` installation is usually not required — the C++ build fetches it via `CMake`.
187
+
188
+
If you prefer a Homebrew fallback:
140
189
141
190
```bash
142
-
# optional: install SIMDe via Homebrew
143
191
brew install simde
144
-
145
-
# optional fallback: ensure the compiler can find Homebrew headers
146
192
export CXXFLAGS="-I/opt/homebrew/include"
147
193
export CFLAGS="-I/opt/homebrew/include"
148
194
```
149
195
150
196
## Development
151
197
152
-
* This project is easier to develop with [just](https://github.com/casey/just#readme), a modern alternative to `make`.
153
-
Install it with `cargo install just`.
154
-
* To get a list of available commands, run `just`.
155
-
* To run tests, use `just test`.
198
+
This project uses [just](https://github.com/casey/just#readme) as a task runner:
199
+
200
+
```bash
201
+
cargo install just # install once
202
+
just # list available commands
203
+
just test# run all tests
204
+
```
156
205
157
206
## License
158
207
159
208
Licensed under either of
160
209
161
210
* Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or <https://www.apache.org/licenses/LICENSE-2.0>)
162
211
* MIT license ([LICENSE-MIT](LICENSE-MIT) or <https://opensource.org/licenses/MIT>)
0 commit comments