Skip to content

Commit fa7fb19

Browse files
Cap DuckDB memory during parquet compaction to 2 GB (#758) (#762)
The in-memory DuckDB connection used for archive compaction was using the default memory_limit (80% of physical RAM), causing multi-GB spikes when decompressing large parquet archives. With 800 MB of compressed archives, this easily hits 5-7 GB. Set memory_limit=2GB so DuckDB spills to disk instead of consuming all available RAM. Also set preserve_insertion_order=false to reduce memory pressure since compaction has no meaningful row order. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 8d682ee commit fa7fb19

1 file changed

Lines changed: 8 additions & 0 deletions

File tree

Lite/Services/ArchiveService.cs

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -309,6 +309,14 @@ private void CompactParquetFiles()
309309
using var con = new DuckDBConnection("DataSource=:memory:");
310310
con.Open();
311311

312+
/* Cap memory to avoid multi-GB spikes decompressing large parquet archives.
313+
DuckDB will spill excess to its temp directory automatically. */
314+
using (var pragma = con.CreateCommand())
315+
{
316+
pragma.CommandText = "SET memory_limit = '2GB'; SET preserve_insertion_order = false;";
317+
pragma.ExecuteNonQuery();
318+
}
319+
312320
var totalMerged = 0;
313321
var totalRemoved = 0;
314322

0 commit comments

Comments
 (0)