Add duckdb.MemoryUsage JFR periodic event for non-blocking per-instance memory sampling

[arouel]

Why

Sampling duckdb_memory() on an active connection blocks for the duration of any in-flight query, so getting a memory snapshot at an arbitrary moment requires either waiting for the query to finish or holding a spare connection open just for diagnostics. In a process running multiple DuckDB instances there is a second problem: duckdb_memory() returns a global per-tag snapshot with no way to attribute usage to a specific instance. A periodic JFR event backed by a dedicated monitor connection solves both: sampling never interferes with application queries, and each event carries dbAddress (the native instance pointer) as a stable attribution key.

What

Adds a duckdb.MemoryUsage JFR periodic event, opt-in per connection via the jdbc_jfr_memory_monitor JDBC property, that samples duckdb_memory() on a dedicated side-channel connection per database instance and emits one event per memory tag per JFR tick.

[staticlibs]

Hi, thanks for the PR! To pass the formatter please run:

python -m pip install --user clang_format==11.0.1
make format

Or just apply the following diff:

diff --git a/src/main/java/org/duckdb/DuckDBConnection.java b/src/main/java/org/duckdb/DuckDBConnection.java
index 24f0105d..fd48e531 100644
--- a/src/main/java/org/duckdb/DuckDBConnection.java
+++ b/src/main/java/org/duckdb/DuckDBConnection.java
@@ -72,8 +72,8 @@ public final class DuckDBConnection implements java.sql.Connection {
         return newConnection(url, readOnly, sessionInitSQL, properties, null);
     }
 
-    public static DuckDBConnection newConnection(String url, boolean readOnly, String sessionInitSQL, Properties properties,
-                                                 String monitorName) throws SQLException {
+    public static DuckDBConnection newConnection(String url, boolean readOnly, String sessionInitSQL,
+                                                 Properties properties, String monitorName) throws SQLException {
         if (null == properties) {
             properties = new Properties();
         }
diff --git a/src/main/java/org/duckdb/DuckDBDriver.java b/src/main/java/org/duckdb/DuckDBDriver.java
index 1883b19b..771cfcf2 100644
--- a/src/main/java/org/duckdb/DuckDBDriver.java
+++ b/src/main/java/org/duckdb/DuckDBDriver.java
@@ -140,7 +140,8 @@ public class DuckDBDriver implements java.sql.Driver {
         String monitorName = removeOption(props, JDBC_JFR_MEMORY_MONITOR);
 
         // Create connection
-        DuckDBConnection conn = DuckDBConnection.newConnection(pp.shortUrl, readOnly, sf.origFileText, props, monitorName);
+        DuckDBConnection conn =
+            DuckDBConnection.newConnection(pp.shortUrl, readOnly, sf.origFileText, props, monitorName);
 
         // Run post-init
         try {
@@ -177,8 +178,9 @@ public class DuckDBDriver implements java.sql.Driver {
                                       "Do not close the DB instance after all connections to it are closed"));
         list.add(createDriverPropInfo(JDBC_IGNORE_UNSUPPORTED_OPTIONS, "",
                                       "Silently discard unsupported connection options"));
-        list.add(createDriverPropInfo(JDBC_JFR_MEMORY_MONITOR, "",
-                                      "User-assigned identifier under which this connection's DuckDB instance is tracked in the duckdb.MemoryUsage JFR event. Leave empty to disable monitoring. JFR controls the event's enabled state and period via recording settings. Requires a JFR-capable JVM."));
+        list.add(createDriverPropInfo(
+            JDBC_JFR_MEMORY_MONITOR, "",
+            "User-assigned identifier under which this connection's DuckDB instance is tracked in the duckdb.MemoryUsage JFR event. Leave empty to disable monitoring. JFR controls the event's enabled state and period via recording settings. Requires a JFR-capable JVM."));
         list.sort((o1, o2) -> o1.name.compareToIgnoreCase(o2.name));
         return list.toArray(new DriverPropertyInfo[0]);
     }
diff --git a/src/main/java/org/duckdb/DuckDBMemoryEvent.java b/src/main/java/org/duckdb/DuckDBMemoryEvent.java
index a2d00383..7577b560 100644
--- a/src/main/java/org/duckdb/DuckDBMemoryEvent.java
+++ b/src/main/java/org/duckdb/DuckDBMemoryEvent.java
@@ -39,7 +39,8 @@ import jdk.jfr.StackTrace;
 public class DuckDBMemoryEvent extends Event {
 
     @Label("Name")
-    @Description("User-assigned identifier of the DuckDB instance (value of the jdbc_jfr_memory_monitor connection property)")
+    @Description(
+        "User-assigned identifier of the DuckDB instance (value of the jdbc_jfr_memory_monitor connection property)")
     String name;
 
     @Label("Tag")
@@ -54,9 +55,7 @@ public class DuckDBMemoryEvent extends Event {
     @Description("Native address of the underlying DuckDB instance; disambiguates databases when names collide")
     long dbAddress;
 
-    @Label("Memory Usage")
-    @Description("Bytes currently allocated for this tag")
-    long memoryUsageBytes;
+    @Label("Memory Usage") @Description("Bytes currently allocated for this tag") long memoryUsageBytes;
 
     @Label("Temporary Storage Usage")
     @Description("Bytes spilled to the temporary storage for this tag")
diff --git a/src/main/java/org/duckdb/DuckDBMemoryMonitor.java b/src/main/java/org/duckdb/DuckDBMemoryMonitor.java
index 325013d5..b64b23d2 100644
--- a/src/main/java/org/duckdb/DuckDBMemoryMonitor.java
+++ b/src/main/java/org/duckdb/DuckDBMemoryMonitor.java
@@ -51,7 +51,8 @@ final class DuckDBMemoryMonitor {
     private static boolean initialized;
 
     // Non-instantiable
-    private DuckDBMemoryMonitor() {}
+    private DuckDBMemoryMonitor() {
+    }
 
     /**
      * Registers the periodic JFR hook for {@link DuckDBMemoryEvent}. Idempotent
@@ -152,7 +153,8 @@ final class DuckDBMemoryMonitor {
                     // Publish monitorConn last so readers see fully-populated state.
                     monitorConn = mc;
                 } catch (SQLException e) {
-                    logger.log(Level.WARNING, "Failed to open JFR memory-monitor connection; will retry on next open()", e);
+                    logger.log(Level.WARNING, "Failed to open JFR memory-monitor connection; will retry on next open()",
+                               e);
                 }
             }
             openConnections++;
@@ -196,8 +198,7 @@ final class DuckDBMemoryMonitor {
             String nameSnap = name;
             String url = dbUrl;
             long addr = dbAddress;
-            try (Statement stmt = mc.createStatement();
-                 ResultSet rs = stmt.executeQuery(QUERY)) {
+            try (Statement stmt = mc.createStatement(); ResultSet rs = stmt.executeQuery(QUERY)) {
                 while (rs.next()) {
                     String tag = rs.getString(1);
                     long memoryUsageBytes = rs.getLong(2);
diff --git a/src/main/java/org/duckdb/JfrMemoryMonitor.java b/src/main/java/org/duckdb/JfrMemoryMonitor.java
index 600f864d..727825ee 100644
--- a/src/main/java/org/duckdb/JfrMemoryMonitor.java
+++ b/src/main/java/org/duckdb/JfrMemoryMonitor.java
@@ -45,7 +45,8 @@ final class JfrMemoryMonitor {
         CONNECTION_CLOSED = closed;
     }
 
-    private JfrMemoryMonitor() {}
+    private JfrMemoryMonitor() {
+    }
 
     static void init() {
         if (INIT == null) {

[staticlibs]

@arouel

Use BufferManager::GetMemoryUsageInfo() instead of a duckdb_memory()

Sorry, we do not want to introduce any new usage of DuckDB C++ API. Currently only a few things are missing in C API to move the whole JDBC driver onto it. Can we continue using duckdb_memory() instead?

[arouel]

@arouel

Use BufferManager::GetMemoryUsageInfo() instead of a duckdb_memory()

Sorry, we do not want to introduce any new usage of DuckDB C++ API. Currently only a few things are missing in C API to move the whole JDBC driver onto it. Can we continue using duckdb_memory() instead?

@staticlibs I made the changes to use duckdb_memory() instead. Any feedback from you is appreciated.

[staticlibs]

Thanks for the update! I will be able to review the changes at the beginning of the next week, with the intention to include this into 1.5.3 planned for May 18.