GH-5648 new benchmark module by hmottestad · Pull Request #5651 · eclipse-rdf4j/rdf4j

hmottestad · 2025-12-26T13:58:26Z

GitHub issue resolved: #

Briefly describe the changes proposed in this PR:

PR Author Checklist (see the contributor guidelines for more details):

my pull request is self-contained
I've added tests for the changes I made
I've applied code formatting (you can use mvn process-resources to format from the command line)
I've squashed my commits where necessary
every commit message starts with the issue number (GH-xxxx) followed by a meaningful description of the change

hmottestad · 2025-12-27T15:06:14Z

Memory Store


Benchmark                              (themeName)  (z_queryIndex)  Mode  Cnt      Score   Error  Units
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               0  avgt    2     16.047          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               1  avgt    2     56.388          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               2  avgt    2     20.214          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               3  avgt    2     37.450          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               4  avgt    2     30.360          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               5  avgt    2     23.120          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               6  avgt    2     24.906          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               7  avgt    2     19.098          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               8  avgt    2     24.154          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               9  avgt    2    102.066          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS              10  avgt    2     48.849          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               0  avgt    2      0.032          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               1  avgt    2      2.454          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               2  avgt    2      0.032          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               3  avgt    2      0.037          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               4  avgt    2      0.037          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               5  avgt    2    467.787          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               6  avgt    2      0.041          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               7  avgt    2      2.118          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               8  avgt    2    129.833          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               9  avgt    2      1.303          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA              10  avgt    2      0.768          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               0  avgt    2    181.356          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               1  avgt    2    105.210          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               2  avgt    2      8.578          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               3  avgt    2     25.425          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               4  avgt    2     50.724          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               5  avgt    2      4.604          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               6  avgt    2   7505.885          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               7  avgt    2  38323.112          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               8  avgt    2     10.281          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               9  avgt    2     19.339          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY              10  avgt    2    279.059          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               0  avgt    2     81.647          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               1  avgt    2    136.936          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               2  avgt    2      0.172          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               3  avgt    2     84.025          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               4  avgt    2     62.754          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               5  avgt    2      0.387          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               6  avgt    2    140.927          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               7  avgt    2      0.788          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               8  avgt    2      0.341          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               9  avgt    2      0.462          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING              10  avgt    2      0.335          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               0  avgt    2    171.116          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               1  avgt    2    587.085          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               2  avgt    2    172.318          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               3  avgt    2    103.961          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               4  avgt    2    123.950          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               5  avgt    2     76.114          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               6  avgt    2    800.341          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               7  avgt    2     93.165          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               8  avgt    2    314.682          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               9  avgt    2    779.763          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED              10  avgt    2  19750.979          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               0  avgt    2     16.480          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               1  avgt    2     27.165          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               2  avgt    2      1.051          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               3  avgt    2     71.859          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               4  avgt    2     81.394          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               5  avgt    2      6.166          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               6  avgt    2     32.009          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               7  avgt    2     24.418          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               8  avgt    2     39.142          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               9  avgt    2    131.986          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN              10  avgt    2    149.680          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               0  avgt    2     20.380          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               1  avgt    2     10.244          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               2  avgt    2      0.791          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               3  avgt    2    167.310          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               4  avgt    2      0.508          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               5  avgt    2      2.446          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               6  avgt    2     31.060          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               7  avgt    2     33.819          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               8  avgt    2      2.053          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               9  avgt    2      1.375          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID              10  avgt    2    189.281          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               0  avgt    2      0.083          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               1  avgt    2      0.355          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               2  avgt    2   1953.850          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               3  avgt    2      1.908          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               4  avgt    2      7.192          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               5  avgt    2      0.101          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               6  avgt    2      0.837          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               7  avgt    2      3.305          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               8  avgt    2      9.997          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               9  avgt    2      3.254          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA              10  avgt    2   2372.338          ms/op

LMDB Store

Benchmark                              (themeName)  (z_queryIndex)  Mode  Cnt       Score   Error  Units
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               0  avgt    2      51.256          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               1  avgt    2     186.645          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               2  avgt    2      50.005          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               3  avgt    2      98.563          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               4  avgt    2     121.535          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               5  avgt    2      64.928          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               6  avgt    2      61.191          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               7  avgt    2      51.656          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               8  avgt    2      58.814          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS               9  avgt    2     272.421          ms/op
ThemeQueryBenchmark.executeQuery   MEDICAL_RECORDS              10  avgt    2  293429.747          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               0  avgt    2       0.061          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               1  avgt    2       6.016          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               2  avgt    2       0.076          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               3  avgt    2       0.069          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               4  avgt    2       0.087          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               5  avgt    2     861.699          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               6  avgt    2       0.088          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               7  avgt    2       5.986          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               8  avgt    2     737.210          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA               9  avgt    2       5.751          ms/op
ThemeQueryBenchmark.executeQuery      SOCIAL_MEDIA              10  avgt    2       2.608          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               0  avgt    2     665.671          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               1  avgt    2     254.135          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               2  avgt    2      36.678          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               3  avgt    2      42.249          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               4  avgt    2     120.662          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               5  avgt    2       9.535          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               6  avgt    2   23784.702          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               7  avgt    2   91930.684          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               8  avgt    2      74.209          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY               9  avgt    2     151.046          ms/op
ThemeQueryBenchmark.executeQuery           LIBRARY              10  avgt    2     183.888          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               0  avgt    2     233.185          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               1  avgt    2     303.684          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               2  avgt    2       1.289          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               3  avgt    2     150.705          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               4  avgt    2      60.455          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               5  avgt    2       2.357          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               6  avgt    2     248.904          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               7  avgt    2       4.436          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               8  avgt    2       2.528          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING               9  avgt    2       3.786          ms/op
ThemeQueryBenchmark.executeQuery       ENGINEERING              10  avgt    2       1.922          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               0  avgt    2     323.203          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               1  avgt    2    1165.947          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               2  avgt    2     548.036          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               3  avgt    2      99.026          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               4  avgt    2     248.351          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               5  avgt    2     109.557          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               6  avgt    2    1463.763          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               7  avgt    2     152.188          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               8  avgt    2    1170.030          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED               9  avgt    2    1481.964          ms/op
ThemeQueryBenchmark.executeQuery  HIGHLY_CONNECTED              10  avgt    2  126334.595          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               0  avgt    2      37.937          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               1  avgt    2      86.435          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               2  avgt    2       8.738          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               3  avgt    2     162.916          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               4  avgt    2     138.137          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               5  avgt    2      26.519          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               6  avgt    2      97.552          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               7  avgt    2      51.524          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               8  avgt    2     269.601          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN               9  avgt    2     262.040          ms/op
ThemeQueryBenchmark.executeQuery             TRAIN              10  avgt    2     226.000          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               0  avgt    2      42.103          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               1  avgt    2      87.022          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               2  avgt    2       4.939          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               3  avgt    2     328.922          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               4  avgt    2       4.621          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               5  avgt    2      11.560          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               6  avgt    2      95.600          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               7  avgt    2      19.453          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               8  avgt    2      13.925          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID               9  avgt    2       5.615          ms/op
ThemeQueryBenchmark.executeQuery   ELECTRICAL_GRID              10  avgt    2     524.353          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               0  avgt    2       0.303          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               1  avgt    2       1.736          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               2  avgt    2      39.557          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               3  avgt    2      14.988          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               4  avgt    2      31.892          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               5  avgt    2       0.408          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               6  avgt    2       5.374          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               7  avgt    2      22.562          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               8  avgt    2      32.379          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA               9  avgt    2      18.538          ms/op
ThemeQueryBenchmark.executeQuery            PHARMA              10  avgt    2   13561.666          ms/op

hmottestad · 2025-12-27T16:30:29Z

Comparison of benchmark results between the MemorStore and the LMDB Store

memory_vs_lmdb_benchmark_comparison.numbers-Comparison.pdf

hmottestad added 30 commits December 26, 2025 13:54

create module

2f07228

create module

5e29093

move common large benchmark files to the common module

709b62a

clean up shared queries

9b1a8fd

data generator

8776c68

data generator

57b4bfb

data generator

e424986

data generator

6047e81

data generator

b450052

data generator

a86783e

data generator

457e16c

data generator

4be1d41

data generator

ccd8078

data generator

758b038

data generator

a35a6fb

data generator

28345f7

data generator

99e3be6

data generator

761c824

data generator

e4d2a0c

data generator

86c1f82

data generator

1966287

data generator

106aedb

data generator

25459e4

data generator

6d4768e

data generator

e82ee7b

data generator

66d0869

data generator

79b2d1e

fix copyright headers

05ce4d1

wip

804de96

wip

37273c5

hmottestad added 5 commits December 27, 2025 13:15

wip

266d70f

wip

8dced8e

wip

aa10e08

wip

f564a69

wip

077f808

wip

5908edd

hmottestad enabled auto-merge (squash) December 27, 2025 16:32

hmottestad merged commit a38de98 into main Dec 27, 2025
11 checks passed

hmottestad deleted the GH-5648-new-benchmark-module branch December 27, 2025 16:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-5648 new benchmark module#5651

GH-5648 new benchmark module#5651
hmottestad merged 36 commits intomainfrom
GH-5648-new-benchmark-module

hmottestad commented Dec 26, 2025

Uh oh!

hmottestad commented Dec 27, 2025 •

edited

Loading

Uh oh!

hmottestad commented Dec 27, 2025 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

hmottestad commented Dec 26, 2025

Uh oh!

hmottestad commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Memory Store

LMDB Store

Uh oh!

hmottestad commented Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

hmottestad commented Dec 27, 2025 •

edited

Loading

hmottestad commented Dec 27, 2025 •

edited

Loading