Unstable benchmark results

[pekeler]

I've been getting fluctuating results running the benchmark on my laptop. So I set up a new server on Hetzner (CCX13 - 2 dedicated vCPU, 8GB) in hopes of more reliable measurements and ran the benchmark 3 times against v0.36.2 with nothing else running on that machine. But it's as bad as running it locally. These are the 4 most fluctuating results:

Scenario	Category	Run1	Run2	Run3	CV	max/min
posts10k - author.id (extra join) check (with index) [reqs:100, conc:10, rule:"author.id = @request.auth.id", query:?perPage=20]	List records	175.706ms	190.224ms	81.707ms	32.24%	2.328x
posts10k - author check (no index) [reqs:100, conc:10, rule:"author = @request.auth.id", query:?perPage=20]	List records	1.026s	583.606ms	1.133s	26.01%	1.941x
posts100k - nested multiple relations lookup (no indexes, at-least-one, skipTotal) [reqs:100, conc:10, rule:"author.permissions.active ?= true", query:?perPage=20&skipTotal=1]	List records	497.643ms	531.385ms	307.014ms	22.18%	1.731x
posts50k - simpleA (many requests, no rules, no concurrency) [reqs:1000, conc:1, rule:"", query:?perPage=20]	List records	6.002s	9.156s	9.019s	18.06%	1.526x

Any idea what might cause this?

[ganigeorgiev]

Suggestions for improvements are welcomed but I'm not sure how this can be "fixed" easily.

The execution of the benchmark tests, especially high level ones as here, could depend on many factors - OS, work that other running processes do, CPU throttling, disk operations, db/connections/runtime cache, etc.

Keep in mind also that the benchmark tests themselves run as part of the same PocketBase process, aka. they could introduce a slight overhead because they have impact on the Go GC and runtime heuristics.

While we can probably produce a more isolated benchmark suite and do some fine tuning to squeeze the best performance possible, that's not really the goal here and I don't want to "game" the benchmarks.

The current benchmark tests are intended to showcase a plain PocketBase setup with minimal extra configuration to give a general idea what people could expect from their PocketBase app out of the box. Even in its non-optimized form the numbers are already good enough for the applications PocketBase is intended for and it is better to be realistic than presenting some heavily optimized numbers that rarely could be reproduced in production.