Skip to content

Commit 32bfbc7

Browse files
feat: document fully_consistent nuances on CockroachDB (#529)
Co-authored-by: Maria Ines Parnisari <maineparnisari@gmail.com>
1 parent 4dfda76 commit 32bfbc7

2 files changed

Lines changed: 35 additions & 1 deletion

File tree

app/spicedb/concepts/consistency/page.mdx

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,9 +99,23 @@ Note that the snapshot used will be loaded at the beginning of the API call, and
9999

100100
<Callout type="warning">
101101
This consistency mode explicitly bypasses caching, dramatically impacting latency.
102+
If you need read-after-write consistency, consider using a [ZedToken].
102103

103-
If you need read-after-write consistency, consider using a [ZedToken]
104+
**CockroachDB users:** `fully_consistent` does not guarantee read-after-write consistency on CockroachDB.
104105

106+
At request time, SpiceDB directs the query towards one node in the CockroachDB cluster, and then runs [`cluster_logical_timestamp()`](https://www.cockroachlabs.com/docs/stable/functions-and-operators#system-info-functions).
107+
However, because CockroachDB is a distributed database, this timestamp can differ from other nodes by up to the cluster's configured [`max_offset`][max-offset] (default: 500ms).
108+
If the node handling the read picks a timestamp older than the one assigned to a recent write on a different node, the read will not see that write even though the data has been committed.
109+
110+
This does not affect CockroachDB's normal read-after-write guarantees -- those are bypassed specifically because SpiceDB chooses its own evaluation revision rather than letting CockroachDB select one.
111+
112+
The [overlap strategy][overlap-strategy] does not mitigate this.
113+
The overlap strategy prevents two concurrent writes from receiving commit timestamps in the reverse order, but it does not address clock skew between nodes during reads.
114+
115+
If you need read-after-write consistency with CockroachDB, use a [ZedToken] with [`at_least_as_fresh`](#at-least-as-fresh) instead.
116+
117+
[max-offset]: https://www.cockroachlabs.com/docs/stable/cockroach-start.html#flags-max-offset
118+
[overlap-strategy]: /spicedb/concepts/datastores#overlap-strategy
105119
[ZedToken]: #zedtokens
106120

107121
</Callout>

app/spicedb/concepts/datastores/page.mdx

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,26 @@ AuthZed has standardized our managed services on CockroachDB, but we give self-h
3030
- Query and data balanced across the CockroachDB
3131
- Setup and operational complexity of running CockroachDB
3232

33+
### Consistency Considerations
34+
35+
<Callout type="warning">
36+
`fully_consistent` does not guarantee read-after-write consistency on CockroachDB.
37+
38+
SpiceDB picks a revision using CockroachDB's `cluster_logical_timestamp()`, which can differ across nodes by up to the cluster's [`max_offset`][crdb-max-offset] (default: 500ms).
39+
A read may not see a recently committed write if the reading node's timestamp is behind the writing node's.
40+
41+
This is specific to CockroachDB's distributed clock model and does not apply to single-node datastores like PostgreSQL.
42+
For read-after-write guarantees, use a [ZedToken] with [`at_least_as_fresh`][at-least-as-fresh].
43+
44+
See [Consistency: Fully Consistent][fully-consistent] for details.
45+
46+
[crdb-max-offset]: https://www.cockroachlabs.com/docs/stable/cockroach-start.html#flags-max-offset
47+
[ZedToken]: /spicedb/concepts/consistency#zedtokens
48+
[at-least-as-fresh]: /spicedb/concepts/consistency#at-least-as-fresh
49+
[fully-consistent]: /spicedb/concepts/consistency#fully-consistent
50+
51+
</Callout>
52+
3353
### Developer Notes
3454

3555
- Code can be found [here][crdb-code]

0 commit comments

Comments
 (0)