MDEV-39323: CROSS JOIN query returns wrong result with NOT BETWEEN

[DaveGosselin-MariaDB]

For 1 NOT BETWEEN c0 AND c2 with c0 float and c2 int, the range optimizer was building a SEL_TREE for only c0 > 1 and dropping the c2 < 1 disjunct. can_optimize_range_const rejected c2 because of a type mismatch.

The case from the ticket returned no rows (instead of the expected six rows) because the index range scan on c0 fetched only matches for c0 > 1 but then they were discarded by the t1.c1 = t1.c2 predicate.

Plan before the fix (range scan on c0, key_len 5):

  +----+-------+-------+------+---------+------+
  | id | table | type  | key  | key_len | rows |
  +----+-------+-------+------+---------+------+
  |  1 | t1    | range | c0   |       5 |    1 |
  |  1 | t2    | index | c0   |       6 |    3 |
  +----+-------+-------+------+---------+------+

Plan after the fix (Item_func_between::get_mm_tree returns no usable range tree, so the optimizer falls back to using the covering index scan on i0):

  +----+-------+-------+------+---------+------+
  | id | table | type  | key  | key_len | rows |
  +----+-------+-------+------+---------+------+
  |  1 | t2    | index | c0   |       6 |    6 |
  |  1 | t1    | index | i0   |      14 |    8 |
  +----+-------+-------+------+---------+------+

In Item_func_between::get_mm_tree, when can_optimize_range_const fails for a range bound argument and the predicate is negated, we now abandon the whole tree instead of continuing the loop. The new code mirrors the existing guard in the same function for non FIELD_ITEM arguments.

The bug was introduced by MDEV-36235 which added
can_optimize_range_const without considering the negated path. While we could fine tune the rejection behavior in can_optimize_range_const, I think that should be a separate patch as it requires careful consideration for each pair of types that could be supplied to [NOT] BETWEEN and that's outside the scope of this ticket (which is intended to fix the wrong result issue).

This behavior matches MySQL 9.

[gemini-code-assist]

Code Review

This pull request addresses MDEV-39323, which caused incorrect results in queries using NOT BETWEEN with mismatched argument types. The fix involves updating Item_func_between::get_mm_tree to ensure that range optimization is aborted if a disjunct cannot be properly constructed for a negated BETWEEN expression, preventing the generation of an incomplete range. New test cases covering these scenarios have been added to the test suite. I have no feedback to provide as no review comments were submitted.