Benchmark `xan count --parallel` and `xan parallel cat -P 'select'`

[Yomguithereal]

Hi,

Thanks for the interesting benchmark. It seems it matches what I could observe on my own. Still I think you could add two things to it to have a broader picture, as xan also knows how to parallelize its computations:

• You can try xan count --parallel, for the count bench
• You can try xan parallel cat -P 'select <columns>' for the select bench

It seems zsv does not know how to parallelize unless the file is on disk so I suspect its approach is similar to the one used by xan, i.e. to chunk the file cleverly in constant time ahead of a map-reduce-like process (here is how xan does it, in any case: https://github.com/medialab/xan/blob/master/docs/blog/csv_base_jumping.md)?

I am curious to know if you ever attempted to leverage avx512. A lot of people are touting it can be even faster but I am skeptical. xan SIMD parser is not branchless in any case so I doubt it would give it an edge just yet, but it might for yours?

Another thing that could be good to bench also is a command that requires unquoting (xan count uses a parser that does not even attempt to separate cells, and xan select uses a zero-copy, non-unquoting parser as it is not required to shuffle columns around). I suspect your parser would be faster there also.

Else you write in your README:

non-4180-compliant data: zsv is fastest across the board (xan and polars are N/A for this input category)

xan can deal with such data also, by using a typical non-SIMD parser exposed by the xan input command, if you need to.

Best,

[liquidaty]

Thank you for all this useful info! Happy to add any/all of these suggestions to the benchmarks-- will review right away if you can open a PR with thee changes (or a portion of thereof) to the related benchmark scripts and/or Makefile, otherwise will get to it in due time.

if you ever attempted to leverage avx512

Years ago yes, but gave up because back then AVX512 was not widely adopted and its future was uncertain-- and it was much more time consuming to do this work than it is now with AI assistance-- worth taking another look now for sure

xan SIMD parser is not branchless in any case so I doubt it would give it an edge just yet, but it might for yours

That is a very astute observation. if I understand correctly the xan scanner reloads/restarts when it finds a dbl-quote so it makes sense that 256 vs 512 might have less impact in particular on data with quotes, whereas the zsv fast parser does not ever rescan the same byte segment

[Yomguithereal]

I will try to open a PR. I attempted to edit the benchmark quickly yesterday but stumbled upon issues related to my /usr/bin/time not having a -l flag.

if I understand correctly the xan scanner reloads/restarts when it finds a dbl-quote

Indeed. When entering a quoted section, xan's parser shift gears, if you will, and enters a faster, unrolled, but less amortized SIMD path. This faster path could maybe benefit from AVX512 in my parser but I don't have access to such hardware anyway.

Ultimately xan requires a "copy" parser most of the time and really needs cell values to be unquoted on the fly. I experimented a lot with xor prefix sums and shuffled writing tricks for this but never found a way to make it faster than my version branching on structural characters. I found a way to amortize writes in the branching version though, so I went with that. Ultimately it does not mean there is no way to make xor prefix/shuffling tricks to work faster, just that I failed to find a solution fitting my needs in the matter.

[liquidaty]

really needs cell values to be unquoted on the fly

I don't know with 100% certainty but I believe this cannot be done more efficiently with SIMD operations (compared to branching) due to the need to shift a dynamic length of bytes, starting at a dynamic offset, and to do that a dynamic number of times where length and offset change each time-- all of which I'm sure you're aware of. So yea, when it comes to outputting the actual (unquoted) data, I am not aware of any way to do this other than traditional branching

[Yomguithereal]

I have thought of an improvement of my design, that would still work without shuffling, lookup tables nor pclmulqdq and that would branch only once per chunk, instead of once per structural character (+ once per chunk, you still need to check if mask is zero). It would still need to "shift gears" each time a quoted section is encountered though. This would probably be the best one can do using SIMD indeed if unquoting is required, while your design should remain fastest for zero-copy parsing. I will experiment with this in the future and report whether this yields any improvement.