ready for CRAN

Author: dipterix

NEWS.md

@@ -1,4 +1,15 @@
-# filearray (development version)
+# filearray 0.1.1
+
+* Added `OpenMP` flag in the `MakeVars`
+* Fixed critical bugs that could cause `segfaults`
+* Re-implemented read/write functions to use memory map
+* Allowed `dimnames` to be set
+* Added generics `subset` to subset using `dimnames`
+* Added vignette to compare performance
+* Added speed comparisons in `README.md`
+* Added `collapse` to calculate marginal summation with little memory overhead
+* Added `fmap`, `fmap2` to apply functions to one or multiple file arrays with little memory overhead (also very fast)
+* Fixed 'unprotected' issues warned by `rchk`
 
 # filearray 0.1.0
 

README.md

@@ -11,7 +11,7 @@ Stores large arrays in files to avoid occupying large memories. Implemented with
 
 ![](https://raw.githubusercontent.com/dipterix/filearray/main/adhoc/readme-speed.png)
 
-<small> *Speed comparisons with `lazyarray` (`zstd`-compressed out-of-memory array), and in-memory operation. `filearray` is uniformly faster than `lazyarray`. Random access has almost the same speed as the native in-memory operation. The speed test was performed on an `MacBook Air (M1, 2020)` with 8GB memory* </small>
+<small> *Speed comparisons with `lazyarray` (`zstd`-compressed out-of-memory array), and in-memory operation. The speed test was conducted on an `MacBook Air (M1, 2020, 8GB RAM)`. `filearray` is uniformly faster than `lazyarray`. Random access has almost the same speed as the native in-memory operation.* </small>
 
 ## Installation
 
@@ -150,4 +150,10 @@ hence use with caution when data needs high precision or the max is super large.
 
 3. `collapse` function: when data range is large (say `x[[1]]=1`, but `x[[2]]=10^20`), `collapse` method might lose precision. This is `double` only uses 8 bytes of memory space. When calculating summations, R internally uses `long double` to prevent precision loss, but current `filearray` implementation uses `double`, causing floating error around 16 decimal place. 
 
+#### III. Cold-start vs warm-start
+
+As of version `0.1.1`, most file read/write operations are switched from `fopen` to memory map for two simplify the logic (buffer size, kernel cache...), and to boost the writing/some types of reading speed. While sacrificing the speed of reading large block of data from 2.4GB/s to 1.7GB/s, the writing speed was boosted from 300MB/s to 700MB/s, and the speed of random accessing small slices of data was increased from 900MB/s to 2.5GB/s. As a result, some functions can reach to really high speed (close to in-memory calls) while using much less memory. 
+
+The additional performance improvements brought by the memory mapping approach might be impacted by "cold" start. When reading/writing files, most modern systems will cache the files so that it can load up these files faster next time. I personally call it a cold start. Memory mapping have a little bit extra overhead during the cold start, resulting in decreased performance (but it's still fast). Accessing the same data after the cold start is called warm start. When operating with warm starts, `filearray` is as fast as native R arrays (sometimes even faster due to the indexing method and fewer garbage collections). This means `filearray` reaches its best performance when the arrays are re-used.
+
 

cran-comments.md

@@ -8,6 +8,19 @@
 
 ## R CMD check results
 
-0 errors | 0 warnings | 1 note
+On R-4.X
+0 errors | 0 warnings | 0 notes
 
-* This is a new release.
+On R-3.6
+0 errors | 1 warning | 0 notes
+
+```
+Codoc mismatches from documentation object 'apply':
+apply
+  Code: function(X, MARGIN, FUN, ...)
+  Docs: function(X, MARGIN, FUN, ..., simplify = TRUE)
+  Argument names in docs not in code:
+    simplify
+```
+
+This is because `simplify` was added to `apply` function since R-4.0.