Skip to content

Commit 1135c5c

Browse files
idelvallvladaionescu
idelvall
and
vladaionescu
authored
Adds +INIT and $OS_RELEASE to cache id (#25)
This PR: - Includes the distro information (from `/etc/os-release`) in the cache id default value. This avoids the scenario of reusing the cache entries across potentially incompatibles shared libraries. - Includes a `+INIT` UDC used for initialize the build environment for the rest UDCs to be invoked. - Exports the cache id to the build environment as `$CARGO_HOME_CACHE_ID` --------- Co-authored-by: Vlad A. Ionescu <446771+vladaionescu@users.noreply.github.com>
1 parent 489a6a4 commit 1135c5c

2 files changed

Lines changed: 146 additions & 80 deletions

File tree

‎rust/Earthfile‎

Lines changed: 84 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,51 @@
11
VERSION --global-cache 0.7
22

3+
# INIT stores the configuration required for the other UDCs in the filesystem, and installs required dependencies.
4+
# - cache_id: Overrides default ID of the global $CARGO_HOME cache. Its value is exported to the build environment under the entry: $CARGO_HOME_CACHE_ID
5+
# - keep_fingerprints (false): Instructs the following +CARGO calls to don't remove the Cargo fingerprints of the source packages. Use only when source packages have been COPYed with --keep-ts option.
6+
# - sweep_days (4): +CARGO calls use cargo-sweep to clean build artifacts that haven't been accessed for this number of days.
7+
INIT:
8+
COMMAND
9+
RUN if [ -f /earthly/cfg/cache_id ]; then \
10+
echo "+INIT has already been called in this build environment" ; \
11+
exit 1; \
12+
fi
13+
DO +INSTALL_CARGO_SWEEP
14+
RUN mkdir -p /earthly/cfg
15+
16+
# cache_id
17+
ARG EARTHLY_TARGET_PROJECT_NO_TAG
18+
ARG OS_RELEASE=$(md5sum /etc/os-release | cut -d ' ' -f 1)
19+
ARG cache_id="${EARTHLY_TARGET_PROJECT_NO_TAG}#${OS_RELEASE}#earthly-cargo-cache"
20+
RUN echo "$cache_id">/earthly/cfg/cache_id
21+
ENV CARGO_HOME_CACHE_ID=$cache_id
22+
23+
#keep_fingerprints
24+
ARG keep_fingerprints=false
25+
RUN echo "$keep_fingerprints">/earthly/cfg/keep_fingerprints
26+
27+
#sweep_days
28+
ARG sweep_days=4
29+
RUN echo "$sweep_days">/earthly/cfg/sweep_days
30+
331
# CARGO runs the cargo command "cargo $args".
4-
# This UDC should be thread safe. Parallel builds of targets using it should be free of race conditions.
32+
# This UDC is thread safe. Parallel builds of targets calling this UDC should be free of race conditions.
33+
# Notice that in order to run this UDC, +INIT must be called first.
534
# Arguments:
635
# - args: Cargo subcommand and its arguments. Required.
7-
# - keep_fingerprints (false): Do not remove source packages fingerprints. Use only when source packages have been COPYed with --keep-ts option.
8-
# - sweep_days (4): The UDC uses cargo-sweep to clean build artifacts that haven't been accessed for this number of days.
936
# - output: Regex to match the files within the target folder to be copied from the cache to the caller filesystem (image layers).
1037
# Use this argument when you want to SAVE an ARTIFACT from the target folder (mounted cache), always trying to minimize the total size of the copied fileset.
1138
# For example --output="release/[^\./]+" would keep all the files in /target/release that don't have any extension.
1239
CARGO:
1340
COMMAND
41+
DO +CHECK_INITED
1442
ARG --required args
15-
ARG keep_fingerprints=false
16-
ARG sweep_days=4
43+
ARG keep_fingerprints=$(cat /earthly/cfg/keep_fingerprints)
44+
ARG sweep_days=$(cat /earthly/cfg/sweep_days)
1745
ARG output
1846
IF [ "$keep_fingerprints" = "false" ]
1947
DO +REMOVE_SOURCE_FINGERPRINTS
2048
END
21-
DO +INSTALL_CARGO_SWEEP
2249
DO +RUN_WITH_CACHE --command="set -e;
2350
echo \"Running cargo $args\" ;
2451
cargo $args;
@@ -38,34 +65,43 @@ CARGO:
3865
mv /earthly_lib_rust_temp/* target 2>/dev/null || echo "no files found within ./target matching the provided output regexp" ;
3966
END
4067

41-
# REMOVE_SOURCE_FINGERPRINTS removes the Cargo fingerprint folders of the source packages.
42-
# This guarantees Cargo compiles the packages when COPY commands of the source folders have a static timestamp (see --keep-ts).
43-
REMOVE_SOURCE_FINGERPRINTS:
68+
# RUN_WITH_CACHE runs the passed command with the CARGO caches mounted.
69+
# Notice that in order to run this UDC, +INIT must be called first.
70+
# Arguments:
71+
# - command (required): Command to run, can be any expression.
72+
#
73+
RUN_WITH_CACHE:
4474
COMMAND
45-
COPY +get-tomljson/tomljson /tmp/tomljson
46-
COPY +get-jq/jq /tmp/jq
47-
DO +RUN_WITH_CACHE --command="set -e;
48-
source_libs=\$(find . -name Cargo.toml -exec bash -c '/tmp/tomljson {} | /tmp/jq -r .package.name; printf \"\\n\"' \\;) ;
49-
fingerprint_folders=\$(find target -name .fingerprint) ;
50-
echo \"deleting fingerprints:\";
51-
for fingerprint_folder in \$fingerprint_folders; do
52-
cd \$fingerprint_folder;
53-
for source_lib in \$source_libs; do
54-
find . -maxdepth 1 -regex \"\./\$source_lib-[^-]+\" -exec bash -c 'readlink -f {}; rm -rf {}' \; ;
55-
done
56-
done"
75+
DO +CHECK_INITED
76+
ARG --required command
77+
ARG cache_id = $(cat /earthly/cfg/cache_id)
78+
# Save to restore at the end.
79+
ARG ORIGINAL_CARGO_HOME=$CARGO_HOME
80+
ARG ORIGINAL_CARGO_INSTALL_ROOT=$CARGO_INSTALL_ROOT
81+
# Make sure that crates installed though this UDC are stored in the original cargo home, and not in the cargo home within the mount cache.
82+
# This way, if BK garbage-collects them, the build is not broken.
83+
ENV CARGO_INSTALL_ROOT=$ORIGINAL_CARGO_HOME
84+
# We change $CARGO_HOME while keeping $ORIGINAL_CARGO_HOME/bin directory in the path. This way, the Cargo binary is still accessible and the whole $CARGO_HOME is within the global cache
85+
# ($CARGO_HOME/.package-cache has to be in the cache so Cargo can properly synchronize parallel access to $CARGO_HOME resources).
86+
ENV CARGO_HOME="/earthly/.cargo"
87+
RUN --mount=type=cache,mode=0777,id=$cache_id,sharing=shared,target=$CARGO_HOME \
88+
--mount=type=cache,mode=0777,target=target \
89+
set -e; \
90+
mkdir -p $CARGO_HOME; \
91+
printf "Running:\n $command\n"; \
92+
eval $command
93+
ENV CARGO_HOME=$ORIGINAL_CARGO_HOME
94+
ENV CARGO_INSTALL_ROOT=$ORIGINAL_CARGO_INSTALL_ROOT
5795

58-
# get-tomljson gets the portable tomljson binary.
5996
get-tomljson:
6097
FROM alpine:3.18.3
6198
ARG USERARCH
6299
ARG version=2.1.0
63100
RUN wget -O tomljson.tar.xz https://github.com/pelletier/go-toml/releases/download/v${version}/tomljson_${version}_linux_${USERARCH}.tar.xz && \
64-
tar -xf tomljson.tar.xz && \
101+
tar -xf tomljson.tar.xz; \
65102
chmod +x tomljson
66103
SAVE ARTIFACT tomljson
67104

68-
# get-jq gets the portable jq binary.
69105
get-jq:
70106
FROM alpine:3.18.3
71107
ARG USERARCH
@@ -74,39 +110,32 @@ get-jq:
74110
chmod +x jq
75111
SAVE ARTIFACT jq
76112

77-
# INSTALL_CARGO_SWEEP installs cargo-sweep if it doesn't exist already.
78113
INSTALL_CARGO_SWEEP:
79114
COMMAND
115+
RUN if [ ! -f $CARGO_HOME/bin/cargo-sweep ]; then \
116+
echo "Installing cargo sweep" ; \
117+
cargo install cargo-sweep --root $CARGO_HOME; \
118+
fi;
119+
120+
REMOVE_SOURCE_FINGERPRINTS:
121+
COMMAND
122+
DO +CHECK_INITED
123+
COPY +get-tomljson/tomljson /tmp/tomljson
124+
COPY +get-jq/jq /tmp/jq
80125
DO +RUN_WITH_CACHE --command="set -e;
81-
if [ ! -f \$CARGO_HOME/bin/cargo-sweep ]; then
82-
cargo install cargo-sweep --root \$CARGO_HOME;
83-
fi;"
126+
source_libs=\$(find . -name Cargo.toml -exec bash -c '/tmp/tomljson {} | /tmp/jq -r .package.name; printf \"\\n\"' \\;) ;
127+
fingerprint_folders=\$(find target -name .fingerprint) ;
128+
echo \"deleting fingerprints:\";
129+
for fingerprint_folder in \$fingerprint_folders; do
130+
cd \$fingerprint_folder;
131+
for source_lib in \$source_libs; do
132+
find . -maxdepth 1 -regex \"\./\$source_lib-[^-]+\" -exec bash -c 'readlink -f {}; rm -rf {}' \; ;
133+
done
134+
done"
84135

85-
# RUN_WITH_CACHE runs the passed command with the CARGO caches mounted.
86-
# Arguments:
87-
# - command (required): Command to run, can be any expression.
88-
#
89-
# This implementation is not expected to significantly change. Prefer using the `CARGO` UDC if you can, so you can get future improvements transparently.
90-
RUN_WITH_CACHE:
136+
CHECK_INITED:
91137
COMMAND
92-
ARG --required command
93-
ARG EARTHLY_TARGET_PROJECT_NO_TAG
94-
ARG CACHE_ID="${EARTHLY_TARGET_PROJECT_NO_TAG}#earthly-cargo-cache"
95-
# $ORIGINAL_CARGO_HOME/bin will contain the cargo and rust binaries, as well as the installed crates.
96-
# We save it to reset it back at the end. This location is presumed to be in the calling target layers filesystem rather than in other mount cache.
97-
ARG ORIGINAL_CARGO_HOME=$CARGO_HOME
98-
ARG ORIGINAL_CARGO_INSTALL_ROOT=$CARGO_INSTALL_ROOT
99-
# Make sure that crates installed though this UDC are stored in the original cargo home, and not in the cargo home within the mount cache.
100-
# This way, if BK garbage-collects them, the build is not broken
101-
ENV CARGO_INSTALL_ROOT=$ORIGINAL_CARGO_HOME
102-
# We need $CARGO_HOME/.package-cache within the earthly shared-mount-cache, so cargo can properly synchronize parallel access to $CARGO_HOME resources.
103-
ENV CARGO_HOME="/earthly/.cargo"
104-
RUN echo $CACHE_ID
105-
RUN --mount=type=cache,mode=0777,id=$CACHE_ID,sharing=shared,target=/earthly \
106-
--mount=type=cache,mode=0777,target=target \
107-
set -e; \
108-
mkdir -p $CARGO_HOME; \
109-
printf "Running:\n $command\n"; \
110-
eval $command
111-
ENV CARGO_HOME=$ORIGINAL_CARGO_HOME
112-
ENV CARGO_INSTALL_ROOT=$ORIGINAL_CARGO_INSTALL_ROOT
138+
RUN if [ ! -f /earthly/cfg/cache_id ]; then \
139+
echo "+INIT has not been called yet in this build environment" ; \
140+
exit 1; \
141+
fi;

‎rust/README.md‎

Lines changed: 62 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -2,40 +2,55 @@
22

33
Earthly's official collection of rust [UDCs](https://docs.earthly.dev/docs/guides/udc).
44

5-
## +CARGO
6-
7-
This UDC runs the cargo command `cargo $args` caching the contents of `$CARGO_HOME/registry`, `$CARGO_HOME/git` and `target` for future builds of the same calling target.
8-
9-
### Usage
10-
115
First, import the UDC up in your Earthfile:
126
```earthfile
13-
VERSION 0.7
7+
VERSION --global-cache 0.7
148
IMPORT github.com/earthly/lib/rust:<version/commit> AS rust
159
```
10+
> :warning: Due to [this issue](https://github.com/earthly/earthly/issues/3490), make sure to enable `--global-cache` in the calling Earthfile, as shown above.
1611
17-
Then, just use it in your own targets and UDCs:
18-
```earthfile
19-
DO rust+CARGO ...
20-
```
12+
## +INIT
2113

22-
### Thread safety
23-
This UDC should be thread safe. Parallel builds of targets using it should be free of race conditions.
14+
This UDC stores the configuration required by the other UDCs in the build environment filesystem, and installs required dependencies.
2415

25-
### Arguments
16+
It must be called once per build environment, to avoid passing repetitive arguments to the UDCs called after it, and to install required dependencies before the source files are copied from the build context.
2617

27-
#### `args`
28-
Cargo subcommand and its arguments. Required.
18+
### Usage
2919

30-
#### `keep_fingerprints (false)`
31-
Do not remove source packages fingerprints. Use only when source packages have been `COPY`ed with `--keep-ts` option.
20+
Call once per build environment:
21+
```earthfile
22+
DO rust+INIT ...
23+
```
3224

33-
Cargo caches compilations of packages in `target` folder based on their last modification timestamps.
25+
### Arguments
26+
#### `cache_id`
27+
Overrides default ID of the global `$CARGO_HOME` cache. Its value is exported to the build environment under the entry: `$CARGO_HOME_CACHE_ID`.
3428

29+
#### `keep_fingerprints (false)`
30+
Instructs the following `+CARGO` calls to don't remove the Cargo fingerprints of the source packages. Use only when source packages have been COPYed with `--keep-ts `option.
31+
Cargo caches compilations of packages in `target` folder based on their last modification timestamps.
3532
By default, this UDC removes the fingerprints of the packages found in the source code, to force their recompilation and work even when the Earthly `COPY` commands used overwrote the timestamps.
3633

3734
#### `sweep_days (4)`
38-
The UDC uses [cargo-sweep](https://github.com/holmgr/cargo-sweep) to clean build artifacts that haven't been accessed for this number of days.
35+
`+CARGO` calls use cargo-sweep to clean build artifacts that haven't been accessed for this number of days.
36+
37+
## +CARGO
38+
39+
This UDC runs the cargo command `cargo $args` caching the contents of `$CARGO_HOME` and `target` for future builds of the same calling target.
40+
41+
Notice that in order to run this UDC, [+INIT](#init) must be called first.
42+
43+
### Usage
44+
45+
After calling `+INIT`, use it to wrap cargo commands:
46+
47+
```earthfile
48+
DO rust+CARGO ...
49+
```
50+
### Arguments
51+
52+
#### `args`
53+
Cargo subcommand and its arguments. Required.
3954

4055
#### `output`
4156
Regex to match the files within the target folder to be copied from the cache to the caller filesystem (image layers).
@@ -44,7 +59,27 @@ Use this argument when you want to `SAVE ARTIFACT` from the target folder (mount
4459

4560
For example `--output="release/[^\./]+"` would keep all the files in `/target/release` that don't have any extension.
4661

47-
### Examples:
62+
### Thread safety
63+
This UDC is thread safe. Parallel builds of targets calling this UDC should be free of race conditions.
64+
65+
## +RUN_WITH_CACHE
66+
67+
`+RUN_WITH_CACHE` runs the passed command with the CARGO caches mounted.
68+
69+
Notice that in order to run this UDC, [+INIT](#init) must be called first.
70+
71+
### Arguments
72+
#### `command (required)`
73+
Command to run, can be any expression.
74+
75+
### Example
76+
Show `$CARGO_HOME` cached-entries size:
77+
78+
```earthfile
79+
DO rust-udc+RUN_WITH_CACHE --command "du \$CARGO_HOME"
80+
```
81+
82+
## Complete example
4883

4984
Suppose the following project:
5085
```
@@ -66,25 +101,27 @@ Suppose the following project:
66101
The Earthfile would look like:
67102

68103
```earthfile
69-
VERSION 0.7
70-
ARG --global debian=bookworm
104+
VERSION --global-cache 0.7
71105
72106
# Importing UDC definition from default branch (in a real case, specify version or commit to guarantee immutability)
73107
IMPORT github.com/earthly/lib/rust AS rust
74108
75109
install:
76-
FROM rust:1.73.0-$debian
110+
FROM rust:1.73.0-bookworm
77111
RUN apt-get update -qq
78112
RUN apt-get install --no-install-recommends -qq autoconf autotools-dev libtool-bin clang cmake bsdmainutils
79113
RUN cargo install --locked cargo-deny
80114
RUN rustup component add clippy
81115
RUN rustup component add rustfmt
116+
# Call +INIT before copying the source file to avoid installing depencies every time source code changes.
117+
# This parametrization will be used in future calls to UDCs of the library
118+
DO rust+INIT --keep_fingerprints=true
82119
83120
source:
84121
FROM +install
85122
COPY --keep-ts Cargo.toml Cargo.lock ./
86123
COPY --keep-ts deny.toml ./
87-
COPY --dir package1 package2 ./
124+
COPY --keep-ts --dir package1 package2 ./
88125
89126
# build builds with the Cargo release profile
90127
build:

0 commit comments

Comments
 (0)