Skip to content

Commit 42e23e3

Browse files
xorpsexorpse
authored
add PyPI based distribution and plugin for hcli (#1)
1 parent 3220b73 commit 42e23e3

12 files changed

Lines changed: 678 additions & 2 deletions

File tree

‎.github/workflows/build.yml‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
name: build
2+
3+
permissions:
4+
contents: read
5+
6+
on:
7+
push:
8+
branches: [ master ]
9+
workflow_dispatch:
10+
11+
env:
12+
CARGO_TERM_COLOR: always
13+
14+
jobs:
15+
build-linux:
16+
runs-on: ubuntu-latest
17+
container:
18+
image: quay.io/pypa/manylinux_2_28_x86_64
19+
20+
env:
21+
CARGO_TERM_COLOR: always
22+
PYTHON_BIN: /opt/python/cp311-cp311/bin/python
23+
24+
steps:
25+
- name: checkout
26+
uses: actions/checkout@v4
27+
with:
28+
submodules: true
29+
30+
- name: Install Rust (via rustup)
31+
shell: bash
32+
run: |
33+
curl https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal --default-toolchain stable
34+
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
35+
36+
- name: Install build dependencies
37+
shell: bash
38+
run: |
39+
yum install -y jq llvm clang
40+
41+
- name: Install Python build dependencies
42+
shell: bash
43+
run: |
44+
$PYTHON_BIN -m pip install --upgrade pip
45+
$PYTHON_BIN -m pip install wheel auditwheel
46+
47+
- name: Generate lock file
48+
shell: bash
49+
run: cargo generate-lockfile
50+
51+
- name: Compute package version
52+
shell: bash
53+
run: |
54+
VERSION=$(cargo metadata --format-version=1 --no-deps | jq -r '.packages[0].version')
55+
echo "PACKAGE_VERSION=$VERSION" >> "$GITHUB_ENV"
56+
57+
- name: Build Rust binary (release, stub linkage)
58+
shell: bash
59+
env:
60+
IDALIB_FORCE_STUB_LINKAGE: "1"
61+
run: cargo build --release
62+
63+
- name: Build Python wheel skeleton
64+
shell: bash
65+
run: |
66+
mkdir -p target/wheels
67+
$PYTHON_BIN -m pip wheel --no-deps ./dist/pypi -w ./target/wheels
68+
69+
- name: Merge Rust binary into wheel
70+
shell: bash
71+
run: |
72+
cd target/wheels
73+
74+
# unpack the wheel built by the ./dist/pypi package
75+
$PYTHON_BIN -m wheel unpack parascope-${PACKAGE_VERSION}*.whl
76+
77+
# copy the compiled Rust binary into .data/scripts as rparascope
78+
mkdir -p parascope-${PACKAGE_VERSION}/parascope-${PACKAGE_VERSION}.data/scripts
79+
cp ../release/parascope parascope-${PACKAGE_VERSION}/parascope-${PACKAGE_VERSION}.data/scripts/rparascope
80+
81+
# repack
82+
$PYTHON_BIN -m wheel pack parascope-${PACKAGE_VERSION} -d .
83+
84+
- name: Repair wheel with auditwheel (manylinux tagging & bundling)
85+
shell: bash
86+
run: |
87+
cd target/wheels
88+
WHEEL=$(ls parascope-${PACKAGE_VERSION}-*.whl)
89+
echo "Repairing $WHEEL with auditwheel"
90+
$PYTHON_BIN -m auditwheel repair parascope-${PACKAGE_VERSION}-*.whl -w .
91+
echo "Removing original wheel"
92+
rm "$WHEEL"
93+
94+
- name: Upload Linux wheel
95+
uses: actions/upload-artifact@v4
96+
with:
97+
name: wheels-linux
98+
path: target/wheels/*.whl
99+
100+
build-macos:
101+
runs-on: macOS-latest
102+
steps:
103+
- name: checkout
104+
uses: actions/checkout@v4
105+
with:
106+
submodules: true
107+
108+
- name: Install Rust (via rustup)
109+
shell: bash
110+
run: |
111+
curl https://sh.rustup.rs -sSf | sh -s -- -y --profile minimal --default-toolchain stable
112+
echo "$HOME/.cargo/bin" >> $GITHUB_PATH
113+
114+
- name: Setup Python
115+
uses: actions/setup-python@v5
116+
with:
117+
python-version: "3.11"
118+
119+
- name: Install Python build dependencies
120+
run: |
121+
python -m pip install --upgrade pip
122+
python -m pip install wheel
123+
124+
- name: Generate lock file
125+
shell: bash
126+
run: cargo generate-lockfile
127+
128+
- name: Compute package version
129+
shell: bash
130+
run: |
131+
VERSION=$(cargo metadata --format-version=1 --no-deps | jq -r '.packages[0].version')
132+
echo "PACKAGE_VERSION=$VERSION" >> "$GITHUB_ENV"
133+
134+
- name: Build Rust binary (release, stub linkage)
135+
env:
136+
IDALIB_FORCE_STUB_LINKAGE: "1"
137+
run: cargo build --release
138+
139+
- name: Detect macOS arch from binary
140+
shell: bash
141+
run: |
142+
ARCH=$(lipo -info target/release/parascope | grep -oE 'x86_64|arm64')
143+
if [ "$ARCH" = "x86_64" ]; then
144+
PLATFORM_TAG="macosx_10_9_x86_64"
145+
elif [ "$ARCH" = "arm64" ]; then
146+
PLATFORM_TAG="macosx_11_0_arm64"
147+
else
148+
echo "Unknown arch from lipo: $ARCH"
149+
exit 1
150+
fi
151+
echo "PLATFORM_TAG=$PLATFORM_TAG" >> "$GITHUB_ENV"
152+
153+
- name: Build Python wheel skeleton
154+
run: |
155+
mkdir -p target/wheels
156+
python -m pip wheel --no-deps ./dist/pypi -w ./target/wheels
157+
158+
- name: Merge Rust binary into wheel
159+
run: |
160+
cd target/wheels
161+
python -m wheel unpack parascope-${PACKAGE_VERSION}*.whl
162+
163+
mkdir -p parascope-${PACKAGE_VERSION}/parascope-${PACKAGE_VERSION}.data/scripts
164+
cp ../release/parascope parascope-${PACKAGE_VERSION}/parascope-${PACKAGE_VERSION}.data/scripts/rparascope
165+
166+
python -m wheel pack parascope-${PACKAGE_VERSION} -d .
167+
168+
- name: Retag macOS wheel with platform tag
169+
shell: bash
170+
run: |
171+
cd target/wheels
172+
WHEEL=$(ls parascope-${PACKAGE_VERSION}-*.whl)
173+
echo "Retagging $WHEEL to platform $PLATFORM_TAG"
174+
python -m wheel tags --platform-tag "$PLATFORM_TAG" "$WHEEL"
175+
echo "Removing original wheel"
176+
rm "$WHEEL"
177+
178+
- name: Upload macOS wheel
179+
uses: actions/upload-artifact@v4
180+
with:
181+
name: wheels-macos
182+
path: target/wheels/*.whl

‎Cargo.toml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,11 @@ repository = "https://github.com/xorpse/parascope.git"
1111
readme = "./README.md"
1212
edition = "2024"
1313
build = "build.rs"
14-
exclude = ["assets", "rules", "tests"]
14+
exclude = ["assets", "rules", "tests", "python", "plugin"]
15+
16+
[[bin]]
17+
name = "parascope"
18+
path = "src/main.rs"
1519

1620
[dependencies]
1721
anyhow = "1"

‎build.rs‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,25 @@
1+
use std::env;
2+
13
fn main() -> Result<(), Box<dyn std::error::Error>> {
2-
idalib_build::configure_linkage()?;
4+
if env::var("IDALIB_FORCE_STUB_LINKAGE").is_ok() {
5+
idalib_build::configure_idasdk_linkage();
6+
7+
#[cfg(target_os = "linux")]
8+
{
9+
let (_, stub_path, _, _) = idalib_build::idalib_sdk_paths();
10+
println!(
11+
"cargo::rustc-link-arg=-Wl,-rpath,{},-L{},-l:libida.so",
12+
stub_path.display(),
13+
stub_path.display(),
14+
);
15+
println!(
16+
"cargo::rustc-link-arg=-Wl,-rpath,{},-L{},-l:libidalib.so",
17+
stub_path.display(),
18+
stub_path.display(),
19+
);
20+
}
21+
} else {
22+
idalib_build::configure_linkage()?;
23+
}
324
Ok(())
425
}

‎dist/pypi/README.md‎

Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
# parascope
2+
3+
[![crates.io](https://img.shields.io/crates/v/parascope)](https://crates.io/crates/parascope)
4+
[![license](https://img.shields.io/crates/l/parascope)](https://github.com/xorpse/parascope)
5+
[![crates.io downloads](https://img.shields.io/crates/d/parascope)](https://crates.io/crates/parascope)
6+
7+
<!--
8+
<p align="center">
9+
<img src="https://raw.githubusercontent.com/xorpse/parascope/refs/heads/master/assets/parascope-logo.svg" width="200" height="200" alt="parascope logo">
10+
</p>
11+
-->
12+
13+
Weggli ruleset scanner for binaries and source code. Organise your weggli
14+
rules and scan source code and binaries in parallel!
15+
16+
<p align="center">
17+
<img src="https://raw.githubusercontent.com/xorpse/parascope/refs/heads/master/assets/parascope-demo.gif" width="800" alt="parascope demo">
18+
</p>
19+
20+
## Build/installation
21+
22+
To build and install parascope requires IDA Pro v9.2 and access to the
23+
latest SDK.
24+
25+
Install via crates.io:
26+
27+
```sh
28+
export IDADIR=/path/to/ida # optional
29+
cargo install parascope
30+
```
31+
32+
Build/install from source:
33+
34+
```sh
35+
export IDADIR=/path/to/ida # optional
36+
cargo install --path .
37+
```
38+
39+
## Examples and usage
40+
41+
Scan a single binary and output the rule matches to stdout:
42+
```
43+
parascope --display -r rules /path/of/binary
44+
```
45+
46+
Scan all binaries in the given directory and stream rule matches to results.jsonl:
47+
```
48+
parascope -o results.jsonl -r rules /directory/of/binaries
49+
```
50+
51+
Scan the C source code in the given directory and stream rule matches to results.jsonl:
52+
```
53+
parascope -m c -o results.jsonl -r rules /directory/of/source-code
54+
```
55+
56+
Complete set of capabilities:
57+
58+
```sh
59+
Weggli ruleset scanner for source code and binaries
60+
61+
Usage: parascope [OPTIONS] --rules <rules> <INPUT>
62+
63+
Arguments:
64+
<INPUT>
65+
File or directory to scan
66+
67+
Options:
68+
-m, --mode <mode>
69+
Analysis mode
70+
71+
[default: binary]
72+
73+
Possible values:
74+
- binary: Binary analysis mode (using IDA)
75+
- c: Source code analysis mode (C)
76+
- cxx: Source code analysis mode (C++)
77+
78+
--path-filter [<path-filter>...]
79+
Restrict analysis to files matching the given regular expression.
80+
For C/C++ analysis if no path filters are given analysis is restricted
81+
to a set of default file extensions:
82+
83+
C: c, h
84+
C++: C, cc, cxx, cpp, H, hh, hxx, hpp, h
85+
86+
For binary analysis, all files will be analysed. If an existing IDB is
87+
available, e.g., we have both file and file.i64, only the IDB will be
88+
used for analysis irrespective of the path filter.
89+
90+
--display
91+
Render matches to stdout
92+
93+
--display-context <display-context>
94+
Number of lines before/after match to render
95+
96+
[default: 5]
97+
98+
--summary
99+
Render tabular summary to stdout
100+
101+
-r, --rules <rules>
102+
File or directory containing wegglir rules
103+
104+
-o, --output <OUTPUT>
105+
File to write output results (JSONL)
106+
107+
-h, --help
108+
Print help (see a summary with '-h')
109+
110+
-V, --version
111+
Print version
112+
```
113+
114+
## Rules
115+
116+
We use [weggli-ruleset](https://github.com/xorpse/weggli-ruleset.git) to help
117+
manage weggli patterns. It provides a yaml-based rule format that allows
118+
different (related) patterns to be grouped along with metadata useful for
119+
categorising and triaging matches. For example, we can encode the patterns from
120+
[here](https://github.com/0xdea/weggli-patterns?tab=readme-ov-file#call-to-unbounded-copy-functions-cwe-120-cwe-242-cwe-676),
121+
as follows:
122+
123+
```yaml
124+
id: call-to-unbounded-copy-functions
125+
description: call to unbounded copy functions
126+
severity: medium
127+
tags:
128+
- CWE-120
129+
- CWE-242
130+
- CWE-676
131+
check-patterns:
132+
- name: gets
133+
regex: func=^gets$
134+
pattern: |
135+
{ $func(); }
136+
- name: st(r|p)(cpy|cat)
137+
regex: func=st(r|p)(cpy|cat)$
138+
pattern: |
139+
{ $func(); }
140+
- name: wc(r|p)(cpy|cat)
141+
regex: func=wc(r|p)(cpy|cat)$
142+
pattern: |
143+
{ $func(); }
144+
- name: sprintf
145+
regex: func=sprintf$
146+
pattern: |
147+
{ $func(); }
148+
- name: scanf
149+
regex: func=scanf$
150+
pattern: |
151+
{ $func(); }
152+
```
153+
154+
### Rulesets & Resources
155+
156+
Below is a list of resources containing weggli patterns/rules that can easily
157+
be ported to parascope rules:
158+
159+
- [weggli-patterns](https://github.com/0xdea/weggli-patterns) and [A collection of weggli patterns for C/C++ vulnerability research](https://security.humanativaspa.it/a-collection-of-weggli-patterns-for-c-cpp-vulnerability-research/) by [raptor/@0xdea](https://github.com/0xdea)
160+
- [weggli-patterns](https://github.com/plowsec/weggli-patterns) by [volodya/@plowsec](https://github.com/plowsec)
161+
- [Playing with Weggli](https://dustri.org/b/playing-with-weggli.html) by [Julien Voisin](https://dustri.org/)
162+
- [Weggli rules (SSTIC 2013)](https://github.com/synacktiv/Weggli_rules_SSTIC2023) by [Synacktiv](https://github.com/synacktiv)

‎dist/pypi/parascope/__init__.py‎

Whitespace-only changes.

0 commit comments

Comments
 (0)