Skip to content

Commit ba2e02e

Browse files
welchrwelchr
committed
README updates
1 parent 618ce61 commit ba2e02e

1 file changed

Lines changed: 95 additions & 159 deletions

File tree

‎README.md‎

Lines changed: 95 additions & 159 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,30 @@ Visualize and explore fine-mapped signals and their colocalizations
44

55
To see an example of a running instance of Colocus, try: https://amp.colocus.app/.
66

7+
This repository contains the code for the backend server component of Colocus, as well as a docker compose stack to help deploy it.
8+
9+
:warning: This project is still in active development and has not yet reached a stable release. Breaking changes may occur. :warning:
10+
711
<!-- @import "[TOC]" {cmd="toc" depthFrom=1 depthTo=6 orderedList=false} -->
812

9-
This repository contains the code for the backend server component of Colocus, as well as a docker compose stack to help
10-
deploy it.
13+
<!-- code_chunk_output -->
14+
15+
- [colocus](#colocus)
16+
- [Deployment](#deployment)
17+
- [Docker](#docker)
18+
- [Sentry](#sentry)
19+
- [CSG](#csg)
20+
- [Required data](#required-data)
21+
- [Marginal and conditional / fine-mapping analyses](#marginal-and-conditional--fine-mapping-analyses)
22+
- [Linkage disequilibrium (LD)](#linkage-disequilibrium-ld)
23+
- [Colocalization](#colocalization)
24+
- [Development](#development)
25+
- [Data](#data)
26+
- [Docker](#docker-1)
27+
- [Running all code checks](#running-all-code-checks)
28+
- [Running tests](#running-tests)
29+
30+
<!-- /code_chunk_output -->
1131

1232
## Deployment
1333

@@ -47,16 +67,7 @@ docker compose up -d --build --force-recreate django
4767
# If you're just reloading an existing dataset in the same DATA_PATH as before, you can start here:
4868
docker compose exec db bash -c 'psql -U colocus -c "DROP DATABASE core WITH (FORCE)"'
4969
docker compose exec db bash /docker-entrypoint-initdb.d/init-db.sh
50-
docker compose exec django bash -c 'source .venv/bin/activate && python3 manage.py migrate --database=core'
51-
docker compose exec django bash -c "
52-
source .venv/bin/activate
53-
for subdir in \$(find /data -mindepth 1 -maxdepth 1 -type d); do
54-
subdir=\"\${subdir%/}\"
55-
if [ -d \"\$subdir\" ]; then
56-
uv run python3 scripts/load_dataset.py \"\${subdir}\"
57-
fi
58-
done
59-
"
70+
docker compose exec django bash /opt/colocus/bin/entrypoint-migrate-and-load.sh
6071
```
6172

6273
For debugging a new dataset load:
@@ -67,67 +78,14 @@ docker compose exec django bash
6778

6879
# From inside container
6980
source .venv/bin/activate
70-
python3 -m pdb scripts/load_dataset.py /data
71-
```
72-
73-
While developing you may want the containers to rebuild or resync with your source files changing automatically. The following can be placed in a `docker-compose.override.yml` file:
74-
75-
```yml
76-
services:
77-
django:
78-
build: .
79-
command: --reload
80-
develop:
81-
watch:
82-
- action: sync
83-
path: ./colocus
84-
target: /opt/colocus/colocus
85-
- action: sync
86-
path: ./colocus/core/migrations
87-
target: /opt/colocus/colocus/core/migrations
88-
- action: sync
89-
path: ./scripts
90-
target: /opt/colocus/scripts
91-
- action: rebuild
92-
path: ./Dockerfile
93-
94-
ui:
95-
build:
96-
context: ../colocus-ui-vue3
97-
dockerfile: Dockerfile.dev
98-
ports:
99-
- "${VITE_PORT}:${VITE_PORT}"
100-
healthcheck:
101-
test: ["CMD", "curl", "-f", "http://localhost:${VITE_PORT}"]
102-
interval: 5s
103-
timeout: 5s
104-
retries: 5
105-
develop:
106-
watch:
107-
- action: rebuild
108-
path: ../colocus-ui-vue3/package.json
109-
- action: rebuild
110-
path: ../colocus-ui-vue3/vite.config.mjs
111-
- action: sync
112-
path: ../colocus-ui-vue3/src
113-
target: /app/src
114-
- action: sync
115-
path: ../colocus-ui-vue3/etc
116-
target: /app/etc
117-
- action: rebuild
118-
path: ../colocus-ui-vue3/Dockerfile.dev
81+
python3 -m pdb scripts/load_dataset.py /data/your-dataset
11982
```
12083

121-
This assumes you have `colocus` and `colocus-ui-vue3` repositories checked out and next to each other in the directory hierarchy.
122-
123-
You'll also want the following in your `.env` file:
84+
#### Sentry
12485

125-
```bash
126-
VITE_HOST=0.0.0.0
127-
VITE_PORT=5173
128-
VITE_API_URL=http://django:${UVICORN_PORT}
129-
```
86+
Sentry is an error logging aggregator service. You can sign up for a free account at <https://sentry.io/signup/> or download and host it yourself. The system is set up with reasonable defaults, including 404 logging and integration with the WSGI application.
13087

88+
You must set the DSN url in `SENTRY_DSN` in your `.env` file.
13189

13290
### CSG
13391

@@ -519,100 +477,17 @@ The fields are:
519477

520478
## Development
521479

522-
### Bare metal
523-
524-
This development setup deploys the colocus server directly to your VM or local machine. For a docker based setup, see below.
525-
526-
#### Database setup
527-
528-
We use `uv` to manage packages and dependencies. [Follow these instructions](https://docs.astral.sh/uv/getting-started/installation/) to install `uv` on your system.
529-
530-
The database can be created by applying relevant migrations, and then loading a pre-packaged dataset (not provided in
531-
this repo, though subsets of data may be provided in the future).
532-
533-
```bash
534-
$ cd /path/to/colocus
535-
$ uv sync
536-
$ mkdir database
537-
$ uv run python manage.py migrate
538-
$ uv run python scripts/load_dataset.py <path/to/dataset> # see below for datasets
539-
```
540-
541-
<details>
542-
<summary><b>Datasets for CSG users</b></summary>
543-
544-
There is an existing dataset on our cluster at
545-
`/net/dumbo/home/welchr/projects/amp-cmd/colocus-pipeline-brotman/data/processed/`. The required files are
546-
approximately 13GB in total. You can quickly sync the required files to your development environment with:
547-
548-
```bash
549-
rsync -avimHP \
550-
user@dumbo.sph.umich.edu:/home/welchr/projects/amp-cmd/colocus-pipeline-brotman/ \
551-
/path/on/your/machine/colocus-pipeline-brotman/ \
552-
--exclude 'data/processed/ld/ukbb_grch37_all/variants' \
553-
--exclude 'data/original-copy' \
554-
--exclude 'data/ukbb/' \
555-
--exclude 'venv' \
556-
--exclude '.snakemake' \
557-
--exclude 'logs'
558-
```
559-
560-
Then give the path to the `data/processed` directory as the argument to `load_dataset.py`:
561-
562-
```bash
563-
$ uv run python scripts/load_dataset.py /path/on/your/machine/colocus-pipeline-brotman/data/processed/
564-
```
565-
</details>
566-
567-
568-
569-
If you have already tried to load the data previously and want a fresh start, you can delete the database and start over:
570-
571-
```bash
572-
rm -f "./database/local.sqlite3"
573-
rm -rf "./colocus/media"
574-
uv run python manage.py migrate
575-
uv run python scripts/load_dataset.py <path/to/dataset>
576-
```
480+
### Data
577481

578482
More information on the required types of data can be found below under [required data](#required-data).
579483

580-
#### Running the django server
581-
582-
This will start uvicorn to serve the django app and REST API. By default, the server runs on port 8000.
583-
584-
```bash
585-
uv run uvicorn config.asgi:application --host 0.0.0.0 --reload
586-
```
587-
588-
#### Running all code checks
589-
590-
The project is setup to use [pre-commit](https://pre-commit.com/) to run all checks at once. You can either install
591-
the pre-commit git hooks, or run pre-commit yourself manually before committing.
592-
593-
To run pre-commit manually:
594-
595-
```bash
596-
uv run pre-commit run --all-files -v
597-
```
598-
599-
This is the same command our Github Actions CI will run when you push a commit.
600-
601-
#### Running tests
602-
603-
```bash
604-
uv run pytest
605-
```
606-
607-
#### Sentry
608-
609-
Sentry is an error logging aggregator service. You can sign up for a free account at <https://sentry.io/signup/> or download and host it yourself. The system is set up with reasonable defaults, including 404 logging and integration with the WSGI application.
610-
611-
You must set the DSN url in `SENTRY_DSN` in your `.env` file.
484+
There is an example test dataset in `colocus/tests/data/` that can be used to test loading data. It is the same dataset used for running test cases.
612485

613486
### Docker
614487

615-
Make a docker compose override that enables watching files and rebuilding container images as needed:
488+
Make a `docker-compose.override.yml` that enables watching files and rebuilding container images as needed:
489+
490+
While developing you may want the containers to rebuild or resync with your source files changing automatically. The following can be placed in a `docker-compose.override.yml` file:
616491

617492
```yml
618493
services:
@@ -624,31 +499,92 @@ services:
624499
- action: sync
625500
path: ./colocus
626501
target: /opt/colocus/colocus
502+
- action: sync
503+
path: ./colocus/core/migrations
504+
target: /opt/colocus/colocus/core/migrations
505+
- action: sync
506+
path: ./scripts
507+
target: /opt/colocus/scripts
508+
- action: rebuild
509+
path: ./Dockerfile
627510
628511
ui:
629512
build:
630513
context: ../colocus-ui-vue3
631-
dockerfile: Dockerfile
514+
dockerfile: Dockerfile.dev
515+
ports:
516+
- "${VITE_PORT}:${VITE_PORT}"
517+
healthcheck:
518+
test: ["CMD", "curl", "-f", "http://localhost:${VITE_PORT}"]
519+
interval: 5s
520+
timeout: 5s
521+
retries: 5
632522
develop:
633523
watch:
634524
- action: rebuild
635525
path: ../colocus-ui-vue3/package.json
636526
- action: rebuild
527+
path: ../colocus-ui-vue3/vite.config.mjs
528+
- action: sync
637529
path: ../colocus-ui-vue3/src
530+
target: /app/src
531+
- action: sync
532+
path: ../colocus-ui-vue3/etc
533+
target: /app/etc
534+
- action: rebuild
535+
path: ../colocus-ui-vue3/Dockerfile.dev
638536
```
639537

640-
To use this, you will need the `colocus-ui-vue3` repository checked out next to colocus. For example, your directory tree should look like this:
538+
This assumes you have `colocus` and `colocus-ui-vue3` repositories checked out and next to each other in the directory hierarchy.
539+
540+
For example, your directory tree should look like this:
641541

642542
```
643543
root
644544
| - colocus-ui-vue3
645545
| - colocus
646546
```
647547

548+
You'll also want the following in your `.env` file:
549+
550+
```bash
551+
VITE_HOST=0.0.0.0
552+
VITE_PORT=5173
553+
VITE_API_URL=http://django:${UVICORN_PORT}
554+
```
555+
648556
Now you can run:
649557

650558
```bash
651559
docker compose up --build --watch
652560
```
653561

654562
This will bring up the containers, building them as needed, and watch to see if source code files change. If any of the source changes, the container image will be rebuilt if necessary, and restarted. In the case of the Django container, it will instead just sync the new source files into the container, and then the uvicorn server will reload them.
563+
564+
#### Running all code checks
565+
566+
To run code checks, you will need to install `uv` first, and then required packages:
567+
568+
```bash
569+
# Install uv
570+
curl -LsSf https://astral.sh/uv/install.sh | sh
571+
572+
# Install packages
573+
uv sync
574+
```
575+
576+
The project is setup to use [pre-commit](https://pre-commit.com/) to run all checks at once. You can either install the pre-commit git hooks, or run pre-commit yourself manually before committing.
577+
578+
To run pre-commit manually:
579+
580+
```bash
581+
uv run pre-commit run --all-files -v
582+
```
583+
584+
This is the same command our Github Actions CI will run when you push a commit.
585+
586+
#### Running tests
587+
588+
```bash
589+
bin/compose-run-pytest.sh
590+
```

0 commit comments

Comments
 (0)