Skip to content

Commit 46b932a

Browse files
pichuanpichuan
committed
Update.
1 parent e344489 commit 46b932a

5 files changed

Lines changed: 110 additions & 35 deletions

File tree

‎deepconsensus/scripts/__init__.py‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Copyright (c) 2021, Google Inc.
2+
# All rights reserved.
3+
#
4+
# Redistribution and use in source and binary forms, with or without
5+
# modification, are permitted provided that the following conditions
6+
# are met:
7+
#
8+
# 1. Redistributions of source code must retain the above copyright notice,
9+
# this list of conditions and the following disclaimer.
10+
#
11+
# 2. Redistributions in binary form must reproduce the above copyright
12+
# notice, this list of conditions and the following disclaimer in the
13+
# documentation and/or other materials provided with the distribution.
14+
#
15+
# 3. Neither the name of Google Inc. nor the names of its
16+
# contributors may be used to endorse or promote products derived from this
17+
# software without specific prior written permission.
18+
#
19+
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
20+
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
21+
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
22+
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
23+
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
24+
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
25+
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
26+
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
27+
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
28+
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
29+
# POSSIBILITY OF SUCH DAMAGE.
30+
"""__init__.py."""

‎docs/quick_start.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,13 @@ You can ignore errors regarding google-nucleus installation, such as:
6666
ERROR: Failed building wheel for google-nucleus
6767
```
6868

69+
(Optional) After `source install.sh`, if you want to run all unit tests, you can
70+
do:
71+
72+
```bash
73+
./run_all_tests.sh
74+
```
75+
6976
## Run DeepConsensus
7077

7178
```bash

‎install.sh‎

Lines changed: 3 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -41,8 +41,6 @@
4141
# Global setting for nucleus builds
4242
# ------------------------------------------------------------------------------
4343

44-
NUCLEUS_BAZEL_VERSION="3.1.0"
45-
NUCLEUS_TENSORFLOW_VERSION="2.4.0"
4644
NUCLEUS_PIP_VERSION=0.5.8
4745

4846
function note_build_stage {
@@ -54,50 +52,20 @@ function note_build_stage {
5452
note_build_stage "Update package list"
5553
sudo -H apt-get -qq -y update
5654

57-
# Install generic dependencies
58-
################################################################################
59-
note_build_stage "Update misc. dependencies"
60-
sudo -H apt-get -y install pkg-config zip g++ zlib1g-dev unzip curl git lsb-release
61-
62-
# <internal>
63-
# Install htslib dependencies
64-
################################################################################
65-
note_build_stage "Install htslib dependencies"
66-
sudo -H apt-get -y install libssl-dev libcurl4-openssl-dev liblz-dev libbz2-dev liblzma-dev
67-
6855
# Install pip
6956
################################################################################
7057
note_build_stage "Update pip"
71-
sudo -H apt-get -y install python3-dev python3-pip python3-wheel python3-setuptools
58+
sudo -H apt-get -y install python3-dev python3-pip
7259
sudo -H apt-get -y update
73-
# TensorFlow 2.0 requires pip >= 19.0
74-
python3 -m pip install --user -U "pip==20.1.1"
60+
python3 -m pip install --upgrade pip
7561

7662
# Update PATH so that newly installed pip is the one we actually use.
7763
export PATH="$HOME/.local/bin:$PATH"
7864
echo "$(pip --version)"
7965

8066
# Install python packages used by DeepConsensus.
8167
################################################################################
82-
python3 -m pip install --user 'wheel>=0.36'
83-
python3 -m pip install --user 'numpy>=1.19'
84-
python3 -m pip install --user 'pandas>=1.1'
85-
python3 -m pip install --user 'tensorflow==2.4.0'
86-
python3 -m pip install --user 'tf-models-official==2.4.0'
87-
python3 -m pip install --user 'ml_collections>=0.1.0'
88-
# Use apache-beam==2.20.0 for compatible protobuf version:
89-
# https://github.com/apache/beam/blob/release-2.20.0/sdks/python/container/base_image_requirements.txt#L38
90-
python3 -m pip install --user 'apache-beam==2.20.0'
91-
python3 -m pip install --user 'apache-beam[test]==2.20.0'
92-
python3 -m pip install --user "google-nucleus==${NUCLEUS_PIP_VERSION}"
93-
python3 -m pip install --user 'absl-py>=0.13.0'
94-
95-
# We need a recent version of setuptools, because pkg_resources is included in
96-
# setuptools, and protobuf's __init__.py contains the line
97-
# __import__('pkg_resources').declare_namespace(__name__)
98-
# and only recent versions of setuptools correctly sort the namespace
99-
# module's __path__ list when declare_namespace is called.
100-
python3 -m pip install --user 'setuptools>=49.6.0'
68+
python3 -m pip install --user -r requirements.txt
10169

10270
# Pre-compile the proto file.
10371
mkdir -p nucleus/protos

‎requirements.txt‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
numpy>=1.19,<1.20
2+
pandas>=1.1,<1.2
3+
tensorflow==2.4.0
4+
tf-models-official==2.3.0 # 2.4.0 seems to require Python3.7.
5+
ml_collections>=0.1.0
6+
# Use apache-beam==2.20.0 for compatible protobuf version:
7+
# https://github.com/apache/beam/blob/release-2.20.0/sdks/python/container/base_image_requirements.txt#L38
8+
apache-beam==2.20.0
9+
google-nucleus==0.5.8 # NUCLEUS_PIP_VERSION
10+
absl-py>=0.13.0

‎run_all_tests.sh‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#!/bin/bash
2+
# Copyright (c) 2021, Google Inc.
3+
# All rights reserved.
4+
#
5+
# Redistribution and use in source and binary forms, with or without
6+
# modification, are permitted provided that the following conditions
7+
# are met:
8+
#
9+
# 1. Redistributions of source code must retain the above copyright notice,
10+
# this list of conditions and the following disclaimer.
11+
#
12+
# 2. Redistributions in binary form must reproduce the above copyright
13+
# notice, this list of conditions and the following disclaimer in the
14+
# documentation and/or other materials provided with the distribution.
15+
#
16+
# 3. Neither the name of Google Inc. nor the names of its
17+
# contributors may be used to endorse or promote products derived from this
18+
# software without specific prior written permission.
19+
#
20+
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
21+
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
22+
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
23+
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
24+
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
25+
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
26+
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
27+
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
28+
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
29+
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
30+
# POSSIBILITY OF SUCH DAMAGE.
31+
#
32+
# Script to test DeepConsensus.
33+
# Tested with Python3.6.
34+
# Before running this, run:
35+
# source install.sh
36+
37+
set -euo pipefail
38+
39+
python3 -m deepconsensus.models.data_providers_test
40+
python3 -m deepconsensus.models.losses_and_metrics_test
41+
python3 -m deepconsensus.models.majority_vote_transforms_test
42+
python3 -m deepconsensus.models.model_inference_test
43+
python3 -m deepconsensus.models.model_inference_transforms_test
44+
python3 -m deepconsensus.models.model_inference_with_beam_test
45+
python3 -m deepconsensus.models.model_train_custom_loop_test
46+
python3 -m deepconsensus.models.model_utils_test
47+
python3 -m deepconsensus.models.networks_test
48+
python3 -m deepconsensus.models.run_majority_vote_model_test
49+
python3 -m deepconsensus.postprocess.stitch_predictions_test
50+
python3 -m deepconsensus.postprocess.stitch_predictions_transforms_test
51+
python3 -m deepconsensus.preprocess.beam_io_test
52+
python3 -m deepconsensus.preprocess.generate_input_test
53+
python3 -m deepconsensus.preprocess.generate_input_transforms_test
54+
python3 -m deepconsensus.preprocess.merge_datasets_test
55+
python3 -m deepconsensus.preprocess.merge_datasets_transforms_test
56+
python3 -m deepconsensus.preprocess.preprocess_utils_test
57+
python3 -m deepconsensus.tf_examples.tf_example_transforms_test
58+
python3 -m deepconsensus.tf_examples.tf_example_utils_test
59+
python3 -m deepconsensus.tf_examples.write_tf_examples_test
60+
python3 -m deepconsensus.utils.utils_test

0 commit comments

Comments
 (0)