Docker Builds¶
This section describes how to build the Docker images and also the VarFish site data tarballs. The intended audience are VarFish developers.
Build Docker Images¶
Building the image:
$ ./docker/build-docker.sh
By default the latest tag is used. You can change this with.
$ GIT_TAG=v0.1.0 ./docker/build-docker.sh
Get varfish-docker-compose¶
The database is built in varfish-docker-compose.
$ git clone git@github.com:bihealth/varfish-docker-compose.git
$ cd varfish-docker-compose
$ ./init.sh
First-Time Container Startup¶
You have to startup the postgres container once to create the Postgres database. Once it has been initialized, shutdown with Ctrl-C.
$ docker-compose up postgres
<Ctrl-C>
Now copy over the postgresql.conf file that has been tuned for the VarFish use cases.
$ cp config/postgres/postgresql.conf volumes/postgres/data/postgresql.conf
Bring up the site again so we can build the database.
$ docker-compose up
Wait until varfish-web is up and running and all migrations have been applied, look for VARFISH MIGRATIONS END in the output of run-docker-compose-up.sh.
Pre-Build Postgres Database¶
Download static data
$ cd /plenty/space
$ wget https://file-public.bihealth.org/transient/varfish/varfish-server-background-db-20201006.tar.gz{,.sha256}
$ sha256sum -c varfish-server-background-db-20201006.tar.gz.sha256
$ tar xzvf varfish-server-background-db-20201006.tar.gz
Adjust the docker-compose.yml file such that /plenty/space is visible in the varfish-web container.
volumes:
- "/plenty/space:/data"
Get the name of the running varfish-web container.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
44be6ece102e minio/minio "/usr/bin/docker-ent…" 11 minutes ago Up About a minute 9000/tcp varfish-docker-compose_minio_1
3b23113e5aa1 quay.io/biocontainers/exomiser-rest-prioritiser:12.1.0--1 "exomiser-rest-prior…" 11 minutes ago Up About a minute varfish-docker-compose_exomiser-rest-prioritiser_1
b8c49e8c24a6 quay.io/biocontainers/jannovar-cli:0.33--0 "jannovar -Xmx6G -Xm…" 11 minutes ago Up About a minute varfish-docker-compose_jannovar_1
409a535b9951 bihealth/varfish-server:0.22.1-0 "docker-entrypoint.s…" 12 minutes ago Up About a minute 8080/tcp varfish-docker-compose_varfish-celerybeat_1
7eb7425c59e2 bihealth/varfish-server:0.22.1-0 "docker-entrypoint.s…" 12 minutes ago Up About a minute 8080/tcp varfish-docker-compose_varfish-celeryd-import_1
020811fde306 bihealth/varfish-server:0.22.1-0 "docker-entrypoint.s…" 12 minutes ago Up About a minute 8080/tcp varfish-docker-compose_varfish-celeryd-query_1
87b03ee0249b bihealth/varfish-server:0.22.1-0 "docker-entrypoint.s…" 12 minutes ago Up About a minute 8080/tcp varfish-docker-compose_varfish-celeryd-default_1
7a3fdb337fae bihealth/varfish-server:0.22.1-0 "docker-entrypoint.s…" 12 minutes ago Up About a minute 8080/tcp varfish-docker-compose_varfish-web_1
9295a101570f postgres:12 "docker-entrypoint.s…" 12 minutes ago Up About a minute 5432/tcp varfish-docker-compose_postgres_1
1c4d6e235074 traefik:v2.3.1 "/entrypoint.sh --pr…" 12 minutes ago Up About a minute 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp varfish-docker-compose_traefik_1
8d72fd096743 redis:6 "docker-entrypoint.s…" 12 minutes ago Up About a minute 6379/tcp varfish-docker-compose_redis_1
Initialize the tables (while at least docker-compose up varfish-web postgres redis is running).
$ docker exec -it -w /usr/src/app varfish-docker-compose_varfish-web_1 python manage.py import_tables --tables-path /data --threads 8
Then, shutdown the docker-compose up, remove the volumes: entry for varfish-web, and create a tarball of the postgres database to have a clean copy.
Add Other Data¶
Copy the other required data for jannovar and exomiser.
The result should look similar to this:
# tree volumes/jannovar volumes/exomiser
volumes/jannovar
├── hg19_ensembl.ser
├── hg19_refseq_curated.ser
└── hg19_refseq.ser
volumes/exomiser
├── 1909_hg19
│ ├── 1909_hg19_clinvar_whitelist.tsv.gz
. . [..]
│ └── 1909_hg19_variants.mv.db
└── 1909_phenotype
├── 1909_phenotype.h2.db
├── phenix
│ ├── 10.out
. . [..]
│ ├── ALL_SOURCES_ALL_FREQUENCIES_genes_to_phenotype.txt
│ ├── hp.obo
│ └── phenotype_annotation.tab
└── rw_string_10.mv
3 directories, 55 files
Create a Superuser¶
While the docker-compose up is running
$ docker exec -it -w /usr/src/app varfish-docker-compose_varfish-web_1 python manage.py createsuperuser
Username: root
Email address:
Password: <changeme>
Password (again): <changeme>
Superuser created successfully.
Setup Initial Data¶
Create test category & project.
Obtain API key and configure varfish-cli.
Import some test data through the API.
$ varfish-cli --no-verify-ssl case create-import-info --resubmit \
92f5d735-0967-4db2-a801-50fe96359f51 \
$(find path/to/variant_export/work/*NA12878* -name '*.tsv.gz' -or -name '*.ped')
Create Data Tarballs¶
Now create the released data tarballs.
tar -cf - volumes | pigz -c > varfish-site-data-v0.22.2-20210212.tar.gz && sha256sum varfish-site-data-v0.22.2-20210212.tar.gz >varfish-site-data-v0.22.2-20210212.tar.gz.sha256 &
tar -cf - test-data | pigz -c > varfish-test-data-v0.22.2-20210212.tar.gz && sha256sum varfish-test-data-v0.22.2-20210212.tar.gz >varfish-test-data-v0.22.2-20210212.tar.gz.sha256