Local deployment

Development setup

For development, it will be helpful to run dserver itself uncontainerized. The folder devel contains helper components to set up a simple development setup. To run a meaningful dserver instance, underlying services like databases and storage infrastructure are necessary. For the development setup described here, they can be provided with

docker compose -f docker/env.yml up -d

We will install all pythonic components of dserver in a virtual environment.

Create and activate virtual environment with

python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip

The container environment provides a PostgreSQL instance. For dserver to talk with PostgreSQL seamlessly, please install

sudo apt install libpq-dev
pip install gunicorn psycopg2

on Ubuntu.

The container environment also provides an S3 server for testing. For allowing dserver to ingest datasets from an S3 bucket,

pip install dtool-s3

dtool.json provides a minimal configuration for a lookup server launched as described in the following. Place it at ${HOME}/.config/dtool/dtool.json with

cp devel/dtool.json ~/.config/dtool/dtool.json

The script devel/create_test_data.sh showcases the creation of two datasets by means of the dtool command line interface (CLI).

To make the dtool CLI available, run

pip install dtool-cli dtool-info dtool-create

Now, run create_test_data.sh to create test data sets and copies them to a test s3 bucket.

For installing dserver by help of this meta package dserver-minimal, run

pip install dserver-minimal

Alternatively, install dserver’s core as well as search and retrieve plugins individually, e.g.

pip install dservercore
pip install dserver-search-plugin-mongo
pip install dserver-retrieve-plugin-mongo

For an extended setup with support for direct mongo queries to the server, dependency graph queries, and ingestion of datasets on S3 notifications, also install

pip install dserver-direct-mongo-plugin
pip install dserver-dependency-graph-plugin
pip install dserver-notification-plugin

For development purposes, you might want to install these core components and plugins listed above in editable mode from local copies of the repositories, i.e.

git clone https://github.com/jic-dtool/dservercore.git
cd dservercore
pip install -e .

for dservercore, and in similar fashion for the plugins.

env.rc provides a default flask app configuration in form of environment variables. Inspect and modify as needed, and export them to your current shell environment with

source devel/env.rc

Prepare dserver with

bash devel/init.sh

to initilize databases, apply database migrations, create the single user test-user, register the base URI s3://test-bucket, and grant permissions.

Eventually, run

bash devel/run.sh

to launch dserver with gunicorn on localhost:5000.

In case of a successful launch, http://localhost:5000/doc/swagger should expose the REST API documentation.

The route /config/versions is accessible without authentication and authorization and should list all installed server-side plugins with their versions.

For testing other routes that require authorization, retrieve a JWT token for test-user with

curl --insecure -H "Content-Type: application/json" \
   -X POST -d '{"username": "test-user", "password": "test-password" }' \
   http://localhost:5001/token

Docker

The folder docker contains example Docker configuration files that allow running dserver.

Development deployment

We provide a containerized development deployment of dserver. The containers are run using docker compose. The compose environment sets up Postgres and Mongo databases and starts a Minio (S3-compatible) object store. The startup process pushes two example datasets to the object store. Furthermore, an LDAP server and an LDAP-backed token generator for the dserver are launched to allow running other components, such as the dtool-lookup-webapp, that depend on authentication by user credentials, against the local lookup server deployment.

Build the compose environment with

docker compose -f docker/devel.yml build

and start it with

docker compose -f docker/devel.yml up

The lookup server is then available at https://localhost:5000. A single user with the name test-user is already registered. To generate a token for this user, run

docker compose -f docker/devel.yml exec dserver flask user token test-user

The LDAP allows authentication of this user with password test-password.

docker/dtool.json contains a sample dtool configuration for accessing lookup server and token generator from localhost.

Development dependencies deployment

Similar to the development setup above, but only provides all services the lookup server depends on, not the lookup server itself. Useful for playing with different plugin constellations.

To share keys between containerized token generator and the host environment, keys are generated on a bind mount when launching the composition. Prepare an empty folder keys within this directory (meaning at docker/keys relative to the repository root) before launching any container composition. Otherwise, docker compose will not launch.

Build the compose environment with

docker compose -f docker/env.yml build

and start it with

docker compose -f docker/env.yml up -d

Make keys generated at container launch readable by current user on host with

sudo chown -R ${USER}:${USER} docker/keys

Token

The container composition provides an LDAP server and a token generator service. When running the services locally, generate a token with

curl --insecure -H "Content-Type: application/json" \
   -X POST -d '{"username": "test-user", "password": "test-password" }' \
   http://localhost:5001/token

Single container

The subfolder single-container contains a complete dserver setup within a single container. Build with

docker build -t livmats/dserver-minimal docker/single-container

and run with

docker run -v $(pwd)/sample-datasets:/tmp/data:ro -p 8888:8888 livmats/dserver-minimal

to build an index oer datasets in local folder $(pwd)/sample-datasets`.