pub-solar/obs-portal
5
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

Use environment variables to configure the portal

Browse source 
- Added an example .env file, where all variables start with `OBS_`
- `OBS_` variables are handled by the portal as configuration variables
- Uncomment some variables in the config.py, since the config.py overrides the env-vars
- Use env-vars and env-file in the docker-compose.yaml
- Add the worker to the docker-compose.yaml
- Add KeyCloak with its own postgres to the docker-compose.yaml
This commit is contained in:
Dennis Boldt 2022-01-03 13:09:54 +01:00 committed by Paul Bienkowski
parent d11d0dbbf0
commit 0c3bda1a89
4 changed files with 430 additions and 102 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

321
deployment/README.md
View file

@ -8,77 +8,209 @@ proxy](https://doc.traefik.io/traefik/) as a reverse proxy, which listens
on port 80 and 443. Based on some labels, traefik routes the domains to the on port 80 and 443. Based on some labels, traefik routes the domains to the
corresponding docker containers. corresponding docker containers.
## Requirements
This guide requires a Linux-system, where `docker` and `docker-compose` are installed.
Ensure, that your system is up to date.
## Before Getting Started ## Before Getting Started
The guide and example configuration assumes one domain, which points to the The example configurations assume two domains, which points to the
server's IP address. This documentation uses `portal.example.com` as an server's IP address. This documentation uses `portal.example.com` and
example. The API is hosted at `https://portal.example.com/api`, while the main `login.example.com`. The API is hosted at `https://portal.example.com/api`,
frontend is reachable at the domain root. while the main frontend is reachable at the domain root.
## Setup instructions ## Setup instructions
First of all, login into your system via SSH.
### Create working directory
Create a folder somewhere in your system, in this guide we use
`/opt/openbikesensor`.
### Clone the repository ### Clone the repository
First create a folder somewhere in your system, in the example we use Clone the repository to `/opt/openbikesensor/`:
`/opt/openbikesensor` and export it as `$ROOT` to more easily refer to it.
Clone the repository to `$ROOT/source`.
```bash ```bash
export ROOT=/opt/openbikesensor cd /opt/openbikesensor/
mkdir -p $ROOT
cd $ROOT
git clone --recursive https://github.com/openbikesensor/portal source/ git clone --recursive https://github.com/openbikesensor/portal source/
# If you accidentally cloned without --recursive, fix it by running: # If you accidentally cloned without --recursive, fix it by running:
# git submodule update --init --recursive # git submodule update --init --recursive
``` ```
Unless otherwise mentioned, commands below assume your current working ### Copy predefined configuration files
directory to be `$ROOT`.
### Configure `traefik.toml`
```bash ```bash
mkdir -p config/ mkdir -p /opt/openbikesensor/config
cd /opt/openbikesensor/
cp source/deployment/examples/docker-compose.yaml docker-compose.yaml
cp source/deployment/examples/.env .env
cp source/deployment/examples/traefik.toml config/traefik.toml cp source/deployment/examples/traefik.toml config/traefik.toml
vim config/traefik.toml cp source/deployment/examples/config.py config/config.py
```
### Create a Docker network
```bash
docker network create gateway
```
### Traefik
#### Configure `traefik.toml`
```bash
cd /opt/openbikesensor/
nano config/traefik.toml
``` ```
Configure your email in the `config/traefik.toml`. This email is used by Configure your email in the `config/traefik.toml`. This email is used by
*Let's Encrypt* to send you some emails regarding your certificates. *Let's Encrypt* to send you some emails regarding your certificates.
#### Start Traefik
### Configure `docker-compose.yaml`
```bash ```bash
cp source/deployment/examples/docker-compose.yaml docker-compose.yaml docker-compose up -d traefik
vim docker-compose.yaml docker-compose logs -f traefik
``` ```
* Change the domain where it occurs, such as in `Host()` rules. ### Generate passwords
* Generate a secure password for the PostgreSQL database user. You will need to
configure this in the application later.
Generate three passords, for example with `pwgen`:
### Create a keycloak instance ```bash
pwgen -2 -n 20
```
Follow the [official guides](https://www.keycloak.org/documentation) to create They will be uses in the next steps.
your own keycloak server. You can run the keycloak in docker and include it in
your `docker-compose.yaml`, if you like.
Documenting the details of this is out of scope for our project. Please make ### KeyCloak
sure to configure:
* An admin account for yourself #### Configure `.env`
* A realm for the portal
* A client in that realm with "Access Type" set to "confidential" and a
redirect URL of this pattern: `https://portal.example.com/login/redirect`
```bash
cd /opt/openbikesensor/
nano .env
```
### Prepare database Configure:
* `OBS_KEYCLOAK_URI`:
* The subdomain of your keycloak
* `OBS_KEYCLOAK_POSTGRES_PASSWORD` and `OBS_KEYCLOAK_ADMIN_PASSWORD`:
* One of the generated passwords for the KeyCloak-postgres
* `OBS_KEYCLOAK_PORTAL_REDIRECT_URI`:
* The Redirect URI, e.g. the subdomain of your portal (ensure, it ends with `/*`)
Run the following two scripts to prepare the database: Wait until postgres and keycloak are started:
* https://login.dennisboldt.de/
#### Configure Realm and Client
Login into your KeyCloak:
```bash
docker-compose exec keycloak /bin/bash
```
Since we configured the `.env`-file we can run the following commands
to create a realm and a client now:
```bash
# Login
/opt/jboss/keycloak/bin/kcadm.sh config credentials --server http://localhost:8080/auth --realm master --user $KEYCLOAK_USER --password $KEYCLOAK_PASSWORD
# Create Realm
/opt/jboss/keycloak/bin/kcadm.sh create realms -s realm=$OBS_KEYCLOAK_REALM -s enabled=true -o
# Create a client and remember the unique id of the client
CID=$(/opt/jboss/keycloak/bin/kcadm.sh create clients -r $OBS_KEYCLOAK_REALM -s clientId=portal -s "redirectUris=[\"$OBS_KEYCLOAK_PORTAL_REDIRECT_URI\"]" -i)
# Create a secret for the client
/opt/jboss/keycloak/bin/kcadm.sh create clients/$CID/client-secret -r $OBS_KEYCLOAK_REALM
# Get the secret of the client
/opt/jboss/keycloak/bin/kcadm.sh get clients/$CID/client-secret -r $OBS_KEYCLOAK_REALM
exit
```
Now, configure the client secret:
```bash
cd /opt/openbikesensor/
nano .env
```
Configure:
* `OBS_KEYCLOAK_CLIENT_SECRET`:
* Use the obtained client secret
#### Create a user
* Login into your Keycloak with the admin user and select the realm obs
* Create a user with username and email (*Hint*: email is required by the portal)
* Configure a password as well
### Portal
#### Configure Postgres
```bash
cd /opt/openbikesensor/
nano .env
```
Configure:
* `OBS_POSTGRES_HOST`:
* The should be the postgres-container, e.g. `postgres`
* `OBS_POSTGRES_USER`:
* The default postgres-user is `obs`
* `OBS_POSTGRES_PASSWORD`:
* Use one of the generated passwords for the postgres
* `OBS_POSTGRES_DB`:
* The default postgres-database is `obs`
* `OBS_POSTGRES_URL`:
* Use the same informations as aboe to configure the `POSTGRES_URL`,
this one is used by the portal.
#### Start Postgres for the portal
```
cd /opt/openbikesensor/
docker-compose up -d postgres
docker-compose logs -f
```
#### Build the portal image
```bash
cd /opt/openbikesensor/
docker-compose build portal
```
*Hint*: This may take up to 10 minutes. In the future, we will provide a prebuild image.
#### Download OpenStreetMap maps
Download the area(s) you would like to import from
[GeoFabrik](https://download.geofabrik.de) into `data/pbf`, for example:
```bash
cd /opt/openbikesensor/
wget https://download.geofabrik.de/europe/germany/schleswig-holstein-latest.osm.pbf -P data/pbf
```
*Hint*: Start with a small region/city, since the import can take some hours for huge areas.
#### Prepare database
Run the following scripts to prepare the database:
```bash ```bash
docker-compose run --rm portal tools/reset_database.py docker-compose run --rm portal tools/reset_database.py
@ -87,15 +219,9 @@ docker-compose run --rm portal tools/prepare_sql_tiles.py
For more details, see [README.md](../README.md) under "Prepare database". For more details, see [README.md](../README.md) under "Prepare database".
### Import OpenStreetMap data #### Import OpenStreetMap data
First of all, download the area(s) you would like to import from [GeoFabrik](https://download.geofabrik.de) into `data/pbf`, for example: Run the following script, to import the OSM data:
```bash
wget https://download.geofabrik.de/europe/germany/schleswig-holstein-latest.osm.pbf -P data/pbf
```
Afterwards,run the following script:
``` ```
docker-compose run --rm portal tools/osm2pgsql.sh docker-compose run --rm portal tools/osm2pgsql.sh
@ -103,62 +229,94 @@ docker-compose run --rm portal tools/osm2pgsql.sh
For more details. see [README.md](../README.md) under "Import OpenStreetMap data". For more details. see [README.md](../README.md) under "Import OpenStreetMap data".
### Configure portal
#### Configure portal
The portal can be configured via env-vars or via the `config.py`.
It's important to know, that the `config.py` overrides the env-vars.
All env-vars start with `OBS_` and will be handled by the application without the prefix.
For example, the env-var `OBS_SECRET` will be same as `SECRET` within the `config.py` and will be `SECRET` within the application.
```bash ```bash
cp source/api/config.py.example config/config.py cd /opt/openbikesensor/
nano .env
``` ```
Then edit `config/config.py` to your heart's content (and matching the Configure:
configuration of the keycloak). Do not forget to generate a secure secret
string.
Also set `PROXIES_COUNT = 1` in your config, even if that option is not * `OBS_PORTAL_URI`:
included in the example file. Read the * The subdomain of your portal
[Sanic docs](https://sanicframework.org/en/guide/advanced/proxy-headers.html) * `OBS_SECRET`:
for why this needs to be done. If your reverse proxy supports it, you can also * Generate a UUID with `uuidgen` and use it as the secret
use a forwarded secret to secure your proxy target from spoofing. This is not * `OBS_POSTGRES_URL`:
required if your application server does not listen on a public interface, but * Should be configured already
it is recommended anyway, if possible. * `OBS_KEYCLOAK_URL`:
* You can find it as the `issuer`, when you click on *OpenID Endpoint Configuration* in the realm obs
* `OBS_KEYCLOAK_CLIENT_SECRET`:
* Should be configured already
* `OBS: DEDICATED_WORKER`
* Should be set to `"True"`, since it the workder will be started with the portal
* `OBS_DATA_DIR`
* The data dir must be the same for the portal and the worer.
The default is `/data` within the containers
* `OBS_PROXIES_COUNT`:
* This sets `PROXIES_COUNT = 1` in your config
* Read the [Sanic docs](https://sanicframework.org/en/guide/advanced/proxy-headers.html)
for why this needs to be done. If your reverse proxy supports it, you can also
use a forwarded secret to secure your proxy target from spoofing. This is not
required if your application server does not listen on a public interface, but
it is recommended anyway, if possible.
### Build container and run them Have a look into the `config.py`, which other variables may affect you.
#### Start the portal
```bash ```bash
docker-compose build portal cd /opt/openbikesensor/
docker-compose up -d portal docker-compose up -d portal
``` ```
## Running a dedicated worker This also starts a dedicated worker container to handle the tracks.
Extend your `docker-compose.yaml` with the following service: #### Test the portal
```yaml * Open: https://obs.example.com/
worker: * Login with the user
image: openbikesensor-portal * Upload a track
build:
context: ./source You should see smth. like:
volumes:
- ./data/api-data:/data > worker_1 | INFO: Track 10b9ulou imported.
- ./config/config.py:/opt/obs/api/config.py
restart: on-failure #### Configre the map position
links:
- postgres Open the tab *Map** an zoom to the desired position. The URL contains the corresponding GPS position,
networks: for example:
- backend
command: > 14/53.86449349032097/10.696108517499198
- python
- tools/process_track.py Configure the map position in the `config.py` and restart the portal:
```
cd /opt/openbikesensor/
nano config/config.py
docker-compose restart portal
``` ```
Change the `DEDICATED_WORKER` option in your config to `True` to stop The tab *Map* should be the selected map section now.
processing tracks in the portal container. Then restart the `portal` service
and start the `worker` service. **Hint**: Probably it's required to disable the browser cache to see the change.
#### Verify osm2pgsql
If you zoom in the tab *Map* at the imported region/city, you should see dark grey lines on the streets.
## Miscellaneous ## Miscellaneous
### Logs ### Logs
To read logs, run To read the logs, run
```bash ```bash
docker-compose logs -f docker-compose logs -f
@ -167,7 +325,6 @@ docker-compose logs -f
If something went wrong, you can reconfigure your config files and rerun: If something went wrong, you can reconfigure your config files and rerun:
```bash ```bash
docker-compose build
docker-compose up -d docker-compose up -d
``` ```

47
deployment/examples/.env Normal file
View file

@ -0,0 +1,47 @@
###################################################
# Keycloak
###################################################
OBS_KEYCLOAK_URI=portal.example.com
# Postgres
OBS_KEYCLOAK_POSTGRES_USER=obs
OBS_KEYCLOAK_POSTGRES_PASSWORD=<<TODO>>
OBS_KEYCLOAK_POSTGRES_DB=obs
# KeyCloak
OBS_KEYCLOAK_POSTGRES_HOST=postgres-keycloak
OBS_KEYCLOAK_ADMIN_USER=admin
OBS_KEYCLOAK_ADMIN_PASSWORD=<<TODO>>
OBS_KEYCLOAK_REALM=obs
OBS_KEYCLOAK_PORTAL_REDIRECT_URI=https://obs.example.com/*
###################################################
# Portal
###################################################
OBS_PORTAL_URI=portal.example.com
# Postgres + osm2pgsql
OBS_POSTGRES_HOST=postgres
OBS_POSTGRES_USER=obs
OBS_POSTGRES_PASSWORD=<<TODO>>
OBS_POSTGRES_DB=obs
# Portal
OBS_HOST=0.0.0.0
OBS_PORT=3000
OBS_SECRET=<<TODO>>
OBS_POSTGRES_URL=postgresql+asyncpg://obs:<<TODO>>@postgres/obs
OBS_KEYCLOAK_URL=https://login.example.com/auth/realms/obs/
OBS_KEYCLOAK_CLIENT_ID=portal
OBS_KEYCLOAK_CLIENT_SECRET=<<TODO>>
OBS_DEDICATED_WORKER="True"
OBS_DATA_DIR=/data
OBS_PROXIES_COUNT=1
###################################################

60
deployment/examples/config.py Normal file
View file

@ -0,0 +1,60 @@
# Bind address of the server
#HOST = "127.0.0.1"
#PORT = 3000
# Extended log output, but slower
DEBUG = False
AUTO_RESTART = DEBUG
# Required to encrypt or sign sessions, cookies, tokens, etc.
#SECRET = "!!!<<<CHANGEME>>>!!!"
# Connection to the database
#POSTGRES_URL = "postgresql+asyncpg://user:pass@host/dbname"
# URL to the keycloak realm, as reachable by the API service. This is not
# necessarily its publicly reachable URL, keycloak advertises that iself.
#KEYCLOAK_URL = "http://localhost:1234/auth/realms/obs/"
# Auth client credentials
#KEYCLOAK_CLIENT_ID = "portal"
#KEYCLOAK_CLIENT_SECRET = "00000000-0000-0000-0000-000000000000"
# Whether the API should run the worker loop, or a dedicated worker is used
#DEDICATED_WORKER = True
# The root of the frontend. Needed for redirecting after login, and for CORS.
# Set to None if frontend is served by the API.
FRONTEND_URL = None
FRONTEND_HTTPS = True
# Where to find the compiled frontend assets (must include index.html), or None
# to disable serving the frontend.
FRONTEND_DIR = "../frontend/build/"
# Can be an object or a JSON string
FRONTEND_CONFIG = {
"imprintUrl": "https://example.com/imprint",
"privacyPolicyUrl": "https://example.com/privacy",
"mapHome": {"zoom": 6, "longitude": 10.2, "latitude": 51.3},
"banner": {"text": "This is a test installation.", "style": "warning"},
}
# If the API should serve generated tiles, this is the path where the tiles are
# built. This is an experimental option and probably very inefficient, a proper
# tileserver should be prefered. Set to None to disable.
TILES_FILE = None
# Path overrides:
# API_ROOT_DIR = "??" # default: api/ inside repository
# DATA_DIR = "??" # default: $API_ROOT_DIR/..
# PROCESSING_DIR = "??" # default: DATA_DIR/processing
# PROCESSING_OUTPUT_DIR = "??" # default: DATA_DIR/processing-output
# TRACKS_DIR = "??" # default: DATA_DIR/tracks
# OBS_FACE_CACHE_DIR = "??" # default: DATA_DIR/obs-face-cache
# Additional allowed origins for CORS headers. The FRONTEND_URL is included by
# default. Python list, or whitespace separated string.
ADDITIONAL_CORS_ORIGINS = None
# vim: set ft=python :

104
deployment/examples/docker-compose.yaml
View file

@ -8,12 +8,17 @@ networks:
internal: true internal: true
services: services:
############################################################
# Portal
############################################################
postgres: postgres:
image: "openmaptiles/postgis:6.0" image: "openmaptiles/postgis:6.0"
environment: environment:
POSTGRES_USER: obs - POSTGRES_DB=${OBS_POSTGRES_DB}
POSTGRES_PASSWORD: obs - POSTGRES_USER=${OBS_POSTGRES_USER}
POSTGRES_DB: obs - POSTGRES_PASSWORD=${OBS_POSTGRES_PASSWORD}
volumes: volumes:
- ./data/postgres/data:/var/lib/postgresql/data - ./data/postgres/data:/var/lib/postgresql/data
networks: networks:
@ -23,32 +28,50 @@ services:
image: openbikesensor-portal image: openbikesensor-portal
build: build:
context: ./source context: ./source
environment: env_file: .env
POSTGRES_HOST: postgres
POSTGRES_USER: obs
POSTGRES_PASSWORD: obs
POSTGRES_DB: obs
volumes: volumes:
- ./data/api-data:/data - ./data/api-data:${OBS_DATA_DIR}
- ./config/config.py:/opt/obs/api/config.py - ./config/config.py:/opt/obs/api/config.py
- ./data/tiles/:/tiles - ./data/tiles/:/tiles
- ./data/pbf/:/pbf - ./data/pbf/:/pbf
restart: on-failure restart: on-failure
depends_on: depends_on:
- traefik
- postgres - postgres
# if you introduce a dockerized keycloak instance within this compose also: - worker
# - keycloak # - keycloak
labels: labels:
- traefik.http.routers.portal.rule=Host(`portal.example.com`) - traefik.http.routers.portal.rule=Host(`${OBS_PORTAL_URI}`)
- traefik.http.routers.portal.entrypoints=websecure - traefik.http.routers.portal.entrypoints=websecure
- traefik.http.routers.portal.tls=true - traefik.http.routers.portal.tls=true
- traefik.http.routers.portal.tls.certresolver=leresolver - traefik.http.routers.portal.tls.certresolver=leresolver
- traefik.docker.network=gateway - traefik.docker.network=gateway
- traefik.http.services.whoami.loadbalancer.server.port=80 # - traefik.http.services.portal.loadbalancer.server.port=3000
networks: networks:
- gateway - gateway
- backend - backend
worker:
image: openbikesensor-portal
build:
context: ./source
env_file: .env
volumes:
- ./data/api-data:${OBS_DATA_DIR}
- ./config/config.py:/opt/obs/api/config.py
restart: on-failure
depends_on:
- postgres
networks:
- backend
command:
- python
- tools/process_track.py
############################################################
# Traefik
############################################################
traefik: traefik:
image: traefik:2.4.8 image: traefik:2.4.8
restart: always restart: always
@ -76,11 +99,52 @@ services:
# Configure middlewares # Configure middlewares
- "traefik.http.middlewares.redirect-http-to-https.redirectscheme.scheme=https" - "traefik.http.middlewares.redirect-http-to-https.redirectscheme.scheme=https"
# Show Traefik Dashboard. Enable the dashboard in traefik.toml if you use these. ############################################################
# - "traefik.http.routers.traefik.rule=Host(`traefik.example.com`)" # Keycloak
# - "traefik.http.routers.traefik.service=api@internal" ############################################################
# - "traefik.http.routers.traefik.tls=true"
# - "traefik.http.routers.traefik.entrypoints=websecure" keycloak:
# - "traefik.http.routers.traefik.tls.certresolver=leresolver" image: jboss/keycloak:15.1.0
# - "traefik.http.routers.traefik.middlewares=basic-auth" restart: always
# - "traefik.http.middlewares.basic-auth.basicauth.usersfile=/usersfile" networks:
- gateway
- backend
env_file: .env
environment:
# database
- DB_VENDOR=postgres
- DB_ADDR=${OBS_KEYCLOAK_POSTGRES_HOST}
- DB_DATABASE=${OBS_KEYCLOAK_POSTGRES_DB}
- DB_USER=${OBS_KEYCLOAK_POSTGRES_USER}
- DB_PASSWORD=${OBS_KEYCLOAK_POSTGRES_PASSWORD}
# admin user
- KEYCLOAK_USER=${OBS_KEYCLOAK_ADMIN_USER}
- KEYCLOAK_PASSWORD=${OBS_KEYCLOAK_ADMIN_PASSWORD}
- PROXY_ADDRESS_FORWARDING=true
- OBS_KEYCLOAK_PORTAL_REDIRECT_URI=${OBS_KEYCLOAK_PORTAL_REDIRECT_URI}
depends_on:
- traefik
- postgres-keycloak
labels:
- "traefik.http.routers.login.rule=Host(`${OBS_KEYCLOAK_URI}`)"
- "traefik.http.routers.login.entrypoints=websecure"
- "traefik.http.routers.login.tls=true"
- "traefik.http.routers.login.tls.certresolver=leresolver"
# This container runs on two ports (8080/tcp, 8443/tcp). Tell traefik, which one to use.
- "traefik.http.services.login.loadbalancer.server.port=8080"
# This container runs on more than one network. Tell traefik, which one to use.
- "traefik.docker.network=gateway"
postgres-keycloak:
image: postgres:13.3
restart: always
networks:
- backend
volumes:
- ./data/postgres-keycloak:/var/lib/postgresql/data
environment:
- POSTGRES_DB=${OBS_KEYCLOAK_POSTGRES_DB}
- POSTGRES_USER=${OBS_KEYCLOAK_POSTGRES_USER}
- POSTGRES_PASSWORD=${OBS_KEYCLOAK_POSTGRES_PASSWORD}
labels:
- traefik.enable=false