Start postgres before preparing the database

1) EXPOSE 3000 in Dockerfile + set loadbalancer port to 3000 in docker-compose.yml
2) Add reference to a manually installation of osm2pgsql
3) Add -W to osm2pgsql, which asks for the postgres password
4) Added the manual creation of the docker network `gateway`
5) Changed deployment order: build portal -> prepeare database -> import Osm -> run the container
6) Add localhost binding of postgres to docker-compose.yml (optional)

Dockerfile

@@ -40,7 +40,7 @@ RUN pip install -e /opt/obs/api/
 
 COPY --from=frontend-builder /opt/obs/frontend/build /opt/obs/frontend/build
 
-EXPOSE 8000
+EXPOSE 3000
 
 CMD ["openbikesensor-api"]
 

README.md

@@ -172,13 +172,14 @@ development, you should choose a small area for testing, such as your local
 county or city, to keep the amount of data small. For production use you have
 to import the whole region you are serving.
 
-* Install `osm2pgsql`. 
+* Install `osm2pgsql`. We tested it with version 1.5.1. If the version of your
+  package manger is older, we recommend to [install it from the soruces](https://github.com/openstreetmap/osm2pgsql).
 * Download the area(s) you would like to import from [GeoFabrik](https://download.geofabrik.de). 
 * Import each file like this:
 
     ```bash
     osm2pgsql --create --hstore --style api/roads_import.lua -O flex \
-      -H localhost -d obs -U obs \
+      -H localhost -d obs -U obs -W \
       path/to/downloaded/myarea-latest.osm.pbf 
     ```
 

deployment/README.md

@@ -36,7 +36,6 @@ git clone --recursive https://github.com/openbikesensor/portal source/
 Unless otherwise mentioned, commands below assume your current working
 directory to be `$ROOT`.
 
-
 ### Configure `traefik.toml`
 
 ```bash
@@ -48,7 +47,6 @@ vim config/traefik.toml
 Configure your email in the `config/traefik.toml`. This email is used by
 *Let's Encrypt* to send you some emails regarding your certificates.
 
-
 ### Configure `docker-compose.yaml`
 
 ```bash
@@ -60,12 +58,43 @@ vim docker-compose.yaml
 * Generate a secure password for the PostgreSQL database user. You will need to
   configure this in the application later.
 
+### Create a docker network
+
+The `docker-compose.yml` [requires an external docker network called](examples/docker-compose.yaml#L4)
+`gateway`.
+Thus, create the docker network:
+
+```
+docker network create gateway
+```
+
+If you don't create it, you will see the following error:
+
+> ERROR: Network gateway declared as external, but could not be found.
+
+Please create the network manually using `docker network create gateway` and
+try again.
+
+### Start traefik
+
+Now is a good time to start and test traefik:
+
+```
+docker-compose up -d traefik
+```
+
+Open your domain (e.g., `https://portal.example.com/`) should work already, but
+it should give you an HTTPS securiy warning, since no service is configured to
+handle the domain yet. Beacuse of this, you should also see a
+`404 page not found` when you confirm the warning.
 
 ### Create a keycloak instance
 
 Follow the [official guides](https://www.keycloak.org/documentation) to create
-your own keycloak server. You can run the keycloak in docker and include it in
+your own keycloak server. If you like, You can run the keycloak in docker as
-your `docker-compose.yaml`, if you like.
+well. Thus, you can include it in your `docker-compose.yaml` and use traefik.
+
+Keep in mind, that your keycloak server requires a separate domain.
 
 Documenting the details of this is out of scope for our project. Please make
 sure to configure:
@@ -75,11 +104,47 @@ sure to configure:
 * A client in that realm with "Access Type" set to "confidential" and a
   redirect URL of this pattern: `https://portal.example.com/login/redirect`
 
+### Configure portal
+
+```bash
+cp source/api/config.py.example config/config.py
+```
+
+Then edit `config/config.py` to your heart's content (and matching the
+configuration of the keycloak). Do not forget to generate a secure secret
+string.
+
+Ensure to set `HOST = "0.0.0.0"`, thus the webserver can be accessed by docker.
+
+Also set `PROXIES_COUNT = 1` in your config, even if that option is not
+included in the example file. Read the 
+[Sanic docs](https://sanicframework.org/en/guide/advanced/proxy-headers.html) 
+for why this needs to be done. If your reverse proxy supports it, you can also
+use a forwarded secret to secure your proxy target from spoofing. This is not
+required if your application server does not listen on a public interface, but
+it is recommended anyway, if possible.
+
+### Build portal container
+
+```bash
+docker-compose build portal
+```
+
+This container is used for the next three steps.
+
+
+#### Start postgres
+
+```bash
+docker-compose up -d postgres
+```
 
 ### Prepare database
 
-Follow the procedure outlined in [README.md](../README.md) under "Prepare
+Follow the procedure outlined in [README.md](../README.md#prepare-database)
-database". Whenever the docker-compose service `api` is referenced, replace it
+under "Prepare database".
+
+Whenever the docker-compose service `api` is referenced, replace it
 with `portal`, which contains the same python code as the development `api`
 service, but also the frontend. For example:
 
@@ -92,31 +157,12 @@ docker-compose run --rm portal tools/prepare_sql_tiles.py
 
 ### Import OpenStreetMap data
 
-Follow the procedure outlined in [README.md](../README.md) under "Import OpenStreetMap data".
+Follow the procedure outlined in [README.md](../README.md#import-openstreetmap-data)
+under "Import OpenStreetMap data".
 
-
+#### Run portal container
-### Configure portal
 
 ```bash
-cp source/api/config.py.example config/config.py
-```
-
-Then edit `config/config.py` to your heart's content (and matching the
-configuration of the keycloak). Do not forget to generate a secure secret
-string.
-
-Also set `PROXIES_COUNT = 1` in your config, even if that option is not
-included in the example file. Read the 
-[Sanic docs](https://sanicframework.org/en/guide/advanced/proxy-headers.html) 
-for why this needs to be done. If your reverse proxy supports it, you can also
-use a forwarded secret to secure your proxy target from spoofing. This is not
-required if your application server does not listen on a public interface, but
-it is recommended anyway, if possible.
-
-### Build container and run them
-
-```bash
-docker-compose build portal
 docker-compose up -d portal
 ```
 
@@ -130,7 +176,7 @@ Extend your `docker-compose.yaml` with the following service:
     build:
       context: ./source
     volumes:
-      - ./data/api-data:/data
+      - ./data/api-data:/opt/obs/data
       - ./config/config.py:/opt/obs/api/config.py
     restart: on-failure
     links:
@@ -142,7 +188,7 @@ Extend your `docker-compose.yaml` with the following service:
       - tools/process_track.py
 ```
 
-Change the `DEDICATED_WORKER` option in your config to `True` to stop
+Change the `DEDICATED_WORKER` option in your `config.py` to `True` to stop
 processing tracks in the portal container. Then restart the `portal` service
 and start the `worker` service.
 

deployment/examples/docker-compose.yaml

@@ -1,4 +1,4 @@
-version: '3'
+version: '3.5'
 
 networks:
   gateway:
@@ -16,13 +16,21 @@ services:
       POSTGRES_DB: obs
     volumes:
       - ./data/postgres/data:/var/lib/postgresql/data
+    labels:
+      - traefik.enable=false
+    networks:
+      - backend
+    # Uncomment, when you want to run osm2pgsql
+    #  - gateway
+    #ports:
+    #  - '127.0.0.1:5432:5432'
 
   portal:
     image: openbikesensor-portal
     build:
       context: ./source
     volumes:
-      - ./data/api-data:/data
+      - ./data/api-data:/opt/obs/data
       - ./config/config.py:/opt/obs/api/config.py
       - ./data/tiles/:/tiles
     restart: on-failure
@@ -34,13 +42,13 @@ services:
       - traefik.http.routers.portal.tls=true
       - traefik.http.routers.portal.tls.certresolver=leresolver
       - traefik.docker.network=gateway
-      - traefik.http.services.whoami.loadbalancer.server.port=80
+      - traefik.http.services.portal.loadbalancer.server.port=3000
     networks:
       - gateway
       - backend
 
   traefik:
-    image: traefik:2.4.8
+    image: traefik:2.5.5
     restart: always
     ports:
       - "80:80"