From c353d2afc17e8d853180d30a2ab3bb2636db27bc Mon Sep 17 00:00:00 2001 From: Paul Bienkowski Date: Sat, 13 Nov 2021 02:46:15 +0100 Subject: [PATCH] Lots of documentation for new development setup as well as upgrade guide --- README.md | 198 +++++++++++++++++++-------------------------------- UPGRADING.md | 31 ++++++++ 2 files changed, 105 insertions(+), 124 deletions(-) create mode 100644 UPGRADING.md diff --git a/README.md b/README.md index 546fa18..af54afe 100644 --- a/README.md +++ b/README.md @@ -25,48 +25,85 @@ git submodule update --init --recursive ## Production setup -**Note:** There is now a guide for a dockerized deployment in the -[deployment](deployment) folder. +There is a guide for a deployment based on docker in the +[deployment](deployment) folder. Lots of non-docker deployment strategy are +possible, but they are not "officially" supported, so please do not expect the +authors of the software to assist in troubleshooting. -You should be familiar with managing a Linux server. If not, find a suitable -guide first. This will only give a rough outline of the steps to take, you must -make sure to properly set up and secure your server yourself. +This is a rather complex application, and it is expected that you know the +basics of deploying a modern web application securely onto a production server. +We are sorry that we cannot guide you through all the details of that, as we +just don't have the capacities to do so. Please research the respective topics +first. If you struggle with application-specific issues, please let us know, we +might be able to assist with those. + +Please note that you will always need to install your own reverse proxy that +terminates TLS for you and handles certificates. We do not support TLS directly +in the application, instead, please use this prefered method. + +### Migrating (Production) + +Migrations are not implemented yet. Once we need them, we'll add them and +document the usage here. + +### Upgrading from v0.2 to v0.3 + +After v0.2 we switched the underlying technology of the API and the database. +We now have no more MongoDB, instead, everything has moved to the PostgreSQL +installation. For development setups, it is advised to just reset the whole +state (remove the `local` folder) and start fresh. For production upgrades, +please follow the relevant section in [`UPGRADING.md`](./UPGRADING.md). -1. Create a user for running the application. It is not recommended to run as a - user that is also used for other things. Do not run as root! -2. Clone the repository. -3. Install `node` of at least version 15, and also `npm`. You can use the - [Node Version Manager (nvm)](https://github.com/nvm-sh/nvm), since this project - contains a `.nvmrc`. Just run `nvm install` followed by `nvm use`. -4. Run `npm ci` in the `api` and `frontend` directories to install dependencies. -5. Install and configure a MongoDB somewhere. -6. Copy `api/config.json.example` to `api/config.json` and change it to suit - your setup. Make sure to only use https URLs. Generate secure secrets. - Customize client IDs. -7. Copy `frontend/src/config.json.example` to `frontend/src/config.json` and - adjust. -8. Run `npm run build` in the frontend directory. This needs to be done after - changing the config, so if you did something wrong, re-run the build. If you - run your frontend on a non-root URL, run the build with the - `PUBLIC_URL=/prefix` environment variable. -9. Create a systemd-user service, or use tmux, or do whatever you like to start - the API service (`npm start` in `api/` directory). It should run as the - dedicated user, and expose `127.0.0.1:3000`. Use `PORT` environment variable - if you want to change the port. Make sure the service starts with - environment `NODE_ENV=production`, too. -10. Configure nginx or your reverse-proxy of choice to forward API requests to - the API port, and serve static files from the `frontend/build/` folder. ## Development setup We've moved the whole development setup into Docker to make it easy for -everyone to get involved. After sucessfully [installing Docker -Engine](https://docs.docker.com/engine/install/) as well as [Docker -Compose](https://docs.docker.com/compose/install/) onto your machine, and -cloning the repository, all you need to do is: +everyone to get involved. + +### Install docker + +Please [install Docker Engine](https://docs.docker.com/engine/install/) as well as +[Docker Compose](https://docs.docker.com/compose/install/) onto your machine. + +Then clone the repository as described above. + +### Configure Keycloak + + +Login will not be possible until you configure the keycloak realm correctly. Boot your keycloak instance: ```bash -docker-compose up -d --build frontend +docker-compose up -d --build keycloak +``` + +Now navigate to http://localhost:3003/ and follow these steps: + +* Click "Administration Console" and log in with `admin` / `admin` +* Hover over the realm name on the top left and click "Add realm" +* Name the Realm `OBS Dev` (spelling matters) and create it +* In the sidebar, navigate to Configure -> Clients, and click "Create" on the top right +* Client ID is `portal`. Hit "Save". +* In the Tab "Settings", edit the new client's "Access Type" to `confidential` + and enter as "Valid Redirect URIs": `http://localhost:3000/login/redirect`, + then "Save" +* Under "Credentials", copy the "Secret" and paste it into `api/config.dev.py` + as `KEYCLOAK_CLIENT_SECRET`. Please do not commit this change to git. +* In the sidebar, navigate to Manage -> Users, and click "Add user" on the top right. +* Give the user a name (e.g. `test`), leave the rest as-is. +* Under the tab "Credentials", set a new password, and make it non-temporary. + Press "Set Password". + +We are going to automate this process. For now, you will have to repeat it +every time you reset the keycloak datbaase, which is inside the PostgreSQL. The +script `api/tools/reset_database.py` does *not* affect the state of the +keycloak database, however, so this should be rather rare. + +### Boot the application + +Now you can run the remaining parts of the application: + +```bash +docker-compose up -d --build api worker frontend ``` If this does not work, please open an issue and describe the problem you're @@ -77,97 +114,10 @@ http://localhost:3000 -- but you probably only need to access the frontend for testing. The frontend dev server also proxies all unknown requests to the API, so the frontend always just requests data at its own URL. -## Running without docker +### Migrating (Development) -If you don't like docker, or want to run this in production without it, you can -do so as well. Our Docker setup is simply a slight wrapper around very simple -JavaScript packages that you can install yourself as usual, with `npm install`. -The API can be started with `npm start` inside its folder. The frontend -development server uses `npm start` as well, while building a production -version of the frontend happens with `npm run build`. - -To connect the parts together, please have a look at what we're doing in the -"official" setup of docker, i.e. in `docker-compose.yaml`, the `Dockerfile`s -and in the respective `package.json` of the service. If you've done this kind -of thing before, it's not that hard. Otherwise, ask on Slack and there will be -somebody to help you ;) - -## Running in production - -You are advised not to use the dockerized mongodb service and instead do a -proper MongoDB setup on a server that is backed up and secured. - -You can run the application in docker, but it is prefered to run it as a restricted -user in its own directory somewhere where it cannot escape ;) - -The frontend should be built using `npm run build` and then served from a -proper web server, such as nginx or apache. See the instructions at -create-react-app concerning [deployment of an app](http://cra.link/deployment). - -You are advised to virtualize your server for security reason, and separate -this whole application from other parts of your server system. - -Also please install a reverse proxy that terminates TLS for you and handles -certificates. We do not support TLS directly in the application, instead, -please use this prefered method. This reverse proxy can also handle static file -serving for the frontend, no need for two separate server processes. - -## Migrating - -Sometimes your database will have to be migrated. The docker setup should do -this automatically, but if it does not work, you can run the following -commands: - -```bash -# if running locally -(cd api/; npm run migrate:up) - -# if running in docker -docker-compose run --rm api npm run migrate:up -```` - -## Custom MongoDB installation - -If you have your own MongoDB instance running somewhere, you can set the -environment variable `MONGODB_URL` when starting the server, and it will read -that URL for connecting. - - export MONGODB_URL=mongodb://user:password@mongodb.example.com/obs-app-database - -This does not work when using docker-compose, in that case, you will have to -modify the `docker-compose.yaml` to include that URL. - - -## E-Mail Setup - -By default in development mode mails are not sent, but instead the mail data is -logged to the console. This can be overriden with the `--devSendMails` flag if -you start the application like so: `npm run dev -- --devSendMails`. - -Mails are also always sent in production mode! - -For actually sending e-mails the mailserver, sender, user and password for the -SMTP server need to be specified as environment variables: - -* `MAILUSER` -- the smtp mailbox login name -* `MAILPW` -- password for the mailbox -* `MAILSERVER` -- the hostname of the SMTP server, e.g. `mail.example.com` -* `MAILSENDER` -- sender name, e.g. `noreply@example.com` - -Full command example: - -```bash -MAILSERVER=mail.example.com MAILSENDER=noreply@example.com \ - MAILUSER=my_mail_login MAILPW=hunter2 \ - npm run dev -- --devSendMails -``` - -All of this of course is not too important if you're developing locally. To get -to the logged email content that *would* have been sent, check your docker log: - -```bash -docker-compose logs -f api -``` +Migrations are not implemented yet. Once we need them, we'll add them and +document the usage here. ## Tileserver generation diff --git a/UPGRADING.md b/UPGRADING.md new file mode 100644 index 0000000..017f3a7 --- /dev/null +++ b/UPGRADING.md @@ -0,0 +1,31 @@ +# Upgrading + +This document describes the general steps to upgrade between major changes. +Simple migrations, e.g. for adding schema changes, are not documented +explicitly. Once we implement them, their usage will be described in the +[README](./README.md). + + +## v0.2 to v0.3 (MongoDB to PostgreSQL) + +* Shut down all services +* Obviously, now is a good time to perform a full backup ;) +* Update the codebase +* Update docker-compose.yaml from the example. Leave the MongoDB service in + place for now, but update all other service descriptions. You can remove + `redis` already. +* Start up the `mongo` and `postgres` services +* Build the new images (e.g. with `docker-compose build`) +* Configure your API. The example config file is `api/config.py.example`, and + it will need to be mounted to `api/config.py` in the container. Ignore the + Keycloak options for now. +* Run the database reset script: `docker-compose run --rm api python tools/reset_database.py` +* Run the database migration script: `docker-compose run --rm api python + tools/import_from_mongodb.py mongodb://mongo/obs`. You might need to adjust + the URL. +* Shut down the `mongo` service, you can now remove it from docker-compose.yaml +* Start `keycloak` and configure it, similarly to how it was configured in the + development setup (but choose more secure options). Update the API config + file to match your keycloak configuration. +* Start `api`, `worker` and `frontend` +