api: Fix track again, using correct filters to get all points

frontend: fix config URL when loading from sub-page
chore: add editorconfig
2021-11-22 09:27:59 +01:00 · 2021-11-22 09:27:59 +01:00 · 2021-11-22 09:27:59 +01:00 · 2021-11-22 09:27:59 +01:00 · 2021-11-22 09:27:59 +01:00 · 2021-11-22 09:27:59 +01:00
230 changed files with 28843 additions and 32400 deletions
--- a/.dockerignore
+++ b/.dockerignore
@ -1,10 +0,0 @@
 local
 *.user
 frontend/node_modules
 api/.pyenv
 .git
 cache
 data
 tile-generator/cache
 tile-generator/data
 tile-generator/build
--- a/.forgejo/workflows/build-image.yml
+++ b/.forgejo/workflows/build-image.yml
@ -1,20 +0,0 @@
 name: Build docker image
 on: [push]
 jobs:
  build-image:
    runs-on: ubuntu-latest
    container:
      image: catthehacker/ubuntu:act-latest
    steps:
      - name: Login to Forgejo docker registry
        uses: docker/login-action@v3.0.0
        with:
          registry: git.pub.solar
          username: hakkonaut
          password: ${{ secrets.GIT_AUTH_TOKEN }}
      - name: Build and push
        uses: docker/build-push-action@v5.1.0
        with:
          push: true
          tags: git.pub.solar/pub-solar/obs-portal:latest
--- a/.gitignore
+++ b/.gitignore
@ -1,3 +1,2 @@
 local
 data
 export
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -1,191 +0,0 @@
 # Changelog
 ## 0.8.1
 ### Improvements
 * The zone (urban/rural) is now also exported with the events GeoJson export.
 ### Bug Fixes
 * Update to a current version of gpstime (python dependency) fix portal startup.
 ## 0.8.0
 ### Features
 * Bulk actions on users owned tracks (reprocess, download, make private, make public, delete) (#269, #38)
 * Easy sorting by device for "multi-device users" (e.g. group lending out OBSes)
 * Region display at higher zoom levels to easily find interesting areas (#112)
 * Export of road statistics on top of the already-existing event statistics (#341)
 ### Improvements
 * Refactored database access to hopefully combat portal crashes (#337)
 * New infrastructure for map imports that makes import of larger maps possible on small VMs (#334)
 * Reference current postgres and postgis versions in docker-compose.yaml files (#286)
 * Configurable terms-and-conditions link (#320)
 * French translation by @cbiteau (#303)
 ### Bug Fixes
 * Logout not working (#285)
 * Duplicate road usage hashes (#335, #253)
 * cannot import name .... (#338)
 ## 0.7.0
 ### Features
 * Add histogram of overtaking distances in road details panel
 * Flip table in road details panel and make it easier to read
 * Implement difference between urban and rural for events and road segments.
 * Better road zone detection in import
 * Make the frontend translatable and add German translation
 * Add time and user filters to map view (for logged-in users only)
 ### Improvements
 * Make raw track not look like a river (#252)
 * Update many dependencies
 ### Bug fixes
 * Overtaking events are now deleted when the parent track is deleted (#206)
 * Remove useless session creation (#192)
 * Remove some error logs for canceled requests (as the map page tends to do that quite a lot)
 * Fix ExportPage bounding box input
 ## 0.6.2
 ### Improvements
 * Prevent directory traversals inside container on python-served frontend.
 ## 0.6.1
 ### Improvements
 * Make road details request (clicking on a road segment in the map) way faster
  by using PostGIS geometry index correctly (#226).
 ## 0.6.0
 Starting in this version, the database schema is created through migrations
 instead of using the `reset_database.py` script. This means that for both the
 initial setup, as well as for upgrades, only the migrations have to be run.
 After updating and migrating, it is good practice to regenerate the SQL tile
 functions (`api/tools/prepare_sql_tiles.py`) as well. It doesn't matter if you
 do this when it is not required, so we've written a simple all-in-one update
 script that you can run to do all upgrade tasks. This is now in
 `api/tools/upgrade.py`.
 Please check [`UPGRADING.md`](./UPGRADING.md) for more details if you're
 upgrading an existing installation. It contains an important note for this
 upgrade in particular.
 ## 0.5.1
 Maintenance release, only includes build, deployment and documentation changes.
 ## 0.5.0
 ### Features
 * Use discrete colors for distances, with greens only above 1.5m
 * Use viridis colormap for roads' count layers
 * Generate usage count information (how often has a road been traveled)
 * Project the whole track to the map, and show both versions
 * Log out of OpenID server when logging out of application
 * Convert speed units to km/h in frontend
 * Pages now have titles (#148)
 * Remove map from home page, it was empty anyway (#120)
 ### Internal
 * Add alembic setup for migrating
 * Build osm2pgsql with -j4
 * Update sqlalchemy[asyncio] requirement from ~=1.4.31 to ~=1.4.32 in /api
 ## 0.4.2
 ### Features
 ### Bugfixes
 * Fix export route, it should be a child of /api 
 ## 0.4.1
 ### Features
 * Add page for exporting data through web frontend
 * Generate GPX track file when importing a track
 * Add GPX track export button on the track page (accessible for anybody who can
  see the track)
 ## 0.4.0
 ### Improvements
 * Retry OpenID Connect connection if it fails on boot
 * Format log outputs with color and improve access log
 * Make pool_size and overflow configurable for worker and portal
 * Add a route for exporting events as GeoJSON/Shapefile
 * Point footer to forum, not slack (fixes #140)
 * Improve wording on profile page ("My" instead of "Your")
 * Show "My tracks" directly in main menu (fixes #136)
 ### Bugfixes
 * Make sure the API can recover from the broken postgresql connection state
 * Remove duplicate events from the same track
 * Fix direction of road segments (fixes #142)
 * Solve a few problems with the colormap scales in the map view 
 ### Docs & deployment
 * Greatly improve deployement docs for a simple follow-along routine
 * Use environment variables (`OBS_*`) for configuration
 * Fix port numbers in example files and expose 3000 in the image
 * Add `LEAN_MODE` configuration to disable `road` database table usage and fall
  back to Overpass API for processing tracks (see
  [docs/lean-mode.md](docs/lean-mode.md)).
 * Read `config.overrides.py` file if it exists
 * Add osm2pgsql to portal image to be able to import OSM data from within the
  container
 * Fix path to roads_import.lua in docs
 * Explain to use the portal service, instead of api, in production
 * Use entrypoint instead of command, so you can run process_track.py one-off tasks
 ### Internals
 * Use custom `get_single_arg` everywhere, remove sanicargs (fixes #193)
 * Update requirements and make them consistent
 * Fix error handling, especially for file uploads
 ## 0.3.4
 ### Features
 * Reintroduce event view (fixes #111)
 * Add layer configuration panel to map page
  - Allow choosing basemap style
  - Add toggles for event and road layers
  - Make untagged roads display optional
  - Show a legend for event color
  - Alow choosing attribute used for coloring road segments
 * Add optional banner to frontend via config entry (solves #128)
 ### Bugfixes
 * Clicking on road without events should not cause 500 error
 * Improve mobile layout a bit (fixes #123)
 ### Technical
 * Allow explicit configuration of api base url via `API_URL` config
 * Remove outdated "mapTileset" frontend config section
--- a/674
+++ b/674
@ -1,674 +0,0 @@
                    GNU GENERAL PUBLIC LICENSE
                       Version 3, 29 June 2007
 Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
 Everyone is permitted to copy and distribute verbatim copies
 of this license document, but changing it is not allowed.
                            Preamble
  The GNU General Public License is a free, copyleft license for
 software and other kinds of works.
  The licenses for most software and other practical works are designed
 to take away your freedom to share and change the works.  By contrast,
 the GNU General Public License is intended to guarantee your freedom to
 share and change all versions of a program--to make sure it remains free
 software for all its users.  We, the Free Software Foundation, use the
 GNU General Public License for most of our software; it applies also to
 any other work released this way by its authors.  You can apply it to
 your programs, too.
  When we speak of free software, we are referring to freedom, not
 price.  Our General Public Licenses are designed to make sure that you
 have the freedom to distribute copies of free software (and charge for
 them if you wish), that you receive source code or can get it if you
 want it, that you can change the software or use pieces of it in new
 free programs, and that you know you can do these things.
  To protect your rights, we need to prevent others from denying you
 these rights or asking you to surrender the rights.  Therefore, you have
 certain responsibilities if you distribute copies of the software, or if
 you modify it: responsibilities to respect the freedom of others.
  For example, if you distribute copies of such a program, whether
 gratis or for a fee, you must pass on to the recipients the same
 freedoms that you received.  You must make sure that they, too, receive
 or can get the source code.  And you must show them these terms so they
 know their rights.
  Developers that use the GNU GPL protect your rights with two steps:
 (1) assert copyright on the software, and (2) offer you this License
 giving you legal permission to copy, distribute and/or modify it.
  For the developers' and authors' protection, the GPL clearly explains
 that there is no warranty for this free software.  For both users' and
 authors' sake, the GPL requires that modified versions be marked as
 changed, so that their problems will not be attributed erroneously to
 authors of previous versions.
  Some devices are designed to deny users access to install or run
 modified versions of the software inside them, although the manufacturer
 can do so.  This is fundamentally incompatible with the aim of
 protecting users' freedom to change the software.  The systematic
 pattern of such abuse occurs in the area of products for individuals to
 use, which is precisely where it is most unacceptable.  Therefore, we
 have designed this version of the GPL to prohibit the practice for those
 products.  If such problems arise substantially in other domains, we
 stand ready to extend this provision to those domains in future versions
 of the GPL, as needed to protect the freedom of users.
  Finally, every program is threatened constantly by software patents.
 States should not allow patents to restrict development and use of
 software on general-purpose computers, but in those that do, we wish to
 avoid the special danger that patents applied to a free program could
 make it effectively proprietary.  To prevent this, the GPL assures that
 patents cannot be used to render the program non-free.
  The precise terms and conditions for copying, distribution and
 modification follow.
                       TERMS AND CONDITIONS
  0. Definitions.
  "This License" refers to version 3 of the GNU General Public License.
  "Copyright" also means copyright-like laws that apply to other kinds of
 works, such as semiconductor masks.
  "The Program" refers to any copyrightable work licensed under this
 License.  Each licensee is addressed as "you".  "Licensees" and
 "recipients" may be individuals or organizations.
  To "modify" a work means to copy from or adapt all or part of the work
 in a fashion requiring copyright permission, other than the making of an
 exact copy.  The resulting work is called a "modified version" of the
 earlier work or a work "based on" the earlier work.
  A "covered work" means either the unmodified Program or a work based
 on the Program.
  To "propagate" a work means to do anything with it that, without
 permission, would make you directly or secondarily liable for
 infringement under applicable copyright law, except executing it on a
 computer or modifying a private copy.  Propagation includes copying,
 distribution (with or without modification), making available to the
 public, and in some countries other activities as well.
  To "convey" a work means any kind of propagation that enables other
 parties to make or receive copies.  Mere interaction with a user through
 a computer network, with no transfer of a copy, is not conveying.
  An interactive user interface displays "Appropriate Legal Notices"
 to the extent that it includes a convenient and prominently visible
 feature that (1) displays an appropriate copyright notice, and (2)
 tells the user that there is no warranty for the work (except to the
 extent that warranties are provided), that licensees may convey the
 work under this License, and how to view a copy of this License.  If
 the interface presents a list of user commands or options, such as a
 menu, a prominent item in the list meets this criterion.
  1. Source Code.
  The "source code" for a work means the preferred form of the work
 for making modifications to it.  "Object code" means any non-source
 form of a work.
  A "Standard Interface" means an interface that either is an official
 standard defined by a recognized standards body, or, in the case of
 interfaces specified for a particular programming language, one that
 is widely used among developers working in that language.
  The "System Libraries" of an executable work include anything, other
 than the work as a whole, that (a) is included in the normal form of
 packaging a Major Component, but which is not part of that Major
 Component, and (b) serves only to enable use of the work with that
 Major Component, or to implement a Standard Interface for which an
 implementation is available to the public in source code form.  A
 "Major Component", in this context, means a major essential component
 (kernel, window system, and so on) of the specific operating system
 (if any) on which the executable work runs, or a compiler used to
 produce the work, or an object code interpreter used to run it.
  The "Corresponding Source" for a work in object code form means all
 the source code needed to generate, install, and (for an executable
 work) run the object code and to modify the work, including scripts to
 control those activities.  However, it does not include the work's
 System Libraries, or general-purpose tools or generally available free
 programs which are used unmodified in performing those activities but
 which are not part of the work.  For example, Corresponding Source
 includes interface definition files associated with source files for
 the work, and the source code for shared libraries and dynamically
 linked subprograms that the work is specifically designed to require,
 such as by intimate data communication or control flow between those
 subprograms and other parts of the work.
  The Corresponding Source need not include anything that users
 can regenerate automatically from other parts of the Corresponding
 Source.
  The Corresponding Source for a work in source code form is that
 same work.
  2. Basic Permissions.
  All rights granted under this License are granted for the term of
 copyright on the Program, and are irrevocable provided the stated
 conditions are met.  This License explicitly affirms your unlimited
 permission to run the unmodified Program.  The output from running a
 covered work is covered by this License only if the output, given its
 content, constitutes a covered work.  This License acknowledges your
 rights of fair use or other equivalent, as provided by copyright law.
  You may make, run and propagate covered works that you do not
 convey, without conditions so long as your license otherwise remains
 in force.  You may convey covered works to others for the sole purpose
 of having them make modifications exclusively for you, or provide you
 with facilities for running those works, provided that you comply with
 the terms of this License in conveying all material for which you do
 not control copyright.  Those thus making or running the covered works
 for you must do so exclusively on your behalf, under your direction
 and control, on terms that prohibit them from making any copies of
 your copyrighted material outside their relationship with you.
  Conveying under any other circumstances is permitted solely under
 the conditions stated below.  Sublicensing is not allowed; section 10
 makes it unnecessary.
  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
  No covered work shall be deemed part of an effective technological
 measure under any applicable law fulfilling obligations under article
 11 of the WIPO copyright treaty adopted on 20 December 1996, or
 similar laws prohibiting or restricting circumvention of such
 measures.
  When you convey a covered work, you waive any legal power to forbid
 circumvention of technological measures to the extent such circumvention
 is effected by exercising rights under this License with respect to
 the covered work, and you disclaim any intention to limit operation or
 modification of the work as a means of enforcing, against the work's
 users, your or third parties' legal rights to forbid circumvention of
 technological measures.
  4. Conveying Verbatim Copies.
  You may convey verbatim copies of the Program's source code as you
 receive it, in any medium, provided that you conspicuously and
 appropriately publish on each copy an appropriate copyright notice;
 keep intact all notices stating that this License and any
 non-permissive terms added in accord with section 7 apply to the code;
 keep intact all notices of the absence of any warranty; and give all
 recipients a copy of this License along with the Program.
  You may charge any price or no price for each copy that you convey,
 and you may offer support or warranty protection for a fee.
  5. Conveying Modified Source Versions.
  You may convey a work based on the Program, or the modifications to
 produce it from the Program, in the form of source code under the
 terms of section 4, provided that you also meet all of these conditions:
    a) The work must carry prominent notices stating that you modified
    it, and giving a relevant date.
    b) The work must carry prominent notices stating that it is
    released under this License and any conditions added under section
    7.  This requirement modifies the requirement in section 4 to
    "keep intact all notices".
    c) You must license the entire work, as a whole, under this
    License to anyone who comes into possession of a copy.  This
    License will therefore apply, along with any applicable section 7
    additional terms, to the whole of the work, and all its parts,
    regardless of how they are packaged.  This License gives no
    permission to license the work in any other way, but it does not
    invalidate such permission if you have separately received it.
    d) If the work has interactive user interfaces, each must display
    Appropriate Legal Notices; however, if the Program has interactive
    interfaces that do not display Appropriate Legal Notices, your
    work need not make them do so.
  A compilation of a covered work with other separate and independent
 works, which are not by their nature extensions of the covered work,
 and which are not combined with it such as to form a larger program,
 in or on a volume of a storage or distribution medium, is called an
 "aggregate" if the compilation and its resulting copyright are not
 used to limit the access or legal rights of the compilation's users
 beyond what the individual works permit.  Inclusion of a covered work
 in an aggregate does not cause this License to apply to the other
 parts of the aggregate.
  6. Conveying Non-Source Forms.
  You may convey a covered work in object code form under the terms
 of sections 4 and 5, provided that you also convey the
 machine-readable Corresponding Source under the terms of this License,
 in one of these ways:
    a) Convey the object code in, or embodied in, a physical product
    (including a physical distribution medium), accompanied by the
    Corresponding Source fixed on a durable physical medium
    customarily used for software interchange.
    b) Convey the object code in, or embodied in, a physical product
    (including a physical distribution medium), accompanied by a
    written offer, valid for at least three years and valid for as
    long as you offer spare parts or customer support for that product
    model, to give anyone who possesses the object code either (1) a
    copy of the Corresponding Source for all the software in the
    product that is covered by this License, on a durable physical
    medium customarily used for software interchange, for a price no
    more than your reasonable cost of physically performing this
    conveying of source, or (2) access to copy the
    Corresponding Source from a network server at no charge.
    c) Convey individual copies of the object code with a copy of the
    written offer to provide the Corresponding Source.  This
    alternative is allowed only occasionally and noncommercially, and
    only if you received the object code with such an offer, in accord
    with subsection 6b.
    d) Convey the object code by offering access from a designated
    place (gratis or for a charge), and offer equivalent access to the
    Corresponding Source in the same way through the same place at no
    further charge.  You need not require recipients to copy the
    Corresponding Source along with the object code.  If the place to
    copy the object code is a network server, the Corresponding Source
    may be on a different server (operated by you or a third party)
    that supports equivalent copying facilities, provided you maintain
    clear directions next to the object code saying where to find the
    Corresponding Source.  Regardless of what server hosts the
    Corresponding Source, you remain obligated to ensure that it is
    available for as long as needed to satisfy these requirements.
    e) Convey the object code using peer-to-peer transmission, provided
    you inform other peers where the object code and Corresponding
    Source of the work are being offered to the general public at no
    charge under subsection 6d.
  A separable portion of the object code, whose source code is excluded
 from the Corresponding Source as a System Library, need not be
 included in conveying the object code work.
  A "User Product" is either (1) a "consumer product", which means any
 tangible personal property which is normally used for personal, family,
 or household purposes, or (2) anything designed or sold for incorporation
 into a dwelling.  In determining whether a product is a consumer product,
 doubtful cases shall be resolved in favor of coverage.  For a particular
 product received by a particular user, "normally used" refers to a
 typical or common use of that class of product, regardless of the status
 of the particular user or of the way in which the particular user
 actually uses, or expects or is expected to use, the product.  A product
 is a consumer product regardless of whether the product has substantial
 commercial, industrial or non-consumer uses, unless such uses represent
 the only significant mode of use of the product.
  "Installation Information" for a User Product means any methods,
 procedures, authorization keys, or other information required to install
 and execute modified versions of a covered work in that User Product from
 a modified version of its Corresponding Source.  The information must
 suffice to ensure that the continued functioning of the modified object
 code is in no case prevented or interfered with solely because
 modification has been made.
  If you convey an object code work under this section in, or with, or
 specifically for use in, a User Product, and the conveying occurs as
 part of a transaction in which the right of possession and use of the
 User Product is transferred to the recipient in perpetuity or for a
 fixed term (regardless of how the transaction is characterized), the
 Corresponding Source conveyed under this section must be accompanied
 by the Installation Information.  But this requirement does not apply
 if neither you nor any third party retains the ability to install
 modified object code on the User Product (for example, the work has
 been installed in ROM).
  The requirement to provide Installation Information does not include a
 requirement to continue to provide support service, warranty, or updates
 for a work that has been modified or installed by the recipient, or for
 the User Product in which it has been modified or installed.  Access to a
 network may be denied when the modification itself materially and
 adversely affects the operation of the network or violates the rules and
 protocols for communication across the network.
  Corresponding Source conveyed, and Installation Information provided,
 in accord with this section must be in a format that is publicly
 documented (and with an implementation available to the public in
 source code form), and must require no special password or key for
 unpacking, reading or copying.
  7. Additional Terms.
  "Additional permissions" are terms that supplement the terms of this
 License by making exceptions from one or more of its conditions.
 Additional permissions that are applicable to the entire Program shall
 be treated as though they were included in this License, to the extent
 that they are valid under applicable law.  If additional permissions
 apply only to part of the Program, that part may be used separately
 under those permissions, but the entire Program remains governed by
 this License without regard to the additional permissions.
  When you convey a copy of a covered work, you may at your option
 remove any additional permissions from that copy, or from any part of
 it.  (Additional permissions may be written to require their own
 removal in certain cases when you modify the work.)  You may place
 additional permissions on material, added by you to a covered work,
 for which you have or can give appropriate copyright permission.
  Notwithstanding any other provision of this License, for material you
 add to a covered work, you may (if authorized by the copyright holders of
 that material) supplement the terms of this License with terms:
    a) Disclaiming warranty or limiting liability differently from the
    terms of sections 15 and 16 of this License; or
    b) Requiring preservation of specified reasonable legal notices or
    author attributions in that material or in the Appropriate Legal
    Notices displayed by works containing it; or
    c) Prohibiting misrepresentation of the origin of that material, or
    requiring that modified versions of such material be marked in
    reasonable ways as different from the original version; or
    d) Limiting the use for publicity purposes of names of licensors or
    authors of the material; or
    e) Declining to grant rights under trademark law for use of some
    trade names, trademarks, or service marks; or
    f) Requiring indemnification of licensors and authors of that
    material by anyone who conveys the material (or modified versions of
    it) with contractual assumptions of liability to the recipient, for
    any liability that these contractual assumptions directly impose on
    those licensors and authors.
  All other non-permissive additional terms are considered "further
 restrictions" within the meaning of section 10.  If the Program as you
 received it, or any part of it, contains a notice stating that it is
 governed by this License along with a term that is a further
 restriction, you may remove that term.  If a license document contains
 a further restriction but permits relicensing or conveying under this
 License, you may add to a covered work material governed by the terms
 of that license document, provided that the further restriction does
 not survive such relicensing or conveying.
  If you add terms to a covered work in accord with this section, you
 must place, in the relevant source files, a statement of the
 additional terms that apply to those files, or a notice indicating
 where to find the applicable terms.
  Additional terms, permissive or non-permissive, may be stated in the
 form of a separately written license, or stated as exceptions;
 the above requirements apply either way.
  8. Termination.
  You may not propagate or modify a covered work except as expressly
 provided under this License.  Any attempt otherwise to propagate or
 modify it is void, and will automatically terminate your rights under
 this License (including any patent licenses granted under the third
 paragraph of section 11).
  However, if you cease all violation of this License, then your
 license from a particular copyright holder is reinstated (a)
 provisionally, unless and until the copyright holder explicitly and
 finally terminates your license, and (b) permanently, if the copyright
 holder fails to notify you of the violation by some reasonable means
 prior to 60 days after the cessation.
  Moreover, your license from a particular copyright holder is
 reinstated permanently if the copyright holder notifies you of the
 violation by some reasonable means, this is the first time you have
 received notice of violation of this License (for any work) from that
 copyright holder, and you cure the violation prior to 30 days after
 your receipt of the notice.
  Termination of your rights under this section does not terminate the
 licenses of parties who have received copies or rights from you under
 this License.  If your rights have been terminated and not permanently
 reinstated, you do not qualify to receive new licenses for the same
 material under section 10.
  9. Acceptance Not Required for Having Copies.
  You are not required to accept this License in order to receive or
 run a copy of the Program.  Ancillary propagation of a covered work
 occurring solely as a consequence of using peer-to-peer transmission
 to receive a copy likewise does not require acceptance.  However,
 nothing other than this License grants you permission to propagate or
 modify any covered work.  These actions infringe copyright if you do
 not accept this License.  Therefore, by modifying or propagating a
 covered work, you indicate your acceptance of this License to do so.
  10. Automatic Licensing of Downstream Recipients.
  Each time you convey a covered work, the recipient automatically
 receives a license from the original licensors, to run, modify and
 propagate that work, subject to this License.  You are not responsible
 for enforcing compliance by third parties with this License.
  An "entity transaction" is a transaction transferring control of an
 organization, or substantially all assets of one, or subdividing an
 organization, or merging organizations.  If propagation of a covered
 work results from an entity transaction, each party to that
 transaction who receives a copy of the work also receives whatever
 licenses to the work the party's predecessor in interest had or could
 give under the previous paragraph, plus a right to possession of the
 Corresponding Source of the work from the predecessor in interest, if
 the predecessor has it or can get it with reasonable efforts.
  You may not impose any further restrictions on the exercise of the
 rights granted or affirmed under this License.  For example, you may
 not impose a license fee, royalty, or other charge for exercise of
 rights granted under this License, and you may not initiate litigation
 (including a cross-claim or counterclaim in a lawsuit) alleging that
 any patent claim is infringed by making, using, selling, offering for
 sale, or importing the Program or any portion of it.
  11. Patents.
  A "contributor" is a copyright holder who authorizes use under this
 License of the Program or a work on which the Program is based.  The
 work thus licensed is called the contributor's "contributor version".
  A contributor's "essential patent claims" are all patent claims
 owned or controlled by the contributor, whether already acquired or
 hereafter acquired, that would be infringed by some manner, permitted
 by this License, of making, using, or selling its contributor version,
 but do not include claims that would be infringed only as a
 consequence of further modification of the contributor version.  For
 purposes of this definition, "control" includes the right to grant
 patent sublicenses in a manner consistent with the requirements of
 this License.
  Each contributor grants you a non-exclusive, worldwide, royalty-free
 patent license under the contributor's essential patent claims, to
 make, use, sell, offer for sale, import and otherwise run, modify and
 propagate the contents of its contributor version.
  In the following three paragraphs, a "patent license" is any express
 agreement or commitment, however denominated, not to enforce a patent
 (such as an express permission to practice a patent or covenant not to
 sue for patent infringement).  To "grant" such a patent license to a
 party means to make such an agreement or commitment not to enforce a
 patent against the party.
  If you convey a covered work, knowingly relying on a patent license,
 and the Corresponding Source of the work is not available for anyone
 to copy, free of charge and under the terms of this License, through a
 publicly available network server or other readily accessible means,
 then you must either (1) cause the Corresponding Source to be so
 available, or (2) arrange to deprive yourself of the benefit of the
 patent license for this particular work, or (3) arrange, in a manner
 consistent with the requirements of this License, to extend the patent
 license to downstream recipients.  "Knowingly relying" means you have
 actual knowledge that, but for the patent license, your conveying the
 covered work in a country, or your recipient's use of the covered work
 in a country, would infringe one or more identifiable patents in that
 country that you have reason to believe are valid.
  If, pursuant to or in connection with a single transaction or
 arrangement, you convey, or propagate by procuring conveyance of, a
 covered work, and grant a patent license to some of the parties
 receiving the covered work authorizing them to use, propagate, modify
 or convey a specific copy of the covered work, then the patent license
 you grant is automatically extended to all recipients of the covered
 work and works based on it.
  A patent license is "discriminatory" if it does not include within
 the scope of its coverage, prohibits the exercise of, or is
 conditioned on the non-exercise of one or more of the rights that are
 specifically granted under this License.  You may not convey a covered
 work if you are a party to an arrangement with a third party that is
 in the business of distributing software, under which you make payment
 to the third party based on the extent of your activity of conveying
 the work, and under which the third party grants, to any of the
 parties who would receive the covered work from you, a discriminatory
 patent license (a) in connection with copies of the covered work
 conveyed by you (or copies made from those copies), or (b) primarily
 for and in connection with specific products or compilations that
 contain the covered work, unless you entered into that arrangement,
 or that patent license was granted, prior to 28 March 2007.
  Nothing in this License shall be construed as excluding or limiting
 any implied license or other defenses to infringement that may
 otherwise be available to you under applicable patent law.
  12. No Surrender of Others' Freedom.
  If conditions are imposed on you (whether by court order, agreement or
 otherwise) that contradict the conditions of this License, they do not
 excuse you from the conditions of this License.  If you cannot convey a
 covered work so as to satisfy simultaneously your obligations under this
 License and any other pertinent obligations, then as a consequence you may
 not convey it at all.  For example, if you agree to terms that obligate you
 to collect a royalty for further conveying from those to whom you convey
 the Program, the only way you could satisfy both those terms and this
 License would be to refrain entirely from conveying the Program.
  13. Use with the GNU Affero General Public License.
  Notwithstanding any other provision of this License, you have
 permission to link or combine any covered work with a work licensed
 under version 3 of the GNU Affero General Public License into a single
 combined work, and to convey the resulting work.  The terms of this
 License will continue to apply to the part which is the covered work,
 but the special requirements of the GNU Affero General Public License,
 section 13, concerning interaction through a network will apply to the
 combination as such.
  14. Revised Versions of this License.
  The Free Software Foundation may publish revised and/or new versions of
 the GNU General Public License from time to time.  Such new versions will
 be similar in spirit to the present version, but may differ in detail to
 address new problems or concerns.
  Each version is given a distinguishing version number.  If the
 Program specifies that a certain numbered version of the GNU General
 Public License "or any later version" applies to it, you have the
 option of following the terms and conditions either of that numbered
 version or of any later version published by the Free Software
 Foundation.  If the Program does not specify a version number of the
 GNU General Public License, you may choose any version ever published
 by the Free Software Foundation.
  If the Program specifies that a proxy can decide which future
 versions of the GNU General Public License can be used, that proxy's
 public statement of acceptance of a version permanently authorizes you
 to choose that version for the Program.
  Later license versions may give you additional or different
 permissions.  However, no additional obligations are imposed on any
 author or copyright holder as a result of your choosing to follow a
 later version.
  15. Disclaimer of Warranty.
  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
 APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
 HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
 OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
 THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
 IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
 ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
  16. Limitation of Liability.
  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
 WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
 THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
 GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
 USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
 DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
 PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
 EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
 SUCH DAMAGES.
  17. Interpretation of Sections 15 and 16.
  If the disclaimer of warranty and limitation of liability provided
 above cannot be given local legal effect according to their terms,
 reviewing courts shall apply local law that most closely approximates
 an absolute waiver of all civil liability in connection with the
 Program, unless a warranty or assumption of liability accompanies a
 copy of the Program in return for a fee.
                     END OF TERMS AND CONDITIONS
            How to Apply These Terms to Your New Programs
  If you develop a new program, and you want it to be of the greatest
 possible use to the public, the best way to achieve this is to make it
 free software which everyone can redistribute and change under these terms.
  To do so, attach the following notices to the program.  It is safest
 to attach them to the start of each source file to most effectively
 state the exclusion of warranty; and each file should have at least
 the "copyright" line and a pointer to where the full notice is found.
    <one line to give the program's name and a brief idea of what it does.>
    Copyright (C) <year>  <name of author>
    This program is free software: you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.
    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.
    You should have received a copy of the GNU General Public License
    along with this program.  If not, see <https://www.gnu.org/licenses/>.
 Also add information on how to contact you by electronic and paper mail.
  If the program does terminal interaction, make it output a short
 notice like this when it starts in an interactive mode:
    <program>  Copyright (C) <year>  <name of author>
    This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
    This is free software, and you are welcome to redistribute it
    under certain conditions; type `show c' for details.
 The hypothetical commands `show w' and `show c' should show the appropriate
 parts of the General Public License.  Of course, your program's commands
 might be different; for a GUI interface, you would use an "about box".
  You should also get your employer (if you work as a programmer) or school,
 if any, to sign a "copyright disclaimer" for the program, if necessary.
 For more information on this, and how to apply and follow the GNU GPL, see
 <https://www.gnu.org/licenses/>.
  The GNU General Public License does not permit incorporating your program
 into proprietary programs.  If your program is a subroutine library, you
 may consider it more useful to permit linking proprietary applications with
 the library.  If this is what you want to do, use the GNU Lesser General
 Public License instead of this License.  But first, please read
 <https://www.gnu.org/licenses/why-not-lgpl.html>.
--- a/61
+++ b/61
@ -1,61 +0,0 @@
 # This dockerfile is for the API + Frontend production image
 #############################################
 # Build the frontend AS builder
 #############################################
 FROM node:18 as frontend-builder
 WORKDIR /opt/obs/frontend
 ADD frontend/package.json frontend/package-lock.json /opt/obs/frontend/
 RUN echo update-notifier=false >> ~/.npmrc
 RUN npm ci
 ADD frontend/tsconfig.json frontend/webpack.config.js /opt/obs/frontend/
 ADD frontend/src /opt/obs/frontend/src/
 ADD frontend/public /opt/obs/frontend/public/
 RUN npm run build
 #############################################
 # Build the API and add the built frontend to it
 #############################################
 FROM python:3.11.3-bullseye
 RUN apt-get update &&\
    apt-get install -y \
    libboost-dev \
    libboost-system-dev \
    libboost-filesystem-dev \
    libexpat1-dev \
    zlib1g-dev \
    libbz2-dev \
    libpq-dev \
    libproj-dev \
    lua5.3 \
    liblua5.3-dev &&\
    rm -rf /var/lib/apt/lists/*
 WORKDIR /opt/obs/api
 ADD api/requirements.txt  /opt/obs/api/
 RUN pip install -r requirements.txt
 ADD tile-generator /opt/obs/tile-generator
 ADD api/scripts /opt/obs/scripts
 RUN pip install -e /opt/obs/scripts
 ADD api/setup.py  /opt/obs/api/
 ADD api/alembic.ini /opt/obs/api/
 ADD api/migrations /opt/obs/api/migrations/
 ADD api/obs /opt/obs/api/obs/
 ADD api/tools /opt/obs/api/tools/
 RUN pip install -e /opt/obs/api/
 COPY --from=frontend-builder /opt/obs/frontend/build /opt/obs/frontend/build
 EXPOSE 3000
 CMD ["openbikesensor-api"]
--- a/README.md
+++ b/README.md
@ -4,23 +4,12 @@ This repository contains the source code required to run the
 [OpenBikeSensor](https://openbikesensor.org) data collection portal. It is
 separated into components:
-* **api**: The backend service, written in Python 3 with
+* **api**: The backend service, written in JavaScript for Node.js, using
-  [Sanic](https://sanicframework.org/),
+  express.js, and a MongoDB for metadata storage.
  [SQLAlchemy](https://www.sqlalchemy.org/), and a PostgreSQL/PostGIS database
  for storage. It also depends highly on
  [OpenMapTiles](https://openmaptiles.org) to generate vector tiles of the
  data.
 * **frontend**: A React single-page application that allows access to the data,
  provides summaries and visualizations, and lets users adjust settings and
  manage and publish their tracks.
 Check out the [Architecture Documentation](docs/architecture.md) for more
 details on what parts the whole application is made of.
 This project follows [semantic versioning](https://semver.org). Refer to [issue
 #44](https://github.com/openbikesensor/portal/issues/44) for a description of
 what that means for our project and what is considered the public interface.
 ## Clone the Project
 First of all, you must clone this project. This project uses submodules,
@ -36,195 +25,146 @@ git submodule update --init --recursive
 ## Production setup
-There is a guide for a deployment based on docker at
+**Note:** There is now a guide for a dockerized deployment in the
-[docs/production-deployment.md](docs/production-deployment.md). Lots of
+[deployment](deployment) folder.
 non-docker deployment strategies are possible, but they are not "officially"
 supported, so please do not expect the authors of the software to assist in
 troubleshooting. 
-This is a rather complex application, and it is expected that you know the
+You should be familiar with managing a Linux server. If not, find a suitable
-basics of deploying a modern web application securely onto a production server.
+guide first. This will only give a rough outline of the steps to take, you must
-We are sorry that we cannot guide you through all the details of that, as we
+make sure to properly set up and secure your server yourself.
 just don't have the capacities to do so. Please research the respective topics
 first. If you struggle with application-specific issues, please let us know, we
 might be able to assist with those.
-Please note that you will always need to install your own reverse proxy that
+1. Create a user for running the application. It is not recommended to run as a
-terminates TLS for you and handles certificates. We do not support TLS directly
+   user that is also used for other things. Do not run as root!
-in the application, instead, please use this prefered method. 
+2. Clone the repository.
-
+3. Install `node` of at least version 15, and also `npm`. You can use the
-Upgrading and migrating is described in [UPGRADING.md](./UPGRADING.md) for each
+   [Node Version Manager (nvm)](https://github.com/nvm-sh/nvm), since this project
-version.
+   contains a `.nvmrc`. Just run `nvm install` followed by `nvm use`.
-
+4. Run `npm ci` in the `api` and `frontend` directories to install dependencies.
-### Migrating (Production)
+5. Install and configure a MongoDB somewhere.
-
+6. Copy `api/config.json.example` to `api/config.json` and change it to suit
-Migrations are done with
+   your setup. Make sure to only use https URLs. Generate secure secrets.
-[Alembic](https://alembic.sqlalchemy.org/en/latest/index.html), please refer to
+   Customize client IDs.
-its documentation for help. Most of the time, running this command will do all
+7. Copy `frontend/src/config.json.example` to `frontend/src/config.json` and
-the migrations you need:
+   adjust.
-
+8. Run `npm run build` in the frontend directory. This needs to be done after
-```bash
+   changing the config, so if you did something wrong, re-run the build. If you
-docker-compose run --rm api tools/upgrade.py
+   run your frontend on a non-root URL, run the build with the
-```
+   `PUBLIC_URL=/prefix` environment variable.
-
+9. Create a systemd-user service, or use tmux, or do whatever you like to start
-This command is equivalent to running migrations through *alembic*, then
+   the API service (`npm start` in `api/` directory). It should run as the
-regenerating the SQL functions that compute vector tiles directly in the
+   dedicated user, and expose `127.0.0.1:3000`. Use `PORT` environment variable
-database:
+   if you want to change the port. Make sure the service starts with
-
+   environment `NODE_ENV=production`, too.
-```bash
+10. Configure nginx or your reverse-proxy of choice to forward API requests to
-# equivalent to the above command, you don't usually run these
+    the API port, and serve static files from the `frontend/build/` folder.
 docker-compose run --rm api alembic upgrade head
 docker-compose run --rm api tools/prepare_sql_tiles
 ```
 ## Development setup
 We've moved the whole development setup into Docker to make it easy for
-everyone to get involved. 
+everyone to get involved. After sucessfully [installing Docker
-
+Engine](https://docs.docker.com/engine/install/) as well as [Docker
-### Install docker
+Compose](https://docs.docker.com/compose/install/) onto your machine, and
-
+cloning the repository, all you need to do is:
 Please [install Docker Engine](https://docs.docker.com/engine/install/) as well as 
 [Docker Compose](https://docs.docker.com/compose/install/) onto your machine. 
 Then clone the repository as described above.
 ### Configure Keycloak
 Login will not be possible until you configure the keycloak realm correctly. Boot your keycloak instance:
 ```bash
-docker-compose up -d keycloak
+docker-compose up -d --build
 ```
-Now navigate to http://localhost:3003/ and follow these steps:
+If this does not work, please open an issue and describe the problem you're
-
+having, as it is important to us that onboarding is super easy :)
 - Click *Administration Console* and log in with `admin` / `admin`.
 - Hover over the realm name on the top left and click *Add realm*.
 - Name the Realm `obs-dev` (spelling matters) and create it.
 - In the sidebar, navigate to *Configure* &rarr; *Clients*, and click *Create* on the top right.
 - *Client ID* should be `portal`. Click *Save*.
 - In the Tab *Settings*, edit the new client's *Access Type* to *confidential*
  and enter as *Valid Redirect URIs*: `http://localhost:3000/login/redirect`,
  then *Save*
 - Under *Credentials*, copy the *Secret*. Create a file at `api/config.overrides.py` with the secret in it:
  ```python
  KEYCLOAK_CLIENT_SECRET="your secret here"
  ```
  You can use this file in development mode to change settings without editing
  the git-controlled default file at `api/config.dev.py`. Options in this file
  take precendence.
 - In the sidebar, navigate to *Manage* &rarr; *Users*, and click *Add user* on the top right.
 - Give the user a name (e.g. `test`), leave the rest as-is.
 - Under the tab *Credentials*, choose a new password, and make it
  non-temporary. Click *Set Password*.
 We are going to automate this process. For now, you will have to repeat it
 every time you reset your keycloak settings, which are stored inside the
 PostgreSQL as well. Luckily, the script `api/tools/reset_database.py` does
 *not* affect the state of the keycloak database, so this should be rather rare.
 ### Prepare database
 Start the PostgreSQL database:
 ```bash
 docker-compose up -d postgres
 ```
 The first time you start postgres, a lot of extensions will be installed. This
 takes a while, so check the logs of the docker container until you see:
 > PostgreSQL init process complete; ready for start up.
 If you don't wait long enough, the following commands might fail. In this case,
 you can always stop the container, remove the data directory (`local/postgres`)
 and restart the process.
 Next, run the upgrade command to generate the database schema:
 ```bash
 docker-compose run --rm api tools/upgrade.py
 ```
 You will need to re-run this command after updates, to migrate the database and
 (re-)create the functions in the SQL database that are used when generating
 vector tiles.
 You should also [import OpenStreetMap data](docs/osm-import.md) now.
 ### Boot the application
 Now you can run the remaining parts of the application:
 ```bash
 docker-compose up -d --build api worker frontend
 ```
 Your frontend should be running at http://localhost:3001 and the API at
 http://localhost:3000 -- but you probably only need to access the frontend for
-testing. 
+testing. The frontend dev server also proxies all unknown requests to the API,
 so the frontend always just requests data at its own URL.
-### Migrating (Development)
+## Running without docker
-Migrations are done with
+If you don't like docker, or want to run this in production without it, you can
-[Alembic](https://alembic.sqlalchemy.org/en/latest/index.html), please refer to
+do so as well. Our Docker setup is simply a slight wrapper around very simple
-its documentation for help. Most of the time, running this command will do all
+JavaScript packages that you can install yourself as usual, with `npm install`.
-the migrations you need:
+The API can be started with `npm start` inside its folder. The frontend
 development server uses `npm start` as well, while building a production
 version of the frontend happens with `npm run build`.
 To connect the parts together, please have a look at what we're doing in the
 "official" setup of docker, i.e. in `docker-compose.yaml`, the `Dockerfile`s
 and in the respective `package.json` of the service. If you've done this kind
 of thing before, it's not that hard. Otherwise, ask on Slack and there will be
 somebody to help you ;)
 ## Running in production
 You are advised not to use the dockerized mongodb service and instead do a
 proper MongoDB setup on a server that is backed up and secured.
 You can run the API in docker, but it is prefered to run it as a restricted
 user in its own directory somewhere where it cannot escape ;)
 The frontend should be built using `npm run build` and then served from a
 proper web server, such as nginx or apache. See the instructions at
 create-react-app concerning [deployment of an app](http://cra.link/deployment).
 You are advised to virtualize your server for security reason, and separate
 this whole application from other parts of your server system.
 Also please install a reverse proxy that terminates TLS for you and handles
 certificates. We do not support TLS directly in the application, instead,
 please use this prefered method. This reverse proxy can also handle static file
 serving for the frontend, no need for two separate server processes.
 ## Migrating
 Sometimes your database will have to be migrated. The docker setup should do
 this automatically, but if it does not work, you can run the following
 commands:
 ```bash
-docker-compose run --rm api alembic upgrade head
+# if running locally
 (cd api/; npm run migrate:up)
 # if running in docker
 docker-compose run --rm api npm run migrate:up
 ````
 ## Custom MongoDB installation
 If you have your own MongoDB instance running somewhere, you can set the
 environment variable `MONGODB_URL` when starting the server, and it will read
 that URL for connecting.
    export MONGODB_URL=mongodb://user:password@mongodb.example.com/obs-app-database
 This does not work when using docker-compose, in that case, you will have to
 modify the `docker-compose.yaml` to include that URL.
 ## E-Mail Setup
 By default in development mode mails are not sent, but instead the mail data is
 logged to the console. This can be overriden with the `--devSendMails` flag if
 you start the application like so: `npm run dev -- --devSendMails`.
 Mails are also always sent in production mode!
 For actually sending e-mails the mailserver, sender, user and password for the
 SMTP server need to be specified as environment variables:
 * `MAILUSER` -- the smtp mailbox login name
 * `MAILPW` -- password for the mailbox
 * `MAILSERVER` -- the hostname of the SMTP server, e.g. `mail.example.com`
 * `MAILSENDER` -- sender name, e.g. `noreply@example.com`
 Full command example:
 ```bash
 MAILSERVER=mail.example.com MAILSENDER=noreply@example.com \
    MAILUSER=my_mail_login MAILPW=hunter2 \
    npm run dev -- --devSendMails
 ```
 All of this of course is not too important if you're developing locally. To get
 to the logged email content that *would* have been sent, check your docker log:
-
+```bash
-## Troubleshooting
+docker-compose logs -f api
 If any step of the instructions does not work for you, please open an issue and
 describe the problem you're having, as it is important to us that onboarding is
 super easy :)
 ### Connecting to the PostgreSQL database
 If you need to connect to your development PostgreSQL database, you should
 install `psql` locally. The port 5432 is already forwarded, so you can connect with:
 ```
 psql -h localhost -U obs -d obs
 ```
 The password is `obs` as well.
 ## License
    Copyright (C) 2020-2021 OpenBikeSensor Contributors
    Contact: https://openbikesensor.org
    The OpenBikeSensor Portal is free software: you can redistribute it
    and/or modify it under the terms of the GNU Lesser General Public License
    as published by the Free Software Foundation, either version 3 of the
    License, or (at your option) any later version.
    The OpenBikeSensor Portal is distributed in the hope that it will be
    useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser
    General Public License for more details.
    You should have received a copy of the GNU Lesser General Public License
    along with the OpenBikeSensor Portal. If not, see
    <http://www.gnu.org/licenses/>.
 See also [`COPYING`](./COPYING) and [`COPYING.LESSER`](./COPYING.LESSER).
 The above does not apply to the files listed below, their respective licenses
 are included in a file next to each of them, named accordingly:
 * `frontend/src/mapstyles/bright.json`
 * `frontend/src/mapstyles/positron.json`
 There are lots of other licenses to consider when using this software,
 especially in conjunction with imported data and other tools. Check out the
 [Licenses Documentation](docs/licenses.md) for an (unofficial) overview of the
 license landscape surrounding this project.
--- a/UPGRADING.md
+++ b/UPGRADING.md
@ -1,170 +0,0 @@
 # Upgrading
 This document describes the general steps to upgrade between major changes.
 Simple migrations, e.g. for adding schema changes, are not documented
 explicitly. Their general usage is described in the [README](./README.md) (for
 development) and [docs/production-deployment.md](docs/production-deployment.md) (for production).
 ## 0.8.1
 - Get the release in your source folder (``git pull; git checkout 0.8.0`` and update submodules ``git submodule update --recursive``)
 - Rebuild images ``docker-compose build``
 - No database upgrade is required, but tile functions need an update:
  ```bash
  docker-compose run --rm portal tools/prepare_sql_tiles.py
  ```
 - Start your portal and worker services. ``docker-compose up -d worker portal``
 ## 0.8.0
 Upgrade to `0.7.x` first. See below for details. Then follow these steps:
 > **Warning** The update includes a reprocessing of tracks after import. Depending on the number of tracks this can take a few hours. The portal is reachable during that time but events disappear and incrementally reappear during reimport.
 > **Info** With this version the import process for OpenStreetMap data has changed: the [new process](docs/osm-import.md) is easier on resources and finally permits to import a full country on a low-end VM.
 - Do your [usual backup](docs/production-deployment.md)
 - get the release in your source folder (``git pull; git checkout 0.8.0`` and update submodules ``git submodule update --recursive``)
 - Rebuild images ``docker-compose build``
 - Stop your portal and worker services ``docker-compose stop worker portal``
 - run upgrade
  ```bash
  docker-compose run --rm portal tools/upgrade.py
  ```
  this automatically does the following
  - Migration of database schema using alembic.
  - Upgrade of SQL tile schema to new schema.
  - Import the nuts-regions from the web into the database.
  - Trigger a re-import of all tracks.
 - Start your portal and worker services. ``docker-compose up -d worker portal``
 ## 0.7.0
 Upgrade to `0.6.x` first. See below for details. Then follow these steps:
 - Rebuild images
 - Stop your portal and worker services.
 - **Migration with alembic**: required 
 - **Prepare SQL Tiles**: required
 - Start your portal and worker services.
 - **Reimport tracks**: no action required 
 - **OSM Import**: required
 - **Config changes**: add `POSTGRES_MAX_OVERFLOW` and `POSTGRES_POOL_SIZE`
  variables, see `api/config.py.example`  
 ## 0.6.0
 **Make sure to upgrade to `0.5.1` first, by checking out that version tag and
 running migrations, then coming back to this version.** This is required
 because the migrations have been edited to create the initial database schema,
 but if you run the 0.5.1 migrations first, your database will remember that it
 already has all the tables created. This is not required if you set up a new
 installation.
 For this update, run these steps:
 - Build new images
 - Stop portal and worker services
 - Run the new upgrade tool:
  ```bash
  docker-compose run --rm portal tools/upgrade.py
  ```
 - Start portal and worker services
 ## 0.5.0
 The upgrade requires the following steps in the given order
 - Rebuild images
 - Stop your portal and worker services.
 - **Migration with alembic**: required 
 - **Prepare SQL Tiles**: required 
 - Start your portal and worker services.
 - **Reimport tracks**: required 
 - **OSM Import**: no action required
 - **Config changes**: none
 ## 0.4.1
 You can, but do not have to, reimport all tracks. This will generate a GPX file
 for each track and allow the users to download those. If a GPX file has not yet
 been created, the download will fail. To reimport all tracks, log in to your
 PostgreSQL database (instructions are in [README.md](./README.md) for
 development and [docs/production-deployment.md](./docs/production-deployment.md) for production)
 and run:
 ```sql
 UPDATE track SET processing_status = 'queued';
 ```
 You can do this selectively with `WHERE` statements.
 Make sure your worker is running to process the queue.
 ## 0.4.0
 * Rebuild your image, this may take longer than usual, as it will compile
  `osm2pgsql` for you. Next time, it should be in your docker build cache and
  be fast again.
 * Add new config flags: `VERBOSE`, `LEAN_MODE`, `POSTGRES_POOL_SIZE`,
  `POSTGRES_MAX_OVERFLOW`. Check the example config for sane default values.
 * Re-run `tools/prepare_sql_tiles.py` again (see README)
 * It has been made easier to import OSM data, check
  [docs/production-deployment.md](./docs/production-deployment.md) for the sections "Download
  OpenStreetMap maps" and "Import OpenStreetMap data". You can now download
  multiple .pbf files and then import them at once, using the docker image
  built with the `Dockerfile`. Alternatively, you can choose to enable [lean
  mode](docs/lean-mode.md). You do not need to reimport data, but setting this
  up now will make your life easier in the long run ;)
 ## v0.2 to v0.3 (MongoDB to PostgreSQL)
 * Shut down all services
 * Obviously, now is a good time to perform a full backup ;)
 * Update the codebase (`git pull`, `git submodule update`).
 * Update your ``docker-compose.yaml`` with the one from the ``deployment/examples`` 
  folder.
  * Leave the MongoDB service in place for now.
  * Update all other service descriptions. 
  * You can remove `redis` already. 
  * Generate a better password than the default for your
    postgres user.
  * Traefik rules have been simplified as all routes are handled
    by the portal service now.
 * Start up the `mongo` and `postgres` services. Wait for postgres to finish
  initializing (see [README](README.md)).
 * Build the new image (e.g. with `docker-compose build portal`)
 * Configure your API. The example config file is `api/config.py.example`, and
  it will need to be mounted to `api/config.py` in the container. Ignore the
  Keycloak options for now.
 * Prepare the database: 
    ```bash
    docker-compose run --rm portal python tools/reset_database.py
    docker-compose run --rm portal python tools/prepare_sql_tiles.py
    ```
 * Import OSM data (see [README](README.md)).
 * Run the database migration script: 
    ```bash
    docker-compose run --rm \
        -v $PWD/export:/export \
        portal \
        python tools/import_from_mongodb.py mongodb://mongo/obs \
        --keycloak-users-file /export/users.json
    ```
  There is an option `--keep-api-keys` which means the users won't have to
  reconfigure the devices they used their API key in. **However**, please try
  to avoid this option if at all possible, as the old keys are *very* insecure.
  The default without this option to generate a new, secure API key for each
  user.
 * Shut down the `mongo` service, you can now remove it from docker-compose.yaml
 * Start `keycloak` and configure it, similarly to how it was configured in the
  development setup (but choose more secure options). Update the API config
  file to match your keycloak configuration. Import the file
  `export/users.json` into your realm, it will re-add all the users from the
  old installation. You should delete the file and `export/` folder afterwards.
 * Start `portal`.
 * Consider configuring a worker service. See [docs/production-deployment.md](./docs/production-deployment.md).
--- a/api/.gitignore
+++ b/api/.gitignore
@ -43,5 +43,3 @@ local/
 # both, because then developers will only update one of them and they'll
 # contradict. For now, npm shall be the canonical default (compare README.md).
 yarn.lock
 config.overrides.py
--- a/api/Dockerfile
+++ b/api/Dockerfile
@ -1,16 +1,28 @@
-FROM python:3.11.3-bullseye
+FROM node:15.14-buster
 # Install python3, pip3, and make them the default for `python` and `pip` commands
 RUN apt-get update && apt-get install -y python3 python3-pip
 RUN ln -s $(which python3) /usr/local/bin/python
 RUN ln -s $(which pip3) /usr/local/bin/pip
 WORKDIR /opt/obs/api
-ADD scripts /opt/obs/scripts
+ADD package.json package-lock.json /opt/obs/api/
-RUN pip install -e /opt/obs/scripts
+RUN echo update-notifier=false >> ~/.npmrc
 RUN npm ci
-ADD requirements.txt /opt/obs/api/
+ADD scripts /opt/obs/api/scripts/
-RUN pip install -r requirements.txt
+RUN cd scripts && pip install -e .
 ADD setup.py /opt/obs/api/
 ADD obs /opt/obs/api/obs/
 RUN pip install -e .
-EXPOSE 8000
+ADD views /opt/obs/api/views/
 ADD src /opt/obs/api/src/
-CMD ["openbikesensor-api"]
+#ADD .migrations.js .
 #ADD migrations .
 EXPOSE 3000
 ENV PORT=3000
 ENV DATA_DIR=/data
 CMD ["npm", "run", "start"]
--- a/COPYING.LESSER
+++ b/COPYING.LESSER
--- a/api/alembic.ini
+++ b/api/alembic.ini
@ -1,102 +0,0 @@
 # A generic, single database configuration.
 [alembic]
 # path to migration scripts
 script_location = migrations
 # template used to generate migration files
 # file_template = %%(rev)s_%%(slug)s
 # sys.path path, will be prepended to sys.path if present.
 # defaults to the current working directory.
 prepend_sys_path = .
 # timezone to use when rendering the date within the migration file
 # as well as the filename.
 # If specified, requires the python-dateutil library that can be
 # installed by adding `alembic[tz]` to the pip requirements
 # string value is passed to dateutil.tz.gettz()
 # leave blank for localtime
 # timezone =
 # max length of characters to apply to the
 # "slug" field
 # truncate_slug_length = 40
 # set to 'true' to run the environment during
 # the 'revision' command, regardless of autogenerate
 # revision_environment = false
 # set to 'true' to allow .pyc and .pyo files without
 # a source .py file to be detected as revisions in the
 # versions/ directory
 # sourceless = false
 # version location specification; This defaults
 # to api/migrations/versions.  When using multiple version
 # directories, initial revisions must be specified with --version-path.
 # The path separator used here should be the separator specified by "version_path_separator" below.
 # version_locations = %(here)s/bar:%(here)s/bat:api/migrations/versions
 # version path separator; As mentioned above, this is the character used to split
 # version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
 # If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
 # Valid values for version_path_separator are:
 #
 # version_path_separator = :
 # version_path_separator = ;
 # version_path_separator = space
 version_path_separator = os  # Use os.pathsep. Default configuration used for new projects.
 # the output encoding used when revision files
 # are written from script.py.mako
 # output_encoding = utf-8
 sqlalchemy.url = driver://user:pass@localhost/dbname
 [post_write_hooks]
 # post_write_hooks defines scripts or Python functions that are run
 # on newly generated revision scripts.  See the documentation for further
 # detail and examples
 # format using "black" - use the console_scripts runner, against the "black" entrypoint
 # hooks = black
 # black.type = console_scripts
 # black.entrypoint = black
 # black.options = -l 79 REVISION_SCRIPT_FILENAME
 # Logging configuration
 [loggers]
 keys = root,sqlalchemy,alembic
 [handlers]
 keys = console
 [formatters]
 keys = generic
 [logger_root]
 level = WARN
 handlers = console
 qualname =
 [logger_sqlalchemy]
 level = WARN
 handlers =
 qualname = sqlalchemy.engine
 [logger_alembic]
 level = INFO
 handlers =
 qualname = alembic
 [handler_console]
 class = StreamHandler
 args = (sys.stderr,)
 level = NOTSET
 formatter = generic
 [formatter_generic]
 format = %(levelname)-5.5s [%(name)s] %(message)s
 datefmt = %H:%M:%S
--- a/api/config.dev.json
+++ b/api/config.dev.json
@ -0,0 +1,31 @@
 {
  "cookieSecret": "CHANGEME!!!!!!!!!!@##@!!$$$$$$$$$$$$$!!",
  "jwtSecret": "CHANGEME??????????????////3212321;312kjbkasjd",
  "baseUrl": "http://localhost:3000/",
  "mainFrontendUrl": "http://localhost:3001/",
  "mail": false,
  "mongodb": {
    "url": "mongodb://mongo/obsTest",
    "debug": true
  },
  "redisUrl": "redis://redis",
  "oAuth2Clients": [
    {
      "clientId": "b730f8d2-d93c-4c68-9ff0-dfac8da76ee2",
      "validRedirectUris": ["http://localhost:3001/redirect"],
      "refreshTokenExpirySeconds": 604800,
      "maxScope": "*",
      "title": "OBS Portal",
      "autoAccept": true
    },
    {
      "clientId": "a2958209-4045-4ec9-8cb3-1156abba7de3",
      "validRedirectUris": ["__LOCAL__"],
      "maxScope": "track.upload",
      "refreshTokenExpirySeconds": 86400000,
      "title": "OpenBikeSensor"
    }
  ],
  "imprintUrl": "https://example.com/imprint",
  "privacyPolicyUrl": "https://example.com/privacy"
 }
--- a/api/config.dev.py
+++ b/api/config.dev.py
@ -1,35 +0,0 @@
 HOST = "0.0.0.0"
 PORT = 3000
 DEBUG = True
 VERBOSE = False
 AUTO_RELOAD = True
 SECRET = "!!!!!!!!!!!!CHANGE ME!!!!!!!!!!!!"
 POSTGRES_URL = "postgresql+asyncpg://obs:obs@postgres/obs"
 POSTGRES_POOL_SIZE = 20
 POSTGRES_MAX_OVERFLOW = 2 * POSTGRES_POOL_SIZE
 KEYCLOAK_URL = "http://keycloak:8080/auth/realms/obs-dev/"
 KEYCLOAK_CLIENT_ID = "portal"
 KEYCLOAK_CLIENT_SECRET = "c385278e-bd2e-4f13-9937-34b0c0f44c2d"
 DEDICATED_WORKER = True
 API_URL = "http://localhost:3000/"
 FRONTEND_URL = "http://localhost:3001/"
 FRONTEND_HTTPS = False
 FRONTEND_DIR = None
 FRONTEND_CONFIG = {
    "imprintUrl": "https://example.com/imprint",
    "privacyPolicyUrl": "https://example.com/privacy",
    # "termsUrl": "https://example.com/terms", # Link is only shown when set
    "mapHome": {"zoom": 6, "longitude": 10.2, "latitude": 51.3},
    # "banner": {"text": "This is a development installation.", "style": "info"},
 }
 TILES_FILE = None  # "/tiles/tiles.mbtiles"
 DATA_DIR = "/data"
 ADDITIONAL_CORS_ORIGINS = [
    "http://localhost:8880/",  # for maputnik on 8880
    "http://localhost:8888/",  # for maputnik on 8888
 ]
 TILE_SEMAPHORE_SIZE = 4
 EXPORT_SEMAPHORE_SIZE = 4
 # vim: set ft=python :
--- a/api/config.json.example
+++ b/api/config.json.example
@ -0,0 +1,39 @@
 {
  "cookieSecret": "!!!<<<CHANGEME>>>!!!",
  "jwtSecret": "!!!<<<CHANGEME>>>!!!",
  "baseUrl": "https://portal.example.com/api",
  "mainFrontendUrl": "https://portal.example.com/",
  "mail": {
    "from": "Sender Name <sender@example.com>",
    "smtp" : {
      "host": "mail.example.com",
      "port": 465,
      "starttls": false,
      "username": "sender@example.com",
      "password": "!!!<<<CHANGEME>>>!!!"
    }
  },
  "mongodb": {
    "url": "mongodb://mongo/obs",
    "debug": false
  },
  "redisUrl": "redis://redis",
  "oAuth2Clients": [
    {
      "clientId": "!!!<<<CHANGEME>>>!!!",
      "validRedirectUris": ["https://portal.example.com/redirect"],
      "refreshTokenExpirySeconds": 604800,
      "maxScope": "*",
      "title": "OBS Portal"
    },
    {
      "clientId": "!!!<<<CHANGEME>>>!!!",
      "validRedirectUris": ["__LOCAL__"],
      "maxScope": "track.upload",
      "refreshTokenExpirySeconds": 86400000,
      "title": "OpenBikeSensor"
    }
  ],
  "imprintUrl": "https://example.com/imprint",
  "privacyPolicyUrl": "https://example.com/privacy"
 }
--- a/api/config.py.example
+++ b/api/config.py.example
@ -1,73 +0,0 @@
 # Bind address of the server
 HOST = "127.0.0.1"
 PORT = 3000
 # Extended log output, but slower
 DEBUG = False
 VERBOSE = DEBUG
 AUTO_RELOAD = DEBUG
 # Required to encrypt or sign sessions, cookies, tokens, etc.
 SECRET = "!!!<<<CHANGEME>>>!!!"
 # Connection to the database
 POSTGRES_URL = "postgresql+asyncpg://user:pass@host/dbname"
 POSTGRES_POOL_SIZE = 20
 POSTGRES_MAX_OVERFLOW = 2 * POSTGRES_POOL_SIZE
 # URL to the keycloak realm, as reachable by the API service. This is not
 # necessarily its publicly reachable URL, keycloak advertises that iself.
 KEYCLOAK_URL = "http://localhost:1234/auth/realms/obs/"
 # Auth client credentials
 KEYCLOAK_CLIENT_ID = "portal"
 KEYCLOAK_CLIENT_SECRET = "00000000-0000-0000-0000-000000000000"
 # Whether the API should run the worker loop, or a dedicated worker is used
 DEDICATED_WORKER = True
 # The root of the frontend. Needed for redirecting after login, and for CORS.
 # Set to None if frontend is served by the API.
 FRONTEND_URL = None
 FRONTEND_HTTPS = True
 # Where to find the compiled frontend assets (must include index.html), or None
 # to disable serving the frontend.
 FRONTEND_DIR = "../frontend/build/"
 # Can be an object or a JSON string
 FRONTEND_CONFIG = {
    "imprintUrl": "https://example.com/imprint",
    "privacyPolicyUrl": "https://example.com/privacy",
    # "termsUrl": "https://example.com/user_terms_and_conditions", # Link is only shown when set
    "mapHome": {"zoom": 6, "longitude": 10.2, "latitude": 51.3},
    "banner": {"text": "This is a test installation.", "style": "warning"},
 }
 # If the API should serve generated tiles, this is the path where the tiles are
 # built. This is an experimental option and probably very inefficient, a proper
 # tileserver should be prefered. Set to None to disable.
 TILES_FILE = None
 # Path overrides:
 # API_ROOT_DIR = "??" # default: api/ inside repository
 # DATA_DIR = "??" # default: $API_ROOT_DIR/..
 # PROCESSING_DIR = "??" # default: DATA_DIR/processing
 # PROCESSING_OUTPUT_DIR = "??"  # default: DATA_DIR/processing-output
 # TRACKS_DIR = "??" # default: DATA_DIR/tracks
 # OBS_FACE_CACHE_DIR = "??" # default: DATA_DIR/obs-face-cache
 # Additional allowed origins for CORS headers. The FRONTEND_URL is included by
 # default. Python list, or whitespace separated string.
 ADDITIONAL_CORS_ORIGINS = None
 # How many asynchronous requests may be sent to the database to generate tile
 # information. Should be less than POSTGRES_POOL_SIZE to leave some connections
 # to the other features of the API ;)
 TILE_SEMAPHORE_SIZE = 4
 # How many asynchronous requests may generate exported data simultaneously.
 # Keep this small.
 EXPORT_SEMAPHORE_SIZE = 1
 # vim: set ft=python :
--- a/api/migrations/2020-12-01-1945-reconstruct-track-body.js
+++ b/api/migrations/2020-12-01-1945-reconstruct-track-body.js
@ -0,0 +1,61 @@
 const Track = require('../src/models/Track');
 const { replaceDollarNewlinesHack, detectFormat, buildObsver1 } = require('../src/logic/tracks');
 function shouldRebuildBody(track) {
  if (!track.trackData || !track.trackData.points.length) {
    return false;
  }
  if (!track.body) {
    return true;
  }
  const body = track.body.trim();
  if (!body) {
    return true;
  }
  const actualBody = replaceDollarNewlinesHack(body).trim();
  if (body !== actualBody) {
    return true;
  }
  const lineCount = (actualBody.match(/\n/g) || []).length + 1;
  const format = detectFormat(body);
  if (format === 'invalid') {
    return true;
  }
  // never reconstruct body of version 2
  if (format > 1) {
    return false;
  }
  // not enough data in the file
  if (lineCount < track.trackData.points.length + 1) {
    return true;
  }
  return false;
 }
 async function up(next) {
  const query = Track.find().populate('trackData');
  for await (const track of query) {
    const rebuild = shouldRebuildBody(track);
    if (rebuild) {
      track.body = buildObsver1(track.trackData.points);
    }
    await track.save();
  }
  next();
 }
 async function down(next) {
  // nothing to do
  next();
 }
 module.exports = { up, down };
--- a/api/migrations/2020-12-01-1950-rebuild-track-data.js
+++ b/api/migrations/2020-12-01-1950-rebuild-track-data.js
@ -0,0 +1,15 @@
 const Track = require('../src/models/Track');
 module.exports = {
  async up(next) {
    for await (const track of Track.find()) {
      await track.rebuildTrackDataAndSave();
    }
    next();
  },
  async down(next) {
    next();
  },
 };
--- a/api/migrations/2020-12-12-1823-original-filename.js
+++ b/api/migrations/2020-12-12-1823-original-filename.js
@ -0,0 +1,21 @@
 const Track = require('../src/models/Track');
 module.exports = {
  async up(next) {
    try {
      for await (const track of Track.find()) {
        track.originalFileName = track.slug + '.csv'
        await track.generateOriginalFilePath();
        await track.save()
      }
      next();
    } catch(err) {
      next(err)
    }
  },
  async down(next) {
    next();
  },
 };
--- a/api/migrations/2020-12-13-2025-move-to-upload-file.js
+++ b/api/migrations/2020-12-13-2025-move-to-upload-file.js
@ -0,0 +1,25 @@
 const Track = require('../src/models/Track');
 module.exports = {
  async up(next) {
    try {
      for await (const track of Track.find()) {
        if (!track.body) {
          continue
        }
        await track.writeToOriginalFile(track.body)
        delete track.body;
        await track.save()
      }
      next();
    } catch(err) {
      next(err)
    }
  },
  async down(next) {
    next();
  },
 };
--- a/api/migrations/README
+++ b/api/migrations/README
@ -1 +0,0 @@
 Generic single-database configuration.
--- a/api/migrations/env.py
+++ b/api/migrations/env.py
@ -1,83 +0,0 @@
 import asyncio
 from logging.config import fileConfig
 from sqlalchemy import engine_from_config
 from sqlalchemy import pool
 from alembic import context
 # this is the Alembic Config object, which provides
 # access to the values within the .ini file in use.
 config = context.config
 # Interpret the config file for Python logging.
 # This line sets up loggers basically.
 if config.config_file_name is not None:
    fileConfig(config.config_file_name)
 # add your model's MetaData object here
 # for 'autogenerate' support
 # from myapp import mymodel
 # target_metadata = mymodel.Base.metadata
 target_metadata = None
 # other values from the config, defined by the needs of env.py,
 # can be acquired:
 # my_important_option = config.get_main_option("my_important_option")
 # ... etc.
 def do_run_migrations(connection):
    context.configure(connection=connection, target_metadata=target_metadata)
    with context.begin_transaction():
        context.run_migrations()
 def run_migrations_offline():
    """Run migrations in 'offline' mode.
    This configures the context with just a URL
    and not an Engine, though an Engine is acceptable
    here as well.  By skipping the Engine creation
    we don't even need a DBAPI to be available.
    Calls to context.execute() here emit the given string to the
    script output.
    """
    from obs.api.app import app
    url = app.config.POSTGRES_URL
    context.configure(
        url=url,
        target_metadata=target_metadata,
        literal_binds=True,
        dialect_opts={"paramstyle": "named"},
    )
    with context.begin_transaction():
        context.run_migrations()
 async def run_migrations_online():
    """Run migrations in 'online' mode.
    In this scenario we need to create an Engine
    and associate a connection with the context.
    """
    from obs.api.app import app, connect_db
    url = app.config.POSTGRES_URL
    async with connect_db(url) as engine:
        async with engine.connect() as connection:
            await connection.run_sync(do_run_migrations)
        await engine.dispose()
 if context.is_offline_mode():
    run_migrations_offline()
 else:
    asyncio.run(run_migrations_online())
--- a/api/migrations/script.py.mako
+++ b/api/migrations/script.py.mako
@ -1,24 +0,0 @@
 """${message}
 Revision ID: ${up_revision}
 Revises: ${down_revision | comma,n}
 Create Date: ${create_date}
 """
 from alembic import op
 import sqlalchemy as sa
 ${imports if imports else ""}
 # revision identifiers, used by Alembic.
 revision = ${repr(up_revision)}
 down_revision = ${repr(down_revision)}
 branch_labels = ${repr(branch_labels)}
 depends_on = ${repr(depends_on)}
 def upgrade():
    ${upgrades if upgrades else "pass"}
 def downgrade():
    ${downgrades if downgrades else "pass"}
--- a/api/migrations/utils.py
+++ b/api/migrations/utils.py
@ -1,16 +0,0 @@
 import sqlalchemy as sa
 def dbtype(name):
    """
    Create a UserDefinedType for use in migrations as the type of a column,
    when the type already exists in the database, but isn't available as a
    proper sqlalchemy type.
    """
    class TheType(sa.types.UserDefinedType):
        def get_col_spec(self):
            return name
    TheType.__name__ = name
    return TheType
--- a/api/migrations/versions/35e7f1768f9b_create_table_road.py
+++ b/api/migrations/versions/35e7f1768f9b_create_table_road.py
@ -1,39 +0,0 @@
 """create table road
 Revision ID: 35e7f1768f9b
 Revises: 5d75febe2d59
 Create Date: 2022-03-30 21:36:48.157457
 """
 from alembic import op
 import sqlalchemy as sa
 from sqlalchemy.dialects import postgresql
 from migrations.utils import dbtype
 # revision identifiers, used by Alembic.
 revision = "35e7f1768f9b"
 down_revision = "920aed1450c9"
 branch_labels = None
 depends_on = None
 def upgrade():
    op.create_table(
        "road",
        sa.Column(
            "way_id", sa.BIGINT, primary_key=True, index=True, autoincrement=False
        ),
        sa.Column("zone", dbtype("zone_type")),
        sa.Column("name", sa.Text),
        sa.Column("geometry", dbtype("geometry(LINESTRING,3857)")),
        sa.Column("directionality", sa.Integer),
        sa.Column("oneway", sa.Boolean),
    )
    op.execute(
        "CREATE INDEX road_geometry_idx ON road USING GIST (geometry) WITH (FILLFACTOR=100);"
    )
 def downgrade():
    op.drop_table("road")
--- a/api/migrations/versions/3856f240bb6d_create_extensions.py
+++ b/api/migrations/versions/3856f240bb6d_create_extensions.py
@ -1,28 +0,0 @@
 """create extensions
 Revision ID: 3856f240bb6d
 Revises: a9627f63fbed
 Create Date: 2022-03-30 21:31:06.282725
 """
 from alembic import op
 import sqlalchemy as sa
 # revision identifiers, used by Alembic.
 revision = "3856f240bb6d"
 down_revision = None
 branch_labels = None
 depends_on = None
 def upgrade():
    op.execute('CREATE EXTENSION IF NOT EXISTS "hstore";')
    op.execute('CREATE EXTENSION IF NOT EXISTS "postgis";')
    op.execute('CREATE EXTENSION IF NOT EXISTS "uuid-ossp";')
 def downgrade():
    op.execute('DROP EXTENSION "hstore";')
    op.execute('DROP EXTENSION "postgis";')
    op.execute('DROP EXTENSION "uuid-ossp";')
--- a/api/migrations/versions/587e69ecb466_transform_overtaking_event_geometry_to_.py
+++ b/api/migrations/versions/587e69ecb466_transform_overtaking_event_geometry_to_.py
@ -1,30 +0,0 @@
 """transform overtaking_event geometry to 3857
 Revision ID: 587e69ecb466
 Revises: f4b0f460254d
 Create Date: 2023-04-01 14:30:49.927505
 """
 from alembic import op
 import sqlalchemy as sa
 # revision identifiers, used by Alembic.
 revision = "587e69ecb466"
 down_revision = "f4b0f460254d"
 branch_labels = None
 depends_on = None
 def upgrade():
    op.execute("UPDATE overtaking_event SET geometry = ST_Transform(geometry, 3857);")
    op.execute(
        "ALTER TABLE overtaking_event ALTER COLUMN geometry TYPE geometry(POINT, 3857);"
    )
 def downgrade():
    op.execute(
        "ALTER TABLE overtaking_event ALTER COLUMN geometry TYPE geometry;"
    )
    op.execute("UPDATE overtaking_event SET geometry = ST_Transform(geometry, 4326);")
--- a/api/migrations/versions/5d75febe2d59_create_table_overtaking_event.py
+++ b/api/migrations/versions/5d75febe2d59_create_table_overtaking_event.py
@ -1,43 +0,0 @@
 """create table overtaking_event
 Revision ID: 5d75febe2d59
 Revises: 920aed1450c9
 Create Date: 2022-03-30 21:36:37.687080
 """
 from alembic import op
 import sqlalchemy as sa
 from migrations.utils import dbtype
 # revision identifiers, used by Alembic.
 revision = "5d75febe2d59"
 down_revision = "9336eef458e7"
 branch_labels = None
 depends_on = None
 def upgrade():
    op.create_table(
        "overtaking_event",
        sa.Column("id", sa.Integer, autoincrement=True, primary_key=True, index=True),
        sa.Column(
            "track_id", sa.Integer, sa.ForeignKey("track.id", ondelete="CASCADE")
        ),
        sa.Column("hex_hash", sa.String, unique=True, index=True),
        sa.Column("way_id", sa.BIGINT, index=True),
        sa.Column("direction_reversed", sa.Boolean),
        sa.Column("geometry", dbtype("GEOMETRY")),
        sa.Column("latitude", sa.Float),
        sa.Column("longitude", sa.Float),
        sa.Column("time", sa.DateTime),
        sa.Column("distance_overtaker", sa.Float),
        sa.Column("distance_stationary", sa.Float),
        sa.Column("course", sa.Float),
        sa.Column("speed", sa.Float),
        sa.Index("road_segment", "way_id", "direction_reversed"),
    )
 def downgrade():
    op.drop_table("overtaking_event")
--- a/api/migrations/versions/7868aed76122_add_overtaking_event_index.py
+++ b/api/migrations/versions/7868aed76122_add_overtaking_event_index.py
@ -1,26 +0,0 @@
 """add_overtaking_event_index
 Revision ID: 7868aed76122
 Revises: 587e69ecb466
 Create Date: 2023-07-16 13:37:17.694079
 """
 from alembic import op
 import sqlalchemy as sa
 # revision identifiers, used by Alembic.
 revision = '7868aed76122'
 down_revision = '587e69ecb466'
 branch_labels = None
 depends_on = None
 def upgrade():
    op.execute("CREATE INDEX IF NOT EXISTS ix_overtaking_event_geometry ON overtaking_event using GIST(geometry);")
 def downgrade():
    op.drop_index("ix_overtaking_event_geometry")
--- a/api/migrations/versions/920aed1450c9_create_enum_processing_status.py
+++ b/api/migrations/versions/920aed1450c9_create_enum_processing_status.py
@ -1,31 +0,0 @@
 """create enum processing_status
 Revision ID: 920aed1450c9
 Revises: 986c6953e431
 Create Date: 2022-03-30 21:36:25.896192
 """
 from alembic import op
 import sqlalchemy as sa
 from sqlalchemy.dialects import postgresql
 # revision identifiers, used by Alembic.
 revision = "920aed1450c9"
 down_revision = "986c6953e431"
 branch_labels = None
 depends_on = None
 def _get_enum_type():
    return postgresql.ENUM(
        "created", "queued", "processing", "complete", "error", name="processing_status"
    )
 def upgrade():
    _get_enum_type().create(op.get_bind(), checkfirst=True)
 def downgrade():
    _get_enum_type().drop(op.get_bind())
--- a/api/migrations/versions/9336eef458e7_create_table_comment.py
+++ b/api/migrations/versions/9336eef458e7_create_table_comment.py
@ -1,42 +0,0 @@
 """create table comment
 Revision ID: 9336eef458e7
 Revises: 9d8c8c38a1d0
 Create Date: 2022-03-30 21:37:02.080429
 """
 from alembic import op
 import sqlalchemy as sa
 from sqlalchemy.dialects.postgresql import UUID
 # revision identifiers, used by Alembic.
 revision = "9336eef458e7"
 down_revision = "d66baafab5ec"
 branch_labels = None
 depends_on = None
 def upgrade():
    NOW = sa.text("NOW()")
    op.create_table(
        "comment",
        sa.Column("id", sa.Integer, autoincrement=True, primary_key=True),
        sa.Column("uid", UUID, server_default=sa.func.uuid_generate_v4()),
        sa.Column("created_at", sa.DateTime, nullable=False, server_default=NOW),
        sa.Column(
            "updated_at", sa.DateTime, nullable=False, server_default=NOW, onupdate=NOW
        ),
        sa.Column("body", sa.TEXT),
        sa.Column(
            "author_id", sa.Integer, sa.ForeignKey("user.id", ondelete="CASCADE")
        ),
        sa.Column(
            "track_id", sa.Integer, sa.ForeignKey("track.id", ondelete="CASCADE")
        ),
    )
 def downgrade():
    op.drop_table("comment")
--- a/api/migrations/versions/986c6953e431_create_enum_zone_type.py
+++ b/api/migrations/versions/986c6953e431_create_enum_zone_type.py
@ -1,29 +0,0 @@
 """create enum zone_type
 Revision ID: 986c6953e431
 Revises: 3856f240bb6d
 Create Date: 2022-03-30 21:36:19.888268
 """
 from alembic import op
 import sqlalchemy as sa
 from sqlalchemy.dialects import postgresql
 # revision identifiers, used by Alembic.
 revision = "986c6953e431"
 down_revision = "3856f240bb6d"
 branch_labels = None
 depends_on = None
 def _get_enum_type():
    return postgresql.ENUM("rural", "urban", "motorway", name="zone_type")
 def upgrade():
    _get_enum_type().create(op.get_bind(), checkfirst=True)
 def downgrade():
    _get_enum_type().drop(op.get_bind())
--- a/api/migrations/versions/99a3d2eb08f9_add_user_display_name.py
+++ b/api/migrations/versions/99a3d2eb08f9_add_user_display_name.py
@ -1,26 +0,0 @@
 """add user display_name
 Revision ID: 99a3d2eb08f9
 Revises: a9627f63fbed
 Create Date: 2022-09-13 07:30:18.747880
 """
 from alembic import op
 import sqlalchemy as sa
 # revision identifiers, used by Alembic.
 revision = "99a3d2eb08f9"
 down_revision = "a9627f63fbed"
 branch_labels = None
 depends_on = None
 def upgrade():
    op.add_column(
        "user", sa.Column("display_name", sa.String, nullable=True), schema="public"
    )
 def downgrade():
    op.drop_column("user", "display_name", schema="public")
--- a/api/migrations/versions/9d8c8c38a1d0_create_table_user.py
+++ b/api/migrations/versions/9d8c8c38a1d0_create_table_user.py
@ -1,45 +0,0 @@
 """create table user
 Revision ID: 9d8c8c38a1d0
 Revises: d66baafab5ec
 Create Date: 2022-03-30 21:36:59.375149
 """
 from alembic import op
 import sqlalchemy as sa
 # revision identifiers, used by Alembic.
 revision = "9d8c8c38a1d0"
 down_revision = "35e7f1768f9b"
 branch_labels = None
 depends_on = None
 def upgrade():
    NOW = sa.text("NOW()")
    op.create_table(
        "user",
        sa.Column("id", sa.Integer, autoincrement=True, primary_key=True),
        sa.Column("created_at", sa.DateTime, nullable=False, server_default=NOW),
        sa.Column(
            "updated_at", sa.DateTime, nullable=False, server_default=NOW, onupdate=NOW
        ),
        sa.Column("sub", sa.String, unique=True, nullable=False),
        sa.Column("username", sa.String, unique=True, nullable=False),
        sa.Column("email", sa.String, nullable=False),
        sa.Column("bio", sa.TEXT),
        sa.Column("image", sa.String),
        sa.Column(
            "are_tracks_visible_for_all",
            sa.Boolean,
            server_default=sa.false(),
            nullable=False,
        ),
        sa.Column("api_key", sa.String),
        sa.Column("match_by_username_email", sa.Boolean, server_default=sa.false()),
    )
 def downgrade():
    op.drop_table("user")
--- a/api/migrations/versions/a049e5eb24dd_create_table_region.py
+++ b/api/migrations/versions/a049e5eb24dd_create_table_region.py
@ -1,35 +0,0 @@
 """create table region
 Revision ID: a049e5eb24dd
 Revises: a9627f63fbed
 Create Date: 2022-04-02 21:28:43.124521
 """
 from alembic import op
 import sqlalchemy as sa
 from migrations.utils import dbtype
 # revision identifiers, used by Alembic.
 revision = "a049e5eb24dd"
 down_revision = "99a3d2eb08f9"
 branch_labels = None
 depends_on = None
 def upgrade():
    op.create_table(
        "region",
        sa.Column("id", sa.String(24), primary_key=True, index=True),
        sa.Column("name", sa.Text),
        sa.Column("geometry", dbtype("GEOMETRY(GEOMETRY,3857)"), index=False),
        sa.Column("admin_level", sa.Integer, index=True),
    )
    op.execute(
        "CREATE INDEX region_geometry_idx ON region USING GIST (geometry) WITH (FILLFACTOR=100);"
    )
 def downgrade():
    op.drop_table("region")
--- a/api/migrations/versions/a9627f63fbed_create_table_road_usage.py
+++ b/api/migrations/versions/a9627f63fbed_create_table_road_usage.py
@ -1,34 +0,0 @@
 """create table road_usage
 Revision ID: a9627f63fbed
 Revises:
 Create Date: 2022-03-16 20:26:17.449569
 """
 from alembic import op
 import sqlalchemy as sa
 # revision identifiers, used by Alembic.
 revision = "a9627f63fbed"
 down_revision = "5d75febe2d59"
 branch_labels = None
 depends_on = None
 def upgrade():
    op.create_table(
        "road_usage",
        sa.Column("id", sa.Integer, autoincrement=True, primary_key=True, index=True),
        sa.Column(
            "track_id", sa.Integer, sa.ForeignKey("track.id", ondelete="CASCADE")
        ),
        sa.Column("hex_hash", sa.String, unique=True, index=True),
        sa.Column("way_id", sa.BIGINT, index=True),
        sa.Column("time", sa.DateTime),
        sa.Column("direction_reversed", sa.Boolean),
        sa.Index("road_usage_segment", "way_id", "direction_reversed"),
    )
 def downgrade():
    op.drop_table("road_usage")
--- a/api/migrations/versions/b8b0fbae50a4_add_import_groups.py
+++ b/api/migrations/versions/b8b0fbae50a4_add_import_groups.py
@ -1,39 +0,0 @@
 """add import groups
 Revision ID: b8b0fbae50a4
 Revises: f7b21148126a
 Create Date: 2023-03-26 09:41:36.621203
 """
 from alembic import op
 import sqlalchemy as sa
 # revision identifiers, used by Alembic.
 revision = "b8b0fbae50a4"
 down_revision = "f7b21148126a"
 branch_labels = None
 depends_on = None
 def upgrade():
    op.add_column(
        "road",
        sa.Column("import_group", sa.String(), nullable=True),
    )
    op.add_column(
        "region",
        sa.Column("import_group", sa.String(), nullable=True),
    )
    # Set existing to "osm2pgsql"
    road = sa.table("road", sa.column("import_group", sa.String))
    op.execute(road.update().values(import_group="osm2pgsql"))
    region = sa.table("region", sa.column("import_group", sa.String))
    op.execute(region.update().values(import_group="osm2pgsql"))
 def downgrade():
    op.drop_column("road", "import_group")
    op.drop_column("region", "import_group")
--- a/api/migrations/versions/d66baafab5ec_create_table_track.py
+++ b/api/migrations/versions/d66baafab5ec_create_table_track.py
@ -1,66 +0,0 @@
 """create table track
 Revision ID: d66baafab5ec
 Revises: 35e7f1768f9b
 Create Date: 2022-03-30 21:36:54.848452
 """
 from alembic import op
 import sqlalchemy as sa
 from sqlalchemy.dialects import postgresql
 from migrations.utils import dbtype
 # revision identifiers, used by Alembic.
 revision = "d66baafab5ec"
 down_revision = "9d8c8c38a1d0"
 branch_labels = None
 depends_on = None
 def upgrade():
    NOW = sa.text("NOW()")
    op.create_table(
        "track",
        sa.Column("id", sa.Integer, primary_key=True, autoincrement=True),
        sa.Column("slug", sa.String, unique=True, nullable=False, index=True),
        sa.Column("created_at", sa.DateTime, nullable=False, server_default=NOW),
        sa.Column(
            "updated_at", sa.DateTime, nullable=False, server_default=NOW, onupdate=NOW
        ),
        sa.Column("title", sa.String),
        sa.Column(
            "processing_status",
            dbtype("processing_status"),
            server_default=sa.literal("created"),
        ),
        sa.Column("processing_queued_at", sa.DateTime),
        sa.Column("processed_at", sa.DateTime),
        sa.Column("processing_log", sa.TEXT),
        sa.Column(
            "customized_title", sa.Boolean, server_default=sa.false(), nullable=False
        ),
        sa.Column("description", sa.TEXT),
        sa.Column("public", sa.Boolean, server_default=sa.false()),
        sa.Column("uploaded_by_user_agent", sa.String),
        sa.Column("original_file_name", sa.String),
        sa.Column("original_file_hash", sa.String, nullable=False),
        sa.Column(
            "author_id",
            sa.Integer,
            sa.ForeignKey("user.id", ondelete="CASCADE"),
            nullable=False,
        ),
        sa.Column("recorded_at", sa.DateTime),
        sa.Column("recorded_until", sa.DateTime),
        sa.Column("duration", sa.Float),
        sa.Column("length", sa.Float),
        sa.Column("segments", sa.Integer),
        sa.Column("num_events", sa.Integer),
        sa.Column("num_measurements", sa.Integer),
        sa.Column("num_valid", sa.Integer),
    )
 def downgrade():
    op.drop_table("track")
--- a/api/migrations/versions/f4b0f460254d_add_osm_id_indexes.py
+++ b/api/migrations/versions/f4b0f460254d_add_osm_id_indexes.py
@ -1,24 +0,0 @@
 """add osm id indexes
 Revision ID: f4b0f460254d
 Revises: b8b0fbae50a4
 Create Date: 2023-03-30 10:56:22.066768
 """
 from alembic import op
 import sqlalchemy as sa
 # revision identifiers, used by Alembic.
 revision = "f4b0f460254d"
 down_revision = "b8b0fbae50a4"
 branch_labels = None
 depends_on = None
 def upgrade():
    op.execute("CREATE INDEX IF NOT EXISTS ix_road_way_id ON road (way_id);")
 def downgrade():
    op.drop_index("ix_road_way_id")
--- a/api/migrations/versions/f7b21148126a_add_user_device.py
+++ b/api/migrations/versions/f7b21148126a_add_user_device.py
@ -1,41 +0,0 @@
 """add user_device
 Revision ID: f7b21148126a
 Revises: a9627f63fbed
 Create Date: 2022-09-15 17:48:06.764342
 """
 from alembic import op
 import sqlalchemy as sa
 # revision identifiers, used by Alembic.
 revision = "f7b21148126a"
 down_revision = "a049e5eb24dd"
 branch_labels = None
 depends_on = None
 def upgrade():
    op.create_table(
        "user_device",
        sa.Column("id", sa.Integer, autoincrement=True, primary_key=True),
        sa.Column("user_id", sa.Integer, sa.ForeignKey("user.id", ondelete="CASCADE")),
        sa.Column("identifier", sa.String, nullable=False),
        sa.Column("display_name", sa.String, nullable=True),
        sa.Index("user_id_identifier", "user_id", "identifier", unique=True),
    )
    op.add_column(
        "track",
        sa.Column(
            "user_device_id",
            sa.Integer,
            sa.ForeignKey("user_device.id", ondelete="RESTRICT"),
            nullable=True,
        ),
    )
 def downgrade():
    op.drop_column("track", "user_device_id")
    op.drop_table("user_device")
--- a/api/obs/init.py
+++ b/api/obs/init.py
@ -1 +0,0 @@
 __path__ = __import__("pkgutil").extend_path(__path__, __name__)
--- a/api/obs/api/init.py
+++ b/api/obs/api/init.py
@ -1 +0,0 @@
 __version__ = "0.8.1"
--- a/api/obs/api/app.py
+++ b/api/obs/api/app.py
@ -1,363 +0,0 @@
 import asyncio
 import logging
 import re
 from json import JSONEncoder, dumps
 from functools import wraps, partial
 from urllib.parse import urlparse
 from os.path import dirname, join, normpath, abspath, isfile
 from datetime import datetime, date
 from sanic import Sanic, Blueprint
 from sanic.response import (
    text,
    json as json_response,
    file as file_response,
    html as html_response,
 )
 from sanic.exceptions import Unauthorized, SanicException
 from sanic_session import Session, InMemorySessionInterface
 from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 from obs.api.db import User, make_session, connect_db
 from obs.api.cors import setup_options, add_cors_headers
 from obs.api.utils import get_single_arg
 log = logging.getLogger(__name__)
 class SanicAccessMessageFilter(logging.Filter):
    """
    A filter that modifies the log message of a sanic.access log entry to
    include useful information.
    """
    def filter(self, record):
        record.msg = f"{record.request} -> {record.status}"
        return True
 def configure_sanic_logging():
    for logger_name in ["sanic.root", "sanic.access", "sanic.error"]:
        logger = logging.getLogger(logger_name)
        for handler in logger.handlers:
            logger.removeHandler(handler)
    logger = logging.getLogger("sanic.access")
    for filter_ in logger.filters:
        logger.removeFilter(filter_)
    logger.addFilter(SanicAccessMessageFilter())
    logging.getLogger("sanic.root").setLevel(logging.WARNING)
 app = Sanic(
    "openbikesensor-api",
    env_prefix="OBS_",
 )
 configure_sanic_logging()
 app.config.update(
    dict(
        DEBUG=False,
        VERBOSE=False,
        AUTO_RELOAD=False,
        POSTGRES_POOL_SIZE=20,
        POSTGRES_MAX_OVERFLOW=40,
        DEDICATED_WORKER=True,
        FRONTEND_URL=None,
        FRONTEND_HTTPS=True,
        TILES_FILE=None,
        TILE_SEMAPHORE_SIZE=4,
        EXPORT_SEMAPHORE_SIZE=1,
    )
 )
 # overwrite from defaults again
 app.config.load_environment_vars("OBS_")
 if isfile("./config.py"):
    app.update_config("./config.py")
 # For developers to override the config without committing it
 if isfile("./config.overrides.py"):
    app.update_config("./config.overrides.py")
 c = app.config
 api = Blueprint("api", url_prefix="/api")
 auth = Blueprint("auth", url_prefix="")
 import re
 TILE_REQUEST_CANCELLED = re.compile(
    r"Connection lost before response written.*GET /tiles"
 )
 class NoConnectionLostFilter(logging.Filter):
    def filter(record):
        return not TILE_REQUEST_CANCELLED.match(record.getMessage())
 logging.getLogger("sanic.error").addFilter(NoConnectionLostFilter)
 def setup_cors(app):
    frontend_url = app.config.get("FRONTEND_URL")
    additional_origins = app.config.get("ADDITIONAL_CORS_ORIGINS")
    if not frontend_url and not additional_origins:
        # No CORS configured
        return
    origins = []
    if frontend_url:
        u = urlparse(frontend_url)
        origins.append(f"{u.scheme}://{u.netloc}")
    if isinstance(additional_origins, str):
        origins += re.split(r"\s+", additional_origins)
    elif isinstance(additional_origins, list):
        origins += additional_origins
    elif additional_origins is not None:
        raise ValueError(
            "invalid option type for ADDITIONAL_CORS_ORIGINS, must be list or space separated str"
        )
    app.ctx.cors_origins = origins
    # Add OPTIONS handlers to any route that is missing it
    app.register_listener(setup_options, "before_server_start")
    # Fill in CORS headers
    app.register_middleware(add_cors_headers, "response")
 setup_cors(app)
@app.exception(SanicException, BaseException)
 async def _handle_sanic_errors(_request, exception):
    if isinstance(exception, asyncio.CancelledError):
        return None
    log.error("Exception in handler: %s", exception, exc_info=True)
    return json_response(
        {
            "errors": {
                type(exception).__name__: str(exception),
            },
        },
        status=exception.status_code if hasattr(exception, "status_code") else 500,
    )
 # Configure paths
 def configure_paths(c):
    c.API_ROOT_DIR = c.get("API_ROOT_DIR") or abspath(
        join(dirname(__file__), "..", "..")
    )
    c.DATA_DIR = c.get("DATA_DIR") or normpath(join(c.API_ROOT_DIR, "../data"))
    c.PROCESSING_DIR = c.get("PROCESSING_DIR") or join(c.DATA_DIR, "processing")
    c.PROCESSING_OUTPUT_DIR = c.get("PROCESSING_OUTPUT_DIR") or join(
        c.DATA_DIR, "processing-output"
    )
    c.TRACKS_DIR = c.get("TRACKS_DIR") or join(c.DATA_DIR, "tracks")
    c.OBS_FACE_CACHE_DIR = c.get("OBS_FACE_CACHE_DIR") or join(
        c.DATA_DIR, "obs-face-cache"
    )
    c.FRONTEND_DIR = c.get("FRONTEND_DIR")
 configure_paths(app.config)
 # TODO: use a different interface, maybe backed by the PostgreSQL, to allow
 # scaling the API
 Session(app, interface=InMemorySessionInterface())
@app.before_server_start
 async def app_connect_db(app, loop):
    app.ctx._db_engine_ctx = connect_db(
        app.config.POSTGRES_URL,
        app.config.POSTGRES_POOL_SIZE,
        app.config.POSTGRES_MAX_OVERFLOW,
    )
    app.ctx._db_engine = await app.ctx._db_engine_ctx.__aenter__()
    if app.config.TILE_SEMAPHORE_SIZE:
        app.ctx.tile_semaphore = asyncio.Semaphore(app.config.TILE_SEMAPHORE_SIZE)
    if app.config.EXPORT_SEMAPHORE_SIZE:
        app.ctx.export_semaphore = asyncio.Semaphore(app.config.EXPORT_SEMAPHORE_SIZE)
@app.after_server_stop
 async def app_disconnect_db(app, loop):
    if hasattr(app.ctx, "_db_engine_ctx"):
        await app.ctx._db_engine_ctx.__aexit__(None, None, None)
 def remove_right(l, r):
    if l.endswith(r):
        return l[: -len(r)]
    return l
@app.middleware("request")
 async def inject_arg_getter(req):
    req.ctx.get_single_arg = partial(get_single_arg, req)
@app.middleware("request")
 async def inject_urls(req):
    if req.app.config.FRONTEND_HTTPS:
        req.ctx.frontend_scheme = "https"
    elif req.app.config.FRONTEND_URL:
        req.ctx.frontend_scheme = (
            "http" if req.app.config.FRONTEND_URL.startswith("http://") else "https"
        )
    else:
        req.ctx.frontend_scheme = req.scheme
    if req.app.config.get("API_URL"):
        req.ctx.api_url = req.app.config.API_URL.rstrip("/")
        api_url_parsed = urlparse(req.ctx.api_url)
        req.ctx.api_scheme = api_url_parsed.scheme  # just use the same for now
        req.ctx.api_base_path = api_url_parsed.path
    else:
        req.ctx.api_scheme = req.ctx.frontend_scheme  # just use the same for now
        req.ctx.api_base_path = remove_right(req.server_path, req.path)
        req.ctx.api_url = (
            f"{req.ctx.frontend_scheme}://{req.host}{req.ctx.api_base_path}"
        )
    if req.app.config.FRONTEND_URL:
        req.ctx.frontend_base_path = "/" + urlparse(
            req.app.config.FRONTEND_URL
        ).path.strip("/")
        req.ctx.frontend_url = req.app.config.FRONTEND_URL.rstrip("/")
    elif app.config.FRONTEND_DIR:
        req.ctx.frontend_base_path = req.ctx.api_base_path
        req.ctx.frontend_url = req.ctx.api_url
    else:
        req.ctx.frontend_base_path = "/"
        req.ctx.frontend_url = (
            f"{req.ctx.frontend_scheme}://{req.host}{req.ctx.frontend_base_path}"
        )
@app.middleware("request")
 async def inject_session(req):
    req.ctx._session_ctx = make_session()
    req.ctx.db = await req.ctx._session_ctx.__aenter__()
@app.middleware("response")
 async def close_session(req, response):
    if hasattr(req.ctx, "_session_ctx"):
        await req.ctx.db.close()
        await req.ctx._session_ctx.__aexit__(None, None, None)
@app.middleware("request")
 async def load_user(req):
    user_id = req.ctx.session.get("user_id")
    user = None
    if user_id:
        user = (
            await req.ctx.db.execute(select(User).where(User.id == user_id))
        ).scalar()
    req.ctx.user = user
 def require_auth(fn):
    @wraps(fn)
    def wrapper(req, *args, **kwargs):
        if not req.ctx.user:
            raise Unauthorized("Login required")
        return fn(req, *args, **kwargs)
    return wrapper
 def read_api_key(fn):
    """
    A middleware decorator to read the API Key of a user. It is an opt-in to
    allow usage with API Keys on certain urls. Combine with require_auth to
    actually check whether a user was authenticated through this. If a login
    session exists, the api key is ignored.
    """
    @wraps(fn)
    async def wrapper(req, *args, **kwargs):
        # try to parse a token if one exists, unless a user is already authenticated
        if (
            not req.ctx.user
            and isinstance(req.token, str)
            and req.token.lower().startswith("obsuserid ")
        ):
            try:
                api_key = req.token.split()[1]
            except LookupError:
                api_key = None
            if api_key:
                user = (
                    await req.ctx.db.execute(
                        select(User).where(User.api_key == api_key.strip())
                    )
                ).scalar()
                if not user:
                    raise Unauthorized("invalid OBSUserId token")
                req.ctx.user = user
        return await fn(req, *args, **kwargs)
    return wrapper
 class CustomJsonEncoder(JSONEncoder):
    def default(self, obj):
        if isinstance(obj, (datetime, date)):
            return obj.isoformat() + "+0000"  # explicit UTC for javascript <3
        # Let the base class default method raise the TypeError
        return super().default(obj)
 def json(*args, **kwargs):
    return json_response(*args, **kwargs, dumps=partial(dumps, cls=CustomJsonEncoder))
 from .routes import (
    info,
    login,
    stats,
    tracks,
    users,
    exports,
 )
 from .routes import tiles, mapdetails
 from .routes import frontend
 app.blueprint(api)
 app.blueprint(auth)
 if not app.config.DEDICATED_WORKER:
    async def worker():
        from obs.api.process import process_tracks_loop
        # run forever
        await process_tracks_loop(10)
    app.add_task(worker())
--- a/api/obs/api/cors.py
+++ b/api/obs/api/cors.py
@ -1,68 +0,0 @@
 from collections import defaultdict
 from typing import Dict, FrozenSet, Iterable
 from sanic import Sanic, response
 from sanic_routing.router import Route
 def _add_cors_headers(request, response, methods: Iterable[str]) -> None:
    allow_methods = list(set(methods))
    if "OPTIONS" not in allow_methods:
        allow_methods.append("OPTIONS")
    origin = request.headers.get("origin")
    if origin in request.app.ctx.cors_origins:
        headers = {
            "Access-Control-Allow-Methods": ",".join(allow_methods),
            "Access-Control-Allow-Origin": origin,
            "Access-Control-Allow-Credentials": "true",
            "Access-Control-Allow-Headers": (
                "origin, content-type, accept, "
                "authorization, x-xsrf-token, x-request-id"
            ),
            "Access-Control-Expose-Headers": "content-disposition",
        }
        response.headers.extend(headers)
 def add_cors_headers(request, response):
    if request.method != "OPTIONS":
        methods = [method for method in request.route.methods]
        _add_cors_headers(request, response, methods)
 def _compile_routes_needing_options(routes: Dict[str, Route]) -> Dict[str, FrozenSet]:
    needs_options = defaultdict(list)
    # This is 21.12 and later. You will need to change this for older versions.
    for route in routes.values():
        if "OPTIONS" not in route.methods:
            needs_options[route.uri].extend(route.methods)
    return {uri: frozenset(methods) for uri, methods in dict(needs_options).items()}
 def _options_wrapper(handler, methods):
    def wrapped_handler(request, *args, **kwargs):
        nonlocal methods
        return handler(request, methods)
    return wrapped_handler
 async def options_handler(request, methods) -> response.HTTPResponse:
    resp = response.empty()
    _add_cors_headers(request, resp, methods)
    return resp
 def setup_options(app: Sanic, _):
    app.router.reset()
    needs_options = _compile_routes_needing_options(app.router.routes_all)
    for uri, methods in needs_options.items():
        app.add_route(
            _options_wrapper(options_handler, methods),
            uri,
            methods=["OPTIONS"],
        )
    app.router.finalize()
--- a/api/obs/api/db.py
+++ b/api/obs/api/db.py
@ -1,578 +0,0 @@
 import hashlib
 from contextvars import ContextVar
 from contextlib import asynccontextmanager
 from datetime import datetime
 import os
 from os.path import exists, join, dirname
 from json import loads
 import re
 import math
 import aiofiles
 import random
 import string
 import secrets
 from slugify import slugify
 import logging
 from sqlalchemy.ext.declarative import declarative_base
 from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy.ext.asyncio import create_async_engine
 from sqlalchemy.orm import sessionmaker as SessionMaker, relationship
 from sqlalchemy.types import UserDefinedType, BIGINT, TEXT
 from sqlalchemy import (
    Boolean,
    Column,
    DateTime,
    Enum as SqlEnum,
    Float,
    ForeignKey,
    Index,
    Integer,
    String,
    false,
    func,
    select,
    text,
    literal,
    Text,
 )
 from sqlalchemy.dialects.postgresql import UUID
 log = logging.getLogger(__name__)
 Base = declarative_base()
 engine = None
 sessionmaker: SessionMaker
@asynccontextmanager
 async def make_session():
    async with sessionmaker(autoflush=True) as session:
        yield session
 async def drop_all():
    async with engine.begin() as conn:
        await conn.run_sync(Base.metadata.drop_all)
 async def init_models():
    async with engine.begin() as conn:
        await conn.execute(text('CREATE EXTENSION IF NOT EXISTS "hstore";'))
        await conn.execute(text('CREATE EXTENSION IF NOT EXISTS "postgis";'))
        await conn.execute(text('CREATE EXTENSION IF NOT EXISTS "uuid-ossp";'))
        await conn.run_sync(Base.metadata.create_all)
 def random_string(length):
    letters = string.ascii_lowercase + string.digits
    return "".join(random.choice(letters) for _ in range(length))
@asynccontextmanager
 async def connect_db(url, pool_size=10, max_overflow=20):
    global engine, sessionmaker
    engine = create_async_engine(
        url, echo=False, pool_size=pool_size, max_overflow=max_overflow
    )
    sessionmaker = SessionMaker(engine, class_=AsyncSession, expire_on_commit=False)
    yield engine
    # for AsyncEngine created in function scope, close and
    # clean-up pooled connections
    await engine.dispose()
    engine = None
    sessionmaker = None
 ZoneType = SqlEnum("rural", "urban", "motorway", name="zone_type")
 ProcessingStatus = SqlEnum(
    "created", "queued", "processing", "complete", "error", name="processing_status"
 )
 class Geometry(UserDefinedType):
    def get_col_spec(self):
        return "GEOMETRY"
    def bind_expression(self, bindvalue):
        return func.ST_GeomFromGeoJSON(bindvalue, type_=self)
    def column_expression(self, col):
        return func.ST_AsGeoJSON(func.ST_Transform(col, 4326), type_=self)
 class LineString(UserDefinedType):
    def get_col_spec(self):
        return "geometry(LineString, 3857)"
    def bind_expression(self, bindvalue):
        return func.ST_GeomFromGeoJSON(bindvalue, type_=self)
    def column_expression(self, col):
        return func.ST_AsGeoJSON(func.ST_Transform(col, 4326), type_=self)
 class GeometryGeometry(UserDefinedType):
    def get_col_spec(self):
        return "geometry(GEOMETRY, 3857)"
    def bind_expression(self, bindvalue):
        return func.ST_GeomFromGeoJSON(bindvalue, type_=self)
    def column_expression(self, col):
        return func.ST_AsGeoJSON(func.ST_Transform(col, 4326), type_=self)
 class OvertakingEvent(Base):
    __tablename__ = "overtaking_event"
    __table_args__ = (Index("road_segment", "way_id", "direction_reversed"),)
    id = Column(Integer, autoincrement=True, primary_key=True, index=True)
    track_id = Column(Integer, ForeignKey("track.id", ondelete="CASCADE"))
    hex_hash = Column(String, unique=True, index=True)
    way_id = Column(BIGINT, index=True)
    # whether we were traveling along the way in reverse direction
    direction_reversed = Column(Boolean)
    geometry = Column(Geometry)
    latitude = Column(Float)
    longitude = Column(Float)
    time = Column(DateTime)
    distance_overtaker = Column(Float)
    distance_stationary = Column(Float)
    course = Column(Float)
    speed = Column(Float)
    def __repr__(self):
        return f"<OvertakingEvent {self.id}>"
 class Road(Base):
    __tablename__ = "road"
    way_id = Column(BIGINT, primary_key=True, index=True, autoincrement=False)
    zone = Column(ZoneType)
    name = Column(Text)
    geometry = Column(LineString)
    directionality = Column(Integer)
    oneway = Column(Boolean)
    import_group = Column(String)
    __table_args__ = (
        # We keep the index name as osm2pgsql created it, way back when.
        Index(
            "road_geometry_idx",
            "geometry",
            postgresql_using="gist",
            postgresql_with={"fillfactor": 100},
        ),
    )
    def to_dict(self):
        return {
            "way_id": self.way_id,
            "zone": self.zone,
            "name": self.name,
            "directionality": self.directionality,
            "oneway": self.oneway,
            "geometry": loads(self.geometry),
        }
 class RoadUsage(Base):
    __tablename__ = "road_usage"
    __table_args__ = (Index("road_usage_segment", "way_id", "direction_reversed"),)
    id = Column(Integer, autoincrement=True, primary_key=True, index=True)
    track_id = Column(Integer, ForeignKey("track.id", ondelete="CASCADE"))
    hex_hash = Column(String, unique=True, index=True)
    way_id = Column(BIGINT, index=True)
    time = Column(DateTime)
    direction_reversed = Column(Boolean)
    def __repr__(self):
        return f"<RoadUsage {self.id}>"
    def __hash__(self):
        return int(self.hex_hash, 16)
    def __eq__(self, other):
        return self.hex_hash == other.hex_hash
 NOW = text("NOW()")
 class DuplicateTrackFileError(ValueError):
    pass
 class Track(Base):
    __tablename__ = "track"
    id = Column(Integer, primary_key=True, autoincrement=True)
    slug = Column(String, unique=True, nullable=False, index=True)
    created_at = Column(DateTime, nullable=False, server_default=NOW)
    updated_at = Column(DateTime, nullable=False, server_default=NOW, onupdate=NOW)
    title = Column(String)
    processing_status = Column(ProcessingStatus, server_default=literal("created"))
    processing_queued_at = Column(DateTime)
    processed_at = Column(DateTime)
    processing_log = Column(TEXT)
    # Set to true if the user customized the title. Disables auto-generating
    # an updated title when the track is (re-)processed.
    customized_title = Column(Boolean, server_default=false(), nullable=False)
    # A user-provided description of the track. May contain markdown.
    description = Column(TEXT)
    # Whether this track is visible (anonymized) in the public track list or not.
    public = Column(Boolean, server_default=false())
    # Whether this track should be exported to the public track database
    # (after anonymization).
    # include_in_public_database = Column(Boolean, server_default=false())
    # The user agent string, or a part thereof, that was used to upload this
    # track. Usually contains only the OBS version, other user agents are
    # discarded due to being irrelevant.
    uploaded_by_user_agent = Column(String)
    # The name of the original file, as provided during upload. Used for
    # providing a download with the same name, and for display in the
    # frontend.
    original_file_name = Column(String)
    # A hash of the original file's contents. Nobody can upload the same track twice.
    original_file_hash = Column(String, nullable=False)
    author_id = Column(
        Integer, ForeignKey("user.id", ondelete="CASCADE"), nullable=False
    )
    user_device_id = Column(
        Integer,
        ForeignKey("user_device.id", ondelete="RESTRICT"),
        nullable=True,
    )
    # Statistics... maybe we'll drop some of this if we can easily compute them from SQL
    recorded_at = Column(DateTime)
    recorded_until = Column(DateTime)
    duration = Column(Float)
    length = Column(Float)
    segments = Column(Integer)
    num_events = Column(Integer)
    num_measurements = Column(Integer)
    num_valid = Column(Integer)
    def to_dict(self, for_user_id=None):
        result = {
            "id": self.id,
            "slug": self.slug,
            "title": self.title,
            "description": self.description,
            "createdAt": self.created_at,
            "updatedAt": self.updated_at,
            "public": self.public,
            "processingStatus": self.processing_status,
            "recordedAt": self.recorded_at,
            "recordedUntil": self.recorded_until,
            "duration": self.duration,
            "length": self.length,
            "numEvents": self.num_events,
            "numValid": self.num_valid,
            "numMeasurements": self.num_measurements,
        }
        if for_user_id is not None and for_user_id == self.author_id:
            result["uploadedByUserAgent"] = self.uploaded_by_user_agent
            result["originalFileName"] = self.original_file_name
            result["userDeviceId"] = self.user_device_id
        if self.author:
            result["author"] = self.author.to_dict(for_user_id=for_user_id)
        return result
    def is_visible_to_private(self, user):
        return user is not None and user.id == self.author_id
    def is_visible_to(self, user):
        return self.is_visible_to_private(user) or self.public
    def generate_slug(self, new_title_or_filename=None):
        input_text = new_title_or_filename or self.title or self.original_file_name
        if input_text is not None:
            self.slug = slugify(input_text, separator="_") + "-"
        else:
            self.slug = ""
        # make unique
        self.slug += random_string(8)
    async def prevent_duplicates(self, session, file_body):
        hex_hash = hashlib.sha512(file_body).hexdigest()
        duplicate_count = await session.scalar(
            select(func.count())
            .select_from(Track)
            .where(
                Track.original_file_hash == hex_hash
                and Track.author_id == self.author_id
                and Track.id != self.id
            )
        )
        if duplicate_count:
            raise DuplicateTrackFileError()
        self.original_file_hash = hex_hash
    async def write_to_original_file(self, config, body):
        mode = "wb" if isinstance(body, bytes) else "wt"
        target = self.get_original_file_path(config)
        os.makedirs(dirname(target), exist_ok=True)
        async with aiofiles.open(target, mode=mode) as f:
            await f.write(body)
    def queue_processing(self):
        self.processing_status = "queued"
        self.processing_queued_at = datetime.utcnow()
    def auto_generate_title(self):
        if self.customized_title:
            return
        # Try to figure out when this file was recorded. Either we have it in then
        # statistics, e.g. after parsing and processing the track, or we can maybe
        # derive it from the filename.
        recorded_at = self.recorded_at
        if not recorded_at and self.original_file_name:
            match = re.match(
                r"^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}\.[0-9]{2}\.[0-9]{2}",
                self.original_file_name,
            )
            if match:
                try:
                    recorded_at = datetime.fromisoformat(match[0])
                except ValueError:
                    pass
        if recorded_at:
            daytime = _get_daytime(recorded_at)
            self.title = f"{daytime} ride on {recorded_at.strftime('%a, %x')}"
            return
        # Detecting recording date failed, use filename
        if self.original_file_name:
            words = self.original_file_name
            words = re.sub(r"(\.obsdata)?\.csv$", "", words)
            words = re.split(r"\W+", words)
            words[0] = words[0][0].upper() + words[0][1:]
            self.title = " ".join(words)
    @property
    def file_path(self):
        return join(self.author.username, self.slug)
    def get_original_file_path(self, config):
        return join(config.TRACKS_DIR, self.file_path, "original.csv")
 class User(Base):
    __tablename__ = "user"
    id = Column(Integer, autoincrement=True, primary_key=True)
    created_at = Column(DateTime, nullable=False, server_default=NOW)
    updated_at = Column(DateTime, nullable=False, server_default=NOW, onupdate=NOW)
    sub = Column(String, unique=True, nullable=False)
    username = Column(String, unique=True, nullable=False)
    display_name = Column(String, nullable=True)
    email = Column(String, nullable=False)
    bio = Column(TEXT)
    image = Column(String)
    are_tracks_visible_for_all = Column(Boolean, server_default=false(), nullable=False)
    api_key = Column(String)
    # This user can be matched by the email address from the auth service
    # instead of having to match by `sub`. If  a matching user logs in, the
    # `sub` is updated to the new sub and this flag is disabled. This is for
    # migrating *to* the external authentication scheme.
    match_by_username_email = Column(Boolean, server_default=false())
    def generate_api_key(self):
        """
        Generates a new :py:obj:`api_key` into this instance. The new key is
        sourced from a secure random source and is urlsafe.
        """
        self.api_key = secrets.token_urlsafe(24)
    def to_dict(self, for_user_id=None):
        result = {
            "id": self.id,
            "displayName": self.display_name or self.username,
            "bio": self.bio,
            "image": self.image,
        }
        if for_user_id == self.id:
            result["username"] = self.username
        return result
    async def rename(self, config, new_name):
        old_name = self.username
        renames = [
            (join(basedir, old_name), join(basedir, new_name))
            for basedir in [config.PROCESSING_OUTPUT_DIR, config.TRACKS_DIR]
        ]
        for src, dst in renames:
            if exists(dst):
                raise FileExistsError(
                    f"cannot move {src!r} to {dst!r}, destination exists"
                )
        for src, dst in renames:
            if not exists(src):
                log.debug("Rename user %s: Not moving %s, not found", self.id, src)
            else:
                log.info("Rename user %s: Moving %s to %s", self.id, src, dst)
                os.rename(src, dst)
        self.username = new_name
 class UserDevice(Base):
    __tablename__ = "user_device"
    id = Column(Integer, autoincrement=True, primary_key=True)
    user_id = Column(Integer, ForeignKey("user.id", ondelete="CASCADE"))
    identifier = Column(String, nullable=False)
    display_name = Column(String, nullable=True)
    __table_args__ = (
        Index("user_id_identifier", "user_id", "identifier", unique=True),
    )
    def to_dict(self, for_user_id=None):
        if for_user_id != self.user_id:
            return {}
        return {
            "id": self.id,
            "identifier": self.identifier,
            "displayName": self.display_name,
        }
 class Comment(Base):
    __tablename__ = "comment"
    id = Column(Integer, autoincrement=True, primary_key=True)
    uid = Column(UUID, server_default=func.uuid_generate_v4())
    created_at = Column(DateTime, nullable=False, server_default=NOW)
    updated_at = Column(DateTime, nullable=False, server_default=NOW, onupdate=NOW)
    body = Column(TEXT)
    author_id = Column(Integer, ForeignKey("user.id", ondelete="CASCADE"))
    track_id = Column(Integer, ForeignKey("track.id", ondelete="CASCADE"))
    def to_dict(self, for_user_id=None):
        return {
            "id": self.uid,
            "body": self.body,
            "author": self.author.to_dict(for_user_id=for_user_id),
            "createdAt": self.created_at,
        }
 class Region(Base):
    __tablename__ = "region"
    id = Column(String(24), primary_key=True, index=True)
    name = Column(Text)
    geometry = Column(GeometryGeometry)
    admin_level = Column(Integer, index=True)
    import_group = Column(String)
    __table_args__ = (
        # We keep the index name as osm2pgsql created it, way back when.
        Index(
            "region_geometry_idx",
            "geometry",
            postgresql_using="gist",
            postgresql_with={"fillfactor": 100},
        ),
    )
 Comment.author = relationship("User", back_populates="authored_comments")
 User.authored_comments = relationship(
    "Comment",
    order_by=Comment.created_at,
    back_populates="author",
    passive_deletes=True,
 )
 Track.author = relationship("User", back_populates="authored_tracks")
 User.authored_tracks = relationship(
    "Track", order_by=Track.created_at, back_populates="author", passive_deletes=True
 )
 Comment.track = relationship("Track", back_populates="comments")
 Track.comments = relationship(
    "Comment", order_by=Comment.created_at, back_populates="track", passive_deletes=True
 )
 OvertakingEvent.track = relationship("Track", back_populates="overtaking_events")
 Track.overtaking_events = relationship(
    "OvertakingEvent",
    order_by=OvertakingEvent.time,
    back_populates="track",
    passive_deletes=True,
 )
 Track.user_device = relationship("UserDevice", back_populates="tracks")
 UserDevice.tracks = relationship(
    "Track",
    order_by=Track.created_at,
    back_populates="user_device",
    passive_deletes=False,
 )
 # 0..4 Night, 4..10 Morning, 10..14 Noon, 14..18 Afternoon, 18..22 Evening, 22..00 Night
 # Two hour intervals
 _DAYTIMES = [
    "Night",  # 0h - 2h
    "Night",  # 2h - 4h
    "Morning",  # 4h - 6h
    "Morning",  # 6h - 8h
    "Morning",  # 8h - 10h
    "Noon",  # 10h - 12h
    "Noon",  # 12h - 14h
    "Afternoon",  # 14h - 16h
    "Afternoon",  # 16h - 18h
    "Evening",  # 18h - 20h
    "Evening",  # 20h - 22h
    "Night",  # 22h - 24h
 ]
 def _get_daytime(d):
    return _DAYTIMES[math.floor((d.hour % 24) / 2)]
--- a/api/obs/api/process.py
+++ b/api/obs/api/process.py
@ -1,377 +0,0 @@
 import logging
 import os
 import json
 import asyncio
 import hashlib
 import struct
 import pytz
 from os.path import join
 from datetime import datetime
 from sqlalchemy import delete, func, select, and_
 from sqlalchemy.orm import joinedload
 from obs.face.importer import ImportMeasurementsCsv
 from obs.face.geojson import ExportMeasurements
 from obs.face.annotate import AnnotateMeasurements
 from obs.face.filter import (
    AnonymizationMode,
    ChainFilter,
    ConfirmedFilter,
    DistanceMeasuredFilter,
    PrivacyFilter,
    PrivacyZone,
    PrivacyZonesFilter,
    RequiredFieldsFilter,
 )
 from obs.face.osm import DataSource, DatabaseTileSource
 from obs.api.db import OvertakingEvent, RoadUsage, Track, UserDevice, make_session
 from obs.api.app import app
 log = logging.getLogger(__name__)
 def get_data_source():
    """
    Creates a data source based on the configuration of the portal. In *lean*
    mode, the OverpassTileSource is used to fetch data on demand. In normal
    mode, the roads database is used.
    """
    return DataSource(DatabaseTileSource())
 async def process_tracks_loop(delay):
    while True:
        try:
            async with make_session() as session:
                track = (
                    await session.execute(
                        select(Track)
                        .where(Track.processing_status == "queued")
                        .order_by(Track.processing_queued_at)
                        .options(joinedload(Track.author))
                    )
                ).scalar()
                if track is None:
                    await asyncio.sleep(delay)
                    continue
                data_source = get_data_source()
                await process_track(session, track, data_source)
        except BaseException:
            log.exception("Failed to process track. Will continue.")
            await asyncio.sleep(1)
            continue
 async def process_tracks(tracks):
    """
    Processes the tracks and writes event data to the database.
    :param tracks: A list of strings which
    """
    data_source = get_data_source()
    async with make_session() as session:
        for track_id_or_slug in tracks:
            track = (
                await session.execute(
                    select(Track)
                    .where(
                        Track.id == track_id_or_slug
                        if isinstance(track_id_or_slug, int)
                        else Track.slug == track_id_or_slug
                    )
                    .options(joinedload(Track.author))
                )
            ).scalar()
            if not track:
                raise ValueError(f"Track {track_id_or_slug!r} not found.")
            await process_track(session, track, data_source)
 def to_naive_utc(t):
    if t is None:
        return None
    return t.astimezone(pytz.UTC).replace(tzinfo=None)
 async def export_gpx(track, filename, name):
    import xml.etree.ElementTree as ET
    gpx = ET.Element("gpx")
    metadata = ET.SubElement(gpx, "metadata")
    ET.SubElement(metadata, "name").text = name
    trk = ET.SubElement(gpx, "trk")
    ET.SubElement(trk, "name").text = name
    ET.SubElement(trk, "type").text = "Cycling"
    trkseg = ET.SubElement(trk, "trkseg")
    for point in track:
        trkpt = ET.SubElement(
            trkseg, "trkpt", lat=str(point["latitude"]), lon=str(point["longitude"])
        )
        ET.SubElement(trkpt, "time").text = point["time"].isoformat()
    et = ET.ElementTree(gpx)
    et.write(filename, encoding="utf-8", xml_declaration=True)
 async def process_track(session, track, data_source):
    try:
        track.processing_status = "complete"
        track.processed_at = datetime.utcnow()
        await session.commit()
        original_file_path = track.get_original_file_path(app.config)
        output_dir = join(
            app.config.PROCESSING_OUTPUT_DIR, track.author.username, track.slug
        )
        os.makedirs(output_dir, exist_ok=True)
        log.info("Annotating and filtering CSV file")
        imported_data, statistics, track_metadata = ImportMeasurementsCsv().read(
            original_file_path,
            user_id="dummy",  # TODO: user username or id or nothing?
            dataset_id=Track.slug,  # TODO: use track id or slug or nothing?
            return_metadata=True,
        )
        annotator = AnnotateMeasurements(
            data_source,
            cache_dir=app.config.OBS_FACE_CACHE_DIR,
            fully_annotate_unconfirmed=True,
        )
        input_data = await annotator.annotate(imported_data)
        track_filter = ChainFilter(
            RequiredFieldsFilter(),
            PrivacyFilter(
                user_id_mode=AnonymizationMode.REMOVE,
                measurement_id_mode=AnonymizationMode.REMOVE,
            ),
            # TODO: load user privacy zones and create a PrivacyZonesFilter() from them
        )
        measurements_filter = DistanceMeasuredFilter()
        overtaking_events_filter = ConfirmedFilter()
        track_points = track_filter.filter(input_data, log=log)
        measurements = measurements_filter.filter(track_points, log=log)
        overtaking_events = overtaking_events_filter.filter(measurements, log=log)
        exporter = ExportMeasurements("measurements.dummy")
        await exporter.add_measurements(measurements)
        measurements_json = exporter.get_data()
        del exporter
        exporter = ExportMeasurements("overtaking_events.dummy")
        await exporter.add_measurements(overtaking_events)
        overtaking_events_json = exporter.get_data()
        del exporter
        track_json = {
            "type": "Feature",
            "geometry": {
                "type": "LineString",
                "coordinates": [[m["longitude"], m["latitude"]] for m in track_points],
            },
        }
        track_raw_json = {
            "type": "Feature",
            "geometry": {
                "type": "LineString",
                "coordinates": [
                    [m["longitude_GPS"], m["latitude_GPS"]] for m in track_points
                ],
            },
        }
        for output_filename, data in [
            ("measurements.json", measurements_json),
            ("overtakingEvents.json", overtaking_events_json),
            ("track.json", track_json),
            ("trackRaw.json", track_raw_json),
        ]:
            target = join(output_dir, output_filename)
            log.debug("Writing file %s", target)
            with open(target, "w") as fp:
                json.dump(data, fp, indent=4)
        await export_gpx(track_points, join(output_dir, "track.gpx"), track.slug)
        log.info("Clearing old track data...")
        await clear_track_data(session, track)
        await session.commit()
        device_identifier = track_metadata.get("DeviceId")
        if device_identifier:
            if isinstance(device_identifier, list):
                device_identifier = device_identifier[0]
            log.info("Finding or creating device %s", device_identifier)
            user_device = (
                await session.execute(
                    select(UserDevice).where(
                        and_(
                            UserDevice.user_id == track.author_id,
                            UserDevice.identifier == device_identifier,
                        )
                    )
                )
            ).scalar()
            log.debug("user_device is %s", user_device)
            if not user_device:
                user_device = UserDevice(
                    user_id=track.author_id, identifier=device_identifier
                )
                log.debug("Create new device for this user")
                session.add(user_device)
            track.user_device = user_device
        else:
            log.info("No DeviceId in track metadata.")
        log.info("Import events into database...")
        await import_overtaking_events(session, track, overtaking_events)
        log.info("import road usages...")
        await import_road_usages(session, track, track_points)
        log.info("Write track statistics and update status...")
        track.recorded_at = to_naive_utc(statistics["t_min"])
        track.recorded_until = to_naive_utc(statistics["t_max"])
        track.duration = statistics["t"]
        track.length = statistics["d"]
        track.segments = statistics["n_segments"]
        track.num_events = statistics["n_confirmed"]
        track.num_measurements = statistics["n_measurements"]
        track.num_valid = statistics["n_valid"]
        track.processing_status = "complete"
        track.processed_at = datetime.utcnow()
        await session.commit()
        log.info("Track %s imported.", track.slug)
    except BaseException as e:
        await clear_track_data(session, track)
        track.processing_status = "error"
        track.processing_log = str(e)
        track.processed_at = datetime.utcnow()
        await session.commit()
        raise
 async def clear_track_data(session, track):
    track.recorded_at = None
    track.recorded_until = None
    track.duration = None
    track.length = None
    track.segments = None
    track.num_events = None
    track.num_measurements = None
    track.num_valid = None
    await session.execute(
        delete(OvertakingEvent).where(OvertakingEvent.track_id == track.id)
    )
    await session.execute(delete(RoadUsage).where(RoadUsage.track_id == track.id))
 async def import_overtaking_events(session, track, overtaking_events):
    # We use a dictionary to prevent per-track hash collisions, ignoring all
    # but the first event of the same hash
    event_models = {}
    for m in overtaking_events:
        hex_hash = hashlib.sha256(
            struct.pack(
                "ddQ", m["latitude"], m["longitude"], int(m["time"].timestamp())
            )
        ).hexdigest()
        event_models[hex_hash] = OvertakingEvent(
            track_id=track.id,
            hex_hash=hex_hash,
            way_id=m.get("OSM_way_id"),
            direction_reversed=m.get("OSM_way_orientation", 0) < 0,
            geometry=func.ST_Transform(
                func.ST_GeomFromGeoJSON(
                    json.dumps(
                        {
                            "type": "Point",
                            "coordinates": [m["longitude"], m["latitude"]],
                        }
                    )
                ),
                3857,
            ),
            latitude=m["latitude"],
            longitude=m["longitude"],
            time=m["time"].astimezone(pytz.utc).replace(tzinfo=None),
            distance_overtaker=m["distance_overtaker"],
            distance_stationary=m["distance_stationary"],
            course=m["course"],
            speed=m["speed"],
        )
    session.add_all(event_models.values())
 def get_road_usages(track_points):
    last_key = None
    last = None
    for p in track_points:
        way_id = p.get("OSM_way_id")
        direction_reversed = p.get("OSM_way_orientation", 0) < 0
        key = (way_id, direction_reversed)
        if last_key is None or last_key[0] is None:
            last = p
            last_key = key
            continue
        if last_key != key:
            if last_key[0] is not None:
                yield last
            last_key = key
            last = p
    if last is not None and last_key[0] is not None:
        yield last
 async def import_road_usages(session, track, track_points):
    usages = set()
    for p in get_road_usages(track_points):
        direction_reversed = p.get("OSM_way_orientation", 0) < 0
        way_id = p.get("OSM_way_id")
        time = p["time"]
        hex_hash = hashlib.sha256(
            struct.pack("dQ", way_id, int(time.timestamp()))
        ).hexdigest()
        usages.add(
            RoadUsage(
                track_id=track.id,
                hex_hash=hex_hash,
                way_id=way_id,
                time=time.astimezone(pytz.utc).replace(tzinfo=None),
                direction_reversed=direction_reversed,
            )
        )
    session.add_all(usages)
--- a/api/obs/api/routes/exports.py
+++ b/api/obs/api/routes/exports.py
@ -1,261 +0,0 @@
 import json
 from enum import Enum
 from contextlib import contextmanager
 import zipfile
 import io
 import re
 import math
 from sqlite3 import connect
 import shapefile
 from obs.api.db import OvertakingEvent
 from sqlalchemy import select, func, text
 from sanic.response import raw
 from sanic.exceptions import InvalidUsage
 from obs.api.app import api, json as json_response
 from obs.api.utils import use_request_semaphore
 import logging
 log = logging.getLogger(__name__)
 class ExportFormat(str, Enum):
    SHAPEFILE = "shapefile"
    GEOJSON = "geojson"
 def parse_bounding_box(input_string):
    left, bottom, right, top = map(float, input_string.split(","))
    return func.ST_SetSRID(
        func.ST_MakeBox2D(
            func.ST_Point(left, bottom),
            func.ST_Point(right, top),
        ),
        4326,
    )
 PROJECTION_4326 = (
    'GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],'
    'AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],'
    'UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]]'
 )
@contextmanager
 def shapefile_zip(shape_type=shapefile.POINT, basename="events"):
    zip_buffer = io.BytesIO()
    shp, shx, dbf = (io.BytesIO() for _ in range(3))
    writer = shapefile.Writer(
        shp=shp, shx=shx, dbf=dbf, shapeType=shape_type, encoding="utf8"
    )
    yield writer, zip_buffer
    writer.balance()
    writer.close()
    zip_file = zipfile.ZipFile(zip_buffer, "a", zipfile.ZIP_DEFLATED, False)
    zip_file.writestr(f"{basename}.shp", shp.getbuffer())
    zip_file.writestr(f"{basename}.shx", shx.getbuffer())
    zip_file.writestr(f"{basename}.dbf", dbf.getbuffer())
    zip_file.writestr(f"{basename}.prj", PROJECTION_4326)
    zip_file.close()
@api.get(r"/export/events")
 async def export_events(req):
    async with use_request_semaphore(req, "export_semaphore", timeout=30):
        bbox = req.ctx.get_single_arg("bbox", default="-180,-90,180,90")
        assert re.match(r"(-?\d+\.?\d+,?){4}", bbox)
        bbox = list(map(float, bbox.split(",")))
        fmt = req.ctx.get_single_arg("fmt", convert=ExportFormat)
        events = await req.ctx.db.stream(
            text(
                """
                SELECT
                    ST_AsGeoJSON(ST_Transform(geometry, 4326)) AS geometry,
                    distance_overtaker,
                    distance_stationary,
                    way_id,
                    direction,
                    speed,
                    time_stamp,
                    course,
                    zone
                FROM
                    layer_obs_events(
                        ST_Transform(ST_MakeEnvelope(:bbox0, :bbox1, :bbox2, :bbox3, 4326), 3857),
                        19,
                        NULL,
                        '1900-01-01'::timestamp,
                        '2100-01-01'::timestamp
                    )
                """
            ).bindparams(bbox0=bbox[0], bbox1=bbox[1], bbox2=bbox[2], bbox3=bbox[3])
        )
        if fmt == ExportFormat.SHAPEFILE:
            with shapefile_zip(basename="events") as (writer, zip_buffer):
                writer.field("distance_overtaker", "N", decimal=4)
                writer.field("distance_stationary", "N", decimal=4)
                writer.field("way_id", "N", decimal=0)
                writer.field("direction", "N", decimal=0)
                writer.field("course", "N", decimal=4)
                writer.field("speed", "N", decimal=4)
                writer.field("zone", "C")
                async for event in events:
                    coords = json.loads(event.geometry)["coordinates"]
                    writer.point(*coords)
                    writer.record(
                        distance_overtaker=event.distance_overtaker,
                        distance_stationary=event.distance_stationary,
                        direction=event.direction,
                        way_id=event.way_id,
                        course=event.course,
                        speed=event.speed,
                        zone=event.zone
                        # "time"=event.time,
                    )
            return raw(zip_buffer.getbuffer())
        if fmt == ExportFormat.GEOJSON:
            features = []
            async for event in events:
                geom = json.loads(event.geometry)
                features.append(
                    {
                        "type": "Feature",
                        "geometry": geom,
                        "properties": {
                            "distance_overtaker": event.distance_overtaker
                            if event.distance_overtaker is not None
                            and not math.isnan(event.distance_overtaker)
                            else None,
                            "distance_stationary": event.distance_stationary
                            if event.distance_stationary is not None
                            and not math.isnan(event.distance_stationary)
                            else None,
                            "direction": event.direction
                            if event.direction is not None
                            and not math.isnan(event.direction)
                            else None,
                            "way_id": event.way_id,
                            "course": event.course
                            if event.course is not None and not math.isnan(event.course)
                            else None,
                            "speed": event.speed
                            if event.speed is not None and not math.isnan(event.speed)
                            else None,
                            "time": event.time_stamp,
                            "zone": event.zone,
                        },
                    }
                )
            geojson = {"type": "FeatureCollection", "features": features}
            return json_response(geojson)
        raise InvalidUsage("unknown export format")
@api.get(r"/export/segments")
 async def export_segments(req):
    async with use_request_semaphore(req, "export_semaphore", timeout=30):
        bbox = req.ctx.get_single_arg("bbox", default="-180,-90,180,90")
        assert re.match(r"(-?\d+\.?\d+,?){4}", bbox)
        bbox = list(map(float, bbox.split(",")))
        fmt = req.ctx.get_single_arg("fmt", convert=ExportFormat)
        segments = await req.ctx.db.stream(
            text(
                """
                SELECT
                    ST_AsGeoJSON(ST_Transform(geometry, 4326)) AS geometry,
                    way_id,
                    distance_overtaker_mean,
                    distance_overtaker_min,
                    distance_overtaker_max,
                    distance_overtaker_median,
                    overtaking_event_count,
                    usage_count,
                    direction,
                    zone,
                    offset_direction,
                    distance_overtaker_array
                FROM
                    layer_obs_roads(
                        ST_Transform(ST_MakeEnvelope(:bbox0, :bbox1, :bbox2, :bbox3, 4326), 3857),
                        11,
                        NULL,
                        '1900-01-01'::timestamp,
                        '2100-01-01'::timestamp
                    )
                WHERE usage_count > 0
                """
            ).bindparams(bbox0=bbox[0], bbox1=bbox[1], bbox2=bbox[2], bbox3=bbox[3])
        )
        if fmt == ExportFormat.SHAPEFILE:
            with shapefile_zip(shape_type=3, basename="segments") as (
                writer,
                zip_buffer,
            ):
                writer.field("distance_overtaker_mean", "N", decimal=4)
                writer.field("distance_overtaker_max", "N", decimal=4)
                writer.field("distance_overtaker_min", "N", decimal=4)
                writer.field("distance_overtaker_median", "N", decimal=4)
                writer.field("overtaking_event_count", "N", decimal=4)
                writer.field("usage_count", "N", decimal=4)
                writer.field("way_id", "N", decimal=0)
                writer.field("direction", "N", decimal=0)
                writer.field("zone", "C")
                async for segment in segments:
                    geom = json.loads(segment.st_asgeojson)
                    writer.line([geom["coordinates"]])
                    writer.record(
                        distance_overtaker_mean=segment.distance_overtaker_mean,
                        distance_overtaker_median=segment.distance_overtaker_median,
                        distance_overtaker_max=segment.distance_overtaker_max,
                        distance_overtaker_min=segment.distance_overtaker_min,
                        usage_count=segment.usage_count,
                        overtaking_event_count=segment.overtaking_event_count,
                        direction=segment.direction,
                        way_id=segment.way_id,
                        zone=segment.zone,
                    )
            return raw(zip_buffer.getbuffer())
        if fmt == ExportFormat.GEOJSON:
            features = []
            async for segment in segments:
                features.append(
                    {
                        "type": "Feature",
                        "geometry": json.loads(segment.geometry),
                        "properties": {
                            "distance_overtaker_mean": segment.distance_overtaker_mean,
                            "distance_overtaker_max": segment.distance_overtaker_max,
                            "distance_overtaker_median": segment.distance_overtaker_median,
                            "overtaking_event_count": segment.overtaking_event_count,
                            "usage_count": segment.usage_count,
                            "distance_overtaker_array": segment.distance_overtaker_array,
                            "direction": segment.direction,
                            "way_id": segment.way_id,
                            "zone": segment.zone,
                        },
                    }
                )
            geojson = {"type": "FeatureCollection", "features": features}
            return json_response(geojson)
        raise InvalidUsage("unknown export format")
--- a/api/obs/api/routes/frontend.py
+++ b/api/obs/api/routes/frontend.py
@ -1,57 +0,0 @@
 from os.path import join, exists, isfile, abspath
 import sanic.response as response
 from sanic.exceptions import NotFound
 from obs.api.app import app
 if app.config.FRONTEND_CONFIG:
    @app.get("/config.json")
    def get_frontend_config(req):
        result = {
            "basename": req.ctx.frontend_base_path,
            **req.app.config.FRONTEND_CONFIG,
            "apiUrl": f"{req.ctx.api_url}/api",
            "loginUrl": f"{req.ctx.api_url}/login",
            "obsMapSource": {
                "type": "vector",
                "tiles": [
                    req.ctx.api_url
                    + req.app.url_for("tiles", zoom="000", x="111", y="222.pbf")
                    .replace("000", "{z}")
                    .replace("111", "{x}")
                    .replace("222", "{y}")
                ],
                "minzoom": 0,
                "maxzoom": 14,
            },
        }
        return response.json(result)
 INDEX_HTML = (
    join(app.config.FRONTEND_DIR, "index.html")
    if app.config.get("FRONTEND_DIR")
    else None
 )
 if INDEX_HTML and exists(INDEX_HTML):
    with open(INDEX_HTML, "rt") as f:
        index_file_contents = f.read()
    @app.get("/<path:path>")
    def get_frontend_static(req, path):
        if path.startswith("api/"):
            raise NotFound()
        file = join(app.config.FRONTEND_DIR, path)
        if not abspath(file).startswith(abspath(app.config.FRONTEND_DIR)):
            raise NotFound()
        if not exists(file) or not path or not isfile(file):
            return response.html(
                index_file_contents.replace("__BASE_HREF__", req.ctx.frontend_url + "/")
            )
        return response.file(file)
--- a/api/obs/api/routes/info.py
+++ b/api/obs/api/routes/info.py
@ -1,18 +0,0 @@
 import logging
 from obs.api.app import api
 from sanic.response import json
 log = logging.getLogger(__name__)
 from obs.api import __version__ as version
@api.route("/info")
 async def info(req):
    return json(
        {
            "version": version,
        }
    )
--- a/api/obs/api/routes/login.py
+++ b/api/obs/api/routes/login.py
@ -1,173 +0,0 @@
 import asyncio
 import logging
 import re
 from requests.exceptions import RequestException
 from sqlalchemy import select
 from oic import rndstr
 from oic.oic import Client
 from oic.oic.message import AuthorizationResponse, RegistrationResponse
 from oic.utils.authn.client import CLIENT_AUTHN_METHOD
 from obs.api.app import auth, api
 from obs.api.db import User
 from sanic.response import json, redirect
 log = logging.getLogger(__name__)
 client = Client(client_authn_method=CLIENT_AUTHN_METHOD)
 # Do not show verbose library output, even when the appliaction is in debug mode
 logging.getLogger("oic").setLevel(logging.INFO)
@auth.before_server_start
 async def connect_auth_client(app, loop):
    client.allow["issuer_mismatch"] = True
    try:
        client.provider_config(app.config.KEYCLOAK_URL)
        client.store_registration_info(
            RegistrationResponse(
                client_id=app.config.KEYCLOAK_CLIENT_ID,
                client_secret=app.config.KEYCLOAK_CLIENT_SECRET,
            )
        )
    except RequestException:
        log.exception(f"could not connect to {app.config.KEYCLOAK_URL}")
        log.info("will retry")
        await asyncio.sleep(2)
        log.info("retrying")
        await connect_auth_client(app, loop)
@auth.route("/login")
 async def login(req):
    next_url = req.ctx.get_single_arg("next", default=None)
    session = req.ctx.session
    session["state"] = rndstr()
    session["nonce"] = rndstr()
    session["next"] = next_url
    args = {
        "client_id": client.client_id,
        "response_type": "code",
        "scope": ["openid"],
        "nonce": session["nonce"],
        "redirect_uri": req.ctx.api_url + "/login/redirect",
        "state": session["state"],
    }
    auth_req = client.construct_AuthorizationRequest(request_args=args)
    login_url = auth_req.request(client.authorization_endpoint)
    return redirect(login_url)
@auth.route("/login/redirect")
 async def login_redirect(req):
    session = req.ctx.session
    auth_response = client.parse_response(
        AuthorizationResponse, info=dict(req.query_args), sformat="dict"
    )
    code = auth_response["code"]
    state = auth_response["state"]
    assert "state" in session
    assert state == session["state"]
    client.do_access_token_request(
        state=state,
        request_args={"code": code},
        authn_method="client_secret_basic",
    )
    userinfo = client.do_user_info_request(state=state)
    # {'sub': '3798e2da-b208-4a1a-98c0-08fecfea1345', 'email_verified': True, 'preferred_username': 'test', 'email': 'test@example.com'}
    sub = userinfo["sub"]
    preferred_username = userinfo["preferred_username"]
    email = userinfo.get("email")
    clean_username = re.sub(r"[^a-zA-Z0-9_.-]", "", preferred_username)
    if clean_username != preferred_username:
        log.warning(
            "Username %r contained invalid characters and was changed to %r",
            preferred_username,
            clean_username,
        )
        preferred_username = clean_username
    if email is None:
        raise ValueError(
            "user has no email set, please configure keycloak to require emails"
        )
    user = (await req.ctx.db.execute(select(User).where(User.sub == sub))).scalar()
    if user is None:
        user = (
            await req.ctx.db.execute(
                select(User).where(
                    User.email == email
                    and User.username == preferred_username
                    and User.match_by_username_email
                )
            )
        ).scalar()
        if user:
            log.info(
                "Re-matched existing user %s (sub: %s) based on email and username (%s)",
                user.id,
                user.sub,
                preferred_username,
            )
            user.match_by_username_email = False
            user.sub = sub
    if user is None:
        log.info(
            "Registering new user with sub %r (preferred username: %s)",
            sub,
            preferred_username,
        )
        user = User(sub=sub, username=preferred_username, email=email)
        req.ctx.db.add(user)
    else:
        log.info(
            "Logged in known user (id: %s, sub: %s, %s).",
            user.id,
            user.sub,
            preferred_username,
        )
        if email != user.email:
            log.debug("Updating user (id: %s) email from auth system.", user.id)
            user.email = email
        if preferred_username != user.username:
            log.debug("Updating user (id: %s) username from auth system.", user.id)
            await user.rename(req.app.config, preferred_username)
    await req.ctx.db.commit()
    session["user_id"] = user.id
    next_ = session.pop("next", "/") or "/"
    return redirect(next_)
@api.route("/logout")
 async def logout(req):
    session = req.ctx.session
    if "user_id" in session:
        del session["user_id"]
    auth_req = client.construct_EndSessionRequest(state=session["state"])
    logout_url = auth_req.request(client.end_session_endpoint)
    return redirect(logout_url + f"&post_logout_redirect_uri={req.ctx.api_url}/logout")
--- a/api/obs/api/routes/mapdetails.py
+++ b/api/obs/api/routes/mapdetails.py
@ -1,147 +0,0 @@
 import json
 from functools import partial
 import logging
 import numpy
 import math
 from sqlalchemy import select, func, column
 import sanic.response as response
 from sanic.exceptions import InvalidUsage
 from obs.api.app import api
 from obs.api.db import Road, OvertakingEvent, Track
 from obs.api.utils import round_to
 round_distance = partial(round_to, multiples=0.001)
 round_speed = partial(round_to, multiples=0.1)
 log = logging.getLogger(__name__)
 def get_bearing(b, a):
    # longitude, latitude
    dL = b[0] - a[0]
    X = numpy.cos(b[1]) * numpy.sin(dL)
    Y = numpy.cos(a[1]) * numpy.sin(b[1]) - numpy.sin(a[1]) * numpy.cos(
        b[1]
    ) * numpy.cos(dL)
    return numpy.arctan2(Y, X) + 0.5 * math.pi
 # Bins for histogram on overtaker distances. 0, 0.25, ... 2.25, infinity
 DISTANCE_BINS = numpy.arange(0, 2.5, 0.25).tolist() + [float('inf')]
@api.route("/mapdetails/road", methods=["GET"])
 async def mapdetails_road(req):
    longitude = req.ctx.get_single_arg("longitude", convert=float)
    latitude = req.ctx.get_single_arg("latitude", convert=float)
    radius = req.ctx.get_single_arg("radius", default=100, convert=float)
    if not (1 <= radius <= 1000):
        raise InvalidUsage("`radius` parameter must be between 1 and 1000")
    road_geometry = Road.geometry
    point = func.ST_Transform(
        func.ST_GeomFromGeoJSON(
            json.dumps(
                {
                    "type": "point",
                    "coordinates": [longitude, latitude],
                }
            )
        ),
        3857,
    )
    road = (
        await req.ctx.db.execute(
            select(Road)
            .where(func.ST_DWithin(road_geometry, point, radius))
            .order_by(func.ST_Distance(road_geometry, point))
            .limit(1)
        )
    ).scalar()
    if road is None:
        return response.json({})
    arrays = (
        await req.ctx.db.execute(
            select(
                [
                    OvertakingEvent.distance_overtaker,
                    OvertakingEvent.distance_stationary,
                    OvertakingEvent.speed,
                    # Keep this as the last entry always for numpy.partition
                    # below to work.
                    OvertakingEvent.direction_reversed,
                ]
            ).where(OvertakingEvent.way_id == road.way_id)
        )
    ).all()
    arrays = numpy.array(arrays).T
    if len(arrays) == 0:
        arrays = numpy.array([[], [], [], []], dtype=float)
    data, mask = arrays[:-1], arrays[-1]
    data = data.astype(numpy.float64)
    mask = mask.astype(bool)
    def partition(arr, cond):
        return arr[:, cond], arr[:, ~cond]
    forwards, backwards = partition(data, ~mask)
    def array_stats(arr, rounder, bins=30):
        if len(arr):
            arr = arr[~numpy.isnan(arr)]
        n = len(arr)
        hist, bins = numpy.histogram(arr, bins=bins)
        return {
            "statistics": {
                "count": n,
                "mean": rounder(numpy.mean(arr)) if n else None,
                "min": rounder(numpy.min(arr)) if n else None,
                "max": rounder(numpy.max(arr)) if n else None,
                "median": rounder(numpy.median(arr)) if n else None,
            },
            "histogram": {
                "bins": [None if math.isinf(b) else b for b in bins.tolist()],
                "counts": hist.tolist(),
                "zone": road.zone
            },
            "values": list(map(rounder, arr.tolist())),
        }
    bearing = None
    geom = json.loads(road.geometry)
    if geom["type"] == "LineString":
        coordinates = geom["coordinates"]
        bearing = get_bearing(coordinates[0], coordinates[-1])
        # convert to degrees, as this is more natural to understand for consumers
        bearing = round_to((bearing / math.pi * 180 + 360) % 360, 1)
    def get_direction_stats(direction_arrays, backwards=False):
        return {
            "bearing": ((bearing + 180) % 360 if backwards else bearing)
            if bearing is not None
            else None,
            "distanceOvertaker": array_stats(direction_arrays[0], round_distance, bins=DISTANCE_BINS),
            "distanceStationary": array_stats(direction_arrays[1], round_distance, bins=DISTANCE_BINS),
            "speed": array_stats(direction_arrays[2], round_speed),
        }
    return response.json(
        {
            "road": road.to_dict(),
            "forwards": get_direction_stats(forwards),
            "backwards": get_direction_stats(backwards, True),
        }
    )
--- a/api/obs/api/routes/stats.py
+++ b/api/obs/api/routes/stats.py
@ -1,206 +0,0 @@
 import logging
 from datetime import datetime
 from typing import Optional
 from operator import and_
 from functools import reduce
 from sqlalchemy import distinct, select, func, desc
 from sanic.response import json
 from obs.api.app import api
 from obs.api.db import Track, OvertakingEvent, User, Region, UserDevice
 from obs.api.utils import round_to
 log = logging.getLogger(__name__)
 # round to this number of meters for privacy reasons
 TRACK_LENGTH_ROUNDING = 1000
 # round to this number of seconds for privacy reasons
 TRACK_DURATION_ROUNDING = 120
 # Everything before this date is probably parsed incorrectly
 MINUMUM_RECORDING_DATE = datetime(2010, 1, 1)
@api.route("/stats")
 async def stats(req):
    user = req.ctx.get_single_arg("user", default=None)
    start = req.ctx.get_single_arg("start", default=None, convert=datetime)
    end = req.ctx.get_single_arg("end", default=None, convert=datetime)
    conditions = [
        Track.recorded_at != None,
        Track.recorded_at > MINUMUM_RECORDING_DATE,
    ]
    if start is not None:
        conditions.append(Track.recorded_at >= start)
    if end is not None:
        conditions.append(Track.recorded_at < end)
    # Only the user can look for their own stats, for now
    by_user = (
        user is not None and req.ctx.user is not None and req.ctx.user.id == int(user)
    )
    if by_user:
        conditions.append(Track.author_id == req.ctx.user.id)
    track_condition = reduce(and_, conditions)
    public_track_condition = Track.public and track_condition
    query = (
        select(
            [
                func.count().label("publicTrackCount"),
                func.sum(Track.duration).label("trackDuration"),
                func.sum(Track.length).label("trackLength"),
            ]
        )
        .select_from(Track)
        .where(public_track_condition)
    )
    public_track_count, track_duration, track_length = (
        await req.ctx.db.execute(query)
    ).first()
    # This is required because SQL returns NULL when the input set to a
    # SUM() aggregation is empty.
    track_duration = track_duration or 0
    track_length = track_length or 0
    user_count = (
        1
        if by_user
        else (await req.ctx.db.execute(select(func.count()).select_from(User))).scalar()
    )
    track_count = (
        await req.ctx.db.execute(
            select(func.count()).select_from(Track).where(track_condition)
        )
    ).scalar()
    event_count = (
        await req.ctx.db.execute(
            select(func.count())
            .select_from(OvertakingEvent)
            .join(OvertakingEvent.track)
            .where(track_condition)
        )
    ).scalar()
    device_count = (
        await req.ctx.db.execute(
            select(func.count(distinct(UserDevice.id)))
            .select_from(UserDevice)
            .join(Track.user_device)
            .where(track_condition)
        )
    ).scalar()
    result = {
        "numEvents": event_count,
        "userCount": user_count,
        "trackLength": round_to(track_length or 0, TRACK_LENGTH_ROUNDING),
        "trackDuration": round_to(track_duration or 0, TRACK_DURATION_ROUNDING),
        "publicTrackCount": public_track_count,
        "trackCount": track_count,
        "deviceCount": device_count,
    }
    return json(result)
 #     const trackCount = await Track.find(trackFilter).count();
 #
 #     const publicTrackCount = await Track.find({
 #       ...trackFilter,
 #       public: true,
 #     }).count();
 #
 #     const userCount = await User.find({
 #       ...(userFilter
 #         ? { _id: userFilter }
 #         : {
 #             createdAt: dateFilter,
 #           }),
 #     }).count();
 #
 #     const trackStats = await Track.aggregate([
 #       { $match: trackFilter },
 #       {
 #         $addFields: {
 #           trackLength: {
 #             $cond: [{ $lt: ['$statistics.length', 500000] }, '$statistics.length', 0],
 #           },
 #           numEvents: '$statistics.numEvents',
 #           trackDuration: {
 #             $cond: [
 #               { $and: ['$statistics.recordedUntil', { $gt: ['$statistics.recordedAt', new Date('2010-01-01')] }] },
 #               { $subtract: ['$statistics.recordedUntil', '$statistics.recordedAt'] },
 #               0,
 #             ],
 #           },
 #         },
 #       },
 #       { $project: { trackLength: true, numEvents: true, trackDuration: true } },
 #       {
 #         $group: {
 #           _id: 'sum',
 #           trackLength: { $sum: '$trackLength' },
 #           numEvents: { $sum: '$numEvents' },
 #           trackDuration: { $sum: '$trackDuration' },
 #         },
 #       },
 #     ]);
 #
 #     const [trackLength, numEvents, trackDuration] =
 #       trackStats.length > 0
 #         ? [trackStats[0].trackLength, trackStats[0].numEvents, trackStats[0].trackDuration]
 #         : [0, 0, 0];
 #
 #     const trackLengthPrivatized = Math.floor(trackLength / TRACK_LENGTH_ROUNDING) * TRACK_LENGTH_ROUNDING;
 #     const trackDurationPrivatized =
 #       Math.round(trackDuration / 1000 / TRACK_DURATION_ROUNDING) * TRACK_DURATION_ROUNDING;
 #
 #     return res.json({
 #       publicTrackCount,
 #       trackLength: trackLengthPrivatized,
 #       trackDuration: trackDurationPrivatized,
 #       numEvents,
 #       trackCount,
 #       userCount,
 #     });
 #   }),
 # );
@api.route("/stats/regions")
 async def stats(req):
    query = (
        select(
            [
                Region.id,
                Region.name,
                func.count(OvertakingEvent.id).label("overtaking_event_count"),
            ]
        )
        .select_from(Region)
        .join(
            OvertakingEvent,
            func.ST_Within(OvertakingEvent.geometry, Region.geometry),
        )
        .group_by(
            Region.id,
            Region.name,
            Region.geometry,
        )
        .having(func.count(OvertakingEvent.id) > 0)
        .order_by(desc("overtaking_event_count"))
    )
    regions = list(map(dict, (await req.ctx.db.execute(query)).all()))
    return json(regions)
--- a/api/obs/api/routes/tiles.py
+++ b/api/obs/api/routes/tiles.py
@ -1,129 +0,0 @@
 from gzip import decompress
 from sqlite3 import connect
 from datetime import datetime, time, timedelta
 from typing import Optional, Tuple
 import dateutil.parser
 from sanic.exceptions import Forbidden, InvalidUsage
 from sanic.response import raw
 from sqlalchemy import text
 from obs.api.app import app
 from obs.api.utils import use_request_semaphore
 def get_tile(filename, zoom, x, y):
    """
    Inspired by:
    https://github.com/TileStache/TileStache/blob/master/TileStache/MBTiles.py
    """
    db = connect(filename)
    db.text_factory = bytes
    fmt = db.execute("SELECT value FROM metadata WHERE name='format'").fetchone()[0]
    if fmt != b"pbf":
        raise ValueError("mbtiles file is in wrong format: %s" % fmt)
    content = db.execute(
        "SELECT tile_data FROM tiles WHERE zoom_level=? AND tile_column=? AND tile_row=?",
        (zoom, x, (2**zoom - 1) - y),
    ).fetchone()
    return content and content[0] or None
 def round_date(date, to="weeks", up=False):
    if to != "weeks":
        raise ValueError(f"cannot round to {to}")
    midnight = time(0, 0, 0, 0)
    start_of_day = date.date()  # ignore time
    weekday = date.weekday()
    is_rounded = date.time() == midnight and weekday == 0
    if is_rounded:
        return date
    if up:
        return datetime.combine(start_of_day + timedelta(days=7 - weekday), midnight)
    else:
        return datetime.combine(start_of_day - timedelta(days=weekday), midnight)
 # regenerate approx. once each day
 TILE_CACHE_MAX_AGE = 3600 * 24
 def get_filter_options(
    req,
 ) -> Tuple[Optional[str], Optional[datetime], Optional[datetime]]:
    """
    Returns parsed, validated and normalized options for filtering map data, a
    tuple of
        * user_id (str|None)
        * start (datetime|None)
        * end (datetime|None)
    """
    user_id = req.ctx.get_single_arg("user", default=None, convert=int)
    if user_id is not None and (req.ctx.user is None or req.ctx.user.id != user_id):
        raise Forbidden()
    parse_date = lambda s: dateutil.parser.parse(s)
    start = req.ctx.get_single_arg("start", default=None, convert=parse_date)
    end = req.ctx.get_single_arg("end", default=None, convert=parse_date)
    start = round_date(start, to="weeks", up=False) if start else None
    end = round_date(end, to="weeks", up=True) if end else None
    if start is not None and end is not None and start >= end:
        raise InvalidUsage(
            "end date must be later than start date (note: dates are rounded to weeks)"
        )
    return user_id, start, end
@app.route(r"/tiles/<zoom:int>/<x:int>/<y:(\d+)\.pbf>")
 async def tiles(req, zoom: int, x: int, y: str):
    async with use_request_semaphore(req, "tile_semaphore"):
        if app.config.get("TILES_FILE"):
            tile = get_tile(req.app.config.TILES_FILE, int(zoom), int(x), int(y))
        else:
            user_id, start, end = get_filter_options(req)
            tile = await req.ctx.db.scalar(
                text(
                    "select data from getmvt(:zoom, :x, :y, :user_id, :min_time, :max_time) as b(data, key);"
                ).bindparams(
                    zoom=int(zoom),
                    x=int(x),
                    y=int(y),
                    user_id=user_id,
                    min_time=start,
                    max_time=end,
                )
            )
    gzip = "gzip" in req.headers["accept-encoding"]
    headers = {}
    headers["Vary"] = "Accept-Encoding"
    if req.app.config.DEBUG:
        headers["Cache-Control"] = "no-cache"
    else:
        headers["Cache-Control"] = f"public, max-age={TILE_CACHE_MAX_AGE}"
    # The tiles in the mbtiles file are gzip-compressed already, so we
    # serve them actually as-is, and only decompress them if the browser
    # doesn't accept gzip
    if gzip:
        headers["Content-Encoding"] = "gzip"
    if not gzip:
        tile = decompress(tile)
    return raw(tile, content_type="application/x-protobuf", headers=headers)
--- a/api/obs/api/routes/tracks.py
+++ b/api/obs/api/routes/tracks.py
@ -1,473 +0,0 @@
 import logging
 import re
 from datetime import date
 from json import load as jsonload
 from os.path import join, exists, isfile
 from sanic.exceptions import InvalidUsage, NotFound, Forbidden
 from sanic.response import file_stream, empty
 from slugify import slugify
 from sqlalchemy import select, func, and_
 from sqlalchemy.orm import joinedload
 from obs.api.app import api, require_auth, read_api_key, json
 from obs.api.db import Track, Comment, DuplicateTrackFileError
 from obs.api.utils import tar_of_tracks
 log = logging.getLogger(__name__)
 def normalize_user_agent(user_agent):
    if not user_agent:
        return None
    m = re.match(r"\bOBS\/[^\s]+", user_agent)
    return m[0] if m else None
 async def _return_tracks(req, extend_query, limit, offset, order_by=None):
    if limit <= 0 or limit > 1000:
        raise InvalidUsage("invalid limit")
    if offset < 0:
        raise InvalidUsage("offset must be positive")
    count_query = extend_query(
        select(func.count()).select_from(Track).join(Track.author)
    )
    track_count = await req.ctx.db.scalar(count_query)
    query = (
        extend_query(select(Track).options(joinedload(Track.author)))
        .limit(limit)
        .offset(offset)
        .order_by(order_by if order_by is not None else Track.created_at)
    )
    tracks = (await req.ctx.db.execute(query)).scalars()
    return json(
        {
            "trackCount": track_count,
            "tracks": list(
                map(
                    lambda t: t.to_dict(
                        for_user_id=req.ctx.user.id if req.ctx.user else None
                    ),
                    tracks,
                )
            ),
        },
    )
@api.get("/tracks")
 async def get_tracks(req):
    limit = req.ctx.get_single_arg("limit", default=20, convert=int)
    offset = req.ctx.get_single_arg("offset", default=0, convert=int)
    # author = req.ctx.get_single_arg("author", default=None, convert=int)
    def extend_query(q):
        q = q.where(Track.public)
        # if author is not None:
        #     q = q.where(Track.author_id == author)
        return q
    return await _return_tracks(req, extend_query, limit, offset)
 def parse_boolean(s):
    if s is None:
        return None
    s = s.lower()
    if s in ("true", "1", "yes", "y", "t"):
        return True
    if s in ("false", "0", "no", "n", "f"):
        return False
    raise ValueError("invalid value for boolean")
@api.get("/tracks/feed")
@require_auth
 async def get_feed(req):
    limit = req.ctx.get_single_arg("limit", default=20, convert=int)
    offset = req.ctx.get_single_arg("offset", default=0, convert=int)
    user_device_id = req.ctx.get_single_arg("user_device_id", default=None, convert=int)
    order_by_columns = {
        "recordedAt": Track.recorded_at,
        "title": Track.title,
        "visibility": Track.public,
        "length": Track.length,
        "duration": Track.duration,
        "user_device_id": Track.user_device_id,
    }
    order_by = req.ctx.get_single_arg(
        "order_by", default=None, convert=order_by_columns.get
    )
    reversed_ = req.ctx.get_single_arg("reversed", convert=parse_boolean, default=False)
    if reversed_:
        order_by = order_by.desc()
    public = req.ctx.get_single_arg("public", convert=parse_boolean, default=None)
    def extend_query(q):
        q = q.where(Track.author_id == req.ctx.user.id)
        if user_device_id is not None:
            q = q.where(Track.user_device_id == user_device_id)
        if public is not None:
            q = q.where(Track.public == public)
        return q
    return await _return_tracks(req, extend_query, limit, offset, order_by)
@api.post("/tracks/bulk")
@require_auth
 async def tracks_bulk_action(req):
    body = req.json
    action = body["action"]
    track_slugs = body["tracks"]
    if action not in ("delete", "makePublic", "makePrivate", "reprocess", "download"):
        raise InvalidUsage("invalid action")
    query = select(Track).where(
        and_(Track.author_id == req.ctx.user.id, Track.slug.in_(track_slugs))
    )
    files = set()
    for track in (await req.ctx.db.execute(query)).scalars():
        if action == "delete":
            await req.ctx.db.delete(track)
        elif action == "makePublic":
            if not track.public:
                track.queue_processing()
            track.public = True
        elif action == "makePrivate":
            if track.public:
                track.queue_processing()
            track.public = False
        elif action == "reprocess":
            track.queue_processing()
        elif action == "download":
            files.add(track.get_original_file_path(req.app.config))
    await req.ctx.db.commit()
    if action == "download":
        username_slug = slugify(req.ctx.user.username, separator="-")
        date_str = date.today().isoformat()
        file_basename = f"tracks_{username_slug}_{date_str}"
        await tar_of_tracks(req, files, file_basename)
        return
    return empty()
@api.post("/tracks")
@read_api_key
@require_auth
 async def post_track(req):
    try:
        file = req.files["body"][0]
    except LookupError as e:
        raise InvalidUsage(
            'Track upload needs a single file in "body" multipart field'
        ) from e
    try:
        body = req.json["track"]
    except (LookupError, InvalidUsage):
        body = {}
    title = body.get("title")
    public = body.get("public")
    track = Track(
        title=title,
        customized_title=bool(title),
        author=req.ctx.user,
        public=public
        if public is not None
        else req.ctx.user.are_tracks_visible_for_all,
    )
    track.generate_slug()
    try:
        await track.prevent_duplicates(req.ctx.db, file.body)
    except DuplicateTrackFileError:
        raise InvalidUsage("Track file is not unique")
    track.uploaded_by_user_agent = normalize_user_agent(req.headers["user-agent"])
    track.original_file_name = file.name
    await track.write_to_original_file(req.app.config, file.body)
    track.queue_processing()
    track.auto_generate_title()
    req.ctx.db.add(track)
    await req.ctx.db.commit()
    return await get_track(req, track.slug)
 async def _load_track(req, slug, raise_not_found=True):
    track = (
        await req.ctx.db.execute(
            select(Track)
            .where(Track.slug == slug)
            .options(joinedload(Track.author))
            .limit(1)
        )
    ).scalar()
    if raise_not_found and track is None:
        raise NotFound()
    if not track.is_visible_to(req.ctx.user):
        raise Forbidden()
    return track
@api.get("/tracks/<slug:str>")
 async def get_track(req, slug: str):
    track = await _load_track(req, slug)
    return json(
        {"track": track.to_dict(for_user_id=req.ctx.user.id if req.ctx.user else None)},
    )
@api.delete("/tracks/<slug:str>")
@require_auth
 async def delete_track(req, slug: str):
    track = await _load_track(req, slug)
    if not track.is_visible_to_private(req.ctx.user):
        raise Forbidden()
    await req.ctx.db.delete(track)
    await req.ctx.db.commit()
    return empty()
@api.get("/tracks/<slug:str>/data")
 async def get_track_data(req, slug: str):
    track = await _load_track(req, slug)
    FILE_BY_KEY = {
        "measurements": "measurements.json",
        "overtakingEvents": "overtakingEvents.json",
        "track": "track.json",
        "trackRaw": "trackRaw.json",
    }
    result = {}
    for key, filename in FILE_BY_KEY.items():
        file_path = join(
            req.app.config.PROCESSING_OUTPUT_DIR, track.file_path, filename
        )
        if not exists(file_path) or not isfile(file_path):
            continue
        with open(file_path) as f:
            result[key] = jsonload(f)
    return json(
        result,
    )
@api.get("/tracks/<slug:str>/download/original.csv")
 async def download_original_file(req, slug: str):
    track = await _load_track(req, slug)
    if not track.is_visible_to_private(req.ctx.user):
        raise Forbidden()
    return await file_stream(
        track.get_original_file_path(req.app.config),
        mime_type="text/csv",
        filename=f"{slug}.csv",
    )
@api.get("/tracks/<slug:str>/download/track.gpx")
 async def download_track_gpx(req, slug: str):
    track = await _load_track(req, slug)
    if not track.is_visible_to(req.ctx.user):
        raise Forbidden()
    file_path = join(req.app.config.PROCESSING_OUTPUT_DIR, track.file_path, "track.gpx")
    if not exists(file_path) or not isfile(file_path):
        raise NotFound()
    return await file_stream(
        file_path,
        mime_type="application/gpx+xml",
        filename=f"{slug}.gpx",
    )
@api.put("/tracks/<slug:str>")
@require_auth
 async def put_track(req, slug: str):
    track = await _load_track(req, slug)
    if track.author_id != req.ctx.user.id:
        raise Forbidden()
    try:
        body = req.json["track"]
    except BaseException:
        body = {}
    if "title" in body:
        track.title = (body["title"] or "").strip() or None
        track.customized_title = track.title is not None
    if "description" in body:
        track.description = (body["description"] or "").strip() or None
    process = False
    if "public" in body:
        public = bool(body["public"])
        process = process or (public != track.public)  # if changed
        track.public = public
    if "body" in req.files:
        try:
            file = req.files["body"][0]
        except LookupError as e:
            raise InvalidUsage(
                'Track upload needs a single file in "body" multipart field'
            ) from e
        await track.prevent_duplicates(req.ctx.db, file.body)
        track.uploaded_by_user_agent = normalize_user_agent(req.headers["user-agent"])
        track.original_file_name = file.name or (track.slug + ".csv")
        await track.write_to_original_file(req.app.config, file.body)
        process = True
    if process:
        track.queue_processing()
    track.auto_generate_title()
    await req.ctx.db.commit()
    track = await _load_track(req, track.slug)
    return json(
        {"track": track.to_dict(for_user_id=req.ctx.user.id)},
    )
@api.get("/tracks/<slug:str>/comments")
 async def get_track_comments(req, slug: str):
    limit = req.ctx.get_single_arg("limit", default=20, convert=int)
    offset = req.ctx.get_single_arg("offset", default=0, convert=int)
    track = await _load_track(req, slug)
    comment_count = await req.ctx.db.scalar(
        select(func.count()).select_from(Comment).where(Comment.track_id == track.id)
    )
    query = (
        select(Comment)
        .options(joinedload(Comment.author))
        .where(Comment.track_id == track.id)
        .order_by(Comment.created_at.desc())
        .limit(limit)
        .offset(offset)
    )
    comments = (await req.ctx.db.execute(query)).scalars()
    return json(
        {
            "commentCount": comment_count,
            "comments": list(
                map(
                    lambda c: c.to_dict(
                        for_user_id=req.ctx.user.id if req.ctx.user else None
                    ),
                    comments,
                )
            ),
        },
    )
@api.post("/tracks/<slug:str>/comments")
@require_auth
 async def post_track_comment(req, slug: str):
    track = await _load_track(req, slug)
    body = req.json.get("comment", {}).get("body")
    if not isinstance(body, str):
        raise InvalidUsage("no comment given")
    # Ensure body is not empty
    body = body.strip()
    if not body:
        raise InvalidUsage("empty comment")
    comment = Comment(
        body=body,
        track_id=track.id,
        author_id=req.ctx.user.id,
    )
    req.ctx.db.add(comment)
    await req.ctx.db.commit()
    await req.ctx.db.refresh(comment)
    comment = (
        await req.ctx.db.execute(
            select(Comment)
            .options(joinedload(Comment.author))
            .where(Comment.id == comment.id)
            .limit(1)
        )
    ).scalar()
    return json({"comment": comment.to_dict(for_user_id=req.ctx.user.id)})
@api.delete("/tracks/<slug:str>/comments/<uid:str>")
@require_auth
 async def delete_track_comment(req, slug: str, uid: str):
    track = await _load_track(req, slug)
    comment = (
        await req.ctx.db.execute(
            select(Comment)
            .options(joinedload(Comment.author))
            .where(Comment.track_id == track.id and Comment.uid == uid)
            .limit(1)
        )
    ).scalar()
    if not comment:
        raise NotFound()
    if comment.author_id != req.ctx.user.id:
        raise Forbidden()
    await req.ctx.db.delete(comment)
    await req.ctx.db.commit()
    return empty()
--- a/api/obs/api/routes/users.py
+++ b/api/obs/api/routes/users.py
@ -1,95 +0,0 @@
 import logging
 from sanic.response import json
 from sanic.exceptions import InvalidUsage, Forbidden, NotFound
 from sqlalchemy import and_, select
 from obs.api.app import api, require_auth
 from obs.api.db import UserDevice
 log = logging.getLogger(__name__)
 from obs.api import __version__ as version
 def user_to_json(user):
    return {
        "id": user.id,
        "username": user.username,
        "displayName": user.display_name,
        "email": user.email,
        "bio": user.bio,
        "image": user.image,
        "areTracksVisibleForAll": user.are_tracks_visible_for_all,
        "apiKey": user.api_key,
    }
@api.get("/user")
 async def get_user(req):
    return json(user_to_json(req.ctx.user) if req.ctx.user else None)
@api.get("/user/devices")
 async def get_user_devices(req):
    if not req.ctx.user:
        raise Forbidden()
    query = (
        select(UserDevice)
        .where(UserDevice.user_id == req.ctx.user.id)
        .order_by(UserDevice.id)
    )
    devices = (await req.ctx.db.execute(query)).scalars()
    return json([device.to_dict(req.ctx.user.id) for device in devices])
@api.put("/user/devices/<device_id:int>")
 async def put_user_device(req, device_id):
    if not req.ctx.user:
        raise Forbidden()
    body = req.json
    query = (
        select(UserDevice)
        .where(and_(UserDevice.user_id == req.ctx.user.id, UserDevice.id == device_id))
        .limit(1)
    )
    device = (await req.ctx.db.execute(query)).scalar()
    if device is None:
        raise NotFound()
    new_name = body.get("displayName", "").strip()
    if new_name and device.display_name != new_name:
        device.display_name = new_name
        await req.ctx.db.commit()
    return json(device.to_dict())
@api.put("/user")
@require_auth
 async def put_user(req):
    user = req.ctx.user
    data = req.json
    for key in ["email", "bio", "image"]:
        if key in data and isinstance(data[key], (str, type(None))):
            setattr(user, key, data[key])
    if "displayName" in data:
        user.display_name = data["displayName"] or None
    if "areTracksVisibleForAll" in data:
        user.are_tracks_visible_for_all = bool(data["areTracksVisibleForAll"])
    if data.get("updateApiKey"):
        user.generate_api_key()
    await req.ctx.db.commit()
    return json(user_to_json(req.ctx.user))
--- a/api/obs/api/utils.py
+++ b/api/obs/api/utils.py
@ -1,162 +0,0 @@
 import asyncio
 from contextlib import asynccontextmanager
 from datetime import datetime
 import logging
 from os.path import commonpath, join, relpath
 import queue
 import tarfile
 import dateutil.parser
 from sanic.exceptions import InvalidUsage, ServiceUnavailable
 log = logging.getLogger(__name__)
 RAISE = object()
 def get_single_arg(req, name, default=RAISE, convert=None):
    try:
        value = req.args[name][0]
    except LookupError as e:
        if default is RAISE:
            raise InvalidUsage(f"missing `{name}`") from e
        value = default
    if convert is not None and value is not None:
        if convert is datetime or convert in ("date", "datetime"):
            convert = lambda s: dateutil.parser.parse(s)
        try:
            value = convert(value)
        except (ValueError, TypeError) as e:
            raise InvalidUsage(f"invalid `{name}`: {str(e)}") from e
    return value
 def round_to(value: float, multiples: float) -> float:
    if value is None:
        return None
    return round(value / multiples) * multiples
 def chunk_list(lst, n):
    for s in range(0, len(lst), n):
        yield lst[s : s + n]
 class chunk:
    def __init__(self, iterable, n):
        self.iterable = iterable
        self.n = n
    def __iter__(self):
        if isinstance(self.iterable, list):
            yield from chunk_list(self.iterable, self.n)
            return
        it = iter(self.iterable)
        while True:
            current = []
            try:
                for _ in range(self.n):
                    current.append(next(it))
                yield current
            except StopIteration:
                if current:
                    yield current
                break
    async def __aiter__(self):
        if hasattr(self.iterable, "__iter__"):
            for item in self:
                yield item
            return
        it = self.iterable.__aiter__()
        while True:
            current = []
            try:
                for _ in range(self.n):
                    current.append(await it.__anext__())
                yield current
            except StopAsyncIteration:
                if len(current):
                    yield current
                break
 async def tar_of_tracks(req, files, file_basename="tracks"):
    response = await req.respond(
        content_type="application/x-gtar",
        headers={
            "content-disposition": f'attachment; filename="{file_basename}.tar.bz2"'
        },
    )
    helper = StreamerHelper(response)
    tar = tarfile.open(name=None, fileobj=helper, mode="w|bz2", bufsize=256 * 512)
    root = commonpath(list(files))
    for fname in files:
        log.info("Write file to tar: %s", fname)
        with open(fname, "rb") as fobj:
            tarinfo = tar.gettarinfo(fname)
            tarinfo.name = join(file_basename, relpath(fname, root))
            tar.addfile(tarinfo, fobj)
            await helper.send_all()
    tar.close()
    await helper.send_all()
    await response.eof()
 class StreamerHelper:
    def __init__(self, response):
        self.response = response
        self.towrite = queue.Queue()
    def write(self, data):
        self.towrite.put(data)
    async def send_all(self):
        while True:
            try:
                tosend = self.towrite.get(block=False)
                await self.response.send(tosend)
            except queue.Empty:
                break
@asynccontextmanager
 async def use_request_semaphore(req, semaphore_name, timeout=10):
    """
    If configured, acquire a semaphore for the map tile request and release it
    after the context has finished.
    If the semaphore cannot be acquired within the timeout, issue a 503 Service
    Unavailable error response that describes that the database is overloaded,
    so users know what the problem is.
    Operates as a noop when the tile semaphore is not enabled.
    """
    semaphore = getattr(req.app.ctx, semaphore_name, None)
    if semaphore is None:
        yield
        return
    try:
        await asyncio.wait_for(semaphore.acquire(), timeout)
        try:
            yield
        finally:
            semaphore.release()
    except asyncio.TimeoutError:
        raise ServiceUnavailable(
            "Too many requests, database overloaded. Please retry later."
        )
--- a/api/obs/bin/init.py
+++ b/api/obs/bin/init.py
@ -1 +0,0 @@
 __path__ = __import__("pkgutil").extend_path(__path__, __name__)
--- a/api/obs/bin/openbikesensor_api.py
+++ b/api/obs/bin/openbikesensor_api.py
@ -1,66 +0,0 @@
 #!/usr/bin/env python3
 import math
 import sys
 import os
 import argparse
 import asyncio
 import logging
 import coloredlogs
 from obs.api.app import app
 from obs.api.db import connect_db
 log = logging.getLogger(__name__)
 def format_size(n, b=1024):
    if n == 0:
        return "0 B"
    if n < 0:
        return "-" + format_size(n, b)
    e = math.floor(math.log(n, b))
    prefixes = ["", "Ki", "Mi", "Gi", "Ti"] if b == 1024 else ["", "K", "M", "G", "T"]
    e = min(e, len(prefixes) - 1)
    r = n / b**e
    s = f"{r:0.2f}" if e > 0 else str(n)
    return f"{s} {prefixes[e]}B"
 class AccessLogFilter(logging.Filter):
    def filter(self, record):
        if not record.msg:
            record.msg = (
                f"{record.request} - {record.status} ({format_size(record.byte)})"
            )
        return True
 def main():
    debug = app.config.DEBUG
    coloredlogs.install(
        level=logging.DEBUG if app.config.get("VERBOSE", debug) else logging.INFO,
        milliseconds=True,
        isatty=True,
    )
    for ln in ["sanic.root", "sanic.error", "sanic.access"]:
        l = logging.getLogger(ln)
        for h in list(l.handlers):
            l.removeHandler(h)
    logging.getLogger("sanic.access").addFilter(AccessLogFilter())
    app.run(
        host=app.config.HOST,
        port=app.config.PORT,
        debug=debug,
        auto_reload=app.config.get("AUTO_RELOAD", debug),
        access_log=True,
    )
 if __name__ == "__main__":
    main()
--- a/api/obs/bin/openbikesensor_transform_osm.py
+++ b/api/obs/bin/openbikesensor_transform_osm.py
@ -1,191 +0,0 @@
 #!/usr/bin/env python3
 import sys
 import re
 import msgpack
 import osmium
 import shapely.wkb as wkb
 from shapely.ops import transform
 HIGHWAY_TYPES = {
    "trunk",
    "primary",
    "secondary",
    "tertiary",
    "unclassified",
    "residential",
    "trunk_link",
    "primary_link",
    "secondary_link",
    "tertiary_link",
    "living_street",
    "service",
    "track",
    "road",
 }
 ZONE_TYPES = {
    "urban",
    "rural",
    "motorway",
 }
 URBAN_TYPES = {
    "residential",
    "living_street",
    "road",
 }
 MOTORWAY_TYPES = {
    "motorway",
    "motorway_link",
 }
 ADMIN_LEVEL_MIN = 2
 ADMIN_LEVEL_MAX = 8
 MINSPEED_RURAL = 60
 ONEWAY_YES = {"yes", "true", "1"}
 ONEWAY_REVERSE = {"reverse", "-1"}
 def parse_number(tag):
    if not tag:
        return None
    match = re.search(r"[0-9]+", tag)
    if not match:
        return None
    digits = match.group(0)
    try:
        return int(digits)
    except ValueError:
        return None
 def determine_zone(tags):
    highway = tags.get("highway")
    zone = tags.get("zone:traffic")
    if zone is not None:
        if "rural" in zone:
            return "rural"
        if "motorway" in zone:
            return "motorway"
        return "urban"
    # From here on we are guessing based on other tags
    if highway in URBAN_TYPES:
        return "urban"
    if highway in MOTORWAY_TYPES:
        return "motorway"
    maxspeed_source = tags.get("source:maxspeed")
    if maxspeed_source and "rural" in maxspeed_source:
        return "rural"
    if maxspeed_source and "urban" in maxspeed_source:
        return "urban"
    for key in ["maxspeed", "maxspeed:forward", "maxspeed:backward"]:
        maxspeed = parse_number(tags.get(key))
        if maxspeed is not None and maxspeed > MINSPEED_RURAL:
            return "rural"
    # default to urban if we have no idea
    return "urban"
 def determine_direction(tags, zone):
    if (
        tags.get("oneway") in ONEWAY_YES
        or tags.get("junction") == "roundabout"
        or zone == "motorway"
    ):
        return 1, True
    if tags.get("oneway") in ONEWAY_REVERSE:
        return -1, True
    return 0, False
 class StreamPacker:
    def __init__(self, stream, *args, **kwargs):
        self.stream = stream
        self.packer = msgpack.Packer(*args, autoreset=False, **kwargs)
    def _write_out(self):
        if hasattr(self.packer, "getbuffer"):
            chunk = self.packer.getbuffer()
        else:
            chunk = self.packer.bytes()
        self.stream.write(chunk)
        self.packer.reset()
    def pack(self, *args, **kwargs):
        self.packer.pack(*args, **kwargs)
        self._write_out()
    def pack_array_header(self, *args, **kwargs):
        self.packer.pack_array_header(*args, **kwargs)
        self._write_out()
    def pack_map_header(self, *args, **kwargs):
        self.packer.pack_map_header(*args, **kwargs)
        self._write_out()
    def pack_map_pairs(self, *args, **kwargs):
        self.packer.pack_map_pairs(*args, **kwargs)
        self._write_out()
 # A global factory that creates WKB from a osmium geometry
 wkbfab = osmium.geom.WKBFactory()
 from pyproj import Transformer
 project = Transformer.from_crs("EPSG:4326", "EPSG:3857", always_xy=True).transform
 class OSMHandler(osmium.SimpleHandler):
    def __init__(self, packer):
        self.packer = packer
        super().__init__()
    def way(self, way):
        tags = way.tags
        highway = tags.get("highway")
        if not highway or highway not in HIGHWAY_TYPES:
            return
        access = tags.get("access", None)
        bicycle = tags.get("bicycle", None)
        if access == "no" and bicycle not in ["designated", "yes", "permissive", "destination"]:
            return
        zone = determine_zone(tags)
        directionality, oneway = determine_direction(tags, zone)
        name = tags.get("name")
        geometry = wkb.loads(wkbfab.create_linestring(way), hex=True)
        geometry = transform(project, geometry)
        geometry = wkb.dumps(geometry)
        self.packer.pack(
            [b"\x01", way.id, name, zone, directionality, oneway, geometry]
        )
 def main():
    with open(sys.argv[2], "wb") as fout:
        packer = StreamPacker(fout)
        osmhandler = OSMHandler(packer)
        osmhandler.apply_file(sys.argv[1], locations=True)
 if __name__ == "__main__":
    main()
--- a/api/package-lock.json
+++ b/api/package-lock.json
--- a/api/package.json
+++ b/api/package.json
@ -0,0 +1,116 @@
  {
  "name": "open-bike-sensor-web-api",
  "version": "1.0.0",
  "description": "Backend API for the OpenBikeSensor web app",
  "main": "app.js",
  "scripts": {
    "mongo:start": "docker run --name realworld-mongo -p 27017:27017 mongo & sleep 5",
    "start": "node src/",
    "start:worker": "node src/worker.js",
    "dev": "nodemon src/",
    "dev:worker": "nodemon -w src/ src/worker.js",
    "mongo:stop": "docker stop realworld-mongo && docker rm realworld-mongo",
    "autoformat": "eslint --fix .",
    "lint": "eslint .",
    "test": "jest",
    "migrate": "mongoose-data-migrate -c .migrations.js",
    "migrate:up": "npm run migrate -- up",
    "migrate:down": "npm run migrate -- down"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/openbikesensor/obsAPI.git"
  },
  "license": "LGPLv3",
  "dependencies": {
    "body-parser": "1.19.0",
    "bull": "^3.22.0",
    "connect-busboy": "0.0.2",
    "cors": "2.8.5",
    "csv-parse": "^4.15.1",
    "csv-stringify": "^5.6.1",
    "ejs": "^3.1.6",
    "errorhandler": "1.5.1",
    "express": "4.17.1",
    "express-jwt": "^6.0.0",
    "express-session": "1.17.1",
    "jest": "^26.6.3",
    "joi": "^17.4.0",
    "jsonwebtoken": "8.5.1",
    "luxon": "^1.26.0",
    "method-override": "3.0.0",
    "methods": "1.1.2",
    "mongoose": "^6.0.5",
    "mongoose-data-migrate": "flashstockinc/mongoose-data-migrate",
    "mongoose-unique-validator": "2.0.3",
    "morgan": "1.10.0",
    "nodemailer": "^6.4.18",
    "oauth2orize": "^1.11.0",
    "passport": "0.4.1",
    "passport-custom": "^1.1.1",
    "passport-http-bearer": "^1.0.1",
    "passport-jwt": "^4.0.0",
    "passport-local": "^1.0.0",
    "pkce": "^1.0.0-beta2",
    "pug": "^3.0.1",
    "request": "2.88.2",
    "sanitize-filename": "^1.6.3",
    "semantic-ui-css": "^2.4.1",
    "slug": "^3.5.2",
    "turf": "^3.0.14",
    "underscore": "^1.12.1"
  },
  "devDependencies": {
    "eslint": "^7.20.0",
    "eslint-config-prettier": "^6.15.0",
    "eslint-config-standard": "^16.0.2",
    "eslint-plugin-import": "^2.22.1",
    "eslint-plugin-jest": "^24.1.5",
    "eslint-plugin-node": "^11.1.0",
    "eslint-plugin-prettier": "^3.3.1",
    "eslint-plugin-promise": "^4.3.1",
    "nodemon": "^2.0.7",
    "prettier": "^2.2.1"
  },
  "jest": {
    "modulePathIgnorePatterns": [
      "local"
    ]
  },
  "prettier": {
    "useTabs": false,
    "trailingComma": "all",
    "tabWidth": 2,
    "semi": true,
    "singleQuote": true,
    "printWidth": 120
  },
  "eslintConfig": {
    "extends": [
      "standard",
      "prettier"
    ],
    "plugins": [
      "jest",
      "prettier"
    ],
    "env": {
      "browser": false,
      "node": true,
      "jest/globals": true
    },
    "rules": {
      "prettier/prettier": "error",
      "standard/array-bracket-even-spacing": 0,
      "standard/computed-property-even-spacing": 0,
      "standard/object-curly-even-spacing": 0
    },
    "root": true,
    "ignorePatterns": [
      "postman-examples/**",
      "public/**",
      "node_modules",
      "local"
    ]
  }
 }
--- a/api/requirements.txt
+++ b/api/requirements.txt
@ -1,22 +0,0 @@
 coloredlogs~=15.0.1
 sanic==22.6.2
 oic~=1.5.0
 sanic-session~=0.8.0
 python-slugify~=6.1.2
 motor~=3.1.1
 pyyaml~=5.3.1
 -e git+https://github.com/openmaptiles/openmaptiles-tools#egg=openmaptiles-tools
 sqlparse~=0.4.3
 sqlalchemy[asyncio]~=1.4.46
 asyncpg~=0.27.0
 pyshp~=2.3.1
 alembic~=1.9.4
 stream-zip~=0.0.50
 msgpack~=1.0.5
 osmium~=3.6.0
 psycopg~=3.1.8
 shapely~=2.0.1
 pyproj~=3.4.1
 aiohttp~=3.8.1
 # sanic requires websocets and chockes on >=10 in 2022.6.2
 websockets<11
--- a/api/scripts
+++ b/api/scripts
@ -1 +1 @@
-Subproject commit 664e4d606416417c0651ea1748d32dd36209be6a
+Subproject commit 6beab2ebfede7e41a1184b5ae1d0be0c83f8f95c
--- a/api/setup.py
+++ b/api/setup.py
@ -1,34 +0,0 @@
 from setuptools import setup, find_packages
 setup(
    name="openbikesensor-api",
    version="0.0.1",
    author="OpenBikeSensor Contributors",
    license="LGPL-3.0",
    description="OpenBikeSensor Portal API",
    url="https://github.com/openbikesensor/portal",
    packages=find_packages(),
    package_data={},
    install_requires=[
        "coloredlogs~=15.0.1",
        "sanic==22.6.2",
        "oic>=1.3.0, <2",
        "sanic-session~=0.8.0",
        "python-slugify>=5.0.2,<6.2.0",
        "motor>=2.5.1,<3.1.2",
        "pyyaml<6",
        "sqlparse~=0.4.3",
        "openmaptiles-tools",  # install from git
        "pyshp>=2.2,<2.4",
        "sqlalchemy[asyncio]~=1.4.46",
        "asyncpg~=0.27.0",
        "alembic~=1.9.4",
        "stream-zip~=0.0.50",
    ],
    entry_points={
        "console_scripts": [
            "openbikesensor-api=obs.bin.openbikesensor_api:main",
            "openbikesensor-transform-osm=obs.bin.openbikesensor_transform_osm:main",
        ]
    },
 )
--- a/api/src/_helpers/generators.js
+++ b/api/src/_helpers/generators.js
@ -0,0 +1,65 @@
 function* pairwise(iter) {
  let last;
  let firstLoop = true;
  for (const it of iter) {
    if (firstLoop) {
      firstLoop = false;
    } else {
      yield [last, it];
    }
    last = it;
  }
 }
 function* enumerate(iter) {
  let i = 0;
  for (const it of iter) {
    yield [i, it];
    i++;
  }
 }
 const map = (fn) =>
  function* (iter) {
    for (const [i, it] of enumerate(iter)) {
      yield fn(it, i);
    }
  };
 const filter = (fn) =>
  function* (iter) {
    for (const it of iter) {
      if (fn(it)) {
        yield it;
      }
    }
  };
 const reduce = (fn, init) => (iter) => {
  let acc = init;
  for (const it of iter) {
    acc = fn(acc, it);
  }
  return acc;
 };
 const scan = (fn) =>
  function* (iter, init) {
    let acc = init;
    for (const it of iter) {
      acc = fn(acc, it);
      yield acc;
    }
  };
 const flow = (...reducers) => (input) => reducers.reduce((c, fn) => fn(c), input);
 module.exports = {
  filter,
  map,
  enumerate,
  pairwise,
  flow,
  reduce,
  scan,
 };
--- a/api/src/_helpers/send-email.js
+++ b/api/src/_helpers/send-email.js
@ -0,0 +1,30 @@
 const nodemailer = require('nodemailer');
 const config = require('../config');
 module.exports = sendEmail;
 async function sendEmail({ to, subject, html }) {
  if (config.mail) {
    const from = config.mail.from;
    const transporter = nodemailer.createTransport({
      host: config.mail.smtp.host,
      port: config.mail.smtp.port,
      secure: !config.mail.smtp.starttls,
      requiretls: config.mail.smtp.starttls,
      auth: {
        user: config.mail.smtp.username,
        pass: config.mail.smtp.password,
      },
    });
    await transporter.sendMail({ from, to, subject, html });
  } else {
    console.log(`========== E-Mail disabled, see contents below =========
 To: ${to}
 Subject: ${subject}
 ${html}
 `)
  }
 }
--- a/api/src/_helpers/wrapRoute.js
+++ b/api/src/_helpers/wrapRoute.js
@ -0,0 +1,9 @@
 const wrapRoute = (fn) => async (req, res, next) => {
  try {
    return await fn(req, res);
  } catch (err) {
    next(err);
  }
 };
 module.exports = wrapRoute;
--- a/api/src/_middleware/error-handler.js
+++ b/api/src/_middleware/error-handler.js
@ -0,0 +1,23 @@
 module.exports = errorHandler;
 function errorHandler(err, req, res, next) {
  switch (true) {
    case typeof err === 'string': {
      // custom application error
      const is404 = err.toLowerCase().endsWith('not found');
      const statusCode = is404 ? 404 : 400;
      return res.status(statusCode).json({ message: err });
    }
    case err.name === 'ValidationError':
      // mongoose validation error
      return res.status(400).json({ message: err.message });
    case err.name === 'UnauthorizedError':
      // jwt authentication error
      return res.status(401).json({ message: 'Unauthorized' });
    default:
      return res.status(500).json({ message: err.message });
  }
 }
--- a/api/src/_middleware/validate-request.js
+++ b/api/src/_middleware/validate-request.js
@ -0,0 +1,19 @@
 const validateRequest = (schema, property = 'body') => (req, res, next) => {
  console.log('validateRequest');
  const options = {
    abortEarly: false, // include all errors
    allowUnknown: true, // ignore unknown props
    stripUnknown: true, // remove unknown props
  };
  const { error, value } = schema.validate(req[property], options);
  if (error) {
    console.log('error: ', error);
    next(`Validation error: ${error.details.map((x) => x.message).join(', ')}`);
  } else {
    req[property] = value;
    next();
  }
 };
 module.exports = validateRequest;
--- a/api/src/accounts/account.service.js
+++ b/api/src/accounts/account.service.js
@ -0,0 +1,146 @@
 const crypto = require('crypto');
 const sendEmail = require('../_helpers/send-email');
 const config = require('../config');
 const { User } = require('../models');
 const baseUrl = config.baseUrl.replace(/\/+$/, '');
 module.exports = {
  register,
  verifyEmail,
  forgotPassword,
  validateResetToken,
  resetPassword,
 };
 async function register(params) {
  const user = await User.findOne({ email: params.email });
  if (user) {
    // send already registered error in email to prevent account enumeration
    return await sendAlreadyRegisteredEmail(params.email);
  }
  const newUser = new User();
  newUser.username = params.username;
  newUser.email = params.email;
  newUser.setPassword(params.password);
  newUser.verificationToken = randomTokenString();
  newUser.needsEmailValidation = true;
  await newUser.save();
  // send email
  await sendVerificationEmail(newUser);
 }
 async function verifyEmail({ token }) {
  const account = await User.findOne({ verificationToken: token });
  if (!account) {
    throw Error('Verification failed');
  }
  account.needsEmailValidation = false;
  account.verificationToken = undefined;
  await account.save();
 }
 async function forgotPassword({ email }) {
  const account = await User.findOne({ email });
  console.log('forgotPassword', account, email);
  // always return ok response to prevent email enumeration
  if (!account) return;
  // create reset token that expires after 24 hours
  account.resetToken = {
    token: randomTokenString(),
    expires: new Date(Date.now() + 24 * 60 * 60 * 1000),
  };
  await account.save();
  console.log('forgotPassword account saved', account);
  // send email
  await sendPasswordResetEmail(account);
 }
 async function validateResetToken({ token }) {
  const account = await User.findOne({
    'resetToken.token': token,
    'resetToken.expires': { $gt: Date.now() },
  });
  if (!account) {
    throw Error('Invalid token');
  }
 }
 async function resetPassword({ token, password }) {
  const account = await User.findOne({
    'resetToken.token': token,
    'resetToken.expires': { $gt: Date.now() },
  });
  if (!account) {
    throw Error('Invalid token');
  }
  // update password and remove reset token
  account.setPassword(password);
  account.resetToken = undefined;
  // Since password recovery happens through email, we can consider this a
  // successful verification of the email address.
  account.needsEmailValidation = false;
  account.verificationToken = undefined;
  await account.save();
 }
 function randomTokenString() {
  return crypto.randomBytes(40).toString('hex');
 }
 async function sendVerificationEmail(account) {
  const verifyUrl = `${baseUrl}/verify-email?token=${account.verificationToken}`;
  const html = [
    '<h4>Verify Email</h4>',
    '<p>Thanks for registering!</p>',
    '<p>Please click the below link to verify your email address:</p>',
    `<p><a href="${verifyUrl}">${verifyUrl}</a></p>`,
  ].join('\n');
  await sendEmail({
    to: account.email,
    subject: 'Sign-up Verification API - Verify Email',
    html,
  });
 }
 async function sendAlreadyRegisteredEmail(email) {
  const message = `<p>If you don't know your password please visit the <a href="${baseUrl}/forgot-password">forgot password</a> page.</p>`;
  await sendEmail({
    to: email,
    subject: 'Sign-up Verification API - Email Already Registered',
    html: `<h4>Email Already Registered</h4>
               <p>Your email <strong>${email}</strong> is already registered.</p>
               ${message}`,
  });
 }
 async function sendPasswordResetEmail(account) {
  const resetUrl = `${baseUrl}/reset-password?token=${account.resetToken.token}`;
  const message = `<p>Please click the below link to reset your password, the link will be valid for 1 day:</p>
                 <p><a href="${resetUrl}">${resetUrl}</a></p>`;
  await sendEmail({
    to: account.email,
    subject: 'Sign-up Verification API - Reset Password',
    html: `<h4>Reset Password Email</h4>
               ${message}`,
  });
 }
--- a/api/src/accounts/accounts.controller.js
+++ b/api/src/accounts/accounts.controller.js
@ -0,0 +1,78 @@
 const express = require('express');
 const router = express.Router();
 const Joi = require('joi');
 const wrapRoute = require('../_helpers/wrapRoute');
 const validateRequest = require('../_middleware/validate-request');
 const accountService = require('./account.service');
 router.post(
  '/register',
  validateRequest(
    Joi.object({
      username: Joi.string().required(),
      email: Joi.string().email().required(),
      password: Joi.string().min(6).required(),
      confirmPassword: Joi.string().valid(Joi.ref('password')).required(),
    }),
  ),
  wrapRoute(async (req, res) => {
    await accountService.register(req.body, req.get('origin'));
    res.json({ message: 'Registration successful, please check your email for verification instructions' });
  }),
 );
 router.post(
  '/verify-email',
  validateRequest(
    Joi.object({
      token: Joi.string().required(),
    }),
  ),
  wrapRoute(async (req, res) => {
    await accountService.verifyEmail(req.body);
    res.json({ message: 'Verification successful, you can now login' });
  }),
 );
 router.post(
  '/forgot-password',
  validateRequest(
    Joi.object({
      email: Joi.string().email().required(),
    }),
  ),
  wrapRoute(async (req, res) => {
    await accountService.forgotPassword(req.body, req.get('origin'));
    res.json({ message: 'Please check your email for password reset instructions' });
  }),
 );
 router.post(
  '/validate-reset-token',
  validateRequest(
    Joi.object({
      token: Joi.string().required(),
    }),
  ),
  wrapRoute(async (req, res) => {
    await accountService.validateResetToken(req.body);
    res.json({ message: 'Token is valid' });
  }),
 );
 router.post(
  '/reset-password',
  validateRequest(
    Joi.object({
      token: Joi.string().required(),
      password: Joi.string().min(6).required(),
      confirmPassword: Joi.string().valid(Joi.ref('password')).required(),
    }),
  ),
  wrapRoute(async (req, res) => {
    await accountService.resetPassword(req.body);
    res.json({ message: 'Password reset successful, you can now login' });
  }),
 );
 module.exports = router;
--- a/api/src/config.js
+++ b/api/src/config.js
@ -0,0 +1,72 @@
 const fs = require('fs');
 const Joi = require('joi');
 const configSchema = Joi.object({
  jwtSecret: Joi.string().min(16).max(128).required(),
  cookieSecret: Joi.string().min(16).max(128).required(),
  imprintUrl: Joi.string(),
  privacyPolicyUrl: Joi.string(),
  baseUrl: Joi.string().required(),
  mainFrontendUrl: Joi.string(), // optional
  mail: Joi.alternatives().try(
    Joi.object({
      from: Joi.string().required(),
      smtp: Joi.object({
        host: Joi.string().required(),
        port: Joi.number().default(465),
        starttls: Joi.boolean().default(false),
        username: Joi.string().required(),
        password: Joi.string().required(),
      }).required(),
    }),
    Joi.boolean().valid(false),
  ),
  mongodb: Joi.object({
    url: Joi.string().required(),
    debug: Joi.boolean().default(process.env.NODE_ENV !== 'production'),
  }).required(),
  redisUrl: Joi.string().required(),
  oAuth2Clients: Joi.array()
    .default([])
    .items(
      Joi.object({
        title: Joi.string().required(),
        clientId: Joi.string().required(),
        validRedirectUris: Joi.array().required().items(Joi.string()),
        // Set `refreshTokenExpirySeconds` to null to issue no refresh tokens.  Set
        // to a number of seconds to issue refresh tokens with that duration. No
        // infinite tokens are ever issued, set to big number to simulate that.
        refreshTokenExpirySeconds: Joi.number()
          .default(null)
          .min(1) // 0 would make no sense, use `null` to issue no token
          .max(1000 * 24 * 60 * 60), // 1000 days, nearly 3 years
        // Set to a scope which cannot be exceeded when requesting client tokens.
        // Clients must manually request a scope that is smaller or equal to this
        // scope to get a valid response. Scopes are not automatically truncated.
        // Leave empty or set to `"*"` for unlimited scopes in this client.
        maxScope: Joi.string().required(),
        autoAccept: Joi.boolean().optional(),
      }),
    ),
 }).required();
 const configFiles = [
  process.env.CONFIG_FILE,
  process.env.NODE_ENV === 'production' ? 'config.prod.json' : 'config.dev.json',
  'config.json',
 ].filter((x) => x && fs.existsSync(x));
 if (!configFiles.length) {
  throw new Error('No config file found.');
 }
 module.exports = Joi.attempt(JSON.parse(fs.readFileSync(configFiles[0], 'utf8')), configSchema);
--- a/api/src/db.js
+++ b/api/src/db.js
@ -0,0 +1,13 @@
 const mongoose = require('mongoose');
 const config = require('./config')
 mongoose.connect(config.mongodb.url);
 mongoose.set('debug', config.mongodb.debug);
 require('./models/User');
 require('./models/Track');
 require('./models/Comment');
 require('./passport');
 module.exports = mongoose;
--- a/api/src/index.js
+++ b/api/src/index.js
@ -0,0 +1,87 @@
 const path = require('path');
 const express = require('express');
 const bodyParser = require('body-parser');
 const session = require('express-session');
 const cors = require('cors');
 const errorhandler = require('errorhandler');
 const passport = require('passport');
 const config = require('./config');
 require('./passport');
 const isProduction = process.env.NODE_ENV === 'production';
 // Create global app object
 const app = express();
 app.use(cors());
 // Express configuration
 app.set('views', path.join(__dirname, '..', 'views'));
 app.set('view engine', 'pug');
 // Normal express config defaults
 app.use(require('morgan')('dev'));
 app.use(bodyParser.json({ limit: '50mb' }));
 app.use(bodyParser.urlencoded({ limit: '50mb', extended: false }));
 app.use(require('method-override')());
 app.use(express.static(path.join(__dirname, 'public')));
 app.use('/semantic-ui', express.static(path.join(__dirname, '..', 'node_modules', 'semantic-ui-css')));
 app.use(session({ secret: config.cookieSecret, cookie: { maxAge: 10 * 60 * 1000 }, resave: false, saveUninitialized: false }));
 app.use(passport.initialize());
 app.use(passport.session());
 if (!isProduction) {
  app.use(errorhandler());
 }
 require('./db');
 require('./models');
 app.use(require('./routes'));
 /// catch 404 and forward to error handler
 app.use(function (req, res, next) {
  const err = new Error('Not Found');
  err.status = 404;
  next(err);
 });
 /// error handlers
 // development error handler
 // will print stacktrace
 if (!isProduction) {
  app.use(function (err, req, res, next) {
    console.log(err.stack);
    res.status(err.status || 500);
    res.json({
      errors: {
        message: err.message,
        error: err,
      },
    });
  });
 }
 // production error handler
 // no stacktraces leaked to user
 app.use(function (err, req, res, next) {
  res.status(err.status || 500);
  res.json({
    errors: {
      message: err.message,
      error: {},
    },
  });
 });
 // finally, let's start our server...
 const port = process.env.PORT || 3000;
 app.listen(port, () => {
  console.log('Listening on port ' + port);
 });
--- a/api/src/logic/_tracks_testdata.js
+++ b/api/src/logic/_tracks_testdata.js
@ -0,0 +1,357 @@
 const TEST_ROWS = [
  'Date;Time;Latitude;Longitude;Course;Speed;Right;Left;Confirmed;insidePrivacyArea;',
  '12.07.2020;09:02:59;0.000000;0.000000;0.000;0.0000;255;255;0;0;',
  '12.07.2020;09:02:59;0.000000;0.000000;0.000;0.0000;255;255;0;0;',
  '12.07.2020;09:03:00;0.000000;0.000000;0.000;0.0000;255;255;0;0;',
  '12.07.2020;09:03:01;48.722205;9.270218;0.000;0.4260;255;255;0;0;',
  '12.07.2020;09:03:02;48.722206;9.270219;0.000;0.5741;255;255;0;0;',
  '12.07.2020;09:03:03;48.722204;9.270221;0.000;0.5371;255;255;0;0;',
  '12.07.2020;09:03:04;48.722198;9.270229;0.000;0.7593;255;255;0;0;',
  '12.07.2020;09:03:05;48.722188;9.270241;0.000;0.5556;255;255;0;0;',
  '12.07.2020;09:03:06;48.722174;9.270259;0.000;0.4815;255;255;0;0;',
  '12.07.2020;09:03:07;48.722158;9.270278;0.000;0.3704;255;255;0;0;',
  '12.07.2020;09:03:08;48.722146;9.270293;0.000;0.5741;255;255;0;0;',
  '12.07.2020;09:03:09;48.722138;9.270305;0.000;1.2594;255;255;0;0;',
  '12.07.2020;09:03:10;48.722129;9.270318;0.000;1.5557;255;255;0;0;',
  '12.07.2020;09:03:11;48.722122;9.270329;0.000;1.5372;255;255;0;0;',
  '12.07.2020;09:03:12;48.722115;9.270339;0.000;0.4630;255;255;0;0;',
  '12.07.2020;09:03:13;48.722107;9.270350;0.000;0.2963;255;255;0;0;',
  '12.07.2020;09:03:14;48.722101;9.270357;0.000;0.2963;255;255;0;0;',
  '12.07.2020;09:03:15;48.722092;9.270367;0.000;0.8149;255;255;0;0;',
  '12.07.2020;09:03:16;48.722084;9.270377;0.000;1.2223;255;255;0;0;',
  '12.07.2020;09:03:17;48.722076;9.270385;0.000;0.0926;255;255;0;0;',
  '12.07.2020;09:03:18;48.722070;9.270391;0.000;1.4816;255;255;0;0;',
  '12.07.2020;09:03:19;48.722070;9.270392;0.000;1.0927;255;255;0;0;',
  '12.07.2020;09:03:20;48.722066;9.270395;0.000;1.6668;255;255;0;0;',
  '12.07.2020;09:03:21;48.722068;9.270391;0.000;2.0742;255;255;0;0;',
  '12.07.2020;09:03:22;48.722064;9.270396;0.000;1.6853;255;255;0;0;',
  '12.07.2020;09:03:23;48.722060;9.270401;0.000;1.0927;255;255;0;0;',
  '12.07.2020;09:03:24;48.722056;9.270406;0.000;0.9445;255;255;0;0;',
  '12.07.2020;09:03:25;48.722052;9.270411;0.000;0.7964;255;255;0;0;',
  '12.07.2020;09:03:26;48.722047;9.270416;0.000;0.6482;255;255;0;0;',
  '12.07.2020;09:03:27;48.722042;9.270419;0.000;1.0556;255;255;0;0;',
  '12.07.2020;09:03:28;48.722031;9.270433;0.000;2.0372;255;255;0;0;',
  '12.07.2020;09:03:29;48.722031;9.270432;0.000;2.4261;255;255;0;0;',
  '12.07.2020;09:03:30;48.722029;9.270433;0.000;0.8704;255;255;0;0;',
  '12.07.2020;09:03:31;48.722029;9.270433;0.000;1.8150;255;255;0;0;',
  '12.07.2020;09:03:32;48.722024;9.270439;0.000;1.2223;255;255;0;0;',
  '12.07.2020;09:03:33;48.722025;9.270439;0.000;0.3889;255;255;0;0;',
  '12.07.2020;09:03:34;48.722022;9.270440;0.000;0.3519;255;255;0;0;',
  '12.07.2020;09:03:35;48.722020;9.270445;0.000;0.9445;255;255;0;0;',
  '12.07.2020;09:03:36;48.722018;9.270447;0.000;0.9260;255;255;0;0;',
  '12.07.2020;09:03:37;48.722020;9.270444;0.000;0.9075;255;255;0;0;',
  '12.07.2020;09:03:38;48.722021;9.270443;0.000;1.9261;255;255;0;0;',
  '12.07.2020;09:03:39;48.722018;9.270447;0.000;0.3334;255;255;0;0;',
  '12.07.2020;09:03:40;48.722020;9.270445;0.000;0.1482;255;255;0;0;',
  '12.07.2020;09:03:41;48.722023;9.270440;0.000;1.2594;255;255;0;0;',
  '12.07.2020;09:03:42;48.722023;9.270442;0.000;0.5000;255;255;0;0;',
  '12.07.2020;09:03:43;48.722025;9.270440;0.000;0.6852;220;255;0;0;',
  '12.07.2020;09:03:44;48.722023;9.270441;0.000;0.8519;199;255;0;0;',
  '12.07.2020;09:03:45;48.722026;9.270438;0.000;1.4075;255;255;0;0;',
  '12.07.2020;09:03:46;48.722029;9.270436;0.000;0.5371;255;255;0;0;',
  '12.07.2020;09:03:47;48.722028;9.270435;0.000;0.8334;97;255;0;0;',
  '12.07.2020;09:03:48;48.722029;9.270435;0.000;0.3704;255;255;0;0;',
  '12.07.2020;09:03:49;48.722029;9.270436;0.000;1.1112;96;255;0;0;',
  '12.07.2020;09:03:50;48.722029;9.270435;0.000;1.8890;255;255;0;0;',
  '12.07.2020;09:03:51;48.722034;9.270429;0.000;1.0186;255;255;0;0;',
  '12.07.2020;09:03:52;48.721942;9.270529;128.450;5.2226;255;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;79;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;178;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;89;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;156;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;168;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;181;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;176;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;186;0;0;',
  '12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
  '12.07.2020;09:04:10;48.721896;9.270602;916.230;0.0556;255;255;0;0;',
  '12.07.2020;09:04:11;48.721894;9.270609;916.230;0.0926;255;192;0;0;',
  '12.07.2020;09:04:12;48.721892;9.270616;916.230;0.0556;255;255;0;0;',
  '12.07.2020;09:04:13;48.721890;9.270623;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:04:14;48.721888;9.270629;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:04:15;48.721886;9.270635;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:04:16;48.721883;9.270640;916.230;0.0556;255;255;0;0;',
  '12.07.2020;09:04:17;48.721881;9.270644;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:04:18;48.721879;9.270649;916.230;0.1111;255;255;0;0;',
  '12.07.2020;09:04:19;48.721877;9.270653;916.230;0.1296;255;255;0;0;',
  '12.07.2020;09:04:20;48.721876;9.270657;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:04:21;48.721874;9.270658;916.230;0.3148;255;255;0;0;',
  '12.07.2020;09:04:22;48.721873;9.270659;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:04:23;48.721872;9.270661;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:04:24;48.721871;9.270661;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:04:25;48.721870;9.270660;916.230;0.3334;255;255;0;0;',
  '12.07.2020;09:04:26;48.721869;9.270658;916.230;0.5000;255;255;0;0;',
  '12.07.2020;09:04:27;48.721866;9.270660;916.230;1.6853;255;255;0;0;',
  '12.07.2020;09:04:28;48.721866;9.270659;916.230;0.8704;255;198;0;0;',
  '12.07.2020;09:04:29;48.721867;9.270659;916.230;0.5741;255;196;0;0;',
  '12.07.2020;09:04:30;48.721867;9.270660;916.230;0.3148;255;196;0;0;',
  '12.07.2020;09:04:31;48.721867;9.270659;916.230;0.1111;255;255;0;0;',
  '12.07.2020;09:04:32;48.721866;9.270659;916.230;0.0556;255;199;0;0;',
  '12.07.2020;09:04:33;48.721867;9.270656;916.230;0.1482;255;199;0;0;',
  '12.07.2020;09:04:34;48.721867;9.270654;916.230;0.0370;255;198;0;0;',
  '12.07.2020;09:04:35;48.721867;9.270653;916.230;0.1296;255;198;0;0;',
  '12.07.2020;09:04:36;48.721867;9.270651;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:04:37;48.721867;9.270650;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:04:38;48.721868;9.270650;916.230;0.1852;255;255;0;0;',
  '12.07.2020;09:04:39;48.721868;9.270649;916.230;0.1667;255;201;0;0;',
  '12.07.2020;09:04:40;48.721868;9.270647;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:04:41;48.721869;9.270644;916.230;0.0185;255;255;0;0;',
  '12.07.2020;09:04:42;48.721869;9.270641;916.230;0.0185;255;198;0;0;',
  '12.07.2020;09:04:43;48.721870;9.270638;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:04:44;48.721870;9.270635;916.230;0.0370;255;199;0;0;',
  '12.07.2020;09:04:45;48.721871;9.270632;916.230;0.1482;255;204;0;0;',
  '12.07.2020;09:04:46;48.721871;9.270630;916.230;0.0185;255;201;0;0;',
  '12.07.2020;09:04:47;48.721873;9.270630;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:04:48;48.721873;9.270629;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:04:49;48.721874;9.270628;916.230;0.4074;255;255;0;0;',
  '12.07.2020;09:04:50;48.721875;9.270627;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:04:51;48.721876;9.270625;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:04:52;48.721877;9.270623;916.230;0.1111;255;255;0;0;',
  '12.07.2020;09:04:53;48.721877;9.270622;916.230;0.2037;255;255;0;0;',
  '12.07.2020;09:04:54;48.721879;9.270621;916.230;0.3148;255;255;0;0;',
  '12.07.2020;09:04:55;48.721881;9.270618;916.230;0.2408;255;255;0;0;',
  '12.07.2020;09:04:56;48.721883;9.270615;916.230;0.3148;255;255;0;0;',
  '12.07.2020;09:04:57;48.721884;9.270612;916.230;0.2778;255;255;0;0;',
  '12.07.2020;09:04:58;48.721885;9.270609;916.230;0.1296;255;255;0;0;',
  '12.07.2020;09:04:59;48.721886;9.270606;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:05:00;48.721888;9.270602;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:05:01;48.721889;9.270598;916.230;0.1111;255;191;0;0;',
  '12.07.2020;09:05:02;48.721890;9.270595;916.230;0.1482;255;193;0;0;',
  '12.07.2020;09:05:03;48.721891;9.270593;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:05:04;48.721891;9.270591;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:05:05;48.721891;9.270589;916.230;0.1296;255;255;0;0;',
  '12.07.2020;09:05:06;48.721891;9.270587;916.230;0.3519;255;199;0;0;',
  '12.07.2020;09:05:07;48.721891;9.270586;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:08;48.721891;9.270588;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:05:09;48.721890;9.270589;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:05:10;48.721889;9.270589;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:11;48.721888;9.270589;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:05:12;48.721887;9.270589;916.230;0.1296;255;255;0;0;',
  '12.07.2020;09:05:13;48.721886;9.270590;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:05:14;48.721885;9.270591;916.230;0.3148;255;255;0;0;',
  '12.07.2020;09:05:15;48.721885;9.270592;916.230;0.2037;255;255;0;0;',
  '12.07.2020;09:05:16;48.721885;9.270596;916.230;0.5556;255;255;0;0;',
  '12.07.2020;09:05:17;48.721885;9.270598;916.230;0.3519;255;255;0;0;',
  '12.07.2020;09:05:18;48.721884;9.270600;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:05:19;48.721882;9.270600;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:05:20;48.721881;9.270602;916.230;0.0556;255;255;0;0;',
  '12.07.2020;09:05:21;48.721879;9.270603;916.230;0.0185;255;206;0;0;',
  '12.07.2020;09:05:22;48.721878;9.270605;916.230;0.0556;255;203;0;0;',
  '12.07.2020;09:05:23;48.721876;9.270606;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:05:24;48.721874;9.270605;916.230;0.0185;255;255;0;0;',
  '12.07.2020;09:05:25;48.721873;9.270605;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:05:26;48.721872;9.270605;916.230;0.1296;255;209;0;0;',
  '12.07.2020;09:05:27;48.721870;9.270606;916.230;0.0556;255;255;0;0;',
  '12.07.2020;09:05:28;48.721869;9.270608;916.230;0.1111;255;206;0;0;',
  '12.07.2020;09:05:29;48.721868;9.270610;916.230;0.3148;255;209;0;0;',
  '12.07.2020;09:05:30;48.721867;9.270610;916.230;0.2593;255;208;0;0;',
  '12.07.2020;09:05:31;48.721866;9.270611;916.230;0.0556;255;210;0;0;',
  '12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:44;48.721855;9.270602;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:44;48.721855;9.270602;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:05:46;48.721854;9.270602;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:05:46;48.721854;9.270602;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:05:48;48.721852;9.270606;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:05:49;48.721851;9.270611;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:05:50;48.721851;9.270615;916.230;0.1852;255;255;0;0;',
  '12.07.2020;09:05:51;48.721851;9.270616;916.230;0.0185;255;255;0;0;',
  '12.07.2020;09:05:52;48.721851;9.270617;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:05:53;48.721852;9.270617;916.230;0.0185;255;255;0;0;',
  '12.07.2020;09:05:54;48.721853;9.270616;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:05:55;48.721855;9.270613;916.230;0.0556;255;255;0;0;',
  '12.07.2020;09:05:56;48.721858;9.270609;916.230;0.6482;255;255;0;0;',
  '12.07.2020;09:05:57;48.721860;9.270606;916.230;0.4260;255;255;0;0;',
  '12.07.2020;09:05:58;48.721864;9.270601;916.230;0.6297;255;255;0;0;',
  '12.07.2020;09:05:59;48.721867;9.270595;916.230;0.4260;255;255;0;0;',
  '12.07.2020;09:06:00;48.721872;9.270589;916.230;0.5000;255;255;0;0;',
  '12.07.2020;09:06:01;48.721875;9.270584;916.230;0.2593;255;255;0;0;',
  '12.07.2020;09:06:02;48.721880;9.270578;916.230;0.5186;255;255;0;0;',
  '12.07.2020;09:06:03;48.721883;9.270574;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:06:04;48.721886;9.270570;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:06:05;48.721890;9.270565;916.230;0.2408;255;255;0;0;',
  '12.07.2020;09:06:06;48.721893;9.270562;916.230;0.2593;255;255;0;0;',
  '12.07.2020;09:06:07;48.721893;9.270560;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:06:08;48.721894;9.270559;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:06:09;48.721894;9.270557;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:06:07;48.721896;9.270556;916.230;0.2778;255;255;0;0;',
  '12.07.2020;09:06:08;48.721896;9.270556;916.230;0.2408;255;255;0;0;',
  '12.07.2020;09:06:09;48.721895;9.270557;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:06:10;48.721894;9.270559;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:06:11;48.721892;9.270560;916.230;0.2037;255;255;0;0;',
  '12.07.2020;09:06:12;48.721891;9.270561;916.230;0.1296;255;255;0;0;',
  '12.07.2020;09:06:13;48.721892;9.270562;916.230;0.1852;255;255;0;0;',
  '12.07.2020;09:06:14;48.721891;9.270564;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:06:15;48.721889;9.270566;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:06:16;48.721888;9.270568;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:06:17;48.721888;9.270570;916.230;0.2037;255;255;0;0;',
  '12.07.2020;09:06:18;48.721888;9.270572;916.230;0.4630;255;255;0;0;',
  '12.07.2020;09:06:19;48.721887;9.270573;916.230;0.4815;255;255;0;0;',
  '12.07.2020;09:06:20;48.721886;9.270574;916.230;0.3334;255;255;0;0;',
  '12.07.2020;09:06:21;48.721885;9.270576;916.230;0.1852;255;255;0;0;',
  '12.07.2020;09:06:22;48.721884;9.270579;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:06:23;48.721882;9.270581;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:06:24;48.721881;9.270584;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:06:25;48.721880;9.270589;916.230;0.2963;255;255;0;0;',
  '12.07.2020;09:06:26;48.721879;9.270596;916.230;0.3519;255;255;0;0;',
  '12.07.2020;09:06:27;48.721878;9.270602;916.230;0.1111;255;255;0;0;',
  '12.07.2020;09:06:28;48.721876;9.270601;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:06:29;48.721874;9.270603;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:06:30;48.721873;9.270607;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:06:31;48.721872;9.270614;916.230;0.2037;255;255;0;0;',
  '12.07.2020;09:06:32;48.721870;9.270613;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:06:33;48.721869;9.270614;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:06:34;48.721868;9.270616;916.230;0.1111;255;255;0;0;',
  '12.07.2020;09:06:35;48.721867;9.270617;916.230;0.2593;255;255;0;0;',
  '12.07.2020;09:06:36;48.721867;9.270618;916.230;0.1852;255;255;0;0;',
  '12.07.2020;09:06:37;48.721867;9.270618;916.230;0.1296;255;255;0;0;',
  '12.07.2020;09:06:38;48.721867;9.270616;916.230;0.2963;255;255;0;0;',
  '12.07.2020;09:06:39;48.721867;9.270613;916.230;0.1296;255;255;0;0;',
  '12.07.2020;09:06:40;48.721867;9.270607;916.230;0.0185;255;255;0;0;',
  '12.07.2020;09:06:41;48.721866;9.270601;916.230;0.5186;255;255;0;0;',
  '12.07.2020;09:06:42;48.721866;9.270593;916.230;0.2963;255;255;0;0;',
  '12.07.2020;09:06:43;48.721866;9.270587;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:06:44;48.721866;9.270581;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:06:45;48.721866;9.270576;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:06:46;48.721866;9.270567;916.230;0.1111;255;255;0;0;',
  '12.07.2020;09:06:47;48.721866;9.270558;916.230;0.4074;255;255;0;0;',
  '12.07.2020;09:06:48;48.721866;9.270550;916.230;0.4260;255;255;0;0;',
  '12.07.2020;09:06:49;48.721866;9.270543;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:06:50;48.721867;9.270537;916.230;0.2778;255;255;0;0;',
  '12.07.2020;09:06:51;48.721867;9.270532;916.230;0.2037;255;255;0;0;',
  '12.07.2020;09:06:52;48.721867;9.270526;916.230;0.3148;255;255;0;0;',
  '12.07.2020;09:06:53;48.721868;9.270522;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:06:54;48.721868;9.270517;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:06:55;48.721869;9.270512;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:06:56;48.721869;9.270506;916.230;0.2408;255;255;0;0;',
  '12.07.2020;09:06:57;48.721869;9.270503;916.230;0.1111;255;255;0;0;',
  '12.07.2020;09:06:58;48.721870;9.270500;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:06:59;48.721870;9.270497;916.230;0.1296;255;255;0;0;',
  '12.07.2020;09:07:00;48.721871;9.270494;916.230;0.1296;255;255;0;0;',
  '12.07.2020;09:07:01;48.721871;9.270493;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:07:02;48.721871;9.270492;916.230;0.2037;255;255;0;0;',
  '12.07.2020;09:07:03;48.721872;9.270490;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:07:04;48.721873;9.270489;916.230;0.6667;255;255;0;0;',
  '12.07.2020;09:07:05;48.721873;9.270487;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:07:06;48.721873;9.270486;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:07:07;48.721873;9.270486;916.230;0.2408;255;255;0;0;',
  '12.07.2020;09:07:08;48.721873;9.270485;916.230;0.1852;255;255;0;0;',
  '12.07.2020;09:07:09;48.721873;9.270485;916.230;0.2037;255;255;0;0;',
  '12.07.2020;09:07:10;48.721872;9.270485;916.230;0.1296;255;255;0;0;',
  '12.07.2020;09:07:11;48.721870;9.270486;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:07:12;48.721869;9.270489;916.230;0.2037;255;255;0;0;',
  '12.07.2020;09:07:13;48.721867;9.270492;916.230;0.3148;255;255;0;0;',
  '12.07.2020;09:07:14;48.721865;9.270494;916.230;0.2037;255;255;0;0;',
  '12.07.2020;09:07:15;48.721863;9.270495;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:07:16;48.721861;9.270497;916.230;0.0000;255;255;0;0;',
  '12.07.2020;09:07:17;48.721860;9.270496;916.230;0.4074;255;255;0;0;',
  '12.07.2020;09:07:18;48.721859;9.270495;916.230;0.4445;255;255;0;0;',
  '12.07.2020;09:07:19;48.721857;9.270496;916.230;0.3889;255;255;0;0;',
  '12.07.2020;09:07:20;48.721856;9.270496;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:07:21;48.721854;9.270494;916.230;0.7593;255;255;0;0;',
  '12.07.2020;09:07:22;48.721851;9.270496;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:07:23;48.721850;9.270497;916.230;0.1667;255;255;0;0;',
  '12.07.2020;09:07:24;48.721848;9.270501;916.230;0.4074;255;255;0;0;',
  '12.07.2020;09:07:25;48.721847;9.270504;916.230;0.4074;255;255;0;0;',
  '12.07.2020;09:07:26;48.721846;9.270505;916.230;0.2037;255;255;0;0;',
  '12.07.2020;09:07:27;48.721844;9.270508;916.230;0.1111;255;255;0;0;',
  '12.07.2020;09:07:28;48.721843;9.270507;916.230;0.4630;255;255;0;0;',
  '12.07.2020;09:07:29;48.721842;9.270509;916.230;0.1111;255;255;0;0;',
  '12.07.2020;09:07:30;48.721841;9.270512;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:07:31;48.721840;9.270515;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:07:32;48.721839;9.270517;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:07:33;48.721838;9.270522;916.230;0.0556;255;255;0;0;',
  '12.07.2020;09:07:34;48.721838;9.270527;916.230;0.3889;255;255;0;0;',
  '12.07.2020;09:07:35;48.721837;9.270530;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:07:36;48.721836;9.270532;916.230;0.1111;255;255;0;0;',
  '12.07.2020;09:07:37;48.721835;9.270536;916.230;0.6112;255;255;0;0;',
  '12.07.2020;09:07:38;48.721835;9.270541;916.230;1.1668;255;255;0;0;',
  '12.07.2020;09:07:39;48.721835;9.270543;916.230;0.3889;255;255;0;0;',
  '12.07.2020;09:07:40;48.721834;9.270545;916.230;0.5000;255;255;0;0;',
  '12.07.2020;09:07:41;48.721834;9.270544;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:07:42;48.721834;9.270545;916.230;0.7593;255;255;0;0;',
  '12.07.2020;09:07:43;48.721834;9.270545;916.230;0.8890;255;255;0;0;',
  '12.07.2020;09:07:44;48.721834;9.270543;916.230;0.4260;255;255;0;0;',
  '12.07.2020;09:07:45;48.721834;9.270541;916.230;0.2408;255;255;0;0;',
  '12.07.2020;09:07:46;48.721834;9.270540;916.230;0.3148;255;255;0;0;',
  '12.07.2020;09:07:47;48.721835;9.270538;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:07:48;48.721835;9.270535;916.230;0.0556;255;255;0;0;',
  '12.07.2020;09:07:49;48.721835;9.270534;916.230;0.8890;255;255;0;0;',
  '12.07.2020;09:07:50;48.721835;9.270534;916.230;0.5926;255;255;0;0;',
  '12.07.2020;09:07:51;48.721835;9.270534;916.230;0.7593;255;255;0;0;',
  '12.07.2020;09:07:52;48.721836;9.270533;916.230;0.2408;255;255;0;0;',
  '12.07.2020;09:07:53;48.721836;9.270531;916.230;0.0741;255;255;0;0;',
  '12.07.2020;09:07:54;48.721836;9.270529;916.230;0.3889;255;255;0;0;',
  '12.07.2020;09:07:55;48.721836;9.270530;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:07:56;48.721836;9.270530;916.230;0.0185;255;255;0;0;',
  '12.07.2020;09:07:57;48.721837;9.270531;916.230;0.0185;255;255;0;0;',
  '12.07.2020;09:07:58;48.721837;9.270530;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:07:59;48.721838;9.270526;916.230;0.3519;255;255;0;0;',
  '12.07.2020;09:08:00;48.721838;9.270521;916.230;0.4260;255;255;0;0;',
  '12.07.2020;09:08:01;48.721839;9.270522;916.230;0.5556;255;255;0;0;',
  '12.07.2020;09:08:02;48.721840;9.270524;916.230;0.3519;255;255;0;0;',
  '12.07.2020;09:08:03;48.721842;9.270525;916.230;0.2963;255;255;0;0;',
  '12.07.2020;09:08:04;48.721843;9.270525;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:08:05;48.721844;9.270524;916.230;0.2222;255;255;0;0;',
  '12.07.2020;09:08:06;48.721846;9.270522;916.230;0.3704;255;255;0;0;',
  '12.07.2020;09:08:07;48.721847;9.270519;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:08:08;48.721848;9.270516;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:08:09;48.721849;9.270514;916.230;0.1296;255;255;0;0;',
  '12.07.2020;09:08:10;48.721850;9.270512;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:08:11;48.721851;9.270513;916.230;0.3334;255;255;0;0;',
  '12.07.2020;09:08:12;48.721851;9.270512;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:08:13;48.721851;9.270512;916.230;0.2593;255;255;0;0;',
  '12.07.2020;09:08:14;48.721852;9.270511;916.230;0.0926;255;255;0;0;',
  '12.07.2020;09:08:15;48.721853;9.270512;916.230;0.0370;255;255;0;0;',
  '12.07.2020;09:08:16;48.721852;9.270515;916.230;0.5371;255;255;0;0;',
  '12.07.2020;09:08:17;48.721853;9.270517;916.230;0.1482;255;255;0;0;',
  '12.07.2020;09:08:18;48.721854;9.270519;916.230;0.3148;255;255;0;0;',
  '12.07.2020;09:08:19;48.721855;9.270520;916.230;0.2408;255;255;0;0;',
  '12.07.2020;09:08:20;48.721856;9.270523;916.230;0.3704;255;255;0;0;',
 ];
 const test1 = TEST_ROWS.join('$');
 const test2 = `OBSFirmwareVersion=v0.3.999&OBSDataFormat=2&DataPerMeasurement=3&MaximumMeasurementsPerLine=60&OffsetLeft=30&OffsetRight=30&NumberOfDefinedPrivacyAreas=3&PrivacyLevelApplied=AbsolutePrivacy&MaximumValidFlightTimeMicroseconds=18560&DistanceSensorsUsed=HC-SR04/JSN-SR04T&DeviceId=ECEC&OBSUserID=32423432342234
 Date;Time;Millis;Comment;Latitude;Longitude;Altitude;Course;Speed;HDOP;Satellites;BatteryLevel;Left;Right;Confirmed;Marked;Invalid;InsidePrivacyArea;Factor;Measurements;Tms1;Lus1;Rus1;Tms2;Lus2;Rus2;Tms3;Lus3;Rus3;Tms4;Lus4;Rus4;Tms5;Lus5;Rus5;Tms6;Lus6;Rus6;Tms7;Lus7;Rus7;Tms8;Lus8;Rus8;Tms9;Lus9;Rus9;Tms10;Lus10;Rus10;Tms11;Lus11;Rus11;Tms12;Lus12;Rus12;Tms13;Lus13;Rus13;Tms14;Lus14;Rus14;Tms15;Lus15;Rus15;Tms16;Lus16;Rus16;Tms17;Lus17;Rus17;Tms18;Lus18;Rus18;Tms19;Lus19;Rus19;Tms20;Lus20;Rus20;Tms21;Lus21;Rus21;Tms22;Lus22;Rus22;Tms23;Lus23;Rus23;Tms24;Lus24;Rus24;Tms25;Lus25;Rus25;Tms26;Lus26;Rus26;Tms27;Lus27;Rus27;Tms28;Lus28;Rus28;Tms29;Lus29;Rus29;Tms30;Lus30;Rus30;Tms31;Lus31;Rus31;Tms32;Lus32;Rus32;Tms33;Lus33;Rus33;Tms34;Lus34;Rus34;Tms35;Lus35;Rus35;Tms36;Lus36;Rus36;Tms37;Lus37;Rus37;Tms38;Lus38;Rus38;Tms39;Lus39;Rus39;Tms40;Lus40;Rus40;Tms41;Lus41;Rus41;Tms42;Lus42;Rus42;Tms43;Lus43;Rus43;Tms44;Lus44;Rus44;Tms45;Lus45;Rus45;Tms46;Lus46;Rus46;Tms47;Lus47;Rus47;Tms48;Lus48;Rus48;Tms49;Lus49;Rus49;Tms50;Lus50;Rus50;Tms51;Lus51;Rus51;Tms52;Lus52;Rus52;Tms53;Lus53;Rus53;Tms54;Lus54;Rus54;Tms55;Lus55;Rus55;Tms56;Lus56;Rus56;Tms57;Lus57;Rus57;Tms58;Lus58;Rus58;Tms59;Lus59;Rus59;Tms60;Lus60;Rus60
 18.11.2020;16:05:59;1265034;;48.723224;9.094103;495.3;189.86;3.2;1.01;7;3.74;770;;0;0;58;54;0;6231;;16;;;36;6350;;52;;;72;6263;;87;;;107;6828;;122;;;143;6836;;158;;;178;6936;;193;;;213;7094;;228;;;248;6822;;263;;;284;7019;;299;;;319;6942;;334;;;354;7110;;370;;;390;7203;;405;;;425;7758;;440;;;461;7266;;476;;;496;7499;;511;;;531;7328;;546;;;567;7354;;582;;;602;7397;;617;;;637;;;664;;;684;16615;;708;;;728;9161;;745;;;765;10238;;783;;;802;8525;;818;;;839;7756;;854;;;875;7580;;890;;;910;7926;;925;;;945;7624;;960;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:00;1266041;DEVELOP:  GPSMessages: 2587 GPS crc errors: 0;48.723205;9.0941;495.4;189.86;2.87;1.01;7;3.74;1020;;0;0;58;53;0;8012;;27;;;47;7999;;62;;;83;7660;;98;;;118;7698;;133;;;158;1252;;169;;;194;1146;;204;;;229;1173;;239;;;264;1173;;274;;;300;1147;;310;;;335;7943;;352;;;371;8713;;387;;;407;8005;;423;;;443;8021;;458;;;478;;;505;;;525;8111;;541;;;560;8074;;576;;;596;8254;;612;;;632;8514;;647;;;667;8195;;682;;;703;8094;;718;;;738;8123;;754;;;774;8330;;789;;;810;8966;;826;;;846;9066;;862;;;882;10553;;899;;;920;8345;;935;;;955;9219;;;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:01;1267037;DEVELOP: Mem: 45k Buffer: 4k last write time: 58;48.723197;9.094089;495.7;189.86;2.93;1.01;7;3.74;1090;;0;0;58;53;0;8164;;18;;;39;8184;;53;;;74;16305;;98;;;118;8658;;135;;;155;8198;;170;;;190;8133;;205;;;226;8536;;241;;;261;8676;;276;;;296;8516;;314;;;334;8114;;350;;;370;8294;;385;;;405;8751;;422;;;441;8163;;457;;;478;8062;;493;;;513;8093;;528;;;549;8060;;564;;;584;8085;;599;;;619;8071;;634;;;655;8262;;671;;;690;8746;;707;;;726;9116;;742;;;762;;;789;;;808;8121;;825;;;845;8113;;860;;;881;8129;;896;;;916;8096;;932;;;952;10617;;;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:02;1268027;;48.723185;9.094076;496.1;189.86;3.02;1.01;7;3.74;980;;0;0;58;55;0;8173;;18;;;37;8535;;53;;;73;8435;;88;;;109;8592;;124;;;144;8012;;159;;;180;8037;;195;;;215;7975;;230;;;250;7970;;265;;;286;7850;;301;;;321;7861;;336;;;356;7826;;371;;;392;8097;;407;;;427;8467;;443;;;463;7763;;478;;;498;7687;;513;;;534;7950;;549;;;569;7806;;584;;;604;8253;;620;;;640;7753;;656;;;676;8188;;692;;;711;7533;;727;;;747;7791;;763;;;783;7460;;798;;;825;9827;;843;;;863;7432;;878;;;904;7646;;919;;;939;7538;;955;;;974;7508;;;;;;;;;;;;;;;;
 18.11.2020;16:06:03;1269096;;48.723177;9.094068;496.2;189.86;3;1.01;7;3.74;920;;0;0;58;51;0;7218;;19;;;38;8144;;54;;;74;7463;;89;;;110;7856;;125;;;145;7869;;161;;;181;7422;;196;;;216;7934;;232;;;252;7363;;267;;;293;7297;;307;;;332;8105;;348;;;367;7468;;383;;;403;7213;;418;;;439;7172;;454;;;478;7184;;489;;;514;7312;;528;;;550;7175;;565;;;585;7180;;600;;;620;7013;;635;;;655;7154;;670;;;691;7240;;706;;;726;7075;;741;;;761;7133;;776;;;801;7511;;815;;;836;7639;;851;;;872;8891;;888;;;908;7070;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:04;1270033;;48.723167;9.094056;496.6;189.86;3.19;1.01;7;3.74;870;;0;0;58;53;0;7617;;19;;;39;6812;;55;;;80;1173;;90;;;116;8173;;133;;;152;7431;;168;;;188;7197;;203;;;223;6984;;238;;;259;7218;;274;;;294;6881;;309;;;329;7111;;344;;;365;7500;;380;;;400;7462;;415;;;435;7094;;450;;;471;6820;;486;;;506;7147;;521;;;541;9156;;558;;;578;6961;;594;;;614;;;641;;;660;7176;;676;;;696;7177;;712;;;732;7199;;747;;;767;7218;;782;;;802;7360;;817;;;838;;;865;;;884;;;904;;;924;;;943;;;962;7252;;;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:05;1271032;;48.723153;9.094046;496.5;189.86;3.48;1.01;7;3.74;940;;0;0;58;54;0;7295;;14;;;35;7183;;50;;;71;7283;;85;;;106;8957;;122;;;142;8178;;158;;;178;7814;;194;;;213;7495;;229;;;249;7713;;265;;;285;7305;;300;;;320;7654;;335;;;356;7687;;371;;;391;7634;;406;;;426;7167;;441;;;461;;;488;;;508;7245;;524;;;544;7283;;559;;;580;7150;;595;;;615;7194;;630;;;650;7410;;665;;;686;7670;;702;;;721;7421;;737;;;757;7588;;772;;;792;7452;;809;;;828;8162;;844;;;865;9078;;881;;;901;7563;;917;;;936;7775;;952;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:06;1272031;;48.723146;9.094036;496.5;189.86;2.44;1.01;7;3.74;1000;;0;0;58;54;0;8193;;18;;;39;7629;;54;;;74;;;102;;;121;7778;;137;;;157;7773;;172;;;193;7922;;208;;;228;7706;;243;;;263;8881;;280;;;299;7776;;315;;;334;7797;;350;;;370;8683;;386;;;406;7863;;422;;;441;7901;;457;;;477;7747;;492;;;513;8246;;529;;;549;7756;;564;;;585;7667;;600;;;620;7657;;635;;;655;;;682;;;702;8193;;719;;;738;7751;;754;;;774;7731;;789;;;809;8109;;825;;;845;7623;;860;;;880;7883;;895;;;916;7579;;931;;;951;7514;;966;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:07;1273044;;48.723134;9.094026;496.7;189.86;3.44;1.01;7;3.74;990;;0;0;58;53;0;7543;;15;;;37;7535;;52;;;72;9628;;90;;;109;8166;;125;;;146;7469;;161;;;181;7923;;197;;;216;7651;;232;;;252;7594;;267;;;288;7796;;303;;;323;7960;;338;;;359;7862;;373;;;394;7633;;409;;;429;7926;;444;;;465;7661;;479;;;500;7546;;515;;;535;7522;;550;;;570;8461;;587;;;606;7520;;622;;;643;;;668;;;688;7495;;704;;;723;7672;;739;;;759;7964;;774;;;795;8725;;811;;;831;7366;;847;;;867;7586;;882;;;902;8634;;919;;;938;;;;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:08;1274030;;48.723126;9.094013;496.7;230.11;3.54;1.01;7;3.74;850;;0;0;58;53;0;7452;;29;;;49;7446;;65;;;91;1147;;102;;;127;7517;;143;;;163;7411;;178;;;204;1148;;214;;;240;7282;;256;;;281;1201;;291;;;316;1144;;326;;;351;1173;;362;;;386;6718;;401;;;422;7303;;437;;;461;7621;;476;;;497;7557;;511;;;532;7451;;547;;;567;7658;;583;;;603;7534;;618;;;638;7306;;653;;;673;7222;;688;;;709;7169;;724;;;744;7115;;759;;;779;7277;;794;;;815;;;841;;;861;;;881;;;900;7403;;916;;;936;7356;;951;;;972;7030;;;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:09;1275038;;48.723121;9.093994;496.9;237.39;3.33;1.01;7;3.74;730;;0;0;58;54;0;7327;;16;;;36;6876;;52;;;72;6953;;87;;;107;7261;;122;;;142;6702;;158;;;178;7286;;193;;;213;6605;;228;;;249;7168;;264;;;284;6641;;299;;;324;7059;;339;;;359;7568;;374;;;394;6476;;409;;;430;6589;;445;;;470;1174;;480;;;505;1173;;515;;;541;1175;;551;;;576;1149;;586;;;611;6222;;626;;;647;6722;;661;;;687;5939;;700;;;723;5989;;735;;;760;6144;;773;;;795;6019;;808;;;830;6306;;844;;;866;6169;;879;;;901;6288;;914;;;936;9882;;954;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:10;1276038;;48.723117;9.093979;497.4;247.62;2.96;1.01;7;3.74;7;69;0;;0;0;58;52;0;;;30;;;50;;;69;;;89;;;109;;6187;124;9730;;144;;6203;160;14558;;182;;6178;195;;;222;;6233;235;;;257;;6323;275;;;295;;6379;311;;;331;;6371;346;8588;;366;;6330;381;2150;;401;;6275;417;1200;;437;;6184;461;;;488;;6033;505;;;525;;5943;543;2550;;561;;5872;579;2563;;596;;5844;614;1225;;631;;5835;650;1173;;667;;5799;685;2243;;702;;5804;720;2275;;737;;5798;759;;;785;;5854;805;;;825;;5984;840;;;860;;5979;875;;;895;;6027;911;7850;;931;;6001;946;13531;;969;;5957;;;;;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:11;1277041;;48.723117;9.093965;497.5;247.62;2.74;1.01;7;3.74;143;72;0;;0;0;58;48;0;;;30;;5966;43;;;65;;5940;78;12209;;101;;5923;113;;;140;;7918;155;18175;;182;;6159;199;;;226;;6047;239;;;261;;6283;274;;;297;;;328;;;348;;6015;364;;;384;;;418;1174;;428;;6166;453;;;480;;6265;498;;;518;;6241;536;9449;;553;;6311;571;17498;;597;;6394;611;;;638;;6380;652;;;673;;6408;687;;;708;;7059;722;7897;;744;;7059;759;10810;;779;;6459;794;;;822;;6680;841;15140;;864;;;891;6403;;906;;9053;930;10084;;948;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:12;1278045;;48.723109;9.093963;498;247.62;2.17;1.79;7;3.74;143;76;0;;0;0;58;48;0;;;30;;6485;52;;;71;;6321;93;;;113;;6283;128;;;148;;6319;164;10355;;184;;6232;199;17561;;225;;6259;238;;;265;;;292;16478;;317;;;344;8916;;361;;6160;379;;;406;;6242;426;;;445;;6318;461;;;481;;6172;496;;;516;;6271;534;;;554;;6184;571;10174;;590;;6204;607;14878;;630;;6333;643;;;670;;6332;683;;;705;;6231;718;;;740;;6227;753;;;776;;;803;;;822;;6469;844;;;864;;6215;879;;;899;;;927;;;946;;6326;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:13;1279029;;48.723109;9.093963;498;247.62;0;1.79;7;3.74;116;79;0;;0;0;58;47;0;;;20;;6783;41;;;61;;6343;81;;;101;;6506;121;11871;;142;;6364;162;16368;;185;;6365;197;;;224;;;250;15312;;274;;6698;295;12786;;315;;6428;330;;;357;;6556;375;17429;;401;;6426;418;16587;;444;;6539;462;;;488;;;515;18278;;542;;6507;556;;;584;;6506;593;;;619;;6774;632;;;654;;6775;668;;;690;;;717;;;737;;;764;;;784;;6708;799;;;819;;6908;840;8503;;856;;;882;1202;;893;;6729;917;13601;;939;;;974;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:14;1280049;;48.723109;9.093963;498;247.62;0;1.1;7;3.74;121;92;0;;0;0;58;45;0;;;33;;7087;56;;;75;;7170;94;;;113;;7158;129;;;149;;7575;164;;;184;;7233;204;13424;;226;;7616;248;18289;;282;;;315;;;342;;7353;357;;;377;;7521;400;;;419;;7510;435;;;455;;;481;;;500;;7484;516;12940;;537;;;571;8777;;588;;;615;11659;;634;;;660;1174;;670;;;696;;;723;;;742;;;762;;7640;778;;;797;;7980;819;;;839;;8759;855;;;875;;7752;890;11740;;910;;7612;929;12291;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:15;1281028;;48.723109;9.093963;498;247.62;0;1.01;7;3.74;133;95;0;;0;0;58;48;0;;;30;;7564;49;12590;;70;;7576;84;;;111;;7584;120;;;146;;7794;161;;;182;;7826;196;;;217;;;244;;;264;;8057;286;;;305;;7525;321;8033;;340;;7657;360;11295;;380;;;407;9480;;423;;;451;12842;;472;;7540;487;16790;;512;;;538;1175;;549;;;574;1175;;584;;7788;609;1175;;619;;;647;;;673;;7354;697;;;716;;7287;738;;;758;;7454;773;;;793;;7286;812;9244;;829;;7388;848;17471;;874;;7246;893;;;919;;;952;8724;;968;;7324;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 18.11.2020;16:06:16;1282037;;48.723109;9.093963;498;247.62;0;1.01;7;3.74;5;89;20;;0;0;58;49;0;;;31;;7290;44;;;66;;7277;80;;;101;;;128;;;148;;7213;164;;;183;;6901;202;10902;;223;;7060;242;2257;;258;;7057;277;2124;;293;;7045;313;1201;;328;;;361;2137;;371;;6931;396;2055;;407;;6910;432;1201;;442;;;468;2042;;478;;6961;503;1201;;513;;;548;12669;;568;;6909;590;;;617;;7063;636;;;656;;7148;672;;;691;;6777;707;;;727;;6903;747;11631;;767;;;793;1174;;803;;7283;828;;;856;;;889;9154;;908;;7489;929;9129;;943;;7430;965;14679;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 `;
 const test3 = `Date;Time;Millis;Comment;Latitude;Longitude;Altitude;Course;Speed;HDOP;Satellites;BatteryLevel;Left;Right;Confirmed;Marked;Invalid;InsidePrivacyArea;Factor;Measurements;Tms1;Lus1;Rus1;Tms2;Lus2;Rus2;Tms3;Lus3;Rus3;Tms4;Lus4;Rus4;Tms5;Lus5;Rus5;Tms6;Lus6;Rus6;Tms7;Lus7;Rus7;Tms8;Lus8;Rus8;Tms9;Lus9;Rus9;Tms10;Lus10;Rus10;Tms11;Lus11;Rus11;Tms12;Lus12;Rus12;Tms13;Lus13;Rus13;Tms14;Lus14;Rus14;Tms15;Lus15;Rus15;Tms16;Lus16;Rus16;Tms17;Lus17;Rus17;Tms18;Lus18;Rus18;Tms19;Lus19;Rus19;Tms20;Lus20;Rus20;Tms21;Lus21;Rus21;Tms22;Lus22;Rus22;Tms23;Lus23;Rus23;Tms24;Lus24;Rus24;Tms25;Lus25;Rus25;Tms26;Lus26;Rus26;Tms27;Lus27;Rus27;Tms28;Lus28;Rus28;Tms29;Lus29;Rus29;Tms30;Lus30;Rus30;Tms31;Lus31;Rus31;Tms32;Lus32;Rus32;Tms33;Lus33;Rus33;Tms34;Lus34;Rus34;Tms35;Lus35;Rus35;Tms36;Lus36;Rus36;Tms37;Lus37;Rus37;Tms38;Lus38;Rus38;Tms39;Lus39;Rus39;Tms40;Lus40;Rus40;Tms41;Lus41;Rus41;Tms42;Lus42;Rus42;Tms43;Lus43;Rus43;Tms44;Lus44;Rus44;Tms45;Lus45;Rus45;Tms46;Lus46;Rus46;Tms47;Lus47;Rus47;Tms48;Lus48;Rus48;Tms49;Lus49;Rus49;Tms50;Lus50;Rus50;Tms51;Lus51;Rus51;Tms52;Lus52;Rus52;Tms53;Lus53;Rus53;Tms54;Lus54;Rus54;Tms55;Lus55;Rus55;Tms56;Lus56;Rus56;Tms57;Lus57;Rus57;Tms58;Lus58;Rus58;Tms59;Lus59;Rus59;Tms60;Lus60;Rus60;
 21.11.2020;14:27:00;66890;;;;;;;3.83;4;3.99;;286;0;;0;0;58;5;0;;;41;;18355;67;;;87;;18374;113;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
 `;
 module.exports = { test1, test2, test3 };
--- a/api/src/logic/tracks.js
+++ b/api/src/logic/tracks.js
@ -0,0 +1,312 @@
 const csvParse = require('csv-parse/lib/sync');
 const csvStringify = require('csv-stringify/lib/sync');
 function _parseFloat(token) {
  if (typeof token !== 'string') {
    return null;
  }
  token = token.trim();
  if (token === '') {
    return null;
  }
  if (/^nan$/i.test(token)) {
    return null;
  }
  let f = parseFloat(token);
  if (isNaN(f)) {
    f = parseFloat(token.substring(0, 10));
  }
  if (isNaN(f)) {
    f = 0.0;
  }
  return f;
 }
 function _parseInt(token) {
  const asFloat = _parseFloat(token);
  if (asFloat !== null) {
    return Math.floor(asFloat);
  } else {
    return asFloat;
  }
 }
 function _parseString(token) {
  if (typeof token !== 'string') {
    return null;
  }
  // This time we do not trim -- because we assume that the quoting mechanism
  // from CSV might have kicked in and we actually want the spacing around the
  // token.
  if (token === '') {
    return null;
  }
  return token;
 }
 function replaceDollarNewlinesHack(body) {
  // see if we are using the hack with $ as newlines, replace them for the csv parser
  if (body.endsWith('$') || /insidePrivacyArea;\$/.test(body)) {
    return body.replace(/\$/g, '\n');
  }
  return body;
 }
 function* parseTrackPoints(body, format = null) {
  if (body instanceof Buffer) {
    body = body.toString('utf-8')
  }
  body = replaceDollarNewlinesHack(body);
  const detectedFormat = format != null ? format : detectFormat(body);
  let parser;
  switch (detectedFormat) {
    case 'invalid':
      throw new Error('track format cannot be detected');
    case 1:
      parser = parseObsver1;
      break;
    case 2:
      parser = parseObsver2;
      break;
  }
  yield* parser(body);
 }
 function detectFormat(body) {
  body = replaceDollarNewlinesHack(body);
  if (!body.length) {
    return 'invalid';
  }
  const firstLinebreakIndex = body.indexOf('\n');
  if (firstLinebreakIndex === -1) {
    // We need at least one linebreak in the whole file, to separate header and
    // data. If the file contains no header, it is in valid.
    return 'invalid';
  }
  const firstLine = body.substring(0, firstLinebreakIndex);
  const match = firstLine.match(/(^|&)OBSDataFormat=([\d]+)($|&)/);
  if (match) {
    return Number(match[2]);
  }
  // If we have no metadata line, but start immediately with a header, AND it contains
  // `;Rus`, it is a version 2
  if (/^Date;Time.*;Rus/.test(firstLine)) {
    return 2;
  }
  // If we have no metadata line, but start immediately with a header, it is
  // format version 1.
  if (/^Date;Time/.test(firstLine)) {
    return 1;
  }
  // If we immediately start with data (a date, formatted as DD.MM.YYYY), then
  // we have an old OBS not sending the header. It must therefore be old
  // format, too.
  if (/^[0-9]{2}\.[0-9]{2}\.[0-9]{4};/.test(firstLine)) {
    return 1;
  }
  return 'invalid';
 }
 function* parseObsver1(body) {
  for (const record of csvParse(body, {
    delimiter: ';',
    encoding: 'utf8',
    // We specify  different column names here, as the order of columns was
    // always the same, but their naming was different. By enforicing these
    // column names we don't have to translate between them. Then we just
    // ignore the first line (or any line that starts with "Date;").
    // Original header usually is:
    // Date;Time;Latitude;Longitude;Course;Speed;Right;Left;Confirmed;insidePrivacyArea
    columns: ['date', 'time', 'latitude', 'longitude', 'course', 'speed', 'd1', 'd2', 'flag', 'private'],
    relax_column_count: true,
    cast(value, { column }) {
      if (['latitude', 'longitude', 'course', 'speed'].includes(column)) {
        return _parseFloat(value);
      } else if (['d1', 'd2', 'flag'].includes(column)) {
        return _parseInt(value);
      } else if (column === 'private') {
        return Boolean(_parseInt(value));
      } else {
        return _parseString(value);
      }
    },
  })) {
    if (record.date === 'Date') {
      // ignore header line
      continue;
    }
    if (!record.latitude && !record.longitude) {
      // invalid record, make sure lat/lng say `null` instead of `0`
      record.latitude = null;
      record.longitude = null;
    }
    // in old format, 255 or 999 means "no measurement"
    if (record.d1 === 255 || record.d1 === 999) {
      record.d1 = null;
    }
    if (record.d2 === 255 || record.d2 === 999) {
      record.d2 = null;
    }
    yield record;
  }
 }
 function* parseObsver2(body) {
  for (const record of csvParse(body, {
    from_line: 2,
    trim: true,
    columns: true,
    skip_empty_lines: true,
    delimiter: ';',
    encoding: 'utf8',
    relax_column_count: true,
    cast(value, context) {
      if (value === '') {
        return null;
      }
      let type;
      switch (context.column) {
        case 'Millis':
        case 'Left':
        case 'Right':
        case 'Confirmed':
        case 'Invalid':
        case 'InsidePrivacyArea':
        case 'Measurements':
        case 'Satellites':
          type = 'int';
          break;
        case 'Date':
        case 'Time':
        case 'Comment':
        case 'Marked':
          type = 'string';
          break;
        case 'Latitude':
        case 'Longitude':
        case 'Altitude':
        case 'Course':
        case 'Speed':
        case 'HDOP':
        case 'BatteryLevel':
        case 'Factor':
          type = 'float';
          break;
        default:
          type = /^(Tms|Lus|Rus)/.test(context.column) ? 'int' : 'string';
      }
      switch (type) {
        case 'int':
          return _parseInt(value);
        case 'float':
          return _parseFloat(value);
        case 'string':
          return _parseString(value);
      }
    },
  })) {
    // We convert the new format back to the old format for storage here, until
    // we upgrade the storage format as well to include all data. But we'll
    // have to upgrade the obsApp first.
    yield {
      date: record.Date,
      time: record.Time,
      latitude: record.Latitude,
      longitude: record.Longitude,
      course: record.Course,
      speed: record.Speed,
      d1: record.Left,
      d2: record.Right,
      flag: Boolean(record.Confirmed),
      private: Boolean(record.InsidePrivacyArea),
    };
  }
 }
 /**
 * This function normalizes a User-Agent header for storage in the database. It
 * make sure that we only store the user-agent if it matches the pattern
 * `OBS/*`, and extracts that part of the user agent, if it contains more
 * information. This is the only part we are interested in, the
 * remainder is too privacy sensitive to keep.
 */
 function normalizeUserAgent(userAgent) {
  if (!userAgent) {
    return null;
  }
  const match = userAgent.match(/\bOBS\/[^\s]+/);
  if (match) {
    return match[0];
  }
  return null;
 }
 function buildObsver1(points) {
  return csvStringify(points, {
    columns: [
      { key: 'date', header: 'Date' },
      { key: 'time', header: 'Time' },
      { key: 'latitude', header: 'Latitude' },
      { key: 'longitude', header: 'Longitude' },
      { key: 'course', header: 'Course' },
      { key: 'speed', header: 'Speed' },
      { key: 'd1', header: 'Right' },
      { key: 'd2', header: 'Left' },
      { key: 'flag', header: 'Confirmed' },
      { key: 'private', header: 'insidePrivacyArea' },
    ],
    cast: {
      boolean: (v) => (v ? '1' : '0'),
    },
    delimiter: ';',
    header: true,
  });
 }
 module.exports = {
  detectFormat,
  normalizeUserAgent,
  parseObsver1,
  parseObsver2,
  parseTrackPoints,
  replaceDollarNewlinesHack,
  buildObsver1,
 };
--- a/api/src/logic/tracks.test.js
+++ b/api/src/logic/tracks.test.js
@ -0,0 +1,171 @@
 const {
  buildObsver1,
  detectFormat,
  normalizeUserAgent,
  parseObsver1,
  parseObsver2,
  parseTrackPoints,
  replaceDollarNewlinesHack,
 } = require('./tracks');
 const { test1, test2, test3 } = require('./_tracks_testdata');
 describe('parseTrackPoints', () => {
  it('is a function', () => {
    expect(typeof parseTrackPoints).toBe('function');
  });
  it('works on the sample data with an empty track', () => {
    const points = Array.from(parseTrackPoints(test1));
    expect(points).toHaveLength(324);
    expect(points[0]).toEqual({
      date: '12.07.2020',
      time: '09:02:59',
      latitude: null,
      longitude: null,
      course: 0,
      speed: 0,
      d1: null,
      d2: null,
      flag: 0,
      private: false,
    });
  });
 });
 describe('parseObsver1', () => {
  it('can parse sample data', () => {
    const points = Array.from(parseObsver1(replaceDollarNewlinesHack(test1)));
    expect(points).toHaveLength(324);
    expect(points[0]).toEqual({
      date: '12.07.2020',
      time: '09:02:59',
      latitude: null,
      longitude: null,
      course: 0,
      speed: 0,
      d1: null,
      d2: null,
      flag: 0,
      private: false,
    });
  });
 });
 describe('parseObsver2', () => {
  it('can parse sample data', () => {
    const points = Array.from(parseObsver2(test2));
    expect(points).toHaveLength(18);
    expect(points[0]).toEqual({
      date: '18.11.2020',
      time: '16:05:59',
      latitude: 48.723224,
      longitude: 9.094103,
      course: 189.86,
      speed: 3.2,
      d1: 770,
      d2: null,
      flag: false,
      private: true,
    });
    // this is a non-private, flagged point (i.e. "Confirmed" overtaking)
    expect(points[17]).toEqual({
      date: '18.11.2020',
      time: '16:06:16',
      latitude: 48.723109,
      longitude: 9.093963,
      course: 247.62,
      speed: 0,
      d1: 5,
      d2: 89,
      flag: true,
      private: false,
    });
  });
 });
 describe('detectFormat', () => {
  it('detects format 1', () => {
    expect(detectFormat(test1)).toBe(1);
  });
  it('detects format 2', () => {
    expect(detectFormat(test2)).toBe(2);
    expect(detectFormat(test3)).toBe(2);
  });
  it('detects invalid format', () => {
    expect(detectFormat('foobar\nbaz')).toBe('invalid');
    expect(detectFormat('')).toBe('invalid');
  });
 });
 describe('normalizeUserAgent', () => {
  it('is a function', () => {
    expect(typeof normalizeUserAgent).toBe('function');
  });
  it('ignores falsy values', () => {
    expect(normalizeUserAgent(null)).toBe(null);
    expect(normalizeUserAgent('')).toBe(null);
  });
  it('ignores normal browser agents', () => {
    const browserAgents = [
      'Mozilla/5.0 (Linux; Android 6.0.1; Nexus 6P Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.83 Mobile Safari/537.36',
      'Mozilla/5.0 (Linux; Android 6.0; HTC One M9 Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.98 Mobile Safari/537.3',
      'Mozilla/5.0 (Linux; Android 8.0.0; SM-G960F Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36',
      'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A5370a Safari/604.1',
      'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1',
    ];
    for (const browserAgent of browserAgents) {
      expect(normalizeUserAgent(browserAgent)).toBe(null);
    }
  });
  it('detects OBS versions', () => {
    const agents = ['OBS/123', 'OBS/2', 'OBS/1.2.3.4.5-rc123'];
    for (const agent of agents) {
      expect(normalizeUserAgent(agent)).toBe(agent);
    }
  });
  it('extracts OBS versions from extended formats', () => {
    const agents = ['foo OBS/123', 'OBS/123 bar', 'foo OBS/123 bar'];
    for (const agent of agents) {
      expect(normalizeUserAgent(agent)).toBe('OBS/123');
    }
  });
 });
 describe('buildObsver1', () => {
  it('is a function', () => {
    expect(typeof normalizeUserAgent).toBe('function');
  });
  it('transforms properly back and forth', () => {
    const inputString = replaceDollarNewlinesHack(test1);
    const points1 = Array.from(parseObsver1(inputString));
    const builtString = buildObsver1(points1);
    const points2 = Array.from(parseObsver1(builtString));
    expect(points2).toEqual(points1);
  });
  it('produces a header', () => {
    const builtString = buildObsver1([]);
    expect(builtString).toBe('Date;Time;Latitude;Longitude;Course;Speed;Right;Left;Confirmed;insidePrivacyArea\n');
  });
  it('produces empty rows', () => {
    const builtString = buildObsver1([{}]);
    expect(builtString).toBe(
      'Date;Time;Latitude;Longitude;Course;Speed;Right;Left;Confirmed;insidePrivacyArea\n;;;;;;;;;\n',
    );
  });
 });
--- a/api/src/models/AccessToken.js
+++ b/api/src/models/AccessToken.js
@ -0,0 +1,49 @@
 const mongoose = require('mongoose');
 const uniqueValidator = require('mongoose-unique-validator');
 const crypto = require('crypto');
 const schema = new mongoose.Schema(
  {
    token: { index: true, type: String, required: true, unique: true },
    user: { type: mongoose.Schema.Types.ObjectId, ref: 'User', required: true },
    clientId: { type: String, required: true },
    expiresAt: { type: Date, required: true },
    scope: { type: String, required: true, defaultValue: '*' },
  },
  { timestamps: true },
 );
 schema.plugin(uniqueValidator, { message: 'reused token' });
 class AccessTokenClass extends mongoose.Model {
  toJSON() {
    return {
      token: this.token,
      expires: this.expires,
    };
  }
  isValid() {
    return this.expiresAt < new Date();
  }
  toHeaderString() {
    return 'Bearer ' + this.token;
  }
  static generate(options, expiresInSeconds = 24 * 60 * 60) {
    const token = crypto.randomBytes(32).toString('hex');
    return new AccessToken({
      ...options,
      token,
      expiresAt: new Date(new Date().getTime() + 1000 * expiresInSeconds),
    });
  }
 }
 schema.loadClass(AccessTokenClass);
 const AccessToken = mongoose.model('AccessToken', schema);
 module.exports = AccessToken;
--- a/api/src/models/AuthorizationCode.js
+++ b/api/src/models/AuthorizationCode.js
@ -0,0 +1,32 @@
 const mongoose = require('mongoose');
 const crypto = require('crypto');
 const schema = new mongoose.Schema(
  {
    code: { type: String, unique: true, required: true },
    user: { type: mongoose.Schema.Types.ObjectId, ref: 'User', required: true },
    clientId: { type: String, required: true },
    scope: { type: String, required: true, defaultValue: '*' },
    redirectUri: { type: String, required: true },
    expiresAt: { type: Date, required: true },
    codeChallenge: { type: String, required: true }, // no need to store the method, it is always "S256"
  },
  { timestamps: true },
 );
 class AuthorizationCodeClass extends mongoose.Model {
  static generate(options, expiresInSeconds = 60) {
    const code = crypto.randomBytes(8).toString('hex');
    return new AuthorizationCode({
      ...options,
      code,
      expiresAt: new Date(new Date().getTime() + 1000 * expiresInSeconds),
    });
  }
 }
 schema.loadClass(AuthorizationCodeClass)
 const AuthorizationCode = mongoose.model('AuthorizationCode', schema);
 module.exports = AuthorizationCode
--- a/api/src/models/Comment.js
+++ b/api/src/models/Comment.js
@ -0,0 +1,24 @@
 const mongoose = require('mongoose');
 const schema = new mongoose.Schema(
  {
    body: String,
    author: { type: mongoose.Schema.Types.ObjectId, ref: 'User' },
    track: { type: mongoose.Schema.Types.ObjectId, ref: 'Track' },
  },
  { timestamps: true },
 );
 class CommentClass extends mongoose.Model {
  toJSONFor(user) {
    return {
      id: this._id,
      body: this.body,
      createdAt: this.createdAt,
      author: this.author.toProfileJSONFor(user),
    };
  }
 }
 schema.loadClass(CommentClass)
 module.exports = mongoose.model('Comment', schema);
--- a/api/src/models/RefreshToken.js
+++ b/api/src/models/RefreshToken.js
@ -0,0 +1,46 @@
 const mongoose = require('mongoose');
 const uniqueValidator = require('mongoose-unique-validator');
 const crypto = require('crypto');
 const AccessToken = require('./AccessToken')
 const schema = new mongoose.Schema(
  {
    token: { index: true, type: String, required: true, unique: true },
    user: { type: mongoose.Schema.Types.ObjectId, ref: 'User', required: true },
    clientId: { type: String, required: true },
    expiresAt: { type: Date, required: false },
    scope: { type: String, required: true, defaultValue: '*' },
  },
  { timestamps: true },
 );
 schema.plugin(uniqueValidator, { message: 'reused token' });
 class RefreshTokenClass extends mongoose.Model {
  toJSON() {
    return {
      token: this.token,
      expires: this.expires,
    };
  }
  isValid() {
    return this.expiresAt == null || this.expiresAt < new Date()
  }
  static generate(options, expiresInSeconds = 24 * 60 * 60) {
    const token = crypto.randomBytes(32).toString('hex');
    return new RefreshToken({
      ...options,
      token,
      expiresAt: new Date(new Date().getTime() + 1000 * expiresInSeconds),
    });
  }
 }
 schema.loadClass(RefreshTokenClass);
 const RefreshToken = mongoose.model('RefreshToken', schema);
 module.exports = RefreshToken
--- a/api/src/models/Track.js
+++ b/api/src/models/Track.js
@ -0,0 +1,306 @@
 const crypto = require('crypto');
 const mongoose = require('mongoose');
 const _ = require('lodash');
 const uniqueValidator = require('mongoose-unique-validator');
 const { DateTime } = require('luxon');
 const slug = require('slug');
 const path = require('path');
 const sanitize = require('sanitize-filename');
 const fs = require('fs');
 const uuid = require('uuid/v4');
 const { TRACKS_DIR } = require('../paths');
 const queue = require('../queue');
 const statisticsSchema = new mongoose.Schema(
  {
    recordedAt: Date,
    recordedUntil: Date,
    duration: Number,
    length: Number,
    segments: Number,
    numEvents: Number,
    numMeasurements: Number,
    numValid: Number,
  },
  { timestamps: false },
 );
 const schema = new mongoose.Schema(
  {
    // A (partially or entirely random generated) string that can be used as a
    // public identifier
    slug: { type: String, lowercase: true, unique: true },
    // The title for this track.
    title: String,
    // The status of this track, whether it is to be processed, is currently
    // being processed, or has completed or errored.
    processingStatus: {
      type: String,
      enum: ['pending', 'processing', 'complete', 'error'],
      default: 'pending',
    },
    processingJobId: String,
    // Output from the proccessing routines regarding this track. Might be
    // displayed to the owner or administrators to help in debugging. Should be
    // set to `null` if no processing has not been finished.
    processingLog: String,
    // Set to true if the user customized the title. Disables auto-generating
    // an updated title when the track is (re-)processed.
    customizedTitle: { type: Boolean, default: false },
    // A user-provided description of the track. May contain markdown.
    description: String,
    // Whether this track is visible (anonymized) in the public track list or not.
    public: { type: Boolean, default: false },
    // Whether this track should be exported to the public track database
    // (after anonymization).
    includeInPublicDatabase: { type: Boolean, default: false },
    // The user agent string, or a part thereof, that was used to upload this
    // track. Usually contains only the OBS version, other user agents are
    // discarded due to being irrelevant.
    uploadedByUserAgent: String,
    // The name of the original file, as provided during upload. Used for
    // providing a download with the same name, and for display in the
    // frontend.
    originalFileName: {
      type: String,
      required: true,
      validate: {
        validator: function (v) {
          // Must be a sane filename, i.e. not change when being sanitized
          return sanitize(v) === v && v.length > 0 && /.+\.csv$/i.test(v);
        },
        message: (props) => `${props.value} is not a valid filename`,
      },
    },
    // A hash of the original file's contents. Nobody can upload the same track twice.
    originalFileHash: {
      type: String,
      required: true,
    },
    // Where the files are stored, relative to a group directory like
    // TRACKS_DIR or PROCESSING_DIR.
    filePath: String,
    comments: [{ type: mongoose.Schema.Types.ObjectId, ref: 'Comment' }],
    author: { type: mongoose.Schema.Types.ObjectId, ref: 'User' },
    statistics: statisticsSchema,
  },
  { timestamps: true },
 );
 schema.index({ author: 1, originalFileHash: 1 }, { unique: true });
 schema.plugin(uniqueValidator, { message: 'is already taken' });
 schema.pre('validate', async function (next) {
  try {
    if (!this.slug) {
      this.slugify();
    }
    if (!this.filePath) {
      await this.generateFilePath();
    }
    next();
  } catch (err) {
    next(err);
  }
 });
 // 0..4 Night, 4..10 Morning, 10..14 Noon, 14..18 Afternoon, 18..22 Evening, 22..00 Night
 // Two hour intervals
 const DAYTIMES = [
  'Night', // 0h - 2h
  'Night', // 2h - 4h
  'Morning', // 4h - 6h
  'Morning', // 6h - 8h
  'Morning', // 8h - 10h
  'Noon', // 10h - 12h
  'Noon', // 12h - 14h
  'Afternoon', // 14h - 16h
  'Afternoon', // 16h - 18h
  'Evening', // 18h - 20h
  'Evening', // 20h - 22h
  'Night', // 22h - 24h
 ];
 function getDaytime(dateTime) {
  return DAYTIMES[Math.floor((dateTime.hour % 24) / 2)];
 }
 class TrackClass extends mongoose.Model {
  slugify() {
    this.slug = slug(this.title || 'track') + '-' + ((Math.random() * Math.pow(36, 6)) | 0).toString(36);
  }
  async generateFilePath() {
    await this.populate('author');
    this.filePath = path.join(this.author.username, this.slug);
  }
  isVisibleTo(user) {
    if (this.public) {
      return true;
    }
    if (!user) {
      return false;
    }
    if (user._id.equals(this.author._id)) {
      return true;
    }
    return false;
  }
  isVisibleToPrivate(user) {
    return user && user._id.equals(this.author._id);
  }
  async _ensureDirectoryExists() {
    if (!this.filePath) {
      await this.generateFilePath();
    }
    const dir = path.dirname(this.getOriginalFilePath());
    await fs.promises.mkdir(dir, { recursive: true });
  }
  getOriginalFilePath() {
    if (!this.filePath) {
      throw new Error('Cannot get original file path, `filePath` is not yet set. Call `generateFilePath()` first.');
    }
    return path.join(TRACKS_DIR, this.filePath, 'original.csv');
  }
  async writeToOriginalFile(fileBody) {
    await this._ensureDirectoryExists();
    await fs.promises.writeFile(this.getOriginalFilePath(), fileBody);
  }
  async validateFileBodyUniqueness(fileBody) {
    // Generate hash
    const hash = crypto.createHash('sha512').update(fileBody).digest('hex');
    const existingTracks = await Track.find({ originalFileHash: hash, author: this.author });
    if (existingTracks.length === 0 || (existingTracks.length === 1 && existingTracks[0]._id.equals(this._id))) {
      this.originalFileHash = hash;
      return;
    }
    throw new Error('Track file already uploaded.');
  }
  /**
   * Marks this track as needing processing.
   *
   * Also deletes all stored information that is derived during processing from
   * the database, such that it may be filled again with correct information
   * during the processing operation.
   *
   * Saves the track as well, so it is up to date when the worker receives it.
   */
  async queueProcessing() {
    this.processingStatus = 'pending';
    this.processingLog = null;
    this.processingJobId = uuid();
    this.statistics = null;
    await this.save();
    await queue.add(
      'processTrack',
      {
        trackId: this._id.toString(),
      },
      {
        jobId: this.processingJobId,
      },
    );
  }
  async readProcessingResults(success = true) {
    // Copies some information into this object from the outputs of the
    // processing step. This allows general statistics to be formed, and other
    // information to be displayed, without having to read individual files
    // from disk. Each field set here should be unsed in `queueProcessing`.
    // This routine also moves the `processingStatus` along.
  }
  async autoGenerateTitle() {
    if (this.customizedTitle) {
      return;
    }
    // Try to figure out when this file was recorded. Either we have it in then
    // statistics, e.g. after parsing and processing the track, or we can maybe
    // derive it from the filename.
    let recordedAt = null;
    if (this.statistics && this.statistics.recordedAt != null) {
      recordedAt = DateTime.fromJSDate(this.statistics.recordedAt);
    } else if (this.originalFileName) {
      const match = this.originalFileName.match(/^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}\.[0-9]{2}\.[0-9]{2}/);
      if (match) {
        recordedAt = DateTime.fromFormat(match[0], "yyyy-MM-dd'T'HH.mm.ss");
        if (!recordedAt.isValid) {
          recordedAt = null;
        }
      }
    }
    if (recordedAt) {
      const daytime = getDaytime(recordedAt);
      this.title = `${daytime} ride on ${recordedAt.toLocaleString(recordedAt.DATE_MED)}`;
      await this.save();
      return;
    }
    // Detecting recording date failed, use filename
    if (this.originalFileName) {
      this.title = _.upperFirst(_.words(this.originalFileName.replace(/(\.obsdata)?\.csv$/, '')).join(' '));
      await this.save();
    }
  }
  toJSONFor(user) {
    const includePrivateFields = user && user._id.equals(this.author._id);
    return {
      slug: this.slug,
      title: this.title,
      description: this.description,
      createdAt: this.createdAt,
      updatedAt: this.updatedAt,
      public: this.public,
      author: this.author.toProfileJSONFor(user),
      statistics: this.statistics,
      processingStatus: this.processingStatus,
      ...(includePrivateFields
        ? {
            uploadedByUserAgent: this.uploadedByUserAgent,
            originalFileName: this.originalFileName,
          }
        : {}),
    };
  }
 }
 schema.loadClass(TrackClass);
 const Track = mongoose.model('Track', schema);
 module.exports = Track;
--- a/api/src/models/User.js
+++ b/api/src/models/User.js
@ -0,0 +1,91 @@
 const mongoose = require('mongoose');
 const uniqueValidator = require('mongoose-unique-validator');
 const crypto = require('crypto');
 const jwt = require('jsonwebtoken');
 const config = require('../config')
 const schema = new mongoose.Schema(
  {
    username: {
      type: String,
      lowercase: true,
      unique: true,
      required: [true, "can't be blank"],
      match: [/^[a-zA-Z0-9]+$/, 'is invalid'],
      index: true,
    },
    email: {
      type: String,
      lowercase: true,
      unique: true,
      required: [true, "can't be blank"],
      match: [/\S+@\S+\.\S+/, 'is invalid'],
      index: true,
    },
    bio: String,
    image: String,
    areTracksVisibleForAll: Boolean,
    hash: String,
    salt: String,
    needsEmailValidation: Boolean,
    verificationToken: String,
    resetToken: {
      token: String,
      expires: Date,
    },
  },
  { timestamps: true },
 );
 schema.plugin(uniqueValidator, { message: 'ist bereits vergeben. Sorry!' });
 class UserClass extends mongoose.Model {
  validPassword(password) {
    const hash = crypto.pbkdf2Sync(password, this.salt, 10000, 512, 'sha512').toString('hex');
    return this.hash === hash;
  }
  setPassword(password) {
    this.salt = crypto.randomBytes(16).toString('hex');
    this.hash = crypto.pbkdf2Sync(password, this.salt, 10000, 512, 'sha512').toString('hex');
  }
  generateJWT() {
    const today = new Date();
    const exp = new Date(today);
    exp.setDate(today.getDate() + 60);
    return jwt.sign(
      {
        id: this._id,
        username: this.username,
        exp: parseInt(exp.getTime() / 1000),
      },
      config.jwtSecret,
    );
  }
  toAuthJSON() {
    return {
      username: this.username,
      email: this.email,
      token: this.generateJWT(),
      bio: this.bio,
      image: this.image,
      areTracksVisibleForAll: this.areTracksVisibleForAll,
      apiKey: this._id,
    };
  }
  toProfileJSONFor(user) {
    return {
      username: this.username,
      bio: this.bio,
      image: this.image,
    };
  }
 }
 schema.loadClass(UserClass);
 const User = mongoose.model('User', schema);
 module.exports = User
--- a/api/src/models/index.js
+++ b/api/src/models/index.js
@ -0,0 +1,6 @@
 module.exports.AccessToken = require('./AccessToken')
 module.exports.AuthorizationCode = require('./AuthorizationCode')
 module.exports.Comment = require('./Comment')
 module.exports.RefreshToken = require('./RefreshToken')
 module.exports.Track = require('./Track')
 module.exports.User = require('./User')
--- a/api/src/passport.js
+++ b/api/src/passport.js
@ -0,0 +1,215 @@
 const passport = require('passport');
 const { Strategy: LocalStrategy } = require('passport-local');
 const { Strategy: BearerStrategy } = require('passport-http-bearer');
 const { Strategy: JwtStrategy } = require('passport-jwt');
 const { Strategy: CustomStrategy } = require('passport-custom');
 const { User, AccessToken, RefreshToken } = require('./models');
 const config = require('./config');
 // used to serialize the user for the session
 passport.serializeUser(function (user, done) {
  done(null, user._id);
 });
 // used to deserialize the user
 passport.deserializeUser(function (id, done) {
  User.findById(id, function (err, user) {
    done(err, user);
  });
 });
 async function loginWithPassword(email, password, done) {
  try {
    const user = await User.findOne({ email: email });
    if (!user || !user.validPassword(password)) {
      return done(new Error('invalid credentials'), false);
    }
    // Regardless of whether login is required, if you're logged in as an
    // unverified user, produce an error.
    if (user.needsEmailValidation) {
      return done(new Error('email not verified'), false);
    }
    return done(null, user);
  } catch (err) {
    done(err);
  }
 }
 passport.use(
  'usernameAndPassword',
  new LocalStrategy(
    {
      usernameField: 'user[email]',
      passwordField: 'user[password]',
      session: false,
    },
    loginWithPassword,
  ),
 );
 passport.use(
  'usernameAndPasswordSession',
  new LocalStrategy(
    {
      usernameField: 'email',
      passwordField: 'password',
      session: true,
    },
    loginWithPassword,
  ),
 );
 function getRequestToken(req, tokenTypes = ['Token', 'Bearer']) {
  const authorization = req.headers.authorization;
  if (typeof authorization !== 'string') {
    return null;
  }
  const [tokenType, token] = authorization.split(' ');
  if (tokenTypes.includes(tokenType)) {
    return token;
  }
  return null;
 }
 passport.use(
  'jwt',
  new JwtStrategy(
    {
      secretOrKey: config.jwtSecret,
      jwtFromRequest: getRequestToken,
      algorithms: ['HS256'],
    },
    async function (token, done) {
      try {
        // we used to put the user ID into the token directly :(
        const { id } = token;
        const user = await User.findById(id);
        return done(null, user || false);
      } catch (err) {
        return done(err);
      }
    },
  ),
 );
 passport.use(
  'accessToken',
  new BearerStrategy(async function (token, done) {
    try {
      const accessToken = await AccessToken.findOne({ token }).populate('user');
      if (accessToken && accessToken.user) {
        // TODO: scope
        return done(null, accessToken.user, { scope: accessToken.scope });
      } else {
        return done(null, false);
      }
    } catch (err) {
      return done(err);
    }
  }),
 );
 passport.use(
  'refreshToken',
  new BearerStrategy(async function (token, done) {
    try {
      const refreshToken = await RefreshToken.findOne({ token }).populate('user');
      if (refreshToken && refreshToken.user) {
        // TODO: scope
        return done(null, refreshToken.user, { scope: 'auth.refresh' });
      } else {
        return done(null, false);
      }
    } catch (err) {
      return done(err);
    }
  }),
 );
 passport.use(
  'userId',
  new CustomStrategy(async (req, callback) => {
    try {
      let userId;
      const headerToken = getRequestToken(req, ['OBSUserId']);
      if (headerToken && headerToken.length === 24) {
        userId = headerToken;
      }
      if (!userId) {
        const bodyId = req.body && req.body.id;
        if (bodyId && bodyId.length === 24) {
          userId = bodyId;
        }
      }
      let user;
      if (userId) {
        user = await User.findById(userId);
      }
      callback(null, user || false);
    } catch (err) {
      callback(err);
    }
  }),
 );
 /**
 * This function creates a middleware that does a passport authentication.
 */
 function createMiddleware(strategies, required = true, session = false) {
  return (req, res, next) => {
    passport.authenticate(strategies, { session }, (err, user, info) => {
      // If this authentication produced an error, throw it. In a chain of
      // multiple strategies, errors are ignored, unless every strategy errors.
      if (err) {
        return next(err);
      }
      // If you *must* be logged in for this action, require a user.
      if (required && !user) {
        return res.sendStatus(403);
      }
      // Regardless of whether login is required, if you're logged in as an
      // unverified user, produce an error.
      if (user && user.needsEmailValidation) {
        return res.status(403).json({ errors: { 'E-Mail-Bestätigung': 'noch nicht erfolgt' } });
      }
      req.user = user;
      req.scope = (info && info.scope) || '*';
      return next();
    })(req, res, next);
  };
 }
 module.exports = {
  // these are the standard authentication mechanisms, for when you want user
  // information in the route, and either require a login, or don't care
  optional: createMiddleware(['jwt', 'accessToken'], false),
  required: createMiddleware(['jwt', 'accessToken'], true),
  // required to check username and passwort for generating a new token, e.g.
  // on the /users/login route, and later on oauth routes
  usernameAndPassword: createMiddleware('usernameAndPassword', true),
  usernameAndPasswordSession: createMiddleware('usernameAndPasswordSession', false, true),
  // will be used to verify a refresh token on the route that will exchange the
  // refresh token for a new access token (not in use yet)
  refreshToken: createMiddleware('refreshToken', true),
  // for track upload, we still allow "userId" for a while
  requiredWithUserId: createMiddleware(['jwt', 'accessToken', 'userId'], true),
 };
--- a/api/src/paths.js
+++ b/api/src/paths.js
@ -0,0 +1,29 @@
 const path = require('path');
 const API_ROOT_DIR = path.resolve(__dirname, '../');
 const DATA_DIR = process.env.DATA_DIR || path.resolve(__dirname, '../../data/');
 // Contains the subtree for processing files
 const PROCESSING_DIR = path.join(DATA_DIR, 'processing');
 const PROCESSING_OUTPUT_DIR = path.join(DATA_DIR, 'processing-output');
 // Contains the subtree for processing files, without privatization techniques,
 // used only for display of tracks to authors
 const PROCESSING_DIR_PRIVATE = path.join(DATA_DIR, 'private');
 // Contains original track files
 const TRACKS_DIR = path.join(DATA_DIR, 'tracks');
 // Cache directory for all obs-face calls
 const OBS_FACE_CACHE_DIR = path.join(DATA_DIR, 'obs-face-cache');
 module.exports = {
  API_ROOT_DIR,
  DATA_DIR,
  PROCESSING_DIR,
  PROCESSING_OUTPUT_DIR,
  PROCESSING_DIR_PRIVATE,
  TRACKS_DIR,
  OBS_FACE_CACHE_DIR,
 };
--- a/api/src/process_track.py
+++ b/api/src/process_track.py
@ -0,0 +1,188 @@
 import argparse
 import logging
 import os
 import tempfile
 import json
 from obs.face.importer import ImportMeasurementsCsv
 from obs.face.annotate import AnnotateMeasurements
 from obs.face.filter import (
    AnonymizationMode,
    ChainFilter,
    ConfirmedFilter,
    DistanceMeasuredFilter,
    PrivacyFilter,
    PrivacyZone,
    PrivacyZonesFilter,
    RequiredFieldsFilter,
 )
 from obs.face.osm import DataSource as OSMDataSource
 log = logging.getLogger(__name__)
 def main():
    logging.basicConfig(level=logging.DEBUG, format="%(levelname)s: %(message)s")
    parser = argparse.ArgumentParser(
        description="processes a single track for use in the portal, "
        "using the obs.face algorithms"
    )
    parser.add_argument(
        "-i", "--input", required=True, action="store", help="path to input CSV file"
    )
    parser.add_argument(
        "-o", "--output", required=True, action="store", help="path to output directory"
    )
    parser.add_argument(
        "--path-cache",
        action="store",
        default=None,
        dest="cache_dir",
        help="path where the visualization data will be stored",
    )
    parser.add_argument(
        "--settings",
        type=argparse.FileType("rt", encoding="utf-8"),
        default=None,
        help="path where the visualization data will be stored",
    )
    args = parser.parse_args()
    if args.cache_dir is None:
        with tempfile.TemporaryDirectory() as cache_dir:
            args.cache_dir = cache_dir
            process(args)
    else:
        process(args)
 def process(args):
    log.info("Loading OpenStreetMap data")
    osm = OSMDataSource(cache_dir=args.cache_dir)
    filename_input = os.path.abspath(args.input)
    dataset_id = os.path.splitext(os.path.basename(args.input))[0]
    os.makedirs(args.output, exist_ok=True)
    log.info("Loading settings")
    settings = json.load(args.settings)
    log.info("Annotating and filtering CSV file")
    measurements, statistics = ImportMeasurementsCsv().read(
        filename_input,
        user_id="dummy",
        dataset_id=dataset_id,
    )
    measurements = AnnotateMeasurements(osm, cache_dir=args.cache_dir).annotate(
        measurements
    )
    filters_from_settings = []
    for filter_description in settings.get("filters", []):
        filter_type = filter_description.get("type")
        if filter_type == "PrivacyZonesFilter":
            privacy_zones = [
                PrivacyZone(
                    latitude=zone.get("latitude"),
                    longitude=zone.get("longitude"),
                    radius=zone.get("radius"),
                )
                for zone in filter_description.get("config", {}).get("privacyZones", [])
            ]
            filters_from_settings.append(PrivacyZonesFilter(privacy_zones))
        else:
            log.warning("Ignoring unknown filter type %r in settings file", filter_type)
    input_filter = ChainFilter(
        RequiredFieldsFilter(),
        PrivacyFilter(
            user_id_mode=AnonymizationMode.REMOVE,
            measurement_id_mode=AnonymizationMode.REMOVE,
        ),
        *filters_from_settings,
    )
    events_filter = DistanceMeasuredFilter()
    confirmed_filter = ChainFilter(
        ConfirmedFilter(),
    )
    track_measurements = input_filter.filter(measurements, log=log)
    event_measurements = events_filter.filter(track_measurements , log=log)
    confirmed_measurements = confirmed_filter.filter(track_measurements, log=log)
    # write out
    confirmed_measurements_json = {
        "type": "FeatureCollection",
        "features": [
            {
                "type": "Feature",
                "geometry": {
                    "type": "Point",
                    "coordinates": [m["latitude"], m["longitude"]],
                },
                "properties": {
                    "distanceOvertaker": m["distance_overtaker"],
                    "distanceStationary": m["distance_stationary"],
                    "confirmed": True,
                },
            }
            for m in confirmed_measurements
        ],
    }
    all_measurements_json = {
        "type": "FeatureCollection",
        "features": [
            {
                "type": "Feature",
                "geometry": {
                    "type": "Point",
                    "coordinates": [m["latitude"], m["longitude"]],
                },
                "properties": {
                    "distanceOvertaker": m["distance_overtaker"],
                    "distanceStationary": m["distance_stationary"],
                    "confirmed": m in confirmed_measurements,
                },
            }
            for m in event_measurements
        ],
    }
    track_json = {
        "type": "Feature",
        "geometry": {
            "type": "LineString",
            "coordinates": [
                [m["latitude"], m["longitude"]] for m in track_measurements
            ],
        },
    }
    statistics_json = {
        "recordedAt": statistics["t_min"].isoformat(),
        "recordedUntil": statistics["t_max"].isoformat(),
        "duration": statistics["t"],
        "length": statistics["d"],
        "segments": statistics["n_segments"],
        "numEvents": statistics["n_confirmed"],
        "numMeasurements": statistics["n_measurements"],
        "numValid": statistics["n_valid"],
    }
    for output_filename, data in [
        ("all_measurements.json", all_measurements_json),
        ("confirmed_measurements.json", confirmed_measurements_json),
        ("track.json", track_json),
        ("statistics.json", statistics_json),
    ]:
        with open(os.path.join(args.output, output_filename), "w") as fp:
            json.dump(data, fp, indent=4)
 if __name__ == "__main__":
    main()
--- a/api/obs/api/routes/init.py
+++ b/api/obs/api/routes/init.py
--- a/api/src/queue.js
+++ b/api/src/queue.js
@ -0,0 +1,12 @@
 const Bull = require('bull');
 const config = require('./config');
 module.exports = new Bull('processQueue', config.redisUrl, {
  settings: {
    // if the worker process is killed and restarted, e.g. due to reboot or
    // upgrade, it is okay to wait for a timeout on the job and restart it
    maxStalledCount: 3,
    lockDuration: 120 * 1000,
  },
 });
--- a/api/src/routes/api/index.js
+++ b/api/src/routes/api/index.js
@ -0,0 +1,25 @@
 const router = require('express').Router();
 router.use('/', require('./users'));
 router.use('/profiles', require('./profiles'));
 router.use('/tracks', require('./tracks'));
 router.use('/tags', require('./tags'));
 router.use('/accounts', require('../../accounts/accounts.controller'));
 router.use('/stats', require('./stats'));
 router.use('/info', require('./info'));
 router.use(function (err, req, res, next) {
  if (err.name === 'ValidationError') {
    return res.status(422).json({
      errors: Object.keys(err.errors).reduce(function (errors, key) {
        errors[key] = err.errors[key].message;
        return errors;
      }, {}),
    });
  }
  return next(err);
 });
 module.exports = router;
--- a/api/src/routes/api/info.js
+++ b/api/src/routes/api/info.js
@ -0,0 +1,11 @@
 const router = require('express').Router();
 const { version } = require('../../../package.json');
 router.route('/').get((req, res) => {
  res.json({
    version,
  });
 });
 module.exports = router;
--- a/api/src/routes/api/profiles.js
+++ b/api/src/routes/api/profiles.js
@ -0,0 +1,30 @@
 const router = require('express').Router();
 const wrapRoute = require('../../_helpers/wrapRoute');
 const auth = require('../../passport');
 const { User } = require('../../models');
 // Preload user profile on routes with ':username'
 router.param('username', async function (req, res, next, username) {
  try {
    const user = await User.findOne({ username: username });
    if (!user) {
      return res.sendStatus(404);
    }
    req.profile = user;
    return next();
  } catch (err) {
    next(err);
  }
 });
 router.get(
  '/:username',
  auth.optional,
  wrapRoute(async (req, res) => {
    return res.json({ profile: req.profile.toProfileJSONFor(req.user) });
  }),
 );
 module.exports = router;
--- a/api/src/routes/api/stats.js
+++ b/api/src/routes/api/stats.js
@ -0,0 +1,107 @@
 const router = require('express').Router();
 const { DateTime } = require('luxon');
 const { Track, User } = require('../../models');
 const wrapRoute = require('../../_helpers/wrapRoute');
 const auth = require('../../passport');
 // round to this number of meters for privacy reasons
 const TRACK_LENGTH_ROUNDING = 1000;
 // round to this number of seconds for privacy reasons
 const TRACK_DURATION_ROUNDING = 120;
 router.get(
  '/',
  auth.optional,
  wrapRoute(async (req, res) => {
    const start = DateTime.fromISO(req.query.start);
    const end = DateTime.fromISO(req.query.end);
    const dateFilter = {
      $ne: null,
      ...(start.isValid ? { $gte: start.toJSDate() } : {}),
      ...(end.isValid ? { $lt: end.toJSDate() } : {}),
    };
    let userFilter;
    if (req.query.user) {
      const user = await User.findOne({ username: req.query.user });
      // Only the user can look for their own stats, for now
      if (req.user && req.user._id.equals(user._id)) {
        userFilter = user._id;
      } else {
        userFilter = { $ne: null };
      }
    }
    const trackFilter = {
      'statistics.recordedAt': dateFilter,
      ...(userFilter ? { author: userFilter } : {}),
    };
    const trackCount = await Track.find(trackFilter).count();
    const publicTrackCount = await Track.find({
      ...trackFilter,
      public: true,
    }).count();
    const userCount = await User.find({
      ...(userFilter
        ? { _id: userFilter }
        : {
            createdAt: dateFilter,
          }),
    }).count();
    const trackStats = await Track.aggregate([
      { $match: trackFilter },
      {
        $addFields: {
          trackLength: {
            $cond: [{ $lt: ['$statistics.length', 500000] }, '$statistics.length', 0],
          },
          numEvents: '$statistics.numEvents',
          trackDuration: {
            $cond: [
              { $and: ['$statistics.recordedUntil', { $gt: ['$statistics.recordedAt', new Date('2010-01-01')] }] },
              { $subtract: ['$statistics.recordedUntil', '$statistics.recordedAt'] },
              0,
            ],
          },
        },
      },
      { $project: { trackLength: true, numEvents: true, trackDuration: true } },
      {
        $group: {
          _id: 'sum',
          trackLength: { $sum: '$trackLength' },
          numEvents: { $sum: '$numEvents' },
          trackDuration: { $sum: '$trackDuration' },
        },
      },
    ]);
    const [trackLength, numEvents, trackDuration] =
      trackStats.length > 0
        ? [trackStats[0].trackLength, trackStats[0].numEvents, trackStats[0].trackDuration]
        : [0, 0, 0];
    const trackLengthPrivatized = Math.floor(trackLength / TRACK_LENGTH_ROUNDING) * TRACK_LENGTH_ROUNDING;
    const trackDurationPrivatized =
      Math.round(trackDuration / 1000 / TRACK_DURATION_ROUNDING) * TRACK_DURATION_ROUNDING;
    return res.json({
      publicTrackCount,
      trackLength: trackLengthPrivatized,
      trackDuration: trackDurationPrivatized,
      numEvents,
      trackCount,
      userCount,
    });
  }),
 );
 module.exports = router;
--- a/api/src/routes/api/tags.js
+++ b/api/src/routes/api/tags.js
@ -0,0 +1,14 @@
 const router = require('express').Router();
 const wrapRoute = require('../../_helpers/wrapRoute');
 const { Track } = require('../../models');
 // return a list of tags
 router.get(
  '/',
  wrapRoute(async (req, res) => {
    const tags = await Track.find().distinct('tagList');
    return res.json({ tags: tags });
  }),
 );
 module.exports = router;
--- a/api/src/routes/api/tracks.js
+++ b/api/src/routes/api/tracks.js
@ -0,0 +1,416 @@
 const fs = require('fs');
 const path = require('path');
 const router = require('express').Router();
 const { Track, User, Comment } = require('../../models');
 const busboy = require('connect-busboy');
 const auth = require('../../passport');
 const { normalizeUserAgent, buildObsver1 } = require('../../logic/tracks');
 const wrapRoute = require('../../_helpers/wrapRoute');
 const { PROCESSING_OUTPUT_DIR } = require('../../paths');
 function preloadByParam(target, getValueFromParam) {
  return async (req, res, next, paramValue) => {
    try {
      const value = await getValueFromParam(paramValue);
      if (!value) {
        return res.sendStatus(404);
      }
      req[target] = value;
      return next();
    } catch (err) {
      return next(err);
    }
  };
 }
 router.param(
  'track',
  preloadByParam('track', (slug) => Track.findOne({ slug }).populate('author')),
 );
 router.param(
  'comment',
  preloadByParam('comment', (id) => Comment.findById(id)),
 );
 router.param('comment', async (req, res, next, id) => {
  try {
    const comment = await Comment.findById(id);
    if (!comment) {
      return res.sendStatus(404);
    }
    req.comment = comment;
    return next();
  } catch (err) {
    return next(err);
  }
 });
 router.get(
  '/',
  auth.optional,
  wrapRoute(async (req, res) => {
    const query = { public: true };
    let limit = 20;
    let offset = 0;
    if (typeof req.query.limit !== 'undefined') {
      limit = req.query.limit;
    }
    if (typeof req.query.offset !== 'undefined') {
      offset = req.query.offset;
    }
    if (typeof req.query.tag !== 'undefined') {
      query.tagList = { $in: [req.query.tag] };
    }
    const author = req.query.author ? await User.findOne({ username: req.query.author }) : null;
    if (author) {
      query.author = author._id;
    }
    const [tracks, tracksCount] = await Promise.all([
      Track.find(query)
        .sort('-createdAt')
        .limit(Number(limit))
        .skip(Number(offset))
        .sort({ createdAt: 'desc' })
        .populate('author')
        .exec(),
      Track.countDocuments(query).exec(),
    ]);
    return res.json({
      tracks: tracks.map((track) => track.toJSONFor(req.user)),
      tracksCount,
    });
  }),
 );
 router.get(
  '/feed',
  auth.required,
  wrapRoute(async (req, res) => {
    let limit = 20;
    let offset = 0;
    if (typeof req.query.limit !== 'undefined') {
      limit = req.query.limit;
    }
    if (typeof req.query.offset !== 'undefined') {
      offset = req.query.offset;
    }
    const query = { author: req.user.id };
    const [tracks, tracksCount] = await Promise.all([
      Track.find(query).sort('-createdAt').limit(Number(limit)).skip(Number(offset)).populate('author').exec(),
      Track.countDocuments(query),
    ]);
    return res.json({
      tracks: tracks.map(function (track) {
        return track.toJSONFor(req.user);
      }),
      tracksCount: tracksCount,
    });
  }),
 );
 async function readFile(file) {
  let fileContent = '';
  file.on('data', function (data) {
    fileContent += data;
  });
  await new Promise((resolve, reject) => {
    file.on('end', resolve);
    file.on('error', reject);
  });
  return fileContent;
 }
 async function getMultipartOrJsonBody(req, mapJsonBody = (x) => x) {
  const fileInfo = {};
  let body;
  if (req.busboy) {
    body = {};
    req.busboy.on('file', async function (fieldname, file, filename, encoding, mimetype) {
      body[fieldname] = await readFile(file);
      fileInfo[fieldname] = { filename, encoding, mimetype };
    });
    req.busboy.on('field', (key, value) => {
      body[key] = value;
    });
    req.pipe(req.busboy);
    await new Promise((resolve, reject) => {
      req.busboy.on('finish', resolve);
      req.busboy.on('error', reject);
    });
  } else if (req.headers['content-type'] === 'application/json') {
    body = mapJsonBody(req.body);
  } else {
    body = { body: await readFile(req), ...req.query };
    fileInfo.body = {
      mimetype: req.headers['content-type'],
      filename: req.headers['content-disposition'],
      encoding: req.headers['content-encoding'],
    };
  }
  return { body, fileInfo };
 }
 router.post(
  '/',
  auth.requiredWithUserId,
  busboy(), // parse multipart body
  wrapRoute(async (req, res) => {
    // Read the whole file into memory. This is not optimal, instead, we should
    // write the file data directly to the target file. However, we first have
    // to parse the rest of the track data to know where to place the file.
    // TODO: Stream into temporary file, then move it later.
    const { body, fileInfo } = await getMultipartOrJsonBody(req, (body) => body.track);
    const { body: fileBody, public, ...trackBody } = body;
    const track = new Track({
      ...trackBody,
      author: req.user,
      public: public == null ? req.user.areTracksVisibleForAll : Boolean(trackBody.public),
    });
    track.customizedTitle = track.title != null;
    track.slugify();
    if (fileBody) {
      await track.validateFileBodyUniqueness(fileBody);
      track.uploadedByUserAgent = normalizeUserAgent(req.headers['user-agent']);
      track.originalFileName = fileInfo.body ? fileInfo.body.filename : track.slug + '.csv';
      await track.writeToOriginalFile(fileBody);
    }
    await track.save();
    if (fileBody) {
      await track.queueProcessing();
    }
    await track.autoGenerateTitle();
    return res.json({ track: track.toJSONFor(req.user) });
  }),
 );
 // return a track
 router.get(
  '/:track',
  auth.optional,
  wrapRoute(async (req, res) => {
    if (!req.track.isVisibleTo(req.user)) {
      return res.sendStatus(403);
    }
    return res.json({ track: req.track.toJSONFor(req.user) });
  }),
 );
 // update track
 router.put(
  '/:track',
  busboy(),
  auth.required,
  wrapRoute(async (req, res) => {
    const track = req.track;
    if (!track.author._id.equals(req.user.id)) {
      return res.sendStatus(403);
    }
    const {
      body: { body: fileBody, ...trackBody },
      fileInfo,
    } = await getMultipartOrJsonBody(req, (body) => body.track);
    if (typeof trackBody.title !== 'undefined') {
      track.title = (trackBody.title || '').trim() || null;
      track.customizedTitle = track.title != null;
    }
    if (typeof trackBody.description !== 'undefined') {
      track.description = (trackBody.description || '').trim() || null;
    }
    let process = false;
    if (trackBody.public != null) {
      const public = Boolean(trackBody.public);
      process |= public !== track.public;
      track.public = public;
    }
    if (fileBody) {
      await track.validateFileBodyUniqueness(fileBody);
      track.originalFileName = fileInfo.body ? fileInfo.body.filename : track.slug + '.csv';
      track.uploadedByUserAgent = normalizeUserAgent(req.headers['user-agent']);
      await track.writeToOriginalFile(fileBody);
      process = true;
    }
    await track.save();
    if (process) {
      await track.queueProcessing();
    }
    await track.autoGenerateTitle();
    return res.json({ track: track.toJSONFor(req.user) });
  }),
 );
 // delete track
 router.delete(
  '/:track',
  auth.required,
  wrapRoute(async (req, res) => {
    if (req.track.author._id.equals(req.user.id)) {
      await req.track.remove();
      return res.sendStatus(204);
    } else {
      return res.sendStatus(403);
    }
  }),
 );
 // return an track's comments
 router.get(
  '/:track/comments',
  auth.optional,
  wrapRoute(async (req, res) => {
    if (!req.track.isVisibleTo(req.user)) {
      return res.sendStatus(403);
    }
    await req.track.populate({
      path: 'comments',
      populate: {
        path: 'author',
      },
      options: {
        sort: {
          createdAt: 'asc',
        },
      },
    });
    return res.json({
      comments: req.track.comments.map(function (comment) {
        return comment.toJSONFor(req.user);
      }),
    });
  }),
 );
 // create a new comment
 router.post(
  '/:track/comments',
  auth.required,
  wrapRoute(async (req, res) => {
    const comment = new Comment(req.body.comment);
    comment.track = req.track;
    comment.author = req.user;
    await comment.save();
    req.track.comments.push(comment);
    await req.track.save();
    return res.json({ comment: comment.toJSONFor(req.user) });
  }),
 );
 router.delete(
  '/:track/comments/:comment',
  auth.required,
  wrapRoute(async (req, res) => {
    if (req.comment.author.equals(req.user.id)) {
      req.track.comments.remove(req.comment._id);
      await req.track.save();
      await Comment.find({ _id: req.comment._id }).remove();
      res.sendStatus(204);
    } else {
      res.sendStatus(403);
    }
  }),
 );
 // return an track's generated data
 router.get(
  '/:track/data',
  auth.optional,
  wrapRoute(async (req, res) => {
    const FILE_BY_KEY = {
      allMeasurements: 'all_measurements.json',
      confirmedMeasurements: 'confirmed_measurements.json',
      track: 'track.json',
    };
    if (!req.track.isVisibleTo(req.user)) {
      return res.sendStatus(403);
    }
    const result = {};
    for (const [key, filename] of Object.entries(FILE_BY_KEY)) {
      const filePath = path.join(PROCESSING_OUTPUT_DIR, req.track.filePath, filename);
      let stats;
      try {
        stats = await fs.promises.stat(filePath);
      } catch (err) {
        continue;
      }
      if (!stats.isFile()) {
        // file does not exist (yet)
        continue;
      }
      const content = await fs.promises.readFile(filePath);
      const contentJson = JSON.parse(content);
      result[key] = contentJson;
    }
    return res.json(result);
  }),
 );
 // download the original file
 router.get(
  '/:track/download/original.csv',
  auth.optional,
  wrapRoute(async (req, res) => {
    if (!req.track.isVisibleToPrivate(req.user)) {
      return res.sendStatus(403);
    }
    return res.download(req.track.getOriginalFilePath(), req.track.originalFileName);
  }),
 );
 module.exports = router;
--- a/api/src/routes/api/users.js
+++ b/api/src/routes/api/users.js
@ -0,0 +1,52 @@
 const router = require('express').Router();
 const wrapRoute = require('../../_helpers/wrapRoute');
 const auth = require('../../passport');
 router.get(
  '/user',
  auth.required,
  wrapRoute(async (req, res) => {
    return res.json({ user: req.user.toAuthJSON() });
  }),
 );
 router.put(
  '/user',
  auth.required,
  wrapRoute(async (req, res) => {
    const user = req.user;
    // only update fields that were actually passed...
    if (typeof req.body.user.username !== 'undefined') {
      user.username = req.body.user.username;
    }
    if (typeof req.body.user.email !== 'undefined') {
      user.email = req.body.user.email;
    }
    if (typeof req.body.user.bio !== 'undefined') {
      user.bio = req.body.user.bio;
    }
    if (typeof req.body.user.image !== 'undefined') {
      user.image = req.body.user.image;
    }
    if (typeof req.body.user.areTracksVisibleForAll !== 'undefined') {
      user.areTracksVisibleForAll = req.body.user.areTracksVisibleForAll;
    }
    if (typeof req.body.user.password === 'string' && req.body.user.password !== '') {
      user.setPassword(req.body.user.password);
    }
    await user.save();
    return res.json({ user: user.toAuthJSON() });
  }),
 );
 // Remove this at some point
 router.post('/users/login',
  auth.usernameAndPassword,
  wrapRoute((req, res) => {
    return res.json({ user: req.user.toAuthJSON() });
  }),
 );
 module.exports = router;
--- a/api/src/routes/auth.js
+++ b/api/src/routes/auth.js
@ -0,0 +1,558 @@
 const router = require('express').Router();
 const passport = require('passport');
 const { URL } = require('url');
 const { createChallenge } = require('pkce');
 const { AuthorizationCode, AccessToken, RefreshToken } = require('../models');
 const auth = require('../passport');
 const wrapRoute = require('../_helpers/wrapRoute');
 const config = require('../config');
 const baseUrl = config.baseUrl.replace(/\/+$/, '');
 // Check whether the "bigScope" fully includes the "smallScope".
 function scopeIncludes(smallScope, bigScope) {
  const smallScopeParts = smallScope.split(/\s/);
  const bigScopeParts = bigScope.split(/\s/);
  return bigScopeParts.includes('*') || smallScopeParts.every((part) => bigScopeParts.includes(part));
 }
 function returnError(res, error, errorDescription = undefined, status = 400) {
  return res
    .status(status)
    .json({ error, ...(errorDescription != null ? { error_description: errorDescription } : {}) });
 }
 function redirectWithParams(res, redirectUri, params) {
  const targetUrl = new URL(redirectUri);
  for (const [key, value] of Object.entries(params)) {
    targetUrl.searchParams.append(key, value);
  }
  return res.redirect(targetUrl.toString());
 }
 const ALL_SCOPE_NAMES = `
  tracks.create
  tracks.update
  tracks.list
  tracks.show
  tracks.delete
  users.update
  users.show
  tracks.comments.create
  tracks.comments.update
  tracks.comments.list
  tracks.comments.show
 `.split(/\s/);
 function isValidScope(scope) {
  return scope === '*' || scopeIncludes(scope, ALL_SCOPE_NAMES.join(' '));
 }
 router.use((req, res, next) => {
  res.locals.user = req.user;
  res.locals.mainFrontendUrl = config.mainFrontendUrl;
  res.locals.imprintUrl = config.imprintUrl;
  res.locals.privacyPolicyUrl = config.privacyPolicyUrl;
  res.locals.baseUrl = baseUrl + '/';
  next();
 });
 router.post(
  '/login',
  passport.authenticate('usernameAndPasswordSession', { session: true }),
  (err, req, res, next) => {
    if (!err) {
      next();
    }
    if (err.message === 'invalid credentials') {
      return res.render('login', { badCredentials: true });
    }
    let description = 'Unknown error while processing your login.';
    if (err.message === 'email not verified') {
      description = 'Your account is not yet verified, please check your email or start the password recovery.';
    }
    return res.render('message', { type: 'negative', title: 'Login failed', description });
  },
  wrapRoute((req, res, next) => {
    if (!req.user) {
      return res.redirect(baseUrl + '/login');
    }
    if (req.session.next) {
      res.redirect(baseUrl + req.session.next);
      req.session.next = null;
      return;
    }
    return res.render('message', { type: 'positive', title: 'You are logged in.', showFrontendLink: true });
  }),
 );
 router.get(
  '/login',
  wrapRoute(async (req, res) => {
    if (req.user) {
      return res.render('message', { type: 'positive', title: 'You are already logged in.' });
    }
    return res.render('login');
  }),
 );
 router
  .route('/logout')
  .post(
    auth.usernameAndPasswordSession,
    wrapRoute(async (req, res) => {
      req.logout();
      return res.redirect(baseUrl + '/login');
    }),
  )
  .get((req, res) => {
    if (req.query.redirectTo) {
      req.logout();
      return res.redirect(req.query.redirectTo);
    }
    return res.render('logout')
  });
 const isIp = (ip) =>
  typeof ip === 'string' &&
  /^([0-9]{1,3}\.)[0-9]{1,3}$/.test(ip) &&
  ip
    .split('.')
    .every(
      (num, idx) =>
        !num.startsWith('0') && Number(num) > (idx === 0 || idx === 3 ? 1 : 0) && Number(num) < (idx === 3 ? 254 : 255),
    );
 const isLocalIp = (ip) => isIp(ip) && (ip.startsWith('10.') || ip.startsWith('172.16.') || ip.startsWith('192.168.'));
 const isValidRedirectUriFor = (redirectUri) => (redirectUriPattern) => {
  // Here we have an exception to the security requirements demanded by
  // https://tools.ietf.org/html/draft-ietf-oauth-security-topics-16#section-2.1,
  // namely, that we do not always require fully specified redirect URIs. This
  // is because we cannot know beforehand which IP the OBS will be running at.
  // But since it is usually accessed via local IP, we can allow all local IP
  // ranges. This special case must only be used in clients that have a very
  // restricted `maxScope` as well, to prevent misuse should an attack through
  // this be successful.
  // This special case does however enforce TLS ("https://"), for it prevents
  // usage in a non-TLS-secured web server. At least passive sniffing of the
  // token is not possible then. A self-signed and manually verified
  // certificate should be used for this (though usually we cannot enforce the
  // actual verification).
  if (redirectUriPattern === '__LOCAL__') {
    const url = new URL(redirectUri);
    if (url.protocol === 'https:' && isLocalIp(url.host) && !url.search && !url.hash) {
      return true;
    }
    return false;
  } else {
    return redirectUriPattern === redirectUri;
  }
 };
 router.get(
  '/authorize',
  passport.authenticate('session'),
  wrapRoute(async (req, res) => {
    if (!req.user) {
      req.session.next = req.url;
      return res.redirect(baseUrl + '/login');
    }
    try {
      const {
        client_id: clientId,
        redirect_uri: redirectUri,
        response_type: responseType,
        scope = '*', // fallback to "all" scope
        // for PKCE
        code_challenge: codeChallenge,
        code_challenge_method: codeChallengeMethod,
      } = req.query;
      // 1. Find our client and check if it exists
      if (!clientId) {
        return returnError(res, 'invalid_request', 'client_id parameter required');
      }
      const client = config.oAuth2Clients.find((c) => c.clientId === clientId);
      if (!client) {
        return returnError(res, 'invalid_client', 'unknown client');
      }
      // 2. Check that we have a redirect_uri. In addition to [RFC6749] we
      // *always* require a redirect_uri.
      if (!redirectUri) {
        return returnError(res, 'invalid_request', 'redirect_uri parameter required');
      }
      // We enforce that the redirectUri exactly matches one of the provided URIs
      const check = isValidRedirectUriFor(redirectUri);
      if (!client.validRedirectUris.some(check)) {
        return returnError(res, 'invalid_request', 'invalid redirect_uri');
      }
      // 3. Find out which type of response to use.  [RFC6749] requires one of
      // "code" or "token", but "token" is implicit grant and we do not support
      // that.
      if (responseType !== 'code') {
        return redirectWithParams(res, redirectUri, {
          error: 'unsupported_grant_type',
          error_description: 'only authorization code flow with PKCE is supported by this server',
        });
      }
      // 4. Verify we're using PKCE with supported (S256) code_challenge_method.
      if (!codeChallenge) {
        return redirectWithParams(res, redirectUri, {
          error: 'invalid_request',
          error_description: 'a code_challenge for PKCE is required',
        });
      }
      if (codeChallengeMethod !== 'S256') {
        return redirectWithParams(res, redirectUri, {
          error: 'invalid_request',
          error_description: 'the code_challenge_method for PKCE must be "S256"',
        });
      }
      // 5. Get the scope.
      if (!isValidScope(scope)) {
        return redirectWithParams(res, redirectUri, {
          error: 'invalid_scope',
          error_description: 'the requested scope is not known',
        });
      }
      if (client.maxScope && !scopeIncludes(scope, client.maxScope)) {
        return redirectWithParams(res, redirectUri, {
          error: 'access_denied',
          error_description: 'the requested scope is not valid for this client',
        });
      }
      // Ok, let's save all this in the session, and show a dialog for the
      // decision to the user.
      //
      if (client.autoAccept) {
        const code = AuthorizationCode.generate({
          clientId,
          user: req.user,
          redirectUri,
          scope,
          codeChallenge,
        });
        await code.save();
        return redirectWithParams(res, redirectUri, { code: code.code, scope });
      } else {
        req.session.authorizationTransaction = {
          responseType,
          clientId,
          redirectUri,
          scope,
          expiresAt: new Date().getTime() + 1000 * 60 * 2, // 2 minute decision time
          codeChallenge,
        };
        res.render('authorize', { clientTitle: client.title, scope, redirectUri });
      }
    } catch (err) {
      res.status(400).json({ error: 'invalid_request', error_description: 'unknown error' });
    }
  }),
 );
 router.post(
  ['/authorize/approve', '/authorize/decline'],
  passport.authenticate('session'),
  wrapRoute(async (req, res) => {
    if (!req.session.authorizationTransaction) {
      return res.sendStatus(400);
    }
    if (!req.user) {
      return res.sendStatus(400);
    }
    const { clientId, redirectUri, scope, expiresAt, codeChallenge } = req.session.authorizationTransaction;
    if (expiresAt < new Date().getTime()) {
      return res.status(400).render('message', {
        type: 'negative',
        title: 'Expired',
        description: 'Your authorization has expired. Please go back and retry the process.',
      });
    }
    // invalidate the transaction
    req.session.authorizationTransaction = null;
    if (req.path === '/authorize/approve') {
      const code = AuthorizationCode.generate({
        clientId,
        user: req.user,
        redirectUri,
        scope,
        codeChallenge,
      });
      await code.save();
      return redirectWithParams(res, redirectUri, { code: code.code, scope });
    } else {
      return redirectWithParams(res, redirectUri, { error: 'access_denied' });
    }
  }),
 );
 /**
 * This function is called when the client presents an authorization code
 * (generated above) and wants it turned into an access (and possibly refresh)
 * token.
 */
 router.get(
  '/token',
  wrapRoute(async (req, res) => {
    const {
      grant_type: grantType,
      code,
      client_id: clientId,
      redirect_uri: redirectUri,
      // for PKCE
      code_verifier: codeVerifier,
    } = req.query;
    if (!grantType || grantType !== 'authorization_code') {
      return returnError(
        res,
        'unsupported_grant_type',
        'only authorization code flow with PKCE is supported by this server',
      );
    }
    if (!code) {
      return returnError(res, 'invalid_request', 'code parameter required');
    }
    // Call this function to destroy the authorization code (if it exists),
    // invalidating it when a single failed request has been received. The
    // whole process must be restarted. No trial and error ;)
    const destroyAuthCode = async () => {
      await AuthorizationCode.deleteOne({ code });
    };
    if (!clientId) {
      await destroyAuthCode();
      return returnError(res, 'invalid_client', 'client_id parameter required');
    }
    if (!redirectUri) {
      await destroyAuthCode();
      return returnError(res, 'invalid_request', 'redirect_uri parameter required');
    }
    if (!codeVerifier) {
      await destroyAuthCode();
      return returnError(res, 'invalid_request', 'code_verifier parameter required');
    }
    const client = config.oAuth2Clients.find((c) => c.clientId === clientId);
    if (!client) {
      await destroyAuthCode();
      return returnError(res, 'invalid_client', 'invalid client_id');
    }
    const authorizationCode = await AuthorizationCode.findOne({ code });
    if (!authorizationCode) {
      await destroyAuthCode();
      return returnError(res, 'invalid_grant', 'invalid authorization code');
    }
    if (authorizationCode.redirectUri !== redirectUri) {
      await destroyAuthCode();
      return returnError(res, 'invalid_grant', 'invalid authorization code');
    }
    if (authorizationCode.expiresAt <= new Date().getTime()) {
      await destroyAuthCode();
      return returnError(res, 'invalid_grant', 'invalid authorization code');
    }
    if (clientId !== authorizationCode.clientId) {
      await destroyAuthCode();
      return returnError(res, 'invalid_grant', 'invalid authorization code');
    }
    if (createChallenge(codeVerifier) !== authorizationCode.codeChallenge) {
      await destroyAuthCode();
      return returnError(res, 'invalid_grant', 'invalid authorization code');
    }
    // invalidate auth code now, before generating tokens
    await AuthorizationCode.deleteOne({ _id: authorizationCode._id });
    const accessToken = AccessToken.generate({
      clientId: authorizationCode.clientId,
      user: authorizationCode.user,
      scope: authorizationCode.scope,
    });
    await accessToken.save();
    let refreshToken;
    if (client.refreshTokenExpirySeconds != null) {
      refreshToken = RefreshToken.generate(
        {
          clientId: authorizationCode.clientId,
          user: authorizationCode.user,
          scope: authorizationCode.scope,
        },
        client.refreshTokenExpirySeconds,
      );
      await refreshToken.save();
    }
    return res.json({
      access_token: accessToken.token,
      token_type: 'Bearer',
      expires_in: Math.round((accessToken.expiresAt - new Date().getTime()) / 1000),
      scope: accessToken.scope,
      ...(refreshToken != null ? { refresh_token: refreshToken.token } : {}),
    });
  }),
 );
 /**
 * Metadata endpoint to inform clients about authorization server capabilities,
 * according to https://tools.ietf.org/html/rfc8414.
 */
 router.get(
  '/.well-known/oauth-authorization-server',
  wrapRoute(async (req, res) => {
    return res.json({
      issuer: baseUrl,
      authorization_endpoint: `${baseUrl}/authorize`,
      token_endpoint: `${baseUrl}/token`,
      token_endpoint_auth_methods_supported: ['none'], // only public clients
      userinfo_endpoint: `${baseUrl}/api/user`,
      // registration_endpoint: `${baseUrl}/register`, // TODO
      // scopes_supported: ALL_SCOPE_NAMES, // TODO
      response_types_supported: ['code'], // only auth code, no implicit flow or
      service_documentation: 'https://github.com/openbikesensor/portal',
      ui_locales_supported: ['en-US', 'en-GB', 'en-CA', 'fr-FR', 'fr-CA'],
      code_challenge_methods_supported: ['S256'],
    });
  }),
 );
 module.exports = router;
 const accountService = require('../accounts/account.service');
 const validateRequest = require('../_middleware/validate-request');
 const Joi = require('joi');
 router
  .route('/register')
  .post(
    validateRequest(
      Joi.object({
        username: Joi.string().required(),
        email: Joi.string().email().required(),
        password: Joi.string().min(6).required(),
        confirmPassword: Joi.string().valid(Joi.ref('password')).required(),
        ...(config.privacyPolicyUrl ? {
          acceptPrivacyPolicy: Joi.boolean().truthy().required(),
        } : {}),
      }),
    ),
    wrapRoute(async (req, res) => {
      await accountService.register(req.body);
      return res.render('message', {
        type: 'positive',
        title: 'Registration successful',
        description: 'Please check your email for verification instructions.',
      });
    }),
  )
  .get((req, res) => res.render('register'));
 router.get(
  '/verify-email',
  validateRequest(
    Joi.object({
      token: Joi.string().required(),
    }),
    'query',
  ),
  wrapRoute(async (req, res) => {
    await accountService.verifyEmail(req.query);
    return res.render('message', {
      type: 'positive',
      title: 'Verification successful',
      description: 'You can now log in.',
      showLoginButton: true,
    });
  }),
 );
 router
  .route('/forgot-password')
  .post(
    validateRequest(
      Joi.object({
        email: Joi.string().email().required(),
      }),
    ),
    wrapRoute(async (req, res) => {
      await accountService.forgotPassword(req.body);
      res.render('message', {
        type: 'positive',
        title: 'Recovery mail sent',
        description: 'Please check your inbox for password recovery instructions.',
      });
    }),
  )
  .get((req, res) => res.render('forgot-password'));
 router
  .route('/reset-password')
  .post(
    validateRequest(
      Joi.object({
        token: Joi.string().required(),
        password: Joi.string().min(6).required(),
        confirmPassword: Joi.string().valid(Joi.ref('password')).required(),
      }),
    ),
    wrapRoute(async (req, res) => {
      await accountService.resetPassword(req.body);
      return res.render('message', {
        type: 'positive',
        title: 'Password reset successful',
        description: 'You can now log in.',
        showLoginButton: true,
      });
    }),
  )
  .get(
    validateRequest(
      Joi.object({
        token: Joi.string().required(),
      }),
      'query',
    ),
    wrapRoute(async (req, res) => {
      const { token } = req.query;
      await accountService.validateResetToken({ token });
      res.render('reset-password', { token });
    }),
  );
 module.exports = router;
--- a/api/src/routes/index.js
+++ b/api/src/routes/index.js
@ -0,0 +1,8 @@
 const router = require('express').Router();
 router.use('/api', require('./api'));
 // no prefix
 router.use(require('./auth'));
 module.exports = router;
--- a/api/src/worker.js
+++ b/api/src/worker.js
@ -0,0 +1,133 @@
 const fs = require('fs');
 const path = require('path');
 const { spawn } = require('child_process');
 const queue = require('./queue');
 require('./db');
 const { Track } = require('./models');
 const { API_ROOT_DIR, PROCESSING_DIR, OBS_FACE_CACHE_DIR, PROCESSING_OUTPUT_DIR } = require('./paths');
 queue.process('processTrack', async (job) => {
  const track = await Track.findById(job.data.trackId);
  if (!track) {
    throw new Error('Cannot find track to process');
  }
  if (track.processingJobId !== job.id) {
    throw new Error('Track is processed by another job');
  }
  if (track.processingJobId !== job.id) {
    throw new Error('Track is processed by another job');
  }
  if (track.processingStatus !== 'pending') {
    throw new Error('Track is not pending processing');
  }
  try {
    const { filePath } = track;
    console.log('Will process track', filePath);
    track.processingStatus = 'processing';
    track.processingLog = '';
    await track.save();
    // Create input directory
    const inputDirectory = path.join(PROCESSING_DIR, filePath);
    await fs.promises.mkdir(inputDirectory, { recursive: true });
    // copy original file to processing dir
    const inputFilePath = path.join(inputDirectory, 'track.csv');
    const originalFilePath = track.getOriginalFilePath();
    console.log(`[${track.slug}] Copy ${originalFilePath} to ${inputFilePath}`);
    await fs.promises.copyFile(originalFilePath, inputFilePath);
    // create track settings file
    const settingsFilePath = path.join(inputDirectory, 'track-settings.json');
    console.log(`[${track.slug}] Create settings at ${settingsFilePath}`);
    const settings = {
      settingsGeneratedAt: new Date().getTime(),
      filters: [
        // TODO: Add actual privacy zones from user database
        /* {
          type: 'PrivacyZonesFilter',
          config: { privacyZones: [{ longitude: 10, latitude: 10, radius: 250 }] },
        }, */
      ],
    };
    await fs.promises.writeFile(settingsFilePath, JSON.stringify(settings));
    // Create output directory
    const outputDirectory = path.join(PROCESSING_OUTPUT_DIR, filePath);
    await fs.promises.mkdir(outputDirectory, { recursive: true });
    const stdoutFile = path.join(outputDirectory, 'stdout.log');
    const stderrFile = path.join(outputDirectory, 'stderr.log');
    const stdout = fs.createWriteStream(stdoutFile);
    const stderr = fs.createWriteStream(stderrFile);
    // TODO: Generate track transformation settings (privacy zones etc)
    // const settingsFilePath = path.join(inputDirectory, 'track-settings.json');
    //
    const child = spawn(
      'python',
      [
        path.join(API_ROOT_DIR, 'src', 'process_track.py'),
        '--input',
        inputFilePath,
        '--output',
        outputDirectory,
        '--path-cache',
        OBS_FACE_CACHE_DIR,
        '--settings',
        settingsFilePath,
        // '--anonymize-user-id', 'remove',
        // '--anonymize-measurement-id', 'remove',
      ],
      {
        cwd: PROCESSING_DIR,
      },
    );
    child.stdout.pipe(process.stdout);
    child.stdout.pipe(stdout);
    child.stderr.pipe(process.stderr);
    child.stderr.pipe(stderr);
    const code = await new Promise((resolve) => child.on('close', resolve));
    track.processingLog += (
      await Promise.all([
        fs.promises.readFile(stdoutFile),
        fs.promises.readFile(stderrFile),
        // split lines
      ])
    )
      .join('\n')
      .trim();
    if (code !== 0) {
      throw new Error(`Track processing failed with status ${code}`);
    }
    // Read some results back into the database for quick access and
    // accumulation
    const statisticsContent = await fs.promises.readFile(path.join(outputDirectory, 'statistics.json'));
    track.statistics = JSON.parse(statisticsContent);
    track.processingStatus = 'complete';
    await track.save();
    // Maybe we have found out the recording date, regenerate the automatic
    // title (if not yet manually set)
    await track.autoGenerateTitle();
  } catch (err) {
    console.error('Processing failed:', err);
    track.processingLog += String(err) + '\n' + err.stack + '\n';
    track.processingStatus = 'error';
    await track.save();
  }
 });
 console.log('Worker started.');
--- a/api/tools/import_from_mongodb.py
+++ b/api/tools/import_from_mongodb.py
@ -1,188 +0,0 @@
 #!/usr/bin/env python3
 import argparse
 import asyncio
 import logging
 import json
 from datetime import datetime
 from uuid import uuid4
 from sqlalchemy import select
 from motor.motor_asyncio import AsyncIOMotorClient
 from obs.api.db import make_session, connect_db, User, Track, Comment
 from obs.api.app import app
 log = logging.getLogger(__name__)
 async def main():
    logging.basicConfig(level=logging.DEBUG, format="%(levelname)s: %(message)s")
    parser = argparse.ArgumentParser(
        description="processes a single track for use in the portal, "
        "using the obs.face algorithms"
    )
    parser.add_argument(
        "mongodb_url",
        metavar="MONGODB_URL",
        help="url to the mongodb, in format mongodb://user:pass@host/dbname",
    )
    parser.add_argument(
        "--keycloak-users-file",
        metavar="FILE",
        type=argparse.FileType("wt", encoding="utf-8"),
        help="a file to write a JSON of all old users to, for importing to keycloak",
        default=None,
    )
    parser.add_argument(
        "--keep-api-keys",
        action="store_true",
        help="keep the old API keys (very insecure!) instead of generating new ones",
        default=False,
    )
    args = parser.parse_args()
    if args.keep_api_keys:
        log.warning(
            "Importing users with their old API keys. These keys are very insecure and "
            "could provide access to user data to third parties. Consider to notify "
            "your users about the need to generate a new API key through their profile pages."
        )
    async with connect_db(app.config.POSTGRES_URL):
        async with make_session() as session:
            mongo = AsyncIOMotorClient(args.mongodb_url).get_default_database()
            log.debug("Connected to mongodb and postgres.")
            user_id_map = await import_users(
                mongo, session, args.keycloak_users_file, args.keep_api_keys
            )
            await import_tracks(mongo, session, user_id_map)
            await session.commit()
 async def import_users(mongo, session, keycloak_users_file, keep_api_keys):
    keycloak_users = []
    old_id_by_email = {}
    async for user in mongo.users.find({}):
        old_id_by_email[user["email"]] = user["_id"]
        new_user = User(
            sub=str(uuid4()),
            email=user["email"],
            username=user["username"],
            bio=user.get("bio"),
            image=user.get("image"),
            are_tracks_visible_for_all=user.get("areTracksVisibleForAll") or False,
            created_at=user.get("createdAt") or datetime.utcnow(),
            updated_at=user.get("updatedAt") or datetime.utcnow(),
            match_by_username_email=True,
        )
        if keep_api_keys:
            new_user.api_key = str(user["_id"])
        else:
            new_user.generate_api_key()
        if keycloak_users_file:
            needs_email_verification = user.get("needsEmailValidation", True)
            required_actions = ["UPDATE_PASSWORD"]
            if needs_email_verification:
                required_actions.append("VERIFY_EMAIL")
            keycloak_users.append(
                {
                    "username": new_user.username,
                    "email": new_user.email,
                    "enabled": True,
                    "requiredActions": required_actions,
                    "emailVerified": not needs_email_verification,
                }
            )
        session.add(new_user)
        log.info("Creating user %s", new_user.username)
    await session.commit()
    id_map = {}
    result = await session.scalars(select(User))
    for user in result:
        id_map[old_id_by_email[user.email]] = user.id
    if keycloak_users_file:
        json.dump({"users": keycloak_users}, keycloak_users_file, indent=4)
        log.info("Wrote keycloak users file to %s.", keycloak_users_file.name)
    return id_map
 def parse_datetime(s):
    if isinstance(s, str):
        return datetime.fromisoformat(s)
    return s
 async def import_tracks(mongo, session, user_id_map):
    track_count = 0
    async for track in mongo.tracks.find({}):
        stats = track.get("statistics") or {}
        new_track = Track(
            created_at=parse_datetime(track.get("createdAt")) or datetime.utcnow(),
            updated_at=parse_datetime(track.get("updatedAt")) or datetime.utcnow(),
            slug=track["slug"],
            title=track.get("title"),
            processing_status=track.get("processingStatus") or "pending",
            processing_log=track.get("processingLog"),
            customized_title=bool(track.get("customizedTitle")),
            description=track.get("description"),
            public=track.get("public"),
            uploaded_by_user_agent=track.get("uploadedByUserAgent"),
            original_file_name=track.get("originalFileName"),
            original_file_hash=track.get("originalFileHash"),
            # statistics
            recorded_at=parse_datetime(stats.get("recordedAt")),
            recorded_until=parse_datetime(stats.get("recordedUntil")),
            duration=stats.get("duration"),
            length=stats.get("length"),
            segments=stats.get("segments"),
            num_events=stats.get("num_events"),
            num_measurements=stats.get("num_measurements"),
            num_valid=stats.get("numValid"),
            author_id=user_id_map[track["author"]],
        )
        session.add(new_track)
        comment_ids = track.get("comments") or []
        if comment_ids:
            async for comment in mongo.comments.find({"_id": {"$in": comment_ids}}):
                new_comment = Comment(
                    created_at=parse_datetime(comment.get("createdAt"))
                    or datetime.utcnow(),
                    updated_at=parse_datetime(comment.get("updatedAt"))
                    or datetime.utcnow(),
                    body=comment.get("body"),
                    author_id=user_id_map[comment["author"]],
                )
                new_track.comments.append(new_comment)
                session.add(new_comment)
        track_count += 1
    log.info("Created %s tracks", track_count)
    await session.commit()
 if __name__ == "__main__":
    asyncio.run(main())
--- a/Show more
+++ b/Show more
Author	SHA1	Message	Date
Paul Bienkowski	b7441741c0	api: Fix track again, using correct filters to get all points	2021-11-22 09:27:59 +01:00
Paul Bienkowski	59b67e5120	frontend: fix config URL when loading from sub-page	2021-11-22 09:27:59 +01:00
Paul Bienkowski	fdf6ad7f06	chore: add editorconfig	2021-11-22 09:27:59 +01:00
Paul Bienkowski	90c2147ebc	api: Generate track-settings file and read it in process_track.py	2021-11-22 09:27:59 +01:00
Paul Bienkowski	e1320b3ff5	chore: use new filter classes	2021-11-22 09:27:59 +01:00
Paul Bienkowski	b152d38edf	chore: move obs_process_track.py into the API source, as it is very specific to the API	2021-11-22 09:27:59 +01:00
		`@ -1 +0,0 @@`
			`__path__ = __import__("pkgutil").extend_path(__path__, __name__)`
		`@ -1 +1 @@`
			`Subproject commit 664e4d606416417c0651ea1748d32dd36209be6a`				`Subproject commit 6beab2ebfede7e41a1184b5ae1d0be0c83f8f95c`