Compare commits
6 commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
b7441741c0 | ||
|
|
59b67e5120 | ||
|
|
fdf6ad7f06 | ||
|
|
90c2147ebc | ||
|
|
e1320b3ff5 | ||
|
|
b152d38edf |
|
|
@ -1,10 +0,0 @@
|
|||
local
|
||||
*.user
|
||||
frontend/node_modules
|
||||
api/.pyenv
|
||||
.git
|
||||
cache
|
||||
data
|
||||
tile-generator/cache
|
||||
tile-generator/data
|
||||
tile-generator/build
|
||||
|
|
@ -1,20 +0,0 @@
|
|||
name: Build docker image
|
||||
on: [push]
|
||||
|
||||
jobs:
|
||||
build-image:
|
||||
runs-on: ubuntu-latest
|
||||
container:
|
||||
image: catthehacker/ubuntu:act-latest
|
||||
steps:
|
||||
- name: Login to Forgejo docker registry
|
||||
uses: docker/login-action@v3.0.0
|
||||
with:
|
||||
registry: git.pub.solar
|
||||
username: hakkonaut
|
||||
password: ${{ secrets.GIT_AUTH_TOKEN }}
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v5.1.0
|
||||
with:
|
||||
push: true
|
||||
tags: git.pub.solar/pub-solar/obs-portal:latest
|
||||
1
.gitignore
vendored
1
.gitignore
vendored
|
|
@ -1,3 +1,2 @@
|
|||
local
|
||||
data
|
||||
export
|
||||
|
|
|
|||
191
CHANGELOG.md
191
CHANGELOG.md
|
|
@ -1,191 +0,0 @@
|
|||
# Changelog
|
||||
|
||||
## 0.8.1
|
||||
|
||||
### Improvements
|
||||
|
||||
* The zone (urban/rural) is now also exported with the events GeoJson export.
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* Update to a current version of gpstime (python dependency) fix portal startup.
|
||||
|
||||
## 0.8.0
|
||||
|
||||
### Features
|
||||
|
||||
* Bulk actions on users owned tracks (reprocess, download, make private, make public, delete) (#269, #38)
|
||||
* Easy sorting by device for "multi-device users" (e.g. group lending out OBSes)
|
||||
* Region display at higher zoom levels to easily find interesting areas (#112)
|
||||
* Export of road statistics on top of the already-existing event statistics (#341)
|
||||
|
||||
### Improvements
|
||||
|
||||
* Refactored database access to hopefully combat portal crashes (#337)
|
||||
* New infrastructure for map imports that makes import of larger maps possible on small VMs (#334)
|
||||
* Reference current postgres and postgis versions in docker-compose.yaml files (#286)
|
||||
* Configurable terms-and-conditions link (#320)
|
||||
* French translation by @cbiteau (#303)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* Logout not working (#285)
|
||||
* Duplicate road usage hashes (#335, #253)
|
||||
* cannot import name .... (#338)
|
||||
|
||||
## 0.7.0
|
||||
|
||||
### Features
|
||||
|
||||
* Add histogram of overtaking distances in road details panel
|
||||
* Flip table in road details panel and make it easier to read
|
||||
* Implement difference between urban and rural for events and road segments.
|
||||
* Better road zone detection in import
|
||||
* Make the frontend translatable and add German translation
|
||||
* Add time and user filters to map view (for logged-in users only)
|
||||
|
||||
### Improvements
|
||||
|
||||
* Make raw track not look like a river (#252)
|
||||
* Update many dependencies
|
||||
|
||||
### Bug fixes
|
||||
|
||||
* Overtaking events are now deleted when the parent track is deleted (#206)
|
||||
* Remove useless session creation (#192)
|
||||
* Remove some error logs for canceled requests (as the map page tends to do that quite a lot)
|
||||
* Fix ExportPage bounding box input
|
||||
|
||||
|
||||
## 0.6.2
|
||||
|
||||
### Improvements
|
||||
|
||||
* Prevent directory traversals inside container on python-served frontend.
|
||||
|
||||
## 0.6.1
|
||||
|
||||
### Improvements
|
||||
|
||||
* Make road details request (clicking on a road segment in the map) way faster
|
||||
by using PostGIS geometry index correctly (#226).
|
||||
|
||||
## 0.6.0
|
||||
|
||||
Starting in this version, the database schema is created through migrations
|
||||
instead of using the `reset_database.py` script. This means that for both the
|
||||
initial setup, as well as for upgrades, only the migrations have to be run.
|
||||
|
||||
After updating and migrating, it is good practice to regenerate the SQL tile
|
||||
functions (`api/tools/prepare_sql_tiles.py`) as well. It doesn't matter if you
|
||||
do this when it is not required, so we've written a simple all-in-one update
|
||||
script that you can run to do all upgrade tasks. This is now in
|
||||
`api/tools/upgrade.py`.
|
||||
|
||||
Please check [`UPGRADING.md`](./UPGRADING.md) for more details if you're
|
||||
upgrading an existing installation. It contains an important note for this
|
||||
upgrade in particular.
|
||||
|
||||
## 0.5.1
|
||||
|
||||
Maintenance release, only includes build, deployment and documentation changes.
|
||||
|
||||
## 0.5.0
|
||||
|
||||
### Features
|
||||
|
||||
* Use discrete colors for distances, with greens only above 1.5m
|
||||
* Use viridis colormap for roads' count layers
|
||||
* Generate usage count information (how often has a road been traveled)
|
||||
* Project the whole track to the map, and show both versions
|
||||
* Log out of OpenID server when logging out of application
|
||||
* Convert speed units to km/h in frontend
|
||||
* Pages now have titles (#148)
|
||||
* Remove map from home page, it was empty anyway (#120)
|
||||
|
||||
### Internal
|
||||
|
||||
* Add alembic setup for migrating
|
||||
* Build osm2pgsql with -j4
|
||||
* Update sqlalchemy[asyncio] requirement from ~=1.4.31 to ~=1.4.32 in /api
|
||||
|
||||
## 0.4.2
|
||||
|
||||
### Features
|
||||
|
||||
### Bugfixes
|
||||
|
||||
* Fix export route, it should be a child of /api
|
||||
|
||||
## 0.4.1
|
||||
|
||||
### Features
|
||||
|
||||
* Add page for exporting data through web frontend
|
||||
* Generate GPX track file when importing a track
|
||||
* Add GPX track export button on the track page (accessible for anybody who can
|
||||
see the track)
|
||||
|
||||
## 0.4.0
|
||||
|
||||
### Improvements
|
||||
|
||||
* Retry OpenID Connect connection if it fails on boot
|
||||
* Format log outputs with color and improve access log
|
||||
* Make pool_size and overflow configurable for worker and portal
|
||||
* Add a route for exporting events as GeoJSON/Shapefile
|
||||
* Point footer to forum, not slack (fixes #140)
|
||||
* Improve wording on profile page ("My" instead of "Your")
|
||||
* Show "My tracks" directly in main menu (fixes #136)
|
||||
|
||||
### Bugfixes
|
||||
|
||||
* Make sure the API can recover from the broken postgresql connection state
|
||||
* Remove duplicate events from the same track
|
||||
* Fix direction of road segments (fixes #142)
|
||||
* Solve a few problems with the colormap scales in the map view
|
||||
|
||||
### Docs & deployment
|
||||
|
||||
* Greatly improve deployement docs for a simple follow-along routine
|
||||
* Use environment variables (`OBS_*`) for configuration
|
||||
* Fix port numbers in example files and expose 3000 in the image
|
||||
* Add `LEAN_MODE` configuration to disable `road` database table usage and fall
|
||||
back to Overpass API for processing tracks (see
|
||||
[docs/lean-mode.md](docs/lean-mode.md)).
|
||||
* Read `config.overrides.py` file if it exists
|
||||
* Add osm2pgsql to portal image to be able to import OSM data from within the
|
||||
container
|
||||
* Fix path to roads_import.lua in docs
|
||||
* Explain to use the portal service, instead of api, in production
|
||||
* Use entrypoint instead of command, so you can run process_track.py one-off tasks
|
||||
|
||||
### Internals
|
||||
|
||||
* Use custom `get_single_arg` everywhere, remove sanicargs (fixes #193)
|
||||
* Update requirements and make them consistent
|
||||
* Fix error handling, especially for file uploads
|
||||
|
||||
|
||||
## 0.3.4
|
||||
|
||||
### Features
|
||||
|
||||
* Reintroduce event view (fixes #111)
|
||||
* Add layer configuration panel to map page
|
||||
- Allow choosing basemap style
|
||||
- Add toggles for event and road layers
|
||||
- Make untagged roads display optional
|
||||
- Show a legend for event color
|
||||
- Alow choosing attribute used for coloring road segments
|
||||
* Add optional banner to frontend via config entry (solves #128)
|
||||
|
||||
### Bugfixes
|
||||
|
||||
* Clicking on road without events should not cause 500 error
|
||||
* Improve mobile layout a bit (fixes #123)
|
||||
|
||||
### Technical
|
||||
|
||||
* Allow explicit configuration of api base url via `API_URL` config
|
||||
* Remove outdated "mapTileset" frontend config section
|
||||
674
COPYING
674
COPYING
|
|
@ -1,674 +0,0 @@
|
|||
GNU GENERAL PUBLIC LICENSE
|
||||
Version 3, 29 June 2007
|
||||
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The GNU General Public License is a free, copyleft license for
|
||||
software and other kinds of works.
|
||||
|
||||
The licenses for most software and other practical works are designed
|
||||
to take away your freedom to share and change the works. By contrast,
|
||||
the GNU General Public License is intended to guarantee your freedom to
|
||||
share and change all versions of a program--to make sure it remains free
|
||||
software for all its users. We, the Free Software Foundation, use the
|
||||
GNU General Public License for most of our software; it applies also to
|
||||
any other work released this way by its authors. You can apply it to
|
||||
your programs, too.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not
|
||||
price. Our General Public Licenses are designed to make sure that you
|
||||
have the freedom to distribute copies of free software (and charge for
|
||||
them if you wish), that you receive source code or can get it if you
|
||||
want it, that you can change the software or use pieces of it in new
|
||||
free programs, and that you know you can do these things.
|
||||
|
||||
To protect your rights, we need to prevent others from denying you
|
||||
these rights or asking you to surrender the rights. Therefore, you have
|
||||
certain responsibilities if you distribute copies of the software, or if
|
||||
you modify it: responsibilities to respect the freedom of others.
|
||||
|
||||
For example, if you distribute copies of such a program, whether
|
||||
gratis or for a fee, you must pass on to the recipients the same
|
||||
freedoms that you received. You must make sure that they, too, receive
|
||||
or can get the source code. And you must show them these terms so they
|
||||
know their rights.
|
||||
|
||||
Developers that use the GNU GPL protect your rights with two steps:
|
||||
(1) assert copyright on the software, and (2) offer you this License
|
||||
giving you legal permission to copy, distribute and/or modify it.
|
||||
|
||||
For the developers' and authors' protection, the GPL clearly explains
|
||||
that there is no warranty for this free software. For both users' and
|
||||
authors' sake, the GPL requires that modified versions be marked as
|
||||
changed, so that their problems will not be attributed erroneously to
|
||||
authors of previous versions.
|
||||
|
||||
Some devices are designed to deny users access to install or run
|
||||
modified versions of the software inside them, although the manufacturer
|
||||
can do so. This is fundamentally incompatible with the aim of
|
||||
protecting users' freedom to change the software. The systematic
|
||||
pattern of such abuse occurs in the area of products for individuals to
|
||||
use, which is precisely where it is most unacceptable. Therefore, we
|
||||
have designed this version of the GPL to prohibit the practice for those
|
||||
products. If such problems arise substantially in other domains, we
|
||||
stand ready to extend this provision to those domains in future versions
|
||||
of the GPL, as needed to protect the freedom of users.
|
||||
|
||||
Finally, every program is threatened constantly by software patents.
|
||||
States should not allow patents to restrict development and use of
|
||||
software on general-purpose computers, but in those that do, we wish to
|
||||
avoid the special danger that patents applied to a free program could
|
||||
make it effectively proprietary. To prevent this, the GPL assures that
|
||||
patents cannot be used to render the program non-free.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow.
|
||||
|
||||
TERMS AND CONDITIONS
|
||||
|
||||
0. Definitions.
|
||||
|
||||
"This License" refers to version 3 of the GNU General Public License.
|
||||
|
||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||
works, such as semiconductor masks.
|
||||
|
||||
"The Program" refers to any copyrightable work licensed under this
|
||||
License. Each licensee is addressed as "you". "Licensees" and
|
||||
"recipients" may be individuals or organizations.
|
||||
|
||||
To "modify" a work means to copy from or adapt all or part of the work
|
||||
in a fashion requiring copyright permission, other than the making of an
|
||||
exact copy. The resulting work is called a "modified version" of the
|
||||
earlier work or a work "based on" the earlier work.
|
||||
|
||||
A "covered work" means either the unmodified Program or a work based
|
||||
on the Program.
|
||||
|
||||
To "propagate" a work means to do anything with it that, without
|
||||
permission, would make you directly or secondarily liable for
|
||||
infringement under applicable copyright law, except executing it on a
|
||||
computer or modifying a private copy. Propagation includes copying,
|
||||
distribution (with or without modification), making available to the
|
||||
public, and in some countries other activities as well.
|
||||
|
||||
To "convey" a work means any kind of propagation that enables other
|
||||
parties to make or receive copies. Mere interaction with a user through
|
||||
a computer network, with no transfer of a copy, is not conveying.
|
||||
|
||||
An interactive user interface displays "Appropriate Legal Notices"
|
||||
to the extent that it includes a convenient and prominently visible
|
||||
feature that (1) displays an appropriate copyright notice, and (2)
|
||||
tells the user that there is no warranty for the work (except to the
|
||||
extent that warranties are provided), that licensees may convey the
|
||||
work under this License, and how to view a copy of this License. If
|
||||
the interface presents a list of user commands or options, such as a
|
||||
menu, a prominent item in the list meets this criterion.
|
||||
|
||||
1. Source Code.
|
||||
|
||||
The "source code" for a work means the preferred form of the work
|
||||
for making modifications to it. "Object code" means any non-source
|
||||
form of a work.
|
||||
|
||||
A "Standard Interface" means an interface that either is an official
|
||||
standard defined by a recognized standards body, or, in the case of
|
||||
interfaces specified for a particular programming language, one that
|
||||
is widely used among developers working in that language.
|
||||
|
||||
The "System Libraries" of an executable work include anything, other
|
||||
than the work as a whole, that (a) is included in the normal form of
|
||||
packaging a Major Component, but which is not part of that Major
|
||||
Component, and (b) serves only to enable use of the work with that
|
||||
Major Component, or to implement a Standard Interface for which an
|
||||
implementation is available to the public in source code form. A
|
||||
"Major Component", in this context, means a major essential component
|
||||
(kernel, window system, and so on) of the specific operating system
|
||||
(if any) on which the executable work runs, or a compiler used to
|
||||
produce the work, or an object code interpreter used to run it.
|
||||
|
||||
The "Corresponding Source" for a work in object code form means all
|
||||
the source code needed to generate, install, and (for an executable
|
||||
work) run the object code and to modify the work, including scripts to
|
||||
control those activities. However, it does not include the work's
|
||||
System Libraries, or general-purpose tools or generally available free
|
||||
programs which are used unmodified in performing those activities but
|
||||
which are not part of the work. For example, Corresponding Source
|
||||
includes interface definition files associated with source files for
|
||||
the work, and the source code for shared libraries and dynamically
|
||||
linked subprograms that the work is specifically designed to require,
|
||||
such as by intimate data communication or control flow between those
|
||||
subprograms and other parts of the work.
|
||||
|
||||
The Corresponding Source need not include anything that users
|
||||
can regenerate automatically from other parts of the Corresponding
|
||||
Source.
|
||||
|
||||
The Corresponding Source for a work in source code form is that
|
||||
same work.
|
||||
|
||||
2. Basic Permissions.
|
||||
|
||||
All rights granted under this License are granted for the term of
|
||||
copyright on the Program, and are irrevocable provided the stated
|
||||
conditions are met. This License explicitly affirms your unlimited
|
||||
permission to run the unmodified Program. The output from running a
|
||||
covered work is covered by this License only if the output, given its
|
||||
content, constitutes a covered work. This License acknowledges your
|
||||
rights of fair use or other equivalent, as provided by copyright law.
|
||||
|
||||
You may make, run and propagate covered works that you do not
|
||||
convey, without conditions so long as your license otherwise remains
|
||||
in force. You may convey covered works to others for the sole purpose
|
||||
of having them make modifications exclusively for you, or provide you
|
||||
with facilities for running those works, provided that you comply with
|
||||
the terms of this License in conveying all material for which you do
|
||||
not control copyright. Those thus making or running the covered works
|
||||
for you must do so exclusively on your behalf, under your direction
|
||||
and control, on terms that prohibit them from making any copies of
|
||||
your copyrighted material outside their relationship with you.
|
||||
|
||||
Conveying under any other circumstances is permitted solely under
|
||||
the conditions stated below. Sublicensing is not allowed; section 10
|
||||
makes it unnecessary.
|
||||
|
||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||
|
||||
No covered work shall be deemed part of an effective technological
|
||||
measure under any applicable law fulfilling obligations under article
|
||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||
similar laws prohibiting or restricting circumvention of such
|
||||
measures.
|
||||
|
||||
When you convey a covered work, you waive any legal power to forbid
|
||||
circumvention of technological measures to the extent such circumvention
|
||||
is effected by exercising rights under this License with respect to
|
||||
the covered work, and you disclaim any intention to limit operation or
|
||||
modification of the work as a means of enforcing, against the work's
|
||||
users, your or third parties' legal rights to forbid circumvention of
|
||||
technological measures.
|
||||
|
||||
4. Conveying Verbatim Copies.
|
||||
|
||||
You may convey verbatim copies of the Program's source code as you
|
||||
receive it, in any medium, provided that you conspicuously and
|
||||
appropriately publish on each copy an appropriate copyright notice;
|
||||
keep intact all notices stating that this License and any
|
||||
non-permissive terms added in accord with section 7 apply to the code;
|
||||
keep intact all notices of the absence of any warranty; and give all
|
||||
recipients a copy of this License along with the Program.
|
||||
|
||||
You may charge any price or no price for each copy that you convey,
|
||||
and you may offer support or warranty protection for a fee.
|
||||
|
||||
5. Conveying Modified Source Versions.
|
||||
|
||||
You may convey a work based on the Program, or the modifications to
|
||||
produce it from the Program, in the form of source code under the
|
||||
terms of section 4, provided that you also meet all of these conditions:
|
||||
|
||||
a) The work must carry prominent notices stating that you modified
|
||||
it, and giving a relevant date.
|
||||
|
||||
b) The work must carry prominent notices stating that it is
|
||||
released under this License and any conditions added under section
|
||||
7. This requirement modifies the requirement in section 4 to
|
||||
"keep intact all notices".
|
||||
|
||||
c) You must license the entire work, as a whole, under this
|
||||
License to anyone who comes into possession of a copy. This
|
||||
License will therefore apply, along with any applicable section 7
|
||||
additional terms, to the whole of the work, and all its parts,
|
||||
regardless of how they are packaged. This License gives no
|
||||
permission to license the work in any other way, but it does not
|
||||
invalidate such permission if you have separately received it.
|
||||
|
||||
d) If the work has interactive user interfaces, each must display
|
||||
Appropriate Legal Notices; however, if the Program has interactive
|
||||
interfaces that do not display Appropriate Legal Notices, your
|
||||
work need not make them do so.
|
||||
|
||||
A compilation of a covered work with other separate and independent
|
||||
works, which are not by their nature extensions of the covered work,
|
||||
and which are not combined with it such as to form a larger program,
|
||||
in or on a volume of a storage or distribution medium, is called an
|
||||
"aggregate" if the compilation and its resulting copyright are not
|
||||
used to limit the access or legal rights of the compilation's users
|
||||
beyond what the individual works permit. Inclusion of a covered work
|
||||
in an aggregate does not cause this License to apply to the other
|
||||
parts of the aggregate.
|
||||
|
||||
6. Conveying Non-Source Forms.
|
||||
|
||||
You may convey a covered work in object code form under the terms
|
||||
of sections 4 and 5, provided that you also convey the
|
||||
machine-readable Corresponding Source under the terms of this License,
|
||||
in one of these ways:
|
||||
|
||||
a) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by the
|
||||
Corresponding Source fixed on a durable physical medium
|
||||
customarily used for software interchange.
|
||||
|
||||
b) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by a
|
||||
written offer, valid for at least three years and valid for as
|
||||
long as you offer spare parts or customer support for that product
|
||||
model, to give anyone who possesses the object code either (1) a
|
||||
copy of the Corresponding Source for all the software in the
|
||||
product that is covered by this License, on a durable physical
|
||||
medium customarily used for software interchange, for a price no
|
||||
more than your reasonable cost of physically performing this
|
||||
conveying of source, or (2) access to copy the
|
||||
Corresponding Source from a network server at no charge.
|
||||
|
||||
c) Convey individual copies of the object code with a copy of the
|
||||
written offer to provide the Corresponding Source. This
|
||||
alternative is allowed only occasionally and noncommercially, and
|
||||
only if you received the object code with such an offer, in accord
|
||||
with subsection 6b.
|
||||
|
||||
d) Convey the object code by offering access from a designated
|
||||
place (gratis or for a charge), and offer equivalent access to the
|
||||
Corresponding Source in the same way through the same place at no
|
||||
further charge. You need not require recipients to copy the
|
||||
Corresponding Source along with the object code. If the place to
|
||||
copy the object code is a network server, the Corresponding Source
|
||||
may be on a different server (operated by you or a third party)
|
||||
that supports equivalent copying facilities, provided you maintain
|
||||
clear directions next to the object code saying where to find the
|
||||
Corresponding Source. Regardless of what server hosts the
|
||||
Corresponding Source, you remain obligated to ensure that it is
|
||||
available for as long as needed to satisfy these requirements.
|
||||
|
||||
e) Convey the object code using peer-to-peer transmission, provided
|
||||
you inform other peers where the object code and Corresponding
|
||||
Source of the work are being offered to the general public at no
|
||||
charge under subsection 6d.
|
||||
|
||||
A separable portion of the object code, whose source code is excluded
|
||||
from the Corresponding Source as a System Library, need not be
|
||||
included in conveying the object code work.
|
||||
|
||||
A "User Product" is either (1) a "consumer product", which means any
|
||||
tangible personal property which is normally used for personal, family,
|
||||
or household purposes, or (2) anything designed or sold for incorporation
|
||||
into a dwelling. In determining whether a product is a consumer product,
|
||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||
product received by a particular user, "normally used" refers to a
|
||||
typical or common use of that class of product, regardless of the status
|
||||
of the particular user or of the way in which the particular user
|
||||
actually uses, or expects or is expected to use, the product. A product
|
||||
is a consumer product regardless of whether the product has substantial
|
||||
commercial, industrial or non-consumer uses, unless such uses represent
|
||||
the only significant mode of use of the product.
|
||||
|
||||
"Installation Information" for a User Product means any methods,
|
||||
procedures, authorization keys, or other information required to install
|
||||
and execute modified versions of a covered work in that User Product from
|
||||
a modified version of its Corresponding Source. The information must
|
||||
suffice to ensure that the continued functioning of the modified object
|
||||
code is in no case prevented or interfered with solely because
|
||||
modification has been made.
|
||||
|
||||
If you convey an object code work under this section in, or with, or
|
||||
specifically for use in, a User Product, and the conveying occurs as
|
||||
part of a transaction in which the right of possession and use of the
|
||||
User Product is transferred to the recipient in perpetuity or for a
|
||||
fixed term (regardless of how the transaction is characterized), the
|
||||
Corresponding Source conveyed under this section must be accompanied
|
||||
by the Installation Information. But this requirement does not apply
|
||||
if neither you nor any third party retains the ability to install
|
||||
modified object code on the User Product (for example, the work has
|
||||
been installed in ROM).
|
||||
|
||||
The requirement to provide Installation Information does not include a
|
||||
requirement to continue to provide support service, warranty, or updates
|
||||
for a work that has been modified or installed by the recipient, or for
|
||||
the User Product in which it has been modified or installed. Access to a
|
||||
network may be denied when the modification itself materially and
|
||||
adversely affects the operation of the network or violates the rules and
|
||||
protocols for communication across the network.
|
||||
|
||||
Corresponding Source conveyed, and Installation Information provided,
|
||||
in accord with this section must be in a format that is publicly
|
||||
documented (and with an implementation available to the public in
|
||||
source code form), and must require no special password or key for
|
||||
unpacking, reading or copying.
|
||||
|
||||
7. Additional Terms.
|
||||
|
||||
"Additional permissions" are terms that supplement the terms of this
|
||||
License by making exceptions from one or more of its conditions.
|
||||
Additional permissions that are applicable to the entire Program shall
|
||||
be treated as though they were included in this License, to the extent
|
||||
that they are valid under applicable law. If additional permissions
|
||||
apply only to part of the Program, that part may be used separately
|
||||
under those permissions, but the entire Program remains governed by
|
||||
this License without regard to the additional permissions.
|
||||
|
||||
When you convey a copy of a covered work, you may at your option
|
||||
remove any additional permissions from that copy, or from any part of
|
||||
it. (Additional permissions may be written to require their own
|
||||
removal in certain cases when you modify the work.) You may place
|
||||
additional permissions on material, added by you to a covered work,
|
||||
for which you have or can give appropriate copyright permission.
|
||||
|
||||
Notwithstanding any other provision of this License, for material you
|
||||
add to a covered work, you may (if authorized by the copyright holders of
|
||||
that material) supplement the terms of this License with terms:
|
||||
|
||||
a) Disclaiming warranty or limiting liability differently from the
|
||||
terms of sections 15 and 16 of this License; or
|
||||
|
||||
b) Requiring preservation of specified reasonable legal notices or
|
||||
author attributions in that material or in the Appropriate Legal
|
||||
Notices displayed by works containing it; or
|
||||
|
||||
c) Prohibiting misrepresentation of the origin of that material, or
|
||||
requiring that modified versions of such material be marked in
|
||||
reasonable ways as different from the original version; or
|
||||
|
||||
d) Limiting the use for publicity purposes of names of licensors or
|
||||
authors of the material; or
|
||||
|
||||
e) Declining to grant rights under trademark law for use of some
|
||||
trade names, trademarks, or service marks; or
|
||||
|
||||
f) Requiring indemnification of licensors and authors of that
|
||||
material by anyone who conveys the material (or modified versions of
|
||||
it) with contractual assumptions of liability to the recipient, for
|
||||
any liability that these contractual assumptions directly impose on
|
||||
those licensors and authors.
|
||||
|
||||
All other non-permissive additional terms are considered "further
|
||||
restrictions" within the meaning of section 10. If the Program as you
|
||||
received it, or any part of it, contains a notice stating that it is
|
||||
governed by this License along with a term that is a further
|
||||
restriction, you may remove that term. If a license document contains
|
||||
a further restriction but permits relicensing or conveying under this
|
||||
License, you may add to a covered work material governed by the terms
|
||||
of that license document, provided that the further restriction does
|
||||
not survive such relicensing or conveying.
|
||||
|
||||
If you add terms to a covered work in accord with this section, you
|
||||
must place, in the relevant source files, a statement of the
|
||||
additional terms that apply to those files, or a notice indicating
|
||||
where to find the applicable terms.
|
||||
|
||||
Additional terms, permissive or non-permissive, may be stated in the
|
||||
form of a separately written license, or stated as exceptions;
|
||||
the above requirements apply either way.
|
||||
|
||||
8. Termination.
|
||||
|
||||
You may not propagate or modify a covered work except as expressly
|
||||
provided under this License. Any attempt otherwise to propagate or
|
||||
modify it is void, and will automatically terminate your rights under
|
||||
this License (including any patent licenses granted under the third
|
||||
paragraph of section 11).
|
||||
|
||||
However, if you cease all violation of this License, then your
|
||||
license from a particular copyright holder is reinstated (a)
|
||||
provisionally, unless and until the copyright holder explicitly and
|
||||
finally terminates your license, and (b) permanently, if the copyright
|
||||
holder fails to notify you of the violation by some reasonable means
|
||||
prior to 60 days after the cessation.
|
||||
|
||||
Moreover, your license from a particular copyright holder is
|
||||
reinstated permanently if the copyright holder notifies you of the
|
||||
violation by some reasonable means, this is the first time you have
|
||||
received notice of violation of this License (for any work) from that
|
||||
copyright holder, and you cure the violation prior to 30 days after
|
||||
your receipt of the notice.
|
||||
|
||||
Termination of your rights under this section does not terminate the
|
||||
licenses of parties who have received copies or rights from you under
|
||||
this License. If your rights have been terminated and not permanently
|
||||
reinstated, you do not qualify to receive new licenses for the same
|
||||
material under section 10.
|
||||
|
||||
9. Acceptance Not Required for Having Copies.
|
||||
|
||||
You are not required to accept this License in order to receive or
|
||||
run a copy of the Program. Ancillary propagation of a covered work
|
||||
occurring solely as a consequence of using peer-to-peer transmission
|
||||
to receive a copy likewise does not require acceptance. However,
|
||||
nothing other than this License grants you permission to propagate or
|
||||
modify any covered work. These actions infringe copyright if you do
|
||||
not accept this License. Therefore, by modifying or propagating a
|
||||
covered work, you indicate your acceptance of this License to do so.
|
||||
|
||||
10. Automatic Licensing of Downstream Recipients.
|
||||
|
||||
Each time you convey a covered work, the recipient automatically
|
||||
receives a license from the original licensors, to run, modify and
|
||||
propagate that work, subject to this License. You are not responsible
|
||||
for enforcing compliance by third parties with this License.
|
||||
|
||||
An "entity transaction" is a transaction transferring control of an
|
||||
organization, or substantially all assets of one, or subdividing an
|
||||
organization, or merging organizations. If propagation of a covered
|
||||
work results from an entity transaction, each party to that
|
||||
transaction who receives a copy of the work also receives whatever
|
||||
licenses to the work the party's predecessor in interest had or could
|
||||
give under the previous paragraph, plus a right to possession of the
|
||||
Corresponding Source of the work from the predecessor in interest, if
|
||||
the predecessor has it or can get it with reasonable efforts.
|
||||
|
||||
You may not impose any further restrictions on the exercise of the
|
||||
rights granted or affirmed under this License. For example, you may
|
||||
not impose a license fee, royalty, or other charge for exercise of
|
||||
rights granted under this License, and you may not initiate litigation
|
||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||
any patent claim is infringed by making, using, selling, offering for
|
||||
sale, or importing the Program or any portion of it.
|
||||
|
||||
11. Patents.
|
||||
|
||||
A "contributor" is a copyright holder who authorizes use under this
|
||||
License of the Program or a work on which the Program is based. The
|
||||
work thus licensed is called the contributor's "contributor version".
|
||||
|
||||
A contributor's "essential patent claims" are all patent claims
|
||||
owned or controlled by the contributor, whether already acquired or
|
||||
hereafter acquired, that would be infringed by some manner, permitted
|
||||
by this License, of making, using, or selling its contributor version,
|
||||
but do not include claims that would be infringed only as a
|
||||
consequence of further modification of the contributor version. For
|
||||
purposes of this definition, "control" includes the right to grant
|
||||
patent sublicenses in a manner consistent with the requirements of
|
||||
this License.
|
||||
|
||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||
patent license under the contributor's essential patent claims, to
|
||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||
propagate the contents of its contributor version.
|
||||
|
||||
In the following three paragraphs, a "patent license" is any express
|
||||
agreement or commitment, however denominated, not to enforce a patent
|
||||
(such as an express permission to practice a patent or covenant not to
|
||||
sue for patent infringement). To "grant" such a patent license to a
|
||||
party means to make such an agreement or commitment not to enforce a
|
||||
patent against the party.
|
||||
|
||||
If you convey a covered work, knowingly relying on a patent license,
|
||||
and the Corresponding Source of the work is not available for anyone
|
||||
to copy, free of charge and under the terms of this License, through a
|
||||
publicly available network server or other readily accessible means,
|
||||
then you must either (1) cause the Corresponding Source to be so
|
||||
available, or (2) arrange to deprive yourself of the benefit of the
|
||||
patent license for this particular work, or (3) arrange, in a manner
|
||||
consistent with the requirements of this License, to extend the patent
|
||||
license to downstream recipients. "Knowingly relying" means you have
|
||||
actual knowledge that, but for the patent license, your conveying the
|
||||
covered work in a country, or your recipient's use of the covered work
|
||||
in a country, would infringe one or more identifiable patents in that
|
||||
country that you have reason to believe are valid.
|
||||
|
||||
If, pursuant to or in connection with a single transaction or
|
||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||
covered work, and grant a patent license to some of the parties
|
||||
receiving the covered work authorizing them to use, propagate, modify
|
||||
or convey a specific copy of the covered work, then the patent license
|
||||
you grant is automatically extended to all recipients of the covered
|
||||
work and works based on it.
|
||||
|
||||
A patent license is "discriminatory" if it does not include within
|
||||
the scope of its coverage, prohibits the exercise of, or is
|
||||
conditioned on the non-exercise of one or more of the rights that are
|
||||
specifically granted under this License. You may not convey a covered
|
||||
work if you are a party to an arrangement with a third party that is
|
||||
in the business of distributing software, under which you make payment
|
||||
to the third party based on the extent of your activity of conveying
|
||||
the work, and under which the third party grants, to any of the
|
||||
parties who would receive the covered work from you, a discriminatory
|
||||
patent license (a) in connection with copies of the covered work
|
||||
conveyed by you (or copies made from those copies), or (b) primarily
|
||||
for and in connection with specific products or compilations that
|
||||
contain the covered work, unless you entered into that arrangement,
|
||||
or that patent license was granted, prior to 28 March 2007.
|
||||
|
||||
Nothing in this License shall be construed as excluding or limiting
|
||||
any implied license or other defenses to infringement that may
|
||||
otherwise be available to you under applicable patent law.
|
||||
|
||||
12. No Surrender of Others' Freedom.
|
||||
|
||||
If conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot convey a
|
||||
covered work so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you may
|
||||
not convey it at all. For example, if you agree to terms that obligate you
|
||||
to collect a royalty for further conveying from those to whom you convey
|
||||
the Program, the only way you could satisfy both those terms and this
|
||||
License would be to refrain entirely from conveying the Program.
|
||||
|
||||
13. Use with the GNU Affero General Public License.
|
||||
|
||||
Notwithstanding any other provision of this License, you have
|
||||
permission to link or combine any covered work with a work licensed
|
||||
under version 3 of the GNU Affero General Public License into a single
|
||||
combined work, and to convey the resulting work. The terms of this
|
||||
License will continue to apply to the part which is the covered work,
|
||||
but the special requirements of the GNU Affero General Public License,
|
||||
section 13, concerning interaction through a network will apply to the
|
||||
combination as such.
|
||||
|
||||
14. Revised Versions of this License.
|
||||
|
||||
The Free Software Foundation may publish revised and/or new versions of
|
||||
the GNU General Public License from time to time. Such new versions will
|
||||
be similar in spirit to the present version, but may differ in detail to
|
||||
address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the
|
||||
Program specifies that a certain numbered version of the GNU General
|
||||
Public License "or any later version" applies to it, you have the
|
||||
option of following the terms and conditions either of that numbered
|
||||
version or of any later version published by the Free Software
|
||||
Foundation. If the Program does not specify a version number of the
|
||||
GNU General Public License, you may choose any version ever published
|
||||
by the Free Software Foundation.
|
||||
|
||||
If the Program specifies that a proxy can decide which future
|
||||
versions of the GNU General Public License can be used, that proxy's
|
||||
public statement of acceptance of a version permanently authorizes you
|
||||
to choose that version for the Program.
|
||||
|
||||
Later license versions may give you additional or different
|
||||
permissions. However, no additional obligations are imposed on any
|
||||
author or copyright holder as a result of your choosing to follow a
|
||||
later version.
|
||||
|
||||
15. Disclaimer of Warranty.
|
||||
|
||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
16. Limitation of Liability.
|
||||
|
||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||
SUCH DAMAGES.
|
||||
|
||||
17. Interpretation of Sections 15 and 16.
|
||||
|
||||
If the disclaimer of warranty and limitation of liability provided
|
||||
above cannot be given local legal effect according to their terms,
|
||||
reviewing courts shall apply local law that most closely approximates
|
||||
an absolute waiver of all civil liability in connection with the
|
||||
Program, unless a warranty or assumption of liability accompanies a
|
||||
copy of the Program in return for a fee.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Programs
|
||||
|
||||
If you develop a new program, and you want it to be of the greatest
|
||||
possible use to the public, the best way to achieve this is to make it
|
||||
free software which everyone can redistribute and change under these terms.
|
||||
|
||||
To do so, attach the following notices to the program. It is safest
|
||||
to attach them to the start of each source file to most effectively
|
||||
state the exclusion of warranty; and each file should have at least
|
||||
the "copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the program's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU General Public License as published by
|
||||
the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU General Public License
|
||||
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
If the program does terminal interaction, make it output a short
|
||||
notice like this when it starts in an interactive mode:
|
||||
|
||||
<program> Copyright (C) <year> <name of author>
|
||||
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
|
||||
This is free software, and you are welcome to redistribute it
|
||||
under certain conditions; type `show c' for details.
|
||||
|
||||
The hypothetical commands `show w' and `show c' should show the appropriate
|
||||
parts of the General Public License. Of course, your program's commands
|
||||
might be different; for a GUI interface, you would use an "about box".
|
||||
|
||||
You should also get your employer (if you work as a programmer) or school,
|
||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||
For more information on this, and how to apply and follow the GNU GPL, see
|
||||
<https://www.gnu.org/licenses/>.
|
||||
|
||||
The GNU General Public License does not permit incorporating your program
|
||||
into proprietary programs. If your program is a subroutine library, you
|
||||
may consider it more useful to permit linking proprietary applications with
|
||||
the library. If this is what you want to do, use the GNU Lesser General
|
||||
Public License instead of this License. But first, please read
|
||||
<https://www.gnu.org/licenses/why-not-lgpl.html>.
|
||||
61
Dockerfile
61
Dockerfile
|
|
@ -1,61 +0,0 @@
|
|||
# This dockerfile is for the API + Frontend production image
|
||||
|
||||
#############################################
|
||||
# Build the frontend AS builder
|
||||
#############################################
|
||||
|
||||
FROM node:18 as frontend-builder
|
||||
|
||||
WORKDIR /opt/obs/frontend
|
||||
ADD frontend/package.json frontend/package-lock.json /opt/obs/frontend/
|
||||
RUN echo update-notifier=false >> ~/.npmrc
|
||||
RUN npm ci
|
||||
|
||||
ADD frontend/tsconfig.json frontend/webpack.config.js /opt/obs/frontend/
|
||||
ADD frontend/src /opt/obs/frontend/src/
|
||||
ADD frontend/public /opt/obs/frontend/public/
|
||||
|
||||
RUN npm run build
|
||||
|
||||
#############################################
|
||||
# Build the API and add the built frontend to it
|
||||
#############################################
|
||||
|
||||
FROM python:3.11.3-bullseye
|
||||
|
||||
RUN apt-get update &&\
|
||||
apt-get install -y \
|
||||
libboost-dev \
|
||||
libboost-system-dev \
|
||||
libboost-filesystem-dev \
|
||||
libexpat1-dev \
|
||||
zlib1g-dev \
|
||||
libbz2-dev \
|
||||
libpq-dev \
|
||||
libproj-dev \
|
||||
lua5.3 \
|
||||
liblua5.3-dev &&\
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /opt/obs/api
|
||||
|
||||
ADD api/requirements.txt /opt/obs/api/
|
||||
RUN pip install -r requirements.txt
|
||||
|
||||
ADD tile-generator /opt/obs/tile-generator
|
||||
|
||||
ADD api/scripts /opt/obs/scripts
|
||||
RUN pip install -e /opt/obs/scripts
|
||||
|
||||
ADD api/setup.py /opt/obs/api/
|
||||
ADD api/alembic.ini /opt/obs/api/
|
||||
ADD api/migrations /opt/obs/api/migrations/
|
||||
ADD api/obs /opt/obs/api/obs/
|
||||
ADD api/tools /opt/obs/api/tools/
|
||||
RUN pip install -e /opt/obs/api/
|
||||
|
||||
COPY --from=frontend-builder /opt/obs/frontend/build /opt/obs/frontend/build
|
||||
|
||||
EXPOSE 3000
|
||||
|
||||
CMD ["openbikesensor-api"]
|
||||
308
README.md
308
README.md
|
|
@ -4,23 +4,12 @@ This repository contains the source code required to run the
|
|||
[OpenBikeSensor](https://openbikesensor.org) data collection portal. It is
|
||||
separated into components:
|
||||
|
||||
* **api**: The backend service, written in Python 3 with
|
||||
[Sanic](https://sanicframework.org/),
|
||||
[SQLAlchemy](https://www.sqlalchemy.org/), and a PostgreSQL/PostGIS database
|
||||
for storage. It also depends highly on
|
||||
[OpenMapTiles](https://openmaptiles.org) to generate vector tiles of the
|
||||
data.
|
||||
* **api**: The backend service, written in JavaScript for Node.js, using
|
||||
express.js, and a MongoDB for metadata storage.
|
||||
* **frontend**: A React single-page application that allows access to the data,
|
||||
provides summaries and visualizations, and lets users adjust settings and
|
||||
manage and publish their tracks.
|
||||
|
||||
Check out the [Architecture Documentation](docs/architecture.md) for more
|
||||
details on what parts the whole application is made of.
|
||||
|
||||
This project follows [semantic versioning](https://semver.org). Refer to [issue
|
||||
#44](https://github.com/openbikesensor/portal/issues/44) for a description of
|
||||
what that means for our project and what is considered the public interface.
|
||||
|
||||
## Clone the Project
|
||||
|
||||
First of all, you must clone this project. This project uses submodules,
|
||||
|
|
@ -36,195 +25,146 @@ git submodule update --init --recursive
|
|||
|
||||
## Production setup
|
||||
|
||||
There is a guide for a deployment based on docker at
|
||||
[docs/production-deployment.md](docs/production-deployment.md). Lots of
|
||||
non-docker deployment strategies are possible, but they are not "officially"
|
||||
supported, so please do not expect the authors of the software to assist in
|
||||
troubleshooting.
|
||||
**Note:** There is now a guide for a dockerized deployment in the
|
||||
[deployment](deployment) folder.
|
||||
|
||||
This is a rather complex application, and it is expected that you know the
|
||||
basics of deploying a modern web application securely onto a production server.
|
||||
We are sorry that we cannot guide you through all the details of that, as we
|
||||
just don't have the capacities to do so. Please research the respective topics
|
||||
first. If you struggle with application-specific issues, please let us know, we
|
||||
might be able to assist with those.
|
||||
You should be familiar with managing a Linux server. If not, find a suitable
|
||||
guide first. This will only give a rough outline of the steps to take, you must
|
||||
make sure to properly set up and secure your server yourself.
|
||||
|
||||
Please note that you will always need to install your own reverse proxy that
|
||||
terminates TLS for you and handles certificates. We do not support TLS directly
|
||||
in the application, instead, please use this prefered method.
|
||||
|
||||
Upgrading and migrating is described in [UPGRADING.md](./UPGRADING.md) for each
|
||||
version.
|
||||
|
||||
### Migrating (Production)
|
||||
|
||||
Migrations are done with
|
||||
[Alembic](https://alembic.sqlalchemy.org/en/latest/index.html), please refer to
|
||||
its documentation for help. Most of the time, running this command will do all
|
||||
the migrations you need:
|
||||
|
||||
```bash
|
||||
docker-compose run --rm api tools/upgrade.py
|
||||
```
|
||||
|
||||
This command is equivalent to running migrations through *alembic*, then
|
||||
regenerating the SQL functions that compute vector tiles directly in the
|
||||
database:
|
||||
|
||||
```bash
|
||||
# equivalent to the above command, you don't usually run these
|
||||
docker-compose run --rm api alembic upgrade head
|
||||
docker-compose run --rm api tools/prepare_sql_tiles
|
||||
```
|
||||
1. Create a user for running the application. It is not recommended to run as a
|
||||
user that is also used for other things. Do not run as root!
|
||||
2. Clone the repository.
|
||||
3. Install `node` of at least version 15, and also `npm`. You can use the
|
||||
[Node Version Manager (nvm)](https://github.com/nvm-sh/nvm), since this project
|
||||
contains a `.nvmrc`. Just run `nvm install` followed by `nvm use`.
|
||||
4. Run `npm ci` in the `api` and `frontend` directories to install dependencies.
|
||||
5. Install and configure a MongoDB somewhere.
|
||||
6. Copy `api/config.json.example` to `api/config.json` and change it to suit
|
||||
your setup. Make sure to only use https URLs. Generate secure secrets.
|
||||
Customize client IDs.
|
||||
7. Copy `frontend/src/config.json.example` to `frontend/src/config.json` and
|
||||
adjust.
|
||||
8. Run `npm run build` in the frontend directory. This needs to be done after
|
||||
changing the config, so if you did something wrong, re-run the build. If you
|
||||
run your frontend on a non-root URL, run the build with the
|
||||
`PUBLIC_URL=/prefix` environment variable.
|
||||
9. Create a systemd-user service, or use tmux, or do whatever you like to start
|
||||
the API service (`npm start` in `api/` directory). It should run as the
|
||||
dedicated user, and expose `127.0.0.1:3000`. Use `PORT` environment variable
|
||||
if you want to change the port. Make sure the service starts with
|
||||
environment `NODE_ENV=production`, too.
|
||||
10. Configure nginx or your reverse-proxy of choice to forward API requests to
|
||||
the API port, and serve static files from the `frontend/build/` folder.
|
||||
|
||||
## Development setup
|
||||
|
||||
We've moved the whole development setup into Docker to make it easy for
|
||||
everyone to get involved.
|
||||
|
||||
### Install docker
|
||||
|
||||
Please [install Docker Engine](https://docs.docker.com/engine/install/) as well as
|
||||
[Docker Compose](https://docs.docker.com/compose/install/) onto your machine.
|
||||
|
||||
Then clone the repository as described above.
|
||||
|
||||
### Configure Keycloak
|
||||
|
||||
Login will not be possible until you configure the keycloak realm correctly. Boot your keycloak instance:
|
||||
everyone to get involved. After sucessfully [installing Docker
|
||||
Engine](https://docs.docker.com/engine/install/) as well as [Docker
|
||||
Compose](https://docs.docker.com/compose/install/) onto your machine, and
|
||||
cloning the repository, all you need to do is:
|
||||
|
||||
```bash
|
||||
docker-compose up -d keycloak
|
||||
docker-compose up -d --build
|
||||
```
|
||||
|
||||
Now navigate to http://localhost:3003/ and follow these steps:
|
||||
|
||||
- Click *Administration Console* and log in with `admin` / `admin`.
|
||||
- Hover over the realm name on the top left and click *Add realm*.
|
||||
- Name the Realm `obs-dev` (spelling matters) and create it.
|
||||
- In the sidebar, navigate to *Configure* → *Clients*, and click *Create* on the top right.
|
||||
- *Client ID* should be `portal`. Click *Save*.
|
||||
- In the Tab *Settings*, edit the new client's *Access Type* to *confidential*
|
||||
and enter as *Valid Redirect URIs*: `http://localhost:3000/login/redirect`,
|
||||
then *Save*
|
||||
- Under *Credentials*, copy the *Secret*. Create a file at `api/config.overrides.py` with the secret in it:
|
||||
|
||||
```python
|
||||
KEYCLOAK_CLIENT_SECRET="your secret here"
|
||||
```
|
||||
|
||||
You can use this file in development mode to change settings without editing
|
||||
the git-controlled default file at `api/config.dev.py`. Options in this file
|
||||
take precendence.
|
||||
- In the sidebar, navigate to *Manage* → *Users*, and click *Add user* on the top right.
|
||||
- Give the user a name (e.g. `test`), leave the rest as-is.
|
||||
- Under the tab *Credentials*, choose a new password, and make it
|
||||
non-temporary. Click *Set Password*.
|
||||
|
||||
We are going to automate this process. For now, you will have to repeat it
|
||||
every time you reset your keycloak settings, which are stored inside the
|
||||
PostgreSQL as well. Luckily, the script `api/tools/reset_database.py` does
|
||||
*not* affect the state of the keycloak database, so this should be rather rare.
|
||||
|
||||
### Prepare database
|
||||
|
||||
Start the PostgreSQL database:
|
||||
|
||||
```bash
|
||||
docker-compose up -d postgres
|
||||
```
|
||||
|
||||
The first time you start postgres, a lot of extensions will be installed. This
|
||||
takes a while, so check the logs of the docker container until you see:
|
||||
|
||||
> PostgreSQL init process complete; ready for start up.
|
||||
|
||||
If you don't wait long enough, the following commands might fail. In this case,
|
||||
you can always stop the container, remove the data directory (`local/postgres`)
|
||||
and restart the process.
|
||||
|
||||
Next, run the upgrade command to generate the database schema:
|
||||
|
||||
```bash
|
||||
docker-compose run --rm api tools/upgrade.py
|
||||
```
|
||||
|
||||
You will need to re-run this command after updates, to migrate the database and
|
||||
(re-)create the functions in the SQL database that are used when generating
|
||||
vector tiles.
|
||||
|
||||
You should also [import OpenStreetMap data](docs/osm-import.md) now.
|
||||
|
||||
### Boot the application
|
||||
|
||||
Now you can run the remaining parts of the application:
|
||||
|
||||
```bash
|
||||
docker-compose up -d --build api worker frontend
|
||||
```
|
||||
If this does not work, please open an issue and describe the problem you're
|
||||
having, as it is important to us that onboarding is super easy :)
|
||||
|
||||
Your frontend should be running at http://localhost:3001 and the API at
|
||||
http://localhost:3000 -- but you probably only need to access the frontend for
|
||||
testing.
|
||||
testing. The frontend dev server also proxies all unknown requests to the API,
|
||||
so the frontend always just requests data at its own URL.
|
||||
|
||||
### Migrating (Development)
|
||||
## Running without docker
|
||||
|
||||
Migrations are done with
|
||||
[Alembic](https://alembic.sqlalchemy.org/en/latest/index.html), please refer to
|
||||
its documentation for help. Most of the time, running this command will do all
|
||||
the migrations you need:
|
||||
If you don't like docker, or want to run this in production without it, you can
|
||||
do so as well. Our Docker setup is simply a slight wrapper around very simple
|
||||
JavaScript packages that you can install yourself as usual, with `npm install`.
|
||||
The API can be started with `npm start` inside its folder. The frontend
|
||||
development server uses `npm start` as well, while building a production
|
||||
version of the frontend happens with `npm run build`.
|
||||
|
||||
To connect the parts together, please have a look at what we're doing in the
|
||||
"official" setup of docker, i.e. in `docker-compose.yaml`, the `Dockerfile`s
|
||||
and in the respective `package.json` of the service. If you've done this kind
|
||||
of thing before, it's not that hard. Otherwise, ask on Slack and there will be
|
||||
somebody to help you ;)
|
||||
|
||||
## Running in production
|
||||
|
||||
You are advised not to use the dockerized mongodb service and instead do a
|
||||
proper MongoDB setup on a server that is backed up and secured.
|
||||
|
||||
You can run the API in docker, but it is prefered to run it as a restricted
|
||||
user in its own directory somewhere where it cannot escape ;)
|
||||
|
||||
The frontend should be built using `npm run build` and then served from a
|
||||
proper web server, such as nginx or apache. See the instructions at
|
||||
create-react-app concerning [deployment of an app](http://cra.link/deployment).
|
||||
|
||||
You are advised to virtualize your server for security reason, and separate
|
||||
this whole application from other parts of your server system.
|
||||
|
||||
Also please install a reverse proxy that terminates TLS for you and handles
|
||||
certificates. We do not support TLS directly in the application, instead,
|
||||
please use this prefered method. This reverse proxy can also handle static file
|
||||
serving for the frontend, no need for two separate server processes.
|
||||
|
||||
## Migrating
|
||||
|
||||
Sometimes your database will have to be migrated. The docker setup should do
|
||||
this automatically, but if it does not work, you can run the following
|
||||
commands:
|
||||
|
||||
```bash
|
||||
docker-compose run --rm api alembic upgrade head
|
||||
# if running locally
|
||||
(cd api/; npm run migrate:up)
|
||||
|
||||
# if running in docker
|
||||
docker-compose run --rm api npm run migrate:up
|
||||
````
|
||||
|
||||
## Custom MongoDB installation
|
||||
|
||||
If you have your own MongoDB instance running somewhere, you can set the
|
||||
environment variable `MONGODB_URL` when starting the server, and it will read
|
||||
that URL for connecting.
|
||||
|
||||
export MONGODB_URL=mongodb://user:password@mongodb.example.com/obs-app-database
|
||||
|
||||
This does not work when using docker-compose, in that case, you will have to
|
||||
modify the `docker-compose.yaml` to include that URL.
|
||||
|
||||
|
||||
## E-Mail Setup
|
||||
|
||||
By default in development mode mails are not sent, but instead the mail data is
|
||||
logged to the console. This can be overriden with the `--devSendMails` flag if
|
||||
you start the application like so: `npm run dev -- --devSendMails`.
|
||||
|
||||
Mails are also always sent in production mode!
|
||||
|
||||
For actually sending e-mails the mailserver, sender, user and password for the
|
||||
SMTP server need to be specified as environment variables:
|
||||
|
||||
* `MAILUSER` -- the smtp mailbox login name
|
||||
* `MAILPW` -- password for the mailbox
|
||||
* `MAILSERVER` -- the hostname of the SMTP server, e.g. `mail.example.com`
|
||||
* `MAILSENDER` -- sender name, e.g. `noreply@example.com`
|
||||
|
||||
Full command example:
|
||||
|
||||
```bash
|
||||
MAILSERVER=mail.example.com MAILSENDER=noreply@example.com \
|
||||
MAILUSER=my_mail_login MAILPW=hunter2 \
|
||||
npm run dev -- --devSendMails
|
||||
```
|
||||
|
||||
All of this of course is not too important if you're developing locally. To get
|
||||
to the logged email content that *would* have been sent, check your docker log:
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
If any step of the instructions does not work for you, please open an issue and
|
||||
describe the problem you're having, as it is important to us that onboarding is
|
||||
super easy :)
|
||||
|
||||
### Connecting to the PostgreSQL database
|
||||
|
||||
If you need to connect to your development PostgreSQL database, you should
|
||||
install `psql` locally. The port 5432 is already forwarded, so you can connect with:
|
||||
|
||||
```bash
|
||||
docker-compose logs -f api
|
||||
```
|
||||
psql -h localhost -U obs -d obs
|
||||
```
|
||||
|
||||
The password is `obs` as well.
|
||||
|
||||
## License
|
||||
|
||||
Copyright (C) 2020-2021 OpenBikeSensor Contributors
|
||||
Contact: https://openbikesensor.org
|
||||
|
||||
The OpenBikeSensor Portal is free software: you can redistribute it
|
||||
and/or modify it under the terms of the GNU Lesser General Public License
|
||||
as published by the Free Software Foundation, either version 3 of the
|
||||
License, or (at your option) any later version.
|
||||
|
||||
The OpenBikeSensor Portal is distributed in the hope that it will be
|
||||
useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser
|
||||
General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Lesser General Public License
|
||||
along with the OpenBikeSensor Portal. If not, see
|
||||
<http://www.gnu.org/licenses/>.
|
||||
|
||||
See also [`COPYING`](./COPYING) and [`COPYING.LESSER`](./COPYING.LESSER).
|
||||
|
||||
The above does not apply to the files listed below, their respective licenses
|
||||
are included in a file next to each of them, named accordingly:
|
||||
|
||||
* `frontend/src/mapstyles/bright.json`
|
||||
* `frontend/src/mapstyles/positron.json`
|
||||
|
||||
There are lots of other licenses to consider when using this software,
|
||||
especially in conjunction with imported data and other tools. Check out the
|
||||
[Licenses Documentation](docs/licenses.md) for an (unofficial) overview of the
|
||||
license landscape surrounding this project.
|
||||
|
|
|
|||
170
UPGRADING.md
170
UPGRADING.md
|
|
@ -1,170 +0,0 @@
|
|||
# Upgrading
|
||||
This document describes the general steps to upgrade between major changes.
|
||||
Simple migrations, e.g. for adding schema changes, are not documented
|
||||
explicitly. Their general usage is described in the [README](./README.md) (for
|
||||
development) and [docs/production-deployment.md](docs/production-deployment.md) (for production).
|
||||
|
||||
|
||||
## 0.8.1
|
||||
|
||||
- Get the release in your source folder (``git pull; git checkout 0.8.0`` and update submodules ``git submodule update --recursive``)
|
||||
- Rebuild images ``docker-compose build``
|
||||
- No database upgrade is required, but tile functions need an update:
|
||||
```bash
|
||||
docker-compose run --rm portal tools/prepare_sql_tiles.py
|
||||
```
|
||||
- Start your portal and worker services. ``docker-compose up -d worker portal``
|
||||
|
||||
|
||||
## 0.8.0
|
||||
Upgrade to `0.7.x` first. See below for details. Then follow these steps:
|
||||
|
||||
> **Warning** The update includes a reprocessing of tracks after import. Depending on the number of tracks this can take a few hours. The portal is reachable during that time but events disappear and incrementally reappear during reimport.
|
||||
|
||||
> **Info** With this version the import process for OpenStreetMap data has changed: the [new process](docs/osm-import.md) is easier on resources and finally permits to import a full country on a low-end VM.
|
||||
|
||||
- Do your [usual backup](docs/production-deployment.md)
|
||||
- get the release in your source folder (``git pull; git checkout 0.8.0`` and update submodules ``git submodule update --recursive``)
|
||||
- Rebuild images ``docker-compose build``
|
||||
- Stop your portal and worker services ``docker-compose stop worker portal``
|
||||
- run upgrade
|
||||
```bash
|
||||
docker-compose run --rm portal tools/upgrade.py
|
||||
```
|
||||
this automatically does the following
|
||||
- Migration of database schema using alembic.
|
||||
- Upgrade of SQL tile schema to new schema.
|
||||
- Import the nuts-regions from the web into the database.
|
||||
- Trigger a re-import of all tracks.
|
||||
- Start your portal and worker services. ``docker-compose up -d worker portal``
|
||||
|
||||
|
||||
## 0.7.0
|
||||
|
||||
Upgrade to `0.6.x` first. See below for details. Then follow these steps:
|
||||
|
||||
- Rebuild images
|
||||
- Stop your portal and worker services.
|
||||
- **Migration with alembic**: required
|
||||
- **Prepare SQL Tiles**: required
|
||||
- Start your portal and worker services.
|
||||
- **Reimport tracks**: no action required
|
||||
- **OSM Import**: required
|
||||
- **Config changes**: add `POSTGRES_MAX_OVERFLOW` and `POSTGRES_POOL_SIZE`
|
||||
variables, see `api/config.py.example`
|
||||
|
||||
## 0.6.0
|
||||
|
||||
**Make sure to upgrade to `0.5.1` first, by checking out that version tag and
|
||||
running migrations, then coming back to this version.** This is required
|
||||
because the migrations have been edited to create the initial database schema,
|
||||
but if you run the 0.5.1 migrations first, your database will remember that it
|
||||
already has all the tables created. This is not required if you set up a new
|
||||
installation.
|
||||
|
||||
For this update, run these steps:
|
||||
|
||||
- Build new images
|
||||
- Stop portal and worker services
|
||||
- Run the new upgrade tool:
|
||||
```bash
|
||||
docker-compose run --rm portal tools/upgrade.py
|
||||
```
|
||||
- Start portal and worker services
|
||||
|
||||
## 0.5.0
|
||||
|
||||
The upgrade requires the following steps in the given order
|
||||
|
||||
- Rebuild images
|
||||
- Stop your portal and worker services.
|
||||
- **Migration with alembic**: required
|
||||
- **Prepare SQL Tiles**: required
|
||||
- Start your portal and worker services.
|
||||
- **Reimport tracks**: required
|
||||
- **OSM Import**: no action required
|
||||
- **Config changes**: none
|
||||
|
||||
## 0.4.1
|
||||
|
||||
You can, but do not have to, reimport all tracks. This will generate a GPX file
|
||||
for each track and allow the users to download those. If a GPX file has not yet
|
||||
been created, the download will fail. To reimport all tracks, log in to your
|
||||
PostgreSQL database (instructions are in [README.md](./README.md) for
|
||||
development and [docs/production-deployment.md](./docs/production-deployment.md) for production)
|
||||
and run:
|
||||
|
||||
```sql
|
||||
UPDATE track SET processing_status = 'queued';
|
||||
```
|
||||
|
||||
You can do this selectively with `WHERE` statements.
|
||||
|
||||
Make sure your worker is running to process the queue.
|
||||
|
||||
## 0.4.0
|
||||
|
||||
* Rebuild your image, this may take longer than usual, as it will compile
|
||||
`osm2pgsql` for you. Next time, it should be in your docker build cache and
|
||||
be fast again.
|
||||
* Add new config flags: `VERBOSE`, `LEAN_MODE`, `POSTGRES_POOL_SIZE`,
|
||||
`POSTGRES_MAX_OVERFLOW`. Check the example config for sane default values.
|
||||
* Re-run `tools/prepare_sql_tiles.py` again (see README)
|
||||
* It has been made easier to import OSM data, check
|
||||
[docs/production-deployment.md](./docs/production-deployment.md) for the sections "Download
|
||||
OpenStreetMap maps" and "Import OpenStreetMap data". You can now download
|
||||
multiple .pbf files and then import them at once, using the docker image
|
||||
built with the `Dockerfile`. Alternatively, you can choose to enable [lean
|
||||
mode](docs/lean-mode.md). You do not need to reimport data, but setting this
|
||||
up now will make your life easier in the long run ;)
|
||||
|
||||
## v0.2 to v0.3 (MongoDB to PostgreSQL)
|
||||
|
||||
* Shut down all services
|
||||
* Obviously, now is a good time to perform a full backup ;)
|
||||
* Update the codebase (`git pull`, `git submodule update`).
|
||||
* Update your ``docker-compose.yaml`` with the one from the ``deployment/examples``
|
||||
folder.
|
||||
* Leave the MongoDB service in place for now.
|
||||
* Update all other service descriptions.
|
||||
* You can remove `redis` already.
|
||||
* Generate a better password than the default for your
|
||||
postgres user.
|
||||
* Traefik rules have been simplified as all routes are handled
|
||||
by the portal service now.
|
||||
* Start up the `mongo` and `postgres` services. Wait for postgres to finish
|
||||
initializing (see [README](README.md)).
|
||||
* Build the new image (e.g. with `docker-compose build portal`)
|
||||
* Configure your API. The example config file is `api/config.py.example`, and
|
||||
it will need to be mounted to `api/config.py` in the container. Ignore the
|
||||
Keycloak options for now.
|
||||
* Prepare the database:
|
||||
|
||||
```bash
|
||||
docker-compose run --rm portal python tools/reset_database.py
|
||||
docker-compose run --rm portal python tools/prepare_sql_tiles.py
|
||||
```
|
||||
* Import OSM data (see [README](README.md)).
|
||||
* Run the database migration script:
|
||||
|
||||
```bash
|
||||
docker-compose run --rm \
|
||||
-v $PWD/export:/export \
|
||||
portal \
|
||||
python tools/import_from_mongodb.py mongodb://mongo/obs \
|
||||
--keycloak-users-file /export/users.json
|
||||
```
|
||||
There is an option `--keep-api-keys` which means the users won't have to
|
||||
reconfigure the devices they used their API key in. **However**, please try
|
||||
to avoid this option if at all possible, as the old keys are *very* insecure.
|
||||
The default without this option to generate a new, secure API key for each
|
||||
user.
|
||||
* Shut down the `mongo` service, you can now remove it from docker-compose.yaml
|
||||
* Start `keycloak` and configure it, similarly to how it was configured in the
|
||||
development setup (but choose more secure options). Update the API config
|
||||
file to match your keycloak configuration. Import the file
|
||||
`export/users.json` into your realm, it will re-add all the users from the
|
||||
old installation. You should delete the file and `export/` folder afterwards.
|
||||
* Start `portal`.
|
||||
* Consider configuring a worker service. See [docs/production-deployment.md](./docs/production-deployment.md).
|
||||
|
||||
2
api/.gitignore
vendored
2
api/.gitignore
vendored
|
|
@ -43,5 +43,3 @@ local/
|
|||
# both, because then developers will only update one of them and they'll
|
||||
# contradict. For now, npm shall be the canonical default (compare README.md).
|
||||
yarn.lock
|
||||
|
||||
config.overrides.py
|
||||
|
|
|
|||
|
|
@ -1,16 +1,28 @@
|
|||
FROM python:3.11.3-bullseye
|
||||
FROM node:15.14-buster
|
||||
|
||||
# Install python3, pip3, and make them the default for `python` and `pip` commands
|
||||
RUN apt-get update && apt-get install -y python3 python3-pip
|
||||
RUN ln -s $(which python3) /usr/local/bin/python
|
||||
RUN ln -s $(which pip3) /usr/local/bin/pip
|
||||
|
||||
WORKDIR /opt/obs/api
|
||||
|
||||
ADD scripts /opt/obs/scripts
|
||||
RUN pip install -e /opt/obs/scripts
|
||||
ADD package.json package-lock.json /opt/obs/api/
|
||||
RUN echo update-notifier=false >> ~/.npmrc
|
||||
RUN npm ci
|
||||
|
||||
ADD requirements.txt /opt/obs/api/
|
||||
RUN pip install -r requirements.txt
|
||||
ADD setup.py /opt/obs/api/
|
||||
ADD obs /opt/obs/api/obs/
|
||||
RUN pip install -e .
|
||||
ADD scripts /opt/obs/api/scripts/
|
||||
RUN cd scripts && pip install -e .
|
||||
|
||||
EXPOSE 8000
|
||||
ADD views /opt/obs/api/views/
|
||||
ADD src /opt/obs/api/src/
|
||||
|
||||
CMD ["openbikesensor-api"]
|
||||
#ADD .migrations.js .
|
||||
#ADD migrations .
|
||||
|
||||
EXPOSE 3000
|
||||
|
||||
ENV PORT=3000
|
||||
ENV DATA_DIR=/data
|
||||
|
||||
CMD ["npm", "run", "start"]
|
||||
102
api/alembic.ini
102
api/alembic.ini
|
|
@ -1,102 +0,0 @@
|
|||
# A generic, single database configuration.
|
||||
|
||||
[alembic]
|
||||
# path to migration scripts
|
||||
script_location = migrations
|
||||
|
||||
# template used to generate migration files
|
||||
# file_template = %%(rev)s_%%(slug)s
|
||||
|
||||
# sys.path path, will be prepended to sys.path if present.
|
||||
# defaults to the current working directory.
|
||||
prepend_sys_path = .
|
||||
|
||||
# timezone to use when rendering the date within the migration file
|
||||
# as well as the filename.
|
||||
# If specified, requires the python-dateutil library that can be
|
||||
# installed by adding `alembic[tz]` to the pip requirements
|
||||
# string value is passed to dateutil.tz.gettz()
|
||||
# leave blank for localtime
|
||||
# timezone =
|
||||
|
||||
# max length of characters to apply to the
|
||||
# "slug" field
|
||||
# truncate_slug_length = 40
|
||||
|
||||
# set to 'true' to run the environment during
|
||||
# the 'revision' command, regardless of autogenerate
|
||||
# revision_environment = false
|
||||
|
||||
# set to 'true' to allow .pyc and .pyo files without
|
||||
# a source .py file to be detected as revisions in the
|
||||
# versions/ directory
|
||||
# sourceless = false
|
||||
|
||||
# version location specification; This defaults
|
||||
# to api/migrations/versions. When using multiple version
|
||||
# directories, initial revisions must be specified with --version-path.
|
||||
# The path separator used here should be the separator specified by "version_path_separator" below.
|
||||
# version_locations = %(here)s/bar:%(here)s/bat:api/migrations/versions
|
||||
|
||||
# version path separator; As mentioned above, this is the character used to split
|
||||
# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
|
||||
# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
|
||||
# Valid values for version_path_separator are:
|
||||
#
|
||||
# version_path_separator = :
|
||||
# version_path_separator = ;
|
||||
# version_path_separator = space
|
||||
version_path_separator = os # Use os.pathsep. Default configuration used for new projects.
|
||||
|
||||
# the output encoding used when revision files
|
||||
# are written from script.py.mako
|
||||
# output_encoding = utf-8
|
||||
|
||||
sqlalchemy.url = driver://user:pass@localhost/dbname
|
||||
|
||||
|
||||
[post_write_hooks]
|
||||
# post_write_hooks defines scripts or Python functions that are run
|
||||
# on newly generated revision scripts. See the documentation for further
|
||||
# detail and examples
|
||||
|
||||
# format using "black" - use the console_scripts runner, against the "black" entrypoint
|
||||
# hooks = black
|
||||
# black.type = console_scripts
|
||||
# black.entrypoint = black
|
||||
# black.options = -l 79 REVISION_SCRIPT_FILENAME
|
||||
|
||||
# Logging configuration
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic
|
||||
|
||||
[handlers]
|
||||
keys = console
|
||||
|
||||
[formatters]
|
||||
keys = generic
|
||||
|
||||
[logger_root]
|
||||
level = WARN
|
||||
handlers = console
|
||||
qualname =
|
||||
|
||||
[logger_sqlalchemy]
|
||||
level = WARN
|
||||
handlers =
|
||||
qualname = sqlalchemy.engine
|
||||
|
||||
[logger_alembic]
|
||||
level = INFO
|
||||
handlers =
|
||||
qualname = alembic
|
||||
|
||||
[handler_console]
|
||||
class = StreamHandler
|
||||
args = (sys.stderr,)
|
||||
level = NOTSET
|
||||
formatter = generic
|
||||
|
||||
[formatter_generic]
|
||||
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||
datefmt = %H:%M:%S
|
||||
31
api/config.dev.json
Normal file
31
api/config.dev.json
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
{
|
||||
"cookieSecret": "CHANGEME!!!!!!!!!!@##@!!$$$$$$$$$$$$$!!",
|
||||
"jwtSecret": "CHANGEME??????????????////3212321;312kjbkasjd",
|
||||
"baseUrl": "http://localhost:3000/",
|
||||
"mainFrontendUrl": "http://localhost:3001/",
|
||||
"mail": false,
|
||||
"mongodb": {
|
||||
"url": "mongodb://mongo/obsTest",
|
||||
"debug": true
|
||||
},
|
||||
"redisUrl": "redis://redis",
|
||||
"oAuth2Clients": [
|
||||
{
|
||||
"clientId": "b730f8d2-d93c-4c68-9ff0-dfac8da76ee2",
|
||||
"validRedirectUris": ["http://localhost:3001/redirect"],
|
||||
"refreshTokenExpirySeconds": 604800,
|
||||
"maxScope": "*",
|
||||
"title": "OBS Portal",
|
||||
"autoAccept": true
|
||||
},
|
||||
{
|
||||
"clientId": "a2958209-4045-4ec9-8cb3-1156abba7de3",
|
||||
"validRedirectUris": ["__LOCAL__"],
|
||||
"maxScope": "track.upload",
|
||||
"refreshTokenExpirySeconds": 86400000,
|
||||
"title": "OpenBikeSensor"
|
||||
}
|
||||
],
|
||||
"imprintUrl": "https://example.com/imprint",
|
||||
"privacyPolicyUrl": "https://example.com/privacy"
|
||||
}
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
HOST = "0.0.0.0"
|
||||
PORT = 3000
|
||||
DEBUG = True
|
||||
VERBOSE = False
|
||||
AUTO_RELOAD = True
|
||||
SECRET = "!!!!!!!!!!!!CHANGE ME!!!!!!!!!!!!"
|
||||
POSTGRES_URL = "postgresql+asyncpg://obs:obs@postgres/obs"
|
||||
POSTGRES_POOL_SIZE = 20
|
||||
POSTGRES_MAX_OVERFLOW = 2 * POSTGRES_POOL_SIZE
|
||||
KEYCLOAK_URL = "http://keycloak:8080/auth/realms/obs-dev/"
|
||||
KEYCLOAK_CLIENT_ID = "portal"
|
||||
KEYCLOAK_CLIENT_SECRET = "c385278e-bd2e-4f13-9937-34b0c0f44c2d"
|
||||
DEDICATED_WORKER = True
|
||||
API_URL = "http://localhost:3000/"
|
||||
FRONTEND_URL = "http://localhost:3001/"
|
||||
FRONTEND_HTTPS = False
|
||||
FRONTEND_DIR = None
|
||||
FRONTEND_CONFIG = {
|
||||
"imprintUrl": "https://example.com/imprint",
|
||||
"privacyPolicyUrl": "https://example.com/privacy",
|
||||
# "termsUrl": "https://example.com/terms", # Link is only shown when set
|
||||
"mapHome": {"zoom": 6, "longitude": 10.2, "latitude": 51.3},
|
||||
# "banner": {"text": "This is a development installation.", "style": "info"},
|
||||
}
|
||||
|
||||
TILES_FILE = None # "/tiles/tiles.mbtiles"
|
||||
DATA_DIR = "/data"
|
||||
ADDITIONAL_CORS_ORIGINS = [
|
||||
"http://localhost:8880/", # for maputnik on 8880
|
||||
"http://localhost:8888/", # for maputnik on 8888
|
||||
]
|
||||
TILE_SEMAPHORE_SIZE = 4
|
||||
EXPORT_SEMAPHORE_SIZE = 4
|
||||
|
||||
# vim: set ft=python :
|
||||
39
api/config.json.example
Normal file
39
api/config.json.example
Normal file
|
|
@ -0,0 +1,39 @@
|
|||
{
|
||||
"cookieSecret": "!!!<<<CHANGEME>>>!!!",
|
||||
"jwtSecret": "!!!<<<CHANGEME>>>!!!",
|
||||
"baseUrl": "https://portal.example.com/api",
|
||||
"mainFrontendUrl": "https://portal.example.com/",
|
||||
"mail": {
|
||||
"from": "Sender Name <sender@example.com>",
|
||||
"smtp" : {
|
||||
"host": "mail.example.com",
|
||||
"port": 465,
|
||||
"starttls": false,
|
||||
"username": "sender@example.com",
|
||||
"password": "!!!<<<CHANGEME>>>!!!"
|
||||
}
|
||||
},
|
||||
"mongodb": {
|
||||
"url": "mongodb://mongo/obs",
|
||||
"debug": false
|
||||
},
|
||||
"redisUrl": "redis://redis",
|
||||
"oAuth2Clients": [
|
||||
{
|
||||
"clientId": "!!!<<<CHANGEME>>>!!!",
|
||||
"validRedirectUris": ["https://portal.example.com/redirect"],
|
||||
"refreshTokenExpirySeconds": 604800,
|
||||
"maxScope": "*",
|
||||
"title": "OBS Portal"
|
||||
},
|
||||
{
|
||||
"clientId": "!!!<<<CHANGEME>>>!!!",
|
||||
"validRedirectUris": ["__LOCAL__"],
|
||||
"maxScope": "track.upload",
|
||||
"refreshTokenExpirySeconds": 86400000,
|
||||
"title": "OpenBikeSensor"
|
||||
}
|
||||
],
|
||||
"imprintUrl": "https://example.com/imprint",
|
||||
"privacyPolicyUrl": "https://example.com/privacy"
|
||||
}
|
||||
|
|
@ -1,73 +0,0 @@
|
|||
# Bind address of the server
|
||||
HOST = "127.0.0.1"
|
||||
PORT = 3000
|
||||
|
||||
# Extended log output, but slower
|
||||
DEBUG = False
|
||||
VERBOSE = DEBUG
|
||||
AUTO_RELOAD = DEBUG
|
||||
|
||||
# Required to encrypt or sign sessions, cookies, tokens, etc.
|
||||
SECRET = "!!!<<<CHANGEME>>>!!!"
|
||||
|
||||
# Connection to the database
|
||||
POSTGRES_URL = "postgresql+asyncpg://user:pass@host/dbname"
|
||||
POSTGRES_POOL_SIZE = 20
|
||||
POSTGRES_MAX_OVERFLOW = 2 * POSTGRES_POOL_SIZE
|
||||
|
||||
# URL to the keycloak realm, as reachable by the API service. This is not
|
||||
# necessarily its publicly reachable URL, keycloak advertises that iself.
|
||||
KEYCLOAK_URL = "http://localhost:1234/auth/realms/obs/"
|
||||
|
||||
# Auth client credentials
|
||||
KEYCLOAK_CLIENT_ID = "portal"
|
||||
KEYCLOAK_CLIENT_SECRET = "00000000-0000-0000-0000-000000000000"
|
||||
|
||||
# Whether the API should run the worker loop, or a dedicated worker is used
|
||||
DEDICATED_WORKER = True
|
||||
|
||||
# The root of the frontend. Needed for redirecting after login, and for CORS.
|
||||
# Set to None if frontend is served by the API.
|
||||
FRONTEND_URL = None
|
||||
FRONTEND_HTTPS = True
|
||||
|
||||
# Where to find the compiled frontend assets (must include index.html), or None
|
||||
# to disable serving the frontend.
|
||||
FRONTEND_DIR = "../frontend/build/"
|
||||
|
||||
# Can be an object or a JSON string
|
||||
FRONTEND_CONFIG = {
|
||||
"imprintUrl": "https://example.com/imprint",
|
||||
"privacyPolicyUrl": "https://example.com/privacy",
|
||||
# "termsUrl": "https://example.com/user_terms_and_conditions", # Link is only shown when set
|
||||
"mapHome": {"zoom": 6, "longitude": 10.2, "latitude": 51.3},
|
||||
"banner": {"text": "This is a test installation.", "style": "warning"},
|
||||
}
|
||||
|
||||
# If the API should serve generated tiles, this is the path where the tiles are
|
||||
# built. This is an experimental option and probably very inefficient, a proper
|
||||
# tileserver should be prefered. Set to None to disable.
|
||||
TILES_FILE = None
|
||||
|
||||
# Path overrides:
|
||||
# API_ROOT_DIR = "??" # default: api/ inside repository
|
||||
# DATA_DIR = "??" # default: $API_ROOT_DIR/..
|
||||
# PROCESSING_DIR = "??" # default: DATA_DIR/processing
|
||||
# PROCESSING_OUTPUT_DIR = "??" # default: DATA_DIR/processing-output
|
||||
# TRACKS_DIR = "??" # default: DATA_DIR/tracks
|
||||
# OBS_FACE_CACHE_DIR = "??" # default: DATA_DIR/obs-face-cache
|
||||
|
||||
# Additional allowed origins for CORS headers. The FRONTEND_URL is included by
|
||||
# default. Python list, or whitespace separated string.
|
||||
ADDITIONAL_CORS_ORIGINS = None
|
||||
|
||||
# How many asynchronous requests may be sent to the database to generate tile
|
||||
# information. Should be less than POSTGRES_POOL_SIZE to leave some connections
|
||||
# to the other features of the API ;)
|
||||
TILE_SEMAPHORE_SIZE = 4
|
||||
|
||||
# How many asynchronous requests may generate exported data simultaneously.
|
||||
# Keep this small.
|
||||
EXPORT_SEMAPHORE_SIZE = 1
|
||||
|
||||
# vim: set ft=python :
|
||||
61
api/migrations/2020-12-01-1945-reconstruct-track-body.js
Normal file
61
api/migrations/2020-12-01-1945-reconstruct-track-body.js
Normal file
|
|
@ -0,0 +1,61 @@
|
|||
const Track = require('../src/models/Track');
|
||||
const { replaceDollarNewlinesHack, detectFormat, buildObsver1 } = require('../src/logic/tracks');
|
||||
|
||||
function shouldRebuildBody(track) {
|
||||
if (!track.trackData || !track.trackData.points.length) {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!track.body) {
|
||||
return true;
|
||||
}
|
||||
const body = track.body.trim();
|
||||
if (!body) {
|
||||
return true;
|
||||
}
|
||||
|
||||
const actualBody = replaceDollarNewlinesHack(body).trim();
|
||||
if (body !== actualBody) {
|
||||
return true;
|
||||
}
|
||||
|
||||
const lineCount = (actualBody.match(/\n/g) || []).length + 1;
|
||||
|
||||
const format = detectFormat(body);
|
||||
if (format === 'invalid') {
|
||||
return true;
|
||||
}
|
||||
|
||||
// never reconstruct body of version 2
|
||||
if (format > 1) {
|
||||
return false;
|
||||
}
|
||||
|
||||
// not enough data in the file
|
||||
if (lineCount < track.trackData.points.length + 1) {
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
async function up(next) {
|
||||
const query = Track.find().populate('trackData');
|
||||
for await (const track of query) {
|
||||
const rebuild = shouldRebuildBody(track);
|
||||
if (rebuild) {
|
||||
track.body = buildObsver1(track.trackData.points);
|
||||
}
|
||||
|
||||
await track.save();
|
||||
}
|
||||
|
||||
next();
|
||||
}
|
||||
|
||||
async function down(next) {
|
||||
// nothing to do
|
||||
next();
|
||||
}
|
||||
|
||||
module.exports = { up, down };
|
||||
15
api/migrations/2020-12-01-1950-rebuild-track-data.js
Normal file
15
api/migrations/2020-12-01-1950-rebuild-track-data.js
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
const Track = require('../src/models/Track');
|
||||
|
||||
module.exports = {
|
||||
async up(next) {
|
||||
for await (const track of Track.find()) {
|
||||
await track.rebuildTrackDataAndSave();
|
||||
}
|
||||
|
||||
next();
|
||||
},
|
||||
|
||||
async down(next) {
|
||||
next();
|
||||
},
|
||||
};
|
||||
21
api/migrations/2020-12-12-1823-original-filename.js
Normal file
21
api/migrations/2020-12-12-1823-original-filename.js
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
const Track = require('../src/models/Track');
|
||||
|
||||
module.exports = {
|
||||
async up(next) {
|
||||
try {
|
||||
for await (const track of Track.find()) {
|
||||
track.originalFileName = track.slug + '.csv'
|
||||
await track.generateOriginalFilePath();
|
||||
await track.save()
|
||||
}
|
||||
next();
|
||||
} catch(err) {
|
||||
next(err)
|
||||
}
|
||||
},
|
||||
|
||||
async down(next) {
|
||||
next();
|
||||
},
|
||||
};
|
||||
|
||||
25
api/migrations/2020-12-13-2025-move-to-upload-file.js
Normal file
25
api/migrations/2020-12-13-2025-move-to-upload-file.js
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
|
||||
const Track = require('../src/models/Track');
|
||||
|
||||
module.exports = {
|
||||
async up(next) {
|
||||
try {
|
||||
for await (const track of Track.find()) {
|
||||
if (!track.body) {
|
||||
continue
|
||||
}
|
||||
|
||||
await track.writeToOriginalFile(track.body)
|
||||
delete track.body;
|
||||
await track.save()
|
||||
}
|
||||
next();
|
||||
} catch(err) {
|
||||
next(err)
|
||||
}
|
||||
},
|
||||
|
||||
async down(next) {
|
||||
next();
|
||||
},
|
||||
};
|
||||
|
|
@ -1 +0,0 @@
|
|||
Generic single-database configuration.
|
||||
|
|
@ -1,83 +0,0 @@
|
|||
import asyncio
|
||||
from logging.config import fileConfig
|
||||
|
||||
from sqlalchemy import engine_from_config
|
||||
from sqlalchemy import pool
|
||||
|
||||
from alembic import context
|
||||
|
||||
# this is the Alembic Config object, which provides
|
||||
# access to the values within the .ini file in use.
|
||||
config = context.config
|
||||
|
||||
# Interpret the config file for Python logging.
|
||||
# This line sets up loggers basically.
|
||||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
|
||||
# add your model's MetaData object here
|
||||
# for 'autogenerate' support
|
||||
# from myapp import mymodel
|
||||
# target_metadata = mymodel.Base.metadata
|
||||
target_metadata = None
|
||||
|
||||
# other values from the config, defined by the needs of env.py,
|
||||
# can be acquired:
|
||||
# my_important_option = config.get_main_option("my_important_option")
|
||||
# ... etc.
|
||||
|
||||
|
||||
def do_run_migrations(connection):
|
||||
context.configure(connection=connection, target_metadata=target_metadata)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
def run_migrations_offline():
|
||||
"""Run migrations in 'offline' mode.
|
||||
|
||||
This configures the context with just a URL
|
||||
and not an Engine, though an Engine is acceptable
|
||||
here as well. By skipping the Engine creation
|
||||
we don't even need a DBAPI to be available.
|
||||
|
||||
Calls to context.execute() here emit the given string to the
|
||||
script output.
|
||||
|
||||
"""
|
||||
from obs.api.app import app
|
||||
|
||||
url = app.config.POSTGRES_URL
|
||||
context.configure(
|
||||
url=url,
|
||||
target_metadata=target_metadata,
|
||||
literal_binds=True,
|
||||
dialect_opts={"paramstyle": "named"},
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
async def run_migrations_online():
|
||||
"""Run migrations in 'online' mode.
|
||||
|
||||
In this scenario we need to create an Engine
|
||||
and associate a connection with the context.
|
||||
|
||||
"""
|
||||
from obs.api.app import app, connect_db
|
||||
|
||||
url = app.config.POSTGRES_URL
|
||||
async with connect_db(url) as engine:
|
||||
async with engine.connect() as connection:
|
||||
await connection.run_sync(do_run_migrations)
|
||||
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
if context.is_offline_mode():
|
||||
run_migrations_offline()
|
||||
else:
|
||||
asyncio.run(run_migrations_online())
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
"""${message}
|
||||
|
||||
Revision ID: ${up_revision}
|
||||
Revises: ${down_revision | comma,n}
|
||||
Create Date: ${create_date}
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = ${repr(up_revision)}
|
||||
down_revision = ${repr(down_revision)}
|
||||
branch_labels = ${repr(branch_labels)}
|
||||
depends_on = ${repr(depends_on)}
|
||||
|
||||
|
||||
def upgrade():
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def downgrade():
|
||||
${downgrades if downgrades else "pass"}
|
||||
|
|
@ -1,16 +0,0 @@
|
|||
import sqlalchemy as sa
|
||||
|
||||
|
||||
def dbtype(name):
|
||||
"""
|
||||
Create a UserDefinedType for use in migrations as the type of a column,
|
||||
when the type already exists in the database, but isn't available as a
|
||||
proper sqlalchemy type.
|
||||
"""
|
||||
|
||||
class TheType(sa.types.UserDefinedType):
|
||||
def get_col_spec(self):
|
||||
return name
|
||||
|
||||
TheType.__name__ = name
|
||||
return TheType
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
"""create table road
|
||||
|
||||
Revision ID: 35e7f1768f9b
|
||||
Revises: 5d75febe2d59
|
||||
Create Date: 2022-03-30 21:36:48.157457
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
from migrations.utils import dbtype
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "35e7f1768f9b"
|
||||
down_revision = "920aed1450c9"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"road",
|
||||
sa.Column(
|
||||
"way_id", sa.BIGINT, primary_key=True, index=True, autoincrement=False
|
||||
),
|
||||
sa.Column("zone", dbtype("zone_type")),
|
||||
sa.Column("name", sa.Text),
|
||||
sa.Column("geometry", dbtype("geometry(LINESTRING,3857)")),
|
||||
sa.Column("directionality", sa.Integer),
|
||||
sa.Column("oneway", sa.Boolean),
|
||||
)
|
||||
op.execute(
|
||||
"CREATE INDEX road_geometry_idx ON road USING GIST (geometry) WITH (FILLFACTOR=100);"
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("road")
|
||||
|
|
@ -1,28 +0,0 @@
|
|||
"""create extensions
|
||||
|
||||
Revision ID: 3856f240bb6d
|
||||
Revises: a9627f63fbed
|
||||
Create Date: 2022-03-30 21:31:06.282725
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "3856f240bb6d"
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.execute('CREATE EXTENSION IF NOT EXISTS "hstore";')
|
||||
op.execute('CREATE EXTENSION IF NOT EXISTS "postgis";')
|
||||
op.execute('CREATE EXTENSION IF NOT EXISTS "uuid-ossp";')
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.execute('DROP EXTENSION "hstore";')
|
||||
op.execute('DROP EXTENSION "postgis";')
|
||||
op.execute('DROP EXTENSION "uuid-ossp";')
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
"""transform overtaking_event geometry to 3857
|
||||
|
||||
Revision ID: 587e69ecb466
|
||||
Revises: f4b0f460254d
|
||||
Create Date: 2023-04-01 14:30:49.927505
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "587e69ecb466"
|
||||
down_revision = "f4b0f460254d"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.execute("UPDATE overtaking_event SET geometry = ST_Transform(geometry, 3857);")
|
||||
op.execute(
|
||||
"ALTER TABLE overtaking_event ALTER COLUMN geometry TYPE geometry(POINT, 3857);"
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.execute(
|
||||
"ALTER TABLE overtaking_event ALTER COLUMN geometry TYPE geometry;"
|
||||
)
|
||||
op.execute("UPDATE overtaking_event SET geometry = ST_Transform(geometry, 4326);")
|
||||
|
|
@ -1,43 +0,0 @@
|
|||
"""create table overtaking_event
|
||||
|
||||
Revision ID: 5d75febe2d59
|
||||
Revises: 920aed1450c9
|
||||
Create Date: 2022-03-30 21:36:37.687080
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from migrations.utils import dbtype
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "5d75febe2d59"
|
||||
down_revision = "9336eef458e7"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"overtaking_event",
|
||||
sa.Column("id", sa.Integer, autoincrement=True, primary_key=True, index=True),
|
||||
sa.Column(
|
||||
"track_id", sa.Integer, sa.ForeignKey("track.id", ondelete="CASCADE")
|
||||
),
|
||||
sa.Column("hex_hash", sa.String, unique=True, index=True),
|
||||
sa.Column("way_id", sa.BIGINT, index=True),
|
||||
sa.Column("direction_reversed", sa.Boolean),
|
||||
sa.Column("geometry", dbtype("GEOMETRY")),
|
||||
sa.Column("latitude", sa.Float),
|
||||
sa.Column("longitude", sa.Float),
|
||||
sa.Column("time", sa.DateTime),
|
||||
sa.Column("distance_overtaker", sa.Float),
|
||||
sa.Column("distance_stationary", sa.Float),
|
||||
sa.Column("course", sa.Float),
|
||||
sa.Column("speed", sa.Float),
|
||||
sa.Index("road_segment", "way_id", "direction_reversed"),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("overtaking_event")
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
"""add_overtaking_event_index
|
||||
|
||||
|
||||
Revision ID: 7868aed76122
|
||||
Revises: 587e69ecb466
|
||||
Create Date: 2023-07-16 13:37:17.694079
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '7868aed76122'
|
||||
down_revision = '587e69ecb466'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.execute("CREATE INDEX IF NOT EXISTS ix_overtaking_event_geometry ON overtaking_event using GIST(geometry);")
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_index("ix_overtaking_event_geometry")
|
||||
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
"""create enum processing_status
|
||||
|
||||
Revision ID: 920aed1450c9
|
||||
Revises: 986c6953e431
|
||||
Create Date: 2022-03-30 21:36:25.896192
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "920aed1450c9"
|
||||
down_revision = "986c6953e431"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _get_enum_type():
|
||||
return postgresql.ENUM(
|
||||
"created", "queued", "processing", "complete", "error", name="processing_status"
|
||||
)
|
||||
|
||||
|
||||
def upgrade():
|
||||
_get_enum_type().create(op.get_bind(), checkfirst=True)
|
||||
|
||||
|
||||
def downgrade():
|
||||
_get_enum_type().drop(op.get_bind())
|
||||
|
|
@ -1,42 +0,0 @@
|
|||
"""create table comment
|
||||
|
||||
Revision ID: 9336eef458e7
|
||||
Revises: 9d8c8c38a1d0
|
||||
Create Date: 2022-03-30 21:37:02.080429
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects.postgresql import UUID
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "9336eef458e7"
|
||||
down_revision = "d66baafab5ec"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
NOW = sa.text("NOW()")
|
||||
|
||||
op.create_table(
|
||||
"comment",
|
||||
sa.Column("id", sa.Integer, autoincrement=True, primary_key=True),
|
||||
sa.Column("uid", UUID, server_default=sa.func.uuid_generate_v4()),
|
||||
sa.Column("created_at", sa.DateTime, nullable=False, server_default=NOW),
|
||||
sa.Column(
|
||||
"updated_at", sa.DateTime, nullable=False, server_default=NOW, onupdate=NOW
|
||||
),
|
||||
sa.Column("body", sa.TEXT),
|
||||
sa.Column(
|
||||
"author_id", sa.Integer, sa.ForeignKey("user.id", ondelete="CASCADE")
|
||||
),
|
||||
sa.Column(
|
||||
"track_id", sa.Integer, sa.ForeignKey("track.id", ondelete="CASCADE")
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("comment")
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
"""create enum zone_type
|
||||
|
||||
Revision ID: 986c6953e431
|
||||
Revises: 3856f240bb6d
|
||||
Create Date: 2022-03-30 21:36:19.888268
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "986c6953e431"
|
||||
down_revision = "3856f240bb6d"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def _get_enum_type():
|
||||
return postgresql.ENUM("rural", "urban", "motorway", name="zone_type")
|
||||
|
||||
|
||||
def upgrade():
|
||||
_get_enum_type().create(op.get_bind(), checkfirst=True)
|
||||
|
||||
|
||||
def downgrade():
|
||||
_get_enum_type().drop(op.get_bind())
|
||||
|
|
@ -1,26 +0,0 @@
|
|||
"""add user display_name
|
||||
|
||||
Revision ID: 99a3d2eb08f9
|
||||
Revises: a9627f63fbed
|
||||
Create Date: 2022-09-13 07:30:18.747880
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "99a3d2eb08f9"
|
||||
down_revision = "a9627f63fbed"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.add_column(
|
||||
"user", sa.Column("display_name", sa.String, nullable=True), schema="public"
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_column("user", "display_name", schema="public")
|
||||
|
|
@ -1,45 +0,0 @@
|
|||
"""create table user
|
||||
|
||||
Revision ID: 9d8c8c38a1d0
|
||||
Revises: d66baafab5ec
|
||||
Create Date: 2022-03-30 21:36:59.375149
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "9d8c8c38a1d0"
|
||||
down_revision = "35e7f1768f9b"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
NOW = sa.text("NOW()")
|
||||
|
||||
op.create_table(
|
||||
"user",
|
||||
sa.Column("id", sa.Integer, autoincrement=True, primary_key=True),
|
||||
sa.Column("created_at", sa.DateTime, nullable=False, server_default=NOW),
|
||||
sa.Column(
|
||||
"updated_at", sa.DateTime, nullable=False, server_default=NOW, onupdate=NOW
|
||||
),
|
||||
sa.Column("sub", sa.String, unique=True, nullable=False),
|
||||
sa.Column("username", sa.String, unique=True, nullable=False),
|
||||
sa.Column("email", sa.String, nullable=False),
|
||||
sa.Column("bio", sa.TEXT),
|
||||
sa.Column("image", sa.String),
|
||||
sa.Column(
|
||||
"are_tracks_visible_for_all",
|
||||
sa.Boolean,
|
||||
server_default=sa.false(),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column("api_key", sa.String),
|
||||
sa.Column("match_by_username_email", sa.Boolean, server_default=sa.false()),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("user")
|
||||
|
|
@ -1,35 +0,0 @@
|
|||
"""create table region
|
||||
|
||||
Revision ID: a049e5eb24dd
|
||||
Revises: a9627f63fbed
|
||||
Create Date: 2022-04-02 21:28:43.124521
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
from migrations.utils import dbtype
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "a049e5eb24dd"
|
||||
down_revision = "99a3d2eb08f9"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"region",
|
||||
sa.Column("id", sa.String(24), primary_key=True, index=True),
|
||||
sa.Column("name", sa.Text),
|
||||
sa.Column("geometry", dbtype("GEOMETRY(GEOMETRY,3857)"), index=False),
|
||||
sa.Column("admin_level", sa.Integer, index=True),
|
||||
)
|
||||
op.execute(
|
||||
"CREATE INDEX region_geometry_idx ON region USING GIST (geometry) WITH (FILLFACTOR=100);"
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("region")
|
||||
|
|
@ -1,34 +0,0 @@
|
|||
"""create table road_usage
|
||||
|
||||
Revision ID: a9627f63fbed
|
||||
Revises:
|
||||
Create Date: 2022-03-16 20:26:17.449569
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "a9627f63fbed"
|
||||
down_revision = "5d75febe2d59"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"road_usage",
|
||||
sa.Column("id", sa.Integer, autoincrement=True, primary_key=True, index=True),
|
||||
sa.Column(
|
||||
"track_id", sa.Integer, sa.ForeignKey("track.id", ondelete="CASCADE")
|
||||
),
|
||||
sa.Column("hex_hash", sa.String, unique=True, index=True),
|
||||
sa.Column("way_id", sa.BIGINT, index=True),
|
||||
sa.Column("time", sa.DateTime),
|
||||
sa.Column("direction_reversed", sa.Boolean),
|
||||
sa.Index("road_usage_segment", "way_id", "direction_reversed"),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("road_usage")
|
||||
|
|
@ -1,39 +0,0 @@
|
|||
"""add import groups
|
||||
|
||||
Revision ID: b8b0fbae50a4
|
||||
Revises: f7b21148126a
|
||||
Create Date: 2023-03-26 09:41:36.621203
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "b8b0fbae50a4"
|
||||
down_revision = "f7b21148126a"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.add_column(
|
||||
"road",
|
||||
sa.Column("import_group", sa.String(), nullable=True),
|
||||
)
|
||||
op.add_column(
|
||||
"region",
|
||||
sa.Column("import_group", sa.String(), nullable=True),
|
||||
)
|
||||
|
||||
# Set existing to "osm2pgsql"
|
||||
road = sa.table("road", sa.column("import_group", sa.String))
|
||||
op.execute(road.update().values(import_group="osm2pgsql"))
|
||||
|
||||
region = sa.table("region", sa.column("import_group", sa.String))
|
||||
op.execute(region.update().values(import_group="osm2pgsql"))
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_column("road", "import_group")
|
||||
op.drop_column("region", "import_group")
|
||||
|
|
@ -1,66 +0,0 @@
|
|||
"""create table track
|
||||
|
||||
Revision ID: d66baafab5ec
|
||||
Revises: 35e7f1768f9b
|
||||
Create Date: 2022-03-30 21:36:54.848452
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.dialects import postgresql
|
||||
from migrations.utils import dbtype
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "d66baafab5ec"
|
||||
down_revision = "9d8c8c38a1d0"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
NOW = sa.text("NOW()")
|
||||
|
||||
op.create_table(
|
||||
"track",
|
||||
sa.Column("id", sa.Integer, primary_key=True, autoincrement=True),
|
||||
sa.Column("slug", sa.String, unique=True, nullable=False, index=True),
|
||||
sa.Column("created_at", sa.DateTime, nullable=False, server_default=NOW),
|
||||
sa.Column(
|
||||
"updated_at", sa.DateTime, nullable=False, server_default=NOW, onupdate=NOW
|
||||
),
|
||||
sa.Column("title", sa.String),
|
||||
sa.Column(
|
||||
"processing_status",
|
||||
dbtype("processing_status"),
|
||||
server_default=sa.literal("created"),
|
||||
),
|
||||
sa.Column("processing_queued_at", sa.DateTime),
|
||||
sa.Column("processed_at", sa.DateTime),
|
||||
sa.Column("processing_log", sa.TEXT),
|
||||
sa.Column(
|
||||
"customized_title", sa.Boolean, server_default=sa.false(), nullable=False
|
||||
),
|
||||
sa.Column("description", sa.TEXT),
|
||||
sa.Column("public", sa.Boolean, server_default=sa.false()),
|
||||
sa.Column("uploaded_by_user_agent", sa.String),
|
||||
sa.Column("original_file_name", sa.String),
|
||||
sa.Column("original_file_hash", sa.String, nullable=False),
|
||||
sa.Column(
|
||||
"author_id",
|
||||
sa.Integer,
|
||||
sa.ForeignKey("user.id", ondelete="CASCADE"),
|
||||
nullable=False,
|
||||
),
|
||||
sa.Column("recorded_at", sa.DateTime),
|
||||
sa.Column("recorded_until", sa.DateTime),
|
||||
sa.Column("duration", sa.Float),
|
||||
sa.Column("length", sa.Float),
|
||||
sa.Column("segments", sa.Integer),
|
||||
sa.Column("num_events", sa.Integer),
|
||||
sa.Column("num_measurements", sa.Integer),
|
||||
sa.Column("num_valid", sa.Integer),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_table("track")
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
"""add osm id indexes
|
||||
|
||||
Revision ID: f4b0f460254d
|
||||
Revises: b8b0fbae50a4
|
||||
Create Date: 2023-03-30 10:56:22.066768
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "f4b0f460254d"
|
||||
down_revision = "b8b0fbae50a4"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.execute("CREATE INDEX IF NOT EXISTS ix_road_way_id ON road (way_id);")
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_index("ix_road_way_id")
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
"""add user_device
|
||||
|
||||
Revision ID: f7b21148126a
|
||||
Revises: a9627f63fbed
|
||||
Create Date: 2022-09-15 17:48:06.764342
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = "f7b21148126a"
|
||||
down_revision = "a049e5eb24dd"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
op.create_table(
|
||||
"user_device",
|
||||
sa.Column("id", sa.Integer, autoincrement=True, primary_key=True),
|
||||
sa.Column("user_id", sa.Integer, sa.ForeignKey("user.id", ondelete="CASCADE")),
|
||||
sa.Column("identifier", sa.String, nullable=False),
|
||||
sa.Column("display_name", sa.String, nullable=True),
|
||||
sa.Index("user_id_identifier", "user_id", "identifier", unique=True),
|
||||
)
|
||||
op.add_column(
|
||||
"track",
|
||||
sa.Column(
|
||||
"user_device_id",
|
||||
sa.Integer,
|
||||
sa.ForeignKey("user_device.id", ondelete="RESTRICT"),
|
||||
nullable=True,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
op.drop_column("track", "user_device_id")
|
||||
op.drop_table("user_device")
|
||||
|
|
@ -1 +0,0 @@
|
|||
__path__ = __import__("pkgutil").extend_path(__path__, __name__)
|
||||
|
|
@ -1 +0,0 @@
|
|||
__version__ = "0.8.1"
|
||||
|
|
@ -1,363 +0,0 @@
|
|||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
|
||||
from json import JSONEncoder, dumps
|
||||
from functools import wraps, partial
|
||||
from urllib.parse import urlparse
|
||||
from os.path import dirname, join, normpath, abspath, isfile
|
||||
from datetime import datetime, date
|
||||
|
||||
from sanic import Sanic, Blueprint
|
||||
from sanic.response import (
|
||||
text,
|
||||
json as json_response,
|
||||
file as file_response,
|
||||
html as html_response,
|
||||
)
|
||||
from sanic.exceptions import Unauthorized, SanicException
|
||||
from sanic_session import Session, InMemorySessionInterface
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from obs.api.db import User, make_session, connect_db
|
||||
from obs.api.cors import setup_options, add_cors_headers
|
||||
from obs.api.utils import get_single_arg
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class SanicAccessMessageFilter(logging.Filter):
|
||||
"""
|
||||
A filter that modifies the log message of a sanic.access log entry to
|
||||
include useful information.
|
||||
"""
|
||||
|
||||
def filter(self, record):
|
||||
record.msg = f"{record.request} -> {record.status}"
|
||||
return True
|
||||
|
||||
|
||||
def configure_sanic_logging():
|
||||
for logger_name in ["sanic.root", "sanic.access", "sanic.error"]:
|
||||
logger = logging.getLogger(logger_name)
|
||||
for handler in logger.handlers:
|
||||
logger.removeHandler(handler)
|
||||
|
||||
logger = logging.getLogger("sanic.access")
|
||||
for filter_ in logger.filters:
|
||||
logger.removeFilter(filter_)
|
||||
logger.addFilter(SanicAccessMessageFilter())
|
||||
logging.getLogger("sanic.root").setLevel(logging.WARNING)
|
||||
|
||||
|
||||
app = Sanic(
|
||||
"openbikesensor-api",
|
||||
env_prefix="OBS_",
|
||||
)
|
||||
configure_sanic_logging()
|
||||
|
||||
app.config.update(
|
||||
dict(
|
||||
DEBUG=False,
|
||||
VERBOSE=False,
|
||||
AUTO_RELOAD=False,
|
||||
POSTGRES_POOL_SIZE=20,
|
||||
POSTGRES_MAX_OVERFLOW=40,
|
||||
DEDICATED_WORKER=True,
|
||||
FRONTEND_URL=None,
|
||||
FRONTEND_HTTPS=True,
|
||||
TILES_FILE=None,
|
||||
TILE_SEMAPHORE_SIZE=4,
|
||||
EXPORT_SEMAPHORE_SIZE=1,
|
||||
)
|
||||
)
|
||||
|
||||
# overwrite from defaults again
|
||||
app.config.load_environment_vars("OBS_")
|
||||
|
||||
if isfile("./config.py"):
|
||||
app.update_config("./config.py")
|
||||
|
||||
# For developers to override the config without committing it
|
||||
if isfile("./config.overrides.py"):
|
||||
app.update_config("./config.overrides.py")
|
||||
|
||||
c = app.config
|
||||
|
||||
api = Blueprint("api", url_prefix="/api")
|
||||
auth = Blueprint("auth", url_prefix="")
|
||||
|
||||
import re
|
||||
|
||||
TILE_REQUEST_CANCELLED = re.compile(
|
||||
r"Connection lost before response written.*GET /tiles"
|
||||
)
|
||||
|
||||
|
||||
class NoConnectionLostFilter(logging.Filter):
|
||||
def filter(record):
|
||||
return not TILE_REQUEST_CANCELLED.match(record.getMessage())
|
||||
|
||||
|
||||
logging.getLogger("sanic.error").addFilter(NoConnectionLostFilter)
|
||||
|
||||
|
||||
def setup_cors(app):
|
||||
frontend_url = app.config.get("FRONTEND_URL")
|
||||
additional_origins = app.config.get("ADDITIONAL_CORS_ORIGINS")
|
||||
if not frontend_url and not additional_origins:
|
||||
# No CORS configured
|
||||
return
|
||||
|
||||
origins = []
|
||||
if frontend_url:
|
||||
u = urlparse(frontend_url)
|
||||
origins.append(f"{u.scheme}://{u.netloc}")
|
||||
|
||||
if isinstance(additional_origins, str):
|
||||
origins += re.split(r"\s+", additional_origins)
|
||||
elif isinstance(additional_origins, list):
|
||||
origins += additional_origins
|
||||
elif additional_origins is not None:
|
||||
raise ValueError(
|
||||
"invalid option type for ADDITIONAL_CORS_ORIGINS, must be list or space separated str"
|
||||
)
|
||||
|
||||
app.ctx.cors_origins = origins
|
||||
|
||||
# Add OPTIONS handlers to any route that is missing it
|
||||
app.register_listener(setup_options, "before_server_start")
|
||||
|
||||
# Fill in CORS headers
|
||||
app.register_middleware(add_cors_headers, "response")
|
||||
|
||||
|
||||
setup_cors(app)
|
||||
|
||||
|
||||
@app.exception(SanicException, BaseException)
|
||||
async def _handle_sanic_errors(_request, exception):
|
||||
if isinstance(exception, asyncio.CancelledError):
|
||||
return None
|
||||
|
||||
log.error("Exception in handler: %s", exception, exc_info=True)
|
||||
return json_response(
|
||||
{
|
||||
"errors": {
|
||||
type(exception).__name__: str(exception),
|
||||
},
|
||||
},
|
||||
status=exception.status_code if hasattr(exception, "status_code") else 500,
|
||||
)
|
||||
|
||||
|
||||
# Configure paths
|
||||
def configure_paths(c):
|
||||
c.API_ROOT_DIR = c.get("API_ROOT_DIR") or abspath(
|
||||
join(dirname(__file__), "..", "..")
|
||||
)
|
||||
c.DATA_DIR = c.get("DATA_DIR") or normpath(join(c.API_ROOT_DIR, "../data"))
|
||||
c.PROCESSING_DIR = c.get("PROCESSING_DIR") or join(c.DATA_DIR, "processing")
|
||||
c.PROCESSING_OUTPUT_DIR = c.get("PROCESSING_OUTPUT_DIR") or join(
|
||||
c.DATA_DIR, "processing-output"
|
||||
)
|
||||
c.TRACKS_DIR = c.get("TRACKS_DIR") or join(c.DATA_DIR, "tracks")
|
||||
c.OBS_FACE_CACHE_DIR = c.get("OBS_FACE_CACHE_DIR") or join(
|
||||
c.DATA_DIR, "obs-face-cache"
|
||||
)
|
||||
c.FRONTEND_DIR = c.get("FRONTEND_DIR")
|
||||
|
||||
|
||||
configure_paths(app.config)
|
||||
|
||||
|
||||
# TODO: use a different interface, maybe backed by the PostgreSQL, to allow
|
||||
# scaling the API
|
||||
Session(app, interface=InMemorySessionInterface())
|
||||
|
||||
|
||||
@app.before_server_start
|
||||
async def app_connect_db(app, loop):
|
||||
app.ctx._db_engine_ctx = connect_db(
|
||||
app.config.POSTGRES_URL,
|
||||
app.config.POSTGRES_POOL_SIZE,
|
||||
app.config.POSTGRES_MAX_OVERFLOW,
|
||||
)
|
||||
app.ctx._db_engine = await app.ctx._db_engine_ctx.__aenter__()
|
||||
|
||||
if app.config.TILE_SEMAPHORE_SIZE:
|
||||
app.ctx.tile_semaphore = asyncio.Semaphore(app.config.TILE_SEMAPHORE_SIZE)
|
||||
|
||||
if app.config.EXPORT_SEMAPHORE_SIZE:
|
||||
app.ctx.export_semaphore = asyncio.Semaphore(app.config.EXPORT_SEMAPHORE_SIZE)
|
||||
|
||||
|
||||
@app.after_server_stop
|
||||
async def app_disconnect_db(app, loop):
|
||||
if hasattr(app.ctx, "_db_engine_ctx"):
|
||||
await app.ctx._db_engine_ctx.__aexit__(None, None, None)
|
||||
|
||||
|
||||
def remove_right(l, r):
|
||||
if l.endswith(r):
|
||||
return l[: -len(r)]
|
||||
return l
|
||||
|
||||
|
||||
@app.middleware("request")
|
||||
async def inject_arg_getter(req):
|
||||
req.ctx.get_single_arg = partial(get_single_arg, req)
|
||||
|
||||
|
||||
@app.middleware("request")
|
||||
async def inject_urls(req):
|
||||
if req.app.config.FRONTEND_HTTPS:
|
||||
req.ctx.frontend_scheme = "https"
|
||||
elif req.app.config.FRONTEND_URL:
|
||||
req.ctx.frontend_scheme = (
|
||||
"http" if req.app.config.FRONTEND_URL.startswith("http://") else "https"
|
||||
)
|
||||
else:
|
||||
req.ctx.frontend_scheme = req.scheme
|
||||
|
||||
if req.app.config.get("API_URL"):
|
||||
req.ctx.api_url = req.app.config.API_URL.rstrip("/")
|
||||
api_url_parsed = urlparse(req.ctx.api_url)
|
||||
req.ctx.api_scheme = api_url_parsed.scheme # just use the same for now
|
||||
req.ctx.api_base_path = api_url_parsed.path
|
||||
else:
|
||||
req.ctx.api_scheme = req.ctx.frontend_scheme # just use the same for now
|
||||
req.ctx.api_base_path = remove_right(req.server_path, req.path)
|
||||
req.ctx.api_url = (
|
||||
f"{req.ctx.frontend_scheme}://{req.host}{req.ctx.api_base_path}"
|
||||
)
|
||||
|
||||
if req.app.config.FRONTEND_URL:
|
||||
req.ctx.frontend_base_path = "/" + urlparse(
|
||||
req.app.config.FRONTEND_URL
|
||||
).path.strip("/")
|
||||
req.ctx.frontend_url = req.app.config.FRONTEND_URL.rstrip("/")
|
||||
elif app.config.FRONTEND_DIR:
|
||||
req.ctx.frontend_base_path = req.ctx.api_base_path
|
||||
req.ctx.frontend_url = req.ctx.api_url
|
||||
|
||||
else:
|
||||
req.ctx.frontend_base_path = "/"
|
||||
req.ctx.frontend_url = (
|
||||
f"{req.ctx.frontend_scheme}://{req.host}{req.ctx.frontend_base_path}"
|
||||
)
|
||||
|
||||
|
||||
@app.middleware("request")
|
||||
async def inject_session(req):
|
||||
req.ctx._session_ctx = make_session()
|
||||
req.ctx.db = await req.ctx._session_ctx.__aenter__()
|
||||
|
||||
|
||||
@app.middleware("response")
|
||||
async def close_session(req, response):
|
||||
if hasattr(req.ctx, "_session_ctx"):
|
||||
await req.ctx.db.close()
|
||||
await req.ctx._session_ctx.__aexit__(None, None, None)
|
||||
|
||||
|
||||
@app.middleware("request")
|
||||
async def load_user(req):
|
||||
user_id = req.ctx.session.get("user_id")
|
||||
user = None
|
||||
if user_id:
|
||||
user = (
|
||||
await req.ctx.db.execute(select(User).where(User.id == user_id))
|
||||
).scalar()
|
||||
|
||||
req.ctx.user = user
|
||||
|
||||
|
||||
def require_auth(fn):
|
||||
@wraps(fn)
|
||||
def wrapper(req, *args, **kwargs):
|
||||
if not req.ctx.user:
|
||||
raise Unauthorized("Login required")
|
||||
return fn(req, *args, **kwargs)
|
||||
|
||||
return wrapper
|
||||
|
||||
|
||||
def read_api_key(fn):
|
||||
"""
|
||||
A middleware decorator to read the API Key of a user. It is an opt-in to
|
||||
allow usage with API Keys on certain urls. Combine with require_auth to
|
||||
actually check whether a user was authenticated through this. If a login
|
||||
session exists, the api key is ignored.
|
||||
"""
|
||||
|
||||
@wraps(fn)
|
||||
async def wrapper(req, *args, **kwargs):
|
||||
# try to parse a token if one exists, unless a user is already authenticated
|
||||
if (
|
||||
not req.ctx.user
|
||||
and isinstance(req.token, str)
|
||||
and req.token.lower().startswith("obsuserid ")
|
||||
):
|
||||
try:
|
||||
api_key = req.token.split()[1]
|
||||
except LookupError:
|
||||
api_key = None
|
||||
|
||||
if api_key:
|
||||
user = (
|
||||
await req.ctx.db.execute(
|
||||
select(User).where(User.api_key == api_key.strip())
|
||||
)
|
||||
).scalar()
|
||||
|
||||
if not user:
|
||||
raise Unauthorized("invalid OBSUserId token")
|
||||
|
||||
req.ctx.user = user
|
||||
|
||||
return await fn(req, *args, **kwargs)
|
||||
|
||||
return wrapper
|
||||
|
||||
|
||||
class CustomJsonEncoder(JSONEncoder):
|
||||
def default(self, obj):
|
||||
if isinstance(obj, (datetime, date)):
|
||||
return obj.isoformat() + "+0000" # explicit UTC for javascript <3
|
||||
|
||||
# Let the base class default method raise the TypeError
|
||||
return super().default(obj)
|
||||
|
||||
|
||||
def json(*args, **kwargs):
|
||||
return json_response(*args, **kwargs, dumps=partial(dumps, cls=CustomJsonEncoder))
|
||||
|
||||
|
||||
from .routes import (
|
||||
info,
|
||||
login,
|
||||
stats,
|
||||
tracks,
|
||||
users,
|
||||
exports,
|
||||
)
|
||||
|
||||
from .routes import tiles, mapdetails
|
||||
from .routes import frontend
|
||||
|
||||
|
||||
app.blueprint(api)
|
||||
app.blueprint(auth)
|
||||
|
||||
if not app.config.DEDICATED_WORKER:
|
||||
|
||||
async def worker():
|
||||
from obs.api.process import process_tracks_loop
|
||||
|
||||
# run forever
|
||||
await process_tracks_loop(10)
|
||||
|
||||
app.add_task(worker())
|
||||
|
|
@ -1,68 +0,0 @@
|
|||
from collections import defaultdict
|
||||
from typing import Dict, FrozenSet, Iterable
|
||||
|
||||
from sanic import Sanic, response
|
||||
from sanic_routing.router import Route
|
||||
|
||||
|
||||
def _add_cors_headers(request, response, methods: Iterable[str]) -> None:
|
||||
allow_methods = list(set(methods))
|
||||
|
||||
if "OPTIONS" not in allow_methods:
|
||||
allow_methods.append("OPTIONS")
|
||||
|
||||
origin = request.headers.get("origin")
|
||||
if origin in request.app.ctx.cors_origins:
|
||||
headers = {
|
||||
"Access-Control-Allow-Methods": ",".join(allow_methods),
|
||||
"Access-Control-Allow-Origin": origin,
|
||||
"Access-Control-Allow-Credentials": "true",
|
||||
"Access-Control-Allow-Headers": (
|
||||
"origin, content-type, accept, "
|
||||
"authorization, x-xsrf-token, x-request-id"
|
||||
),
|
||||
"Access-Control-Expose-Headers": "content-disposition",
|
||||
}
|
||||
response.headers.extend(headers)
|
||||
|
||||
|
||||
def add_cors_headers(request, response):
|
||||
if request.method != "OPTIONS":
|
||||
methods = [method for method in request.route.methods]
|
||||
_add_cors_headers(request, response, methods)
|
||||
|
||||
|
||||
def _compile_routes_needing_options(routes: Dict[str, Route]) -> Dict[str, FrozenSet]:
|
||||
needs_options = defaultdict(list)
|
||||
# This is 21.12 and later. You will need to change this for older versions.
|
||||
for route in routes.values():
|
||||
if "OPTIONS" not in route.methods:
|
||||
needs_options[route.uri].extend(route.methods)
|
||||
|
||||
return {uri: frozenset(methods) for uri, methods in dict(needs_options).items()}
|
||||
|
||||
|
||||
def _options_wrapper(handler, methods):
|
||||
def wrapped_handler(request, *args, **kwargs):
|
||||
nonlocal methods
|
||||
return handler(request, methods)
|
||||
|
||||
return wrapped_handler
|
||||
|
||||
|
||||
async def options_handler(request, methods) -> response.HTTPResponse:
|
||||
resp = response.empty()
|
||||
_add_cors_headers(request, resp, methods)
|
||||
return resp
|
||||
|
||||
|
||||
def setup_options(app: Sanic, _):
|
||||
app.router.reset()
|
||||
needs_options = _compile_routes_needing_options(app.router.routes_all)
|
||||
for uri, methods in needs_options.items():
|
||||
app.add_route(
|
||||
_options_wrapper(options_handler, methods),
|
||||
uri,
|
||||
methods=["OPTIONS"],
|
||||
)
|
||||
app.router.finalize()
|
||||
|
|
@ -1,578 +0,0 @@
|
|||
import hashlib
|
||||
from contextvars import ContextVar
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import datetime
|
||||
import os
|
||||
from os.path import exists, join, dirname
|
||||
from json import loads
|
||||
import re
|
||||
import math
|
||||
import aiofiles
|
||||
import random
|
||||
import string
|
||||
import secrets
|
||||
from slugify import slugify
|
||||
import logging
|
||||
|
||||
from sqlalchemy.ext.declarative import declarative_base
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
from sqlalchemy.ext.asyncio import create_async_engine
|
||||
from sqlalchemy.orm import sessionmaker as SessionMaker, relationship
|
||||
from sqlalchemy.types import UserDefinedType, BIGINT, TEXT
|
||||
from sqlalchemy import (
|
||||
Boolean,
|
||||
Column,
|
||||
DateTime,
|
||||
Enum as SqlEnum,
|
||||
Float,
|
||||
ForeignKey,
|
||||
Index,
|
||||
Integer,
|
||||
String,
|
||||
false,
|
||||
func,
|
||||
select,
|
||||
text,
|
||||
literal,
|
||||
Text,
|
||||
)
|
||||
from sqlalchemy.dialects.postgresql import UUID
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
Base = declarative_base()
|
||||
|
||||
|
||||
engine = None
|
||||
sessionmaker: SessionMaker
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def make_session():
|
||||
async with sessionmaker(autoflush=True) as session:
|
||||
yield session
|
||||
|
||||
|
||||
async def drop_all():
|
||||
async with engine.begin() as conn:
|
||||
await conn.run_sync(Base.metadata.drop_all)
|
||||
|
||||
|
||||
async def init_models():
|
||||
async with engine.begin() as conn:
|
||||
await conn.execute(text('CREATE EXTENSION IF NOT EXISTS "hstore";'))
|
||||
await conn.execute(text('CREATE EXTENSION IF NOT EXISTS "postgis";'))
|
||||
await conn.execute(text('CREATE EXTENSION IF NOT EXISTS "uuid-ossp";'))
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
|
||||
def random_string(length):
|
||||
letters = string.ascii_lowercase + string.digits
|
||||
return "".join(random.choice(letters) for _ in range(length))
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def connect_db(url, pool_size=10, max_overflow=20):
|
||||
global engine, sessionmaker
|
||||
|
||||
engine = create_async_engine(
|
||||
url, echo=False, pool_size=pool_size, max_overflow=max_overflow
|
||||
)
|
||||
sessionmaker = SessionMaker(engine, class_=AsyncSession, expire_on_commit=False)
|
||||
|
||||
yield engine
|
||||
|
||||
# for AsyncEngine created in function scope, close and
|
||||
# clean-up pooled connections
|
||||
await engine.dispose()
|
||||
|
||||
engine = None
|
||||
sessionmaker = None
|
||||
|
||||
|
||||
ZoneType = SqlEnum("rural", "urban", "motorway", name="zone_type")
|
||||
ProcessingStatus = SqlEnum(
|
||||
"created", "queued", "processing", "complete", "error", name="processing_status"
|
||||
)
|
||||
|
||||
|
||||
class Geometry(UserDefinedType):
|
||||
def get_col_spec(self):
|
||||
return "GEOMETRY"
|
||||
|
||||
def bind_expression(self, bindvalue):
|
||||
return func.ST_GeomFromGeoJSON(bindvalue, type_=self)
|
||||
|
||||
def column_expression(self, col):
|
||||
return func.ST_AsGeoJSON(func.ST_Transform(col, 4326), type_=self)
|
||||
|
||||
|
||||
class LineString(UserDefinedType):
|
||||
def get_col_spec(self):
|
||||
return "geometry(LineString, 3857)"
|
||||
|
||||
def bind_expression(self, bindvalue):
|
||||
return func.ST_GeomFromGeoJSON(bindvalue, type_=self)
|
||||
|
||||
def column_expression(self, col):
|
||||
return func.ST_AsGeoJSON(func.ST_Transform(col, 4326), type_=self)
|
||||
|
||||
|
||||
class GeometryGeometry(UserDefinedType):
|
||||
def get_col_spec(self):
|
||||
return "geometry(GEOMETRY, 3857)"
|
||||
|
||||
def bind_expression(self, bindvalue):
|
||||
return func.ST_GeomFromGeoJSON(bindvalue, type_=self)
|
||||
|
||||
def column_expression(self, col):
|
||||
return func.ST_AsGeoJSON(func.ST_Transform(col, 4326), type_=self)
|
||||
|
||||
|
||||
class OvertakingEvent(Base):
|
||||
__tablename__ = "overtaking_event"
|
||||
__table_args__ = (Index("road_segment", "way_id", "direction_reversed"),)
|
||||
|
||||
id = Column(Integer, autoincrement=True, primary_key=True, index=True)
|
||||
track_id = Column(Integer, ForeignKey("track.id", ondelete="CASCADE"))
|
||||
hex_hash = Column(String, unique=True, index=True)
|
||||
way_id = Column(BIGINT, index=True)
|
||||
|
||||
# whether we were traveling along the way in reverse direction
|
||||
direction_reversed = Column(Boolean)
|
||||
|
||||
geometry = Column(Geometry)
|
||||
latitude = Column(Float)
|
||||
longitude = Column(Float)
|
||||
time = Column(DateTime)
|
||||
distance_overtaker = Column(Float)
|
||||
distance_stationary = Column(Float)
|
||||
course = Column(Float)
|
||||
speed = Column(Float)
|
||||
|
||||
def __repr__(self):
|
||||
return f"<OvertakingEvent {self.id}>"
|
||||
|
||||
|
||||
class Road(Base):
|
||||
__tablename__ = "road"
|
||||
way_id = Column(BIGINT, primary_key=True, index=True, autoincrement=False)
|
||||
zone = Column(ZoneType)
|
||||
name = Column(Text)
|
||||
geometry = Column(LineString)
|
||||
directionality = Column(Integer)
|
||||
oneway = Column(Boolean)
|
||||
import_group = Column(String)
|
||||
|
||||
__table_args__ = (
|
||||
# We keep the index name as osm2pgsql created it, way back when.
|
||||
Index(
|
||||
"road_geometry_idx",
|
||||
"geometry",
|
||||
postgresql_using="gist",
|
||||
postgresql_with={"fillfactor": 100},
|
||||
),
|
||||
)
|
||||
|
||||
def to_dict(self):
|
||||
return {
|
||||
"way_id": self.way_id,
|
||||
"zone": self.zone,
|
||||
"name": self.name,
|
||||
"directionality": self.directionality,
|
||||
"oneway": self.oneway,
|
||||
"geometry": loads(self.geometry),
|
||||
}
|
||||
|
||||
|
||||
class RoadUsage(Base):
|
||||
__tablename__ = "road_usage"
|
||||
__table_args__ = (Index("road_usage_segment", "way_id", "direction_reversed"),)
|
||||
|
||||
id = Column(Integer, autoincrement=True, primary_key=True, index=True)
|
||||
track_id = Column(Integer, ForeignKey("track.id", ondelete="CASCADE"))
|
||||
hex_hash = Column(String, unique=True, index=True)
|
||||
way_id = Column(BIGINT, index=True)
|
||||
time = Column(DateTime)
|
||||
direction_reversed = Column(Boolean)
|
||||
|
||||
def __repr__(self):
|
||||
return f"<RoadUsage {self.id}>"
|
||||
|
||||
def __hash__(self):
|
||||
return int(self.hex_hash, 16)
|
||||
|
||||
def __eq__(self, other):
|
||||
return self.hex_hash == other.hex_hash
|
||||
|
||||
|
||||
NOW = text("NOW()")
|
||||
|
||||
|
||||
class DuplicateTrackFileError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class Track(Base):
|
||||
__tablename__ = "track"
|
||||
id = Column(Integer, primary_key=True, autoincrement=True)
|
||||
slug = Column(String, unique=True, nullable=False, index=True)
|
||||
|
||||
created_at = Column(DateTime, nullable=False, server_default=NOW)
|
||||
updated_at = Column(DateTime, nullable=False, server_default=NOW, onupdate=NOW)
|
||||
|
||||
title = Column(String)
|
||||
|
||||
processing_status = Column(ProcessingStatus, server_default=literal("created"))
|
||||
processing_queued_at = Column(DateTime)
|
||||
processed_at = Column(DateTime)
|
||||
|
||||
processing_log = Column(TEXT)
|
||||
|
||||
# Set to true if the user customized the title. Disables auto-generating
|
||||
# an updated title when the track is (re-)processed.
|
||||
customized_title = Column(Boolean, server_default=false(), nullable=False)
|
||||
|
||||
# A user-provided description of the track. May contain markdown.
|
||||
description = Column(TEXT)
|
||||
|
||||
# Whether this track is visible (anonymized) in the public track list or not.
|
||||
public = Column(Boolean, server_default=false())
|
||||
|
||||
# Whether this track should be exported to the public track database
|
||||
# (after anonymization).
|
||||
# include_in_public_database = Column(Boolean, server_default=false())
|
||||
|
||||
# The user agent string, or a part thereof, that was used to upload this
|
||||
# track. Usually contains only the OBS version, other user agents are
|
||||
# discarded due to being irrelevant.
|
||||
uploaded_by_user_agent = Column(String)
|
||||
|
||||
# The name of the original file, as provided during upload. Used for
|
||||
# providing a download with the same name, and for display in the
|
||||
# frontend.
|
||||
original_file_name = Column(String)
|
||||
|
||||
# A hash of the original file's contents. Nobody can upload the same track twice.
|
||||
original_file_hash = Column(String, nullable=False)
|
||||
|
||||
author_id = Column(
|
||||
Integer, ForeignKey("user.id", ondelete="CASCADE"), nullable=False
|
||||
)
|
||||
|
||||
user_device_id = Column(
|
||||
Integer,
|
||||
ForeignKey("user_device.id", ondelete="RESTRICT"),
|
||||
nullable=True,
|
||||
)
|
||||
|
||||
# Statistics... maybe we'll drop some of this if we can easily compute them from SQL
|
||||
recorded_at = Column(DateTime)
|
||||
recorded_until = Column(DateTime)
|
||||
duration = Column(Float)
|
||||
length = Column(Float)
|
||||
segments = Column(Integer)
|
||||
num_events = Column(Integer)
|
||||
num_measurements = Column(Integer)
|
||||
num_valid = Column(Integer)
|
||||
|
||||
def to_dict(self, for_user_id=None):
|
||||
result = {
|
||||
"id": self.id,
|
||||
"slug": self.slug,
|
||||
"title": self.title,
|
||||
"description": self.description,
|
||||
"createdAt": self.created_at,
|
||||
"updatedAt": self.updated_at,
|
||||
"public": self.public,
|
||||
"processingStatus": self.processing_status,
|
||||
"recordedAt": self.recorded_at,
|
||||
"recordedUntil": self.recorded_until,
|
||||
"duration": self.duration,
|
||||
"length": self.length,
|
||||
"numEvents": self.num_events,
|
||||
"numValid": self.num_valid,
|
||||
"numMeasurements": self.num_measurements,
|
||||
}
|
||||
|
||||
if for_user_id is not None and for_user_id == self.author_id:
|
||||
result["uploadedByUserAgent"] = self.uploaded_by_user_agent
|
||||
result["originalFileName"] = self.original_file_name
|
||||
result["userDeviceId"] = self.user_device_id
|
||||
|
||||
if self.author:
|
||||
result["author"] = self.author.to_dict(for_user_id=for_user_id)
|
||||
|
||||
return result
|
||||
|
||||
def is_visible_to_private(self, user):
|
||||
return user is not None and user.id == self.author_id
|
||||
|
||||
def is_visible_to(self, user):
|
||||
return self.is_visible_to_private(user) or self.public
|
||||
|
||||
def generate_slug(self, new_title_or_filename=None):
|
||||
input_text = new_title_or_filename or self.title or self.original_file_name
|
||||
|
||||
if input_text is not None:
|
||||
self.slug = slugify(input_text, separator="_") + "-"
|
||||
else:
|
||||
self.slug = ""
|
||||
|
||||
# make unique
|
||||
self.slug += random_string(8)
|
||||
|
||||
async def prevent_duplicates(self, session, file_body):
|
||||
hex_hash = hashlib.sha512(file_body).hexdigest()
|
||||
|
||||
duplicate_count = await session.scalar(
|
||||
select(func.count())
|
||||
.select_from(Track)
|
||||
.where(
|
||||
Track.original_file_hash == hex_hash
|
||||
and Track.author_id == self.author_id
|
||||
and Track.id != self.id
|
||||
)
|
||||
)
|
||||
|
||||
if duplicate_count:
|
||||
raise DuplicateTrackFileError()
|
||||
|
||||
self.original_file_hash = hex_hash
|
||||
|
||||
async def write_to_original_file(self, config, body):
|
||||
mode = "wb" if isinstance(body, bytes) else "wt"
|
||||
|
||||
target = self.get_original_file_path(config)
|
||||
os.makedirs(dirname(target), exist_ok=True)
|
||||
async with aiofiles.open(target, mode=mode) as f:
|
||||
await f.write(body)
|
||||
|
||||
def queue_processing(self):
|
||||
self.processing_status = "queued"
|
||||
self.processing_queued_at = datetime.utcnow()
|
||||
|
||||
def auto_generate_title(self):
|
||||
if self.customized_title:
|
||||
return
|
||||
|
||||
# Try to figure out when this file was recorded. Either we have it in then
|
||||
# statistics, e.g. after parsing and processing the track, or we can maybe
|
||||
# derive it from the filename.
|
||||
recorded_at = self.recorded_at
|
||||
|
||||
if not recorded_at and self.original_file_name:
|
||||
match = re.match(
|
||||
r"^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}\.[0-9]{2}\.[0-9]{2}",
|
||||
self.original_file_name,
|
||||
)
|
||||
if match:
|
||||
try:
|
||||
recorded_at = datetime.fromisoformat(match[0])
|
||||
except ValueError:
|
||||
pass
|
||||
|
||||
if recorded_at:
|
||||
daytime = _get_daytime(recorded_at)
|
||||
self.title = f"{daytime} ride on {recorded_at.strftime('%a, %x')}"
|
||||
return
|
||||
|
||||
# Detecting recording date failed, use filename
|
||||
if self.original_file_name:
|
||||
words = self.original_file_name
|
||||
words = re.sub(r"(\.obsdata)?\.csv$", "", words)
|
||||
words = re.split(r"\W+", words)
|
||||
words[0] = words[0][0].upper() + words[0][1:]
|
||||
self.title = " ".join(words)
|
||||
|
||||
@property
|
||||
def file_path(self):
|
||||
return join(self.author.username, self.slug)
|
||||
|
||||
def get_original_file_path(self, config):
|
||||
return join(config.TRACKS_DIR, self.file_path, "original.csv")
|
||||
|
||||
|
||||
class User(Base):
|
||||
__tablename__ = "user"
|
||||
id = Column(Integer, autoincrement=True, primary_key=True)
|
||||
created_at = Column(DateTime, nullable=False, server_default=NOW)
|
||||
updated_at = Column(DateTime, nullable=False, server_default=NOW, onupdate=NOW)
|
||||
sub = Column(String, unique=True, nullable=False)
|
||||
username = Column(String, unique=True, nullable=False)
|
||||
display_name = Column(String, nullable=True)
|
||||
email = Column(String, nullable=False)
|
||||
bio = Column(TEXT)
|
||||
image = Column(String)
|
||||
are_tracks_visible_for_all = Column(Boolean, server_default=false(), nullable=False)
|
||||
api_key = Column(String)
|
||||
|
||||
# This user can be matched by the email address from the auth service
|
||||
# instead of having to match by `sub`. If a matching user logs in, the
|
||||
# `sub` is updated to the new sub and this flag is disabled. This is for
|
||||
# migrating *to* the external authentication scheme.
|
||||
match_by_username_email = Column(Boolean, server_default=false())
|
||||
|
||||
def generate_api_key(self):
|
||||
"""
|
||||
Generates a new :py:obj:`api_key` into this instance. The new key is
|
||||
sourced from a secure random source and is urlsafe.
|
||||
"""
|
||||
self.api_key = secrets.token_urlsafe(24)
|
||||
|
||||
def to_dict(self, for_user_id=None):
|
||||
result = {
|
||||
"id": self.id,
|
||||
"displayName": self.display_name or self.username,
|
||||
"bio": self.bio,
|
||||
"image": self.image,
|
||||
}
|
||||
if for_user_id == self.id:
|
||||
result["username"] = self.username
|
||||
return result
|
||||
|
||||
async def rename(self, config, new_name):
|
||||
old_name = self.username
|
||||
|
||||
renames = [
|
||||
(join(basedir, old_name), join(basedir, new_name))
|
||||
for basedir in [config.PROCESSING_OUTPUT_DIR, config.TRACKS_DIR]
|
||||
]
|
||||
|
||||
for src, dst in renames:
|
||||
if exists(dst):
|
||||
raise FileExistsError(
|
||||
f"cannot move {src!r} to {dst!r}, destination exists"
|
||||
)
|
||||
|
||||
for src, dst in renames:
|
||||
if not exists(src):
|
||||
log.debug("Rename user %s: Not moving %s, not found", self.id, src)
|
||||
else:
|
||||
log.info("Rename user %s: Moving %s to %s", self.id, src, dst)
|
||||
os.rename(src, dst)
|
||||
|
||||
self.username = new_name
|
||||
|
||||
|
||||
class UserDevice(Base):
|
||||
__tablename__ = "user_device"
|
||||
id = Column(Integer, autoincrement=True, primary_key=True)
|
||||
user_id = Column(Integer, ForeignKey("user.id", ondelete="CASCADE"))
|
||||
identifier = Column(String, nullable=False)
|
||||
display_name = Column(String, nullable=True)
|
||||
|
||||
__table_args__ = (
|
||||
Index("user_id_identifier", "user_id", "identifier", unique=True),
|
||||
)
|
||||
|
||||
def to_dict(self, for_user_id=None):
|
||||
if for_user_id != self.user_id:
|
||||
return {}
|
||||
|
||||
return {
|
||||
"id": self.id,
|
||||
"identifier": self.identifier,
|
||||
"displayName": self.display_name,
|
||||
}
|
||||
|
||||
|
||||
class Comment(Base):
|
||||
__tablename__ = "comment"
|
||||
id = Column(Integer, autoincrement=True, primary_key=True)
|
||||
uid = Column(UUID, server_default=func.uuid_generate_v4())
|
||||
|
||||
created_at = Column(DateTime, nullable=False, server_default=NOW)
|
||||
updated_at = Column(DateTime, nullable=False, server_default=NOW, onupdate=NOW)
|
||||
|
||||
body = Column(TEXT)
|
||||
|
||||
author_id = Column(Integer, ForeignKey("user.id", ondelete="CASCADE"))
|
||||
|
||||
track_id = Column(Integer, ForeignKey("track.id", ondelete="CASCADE"))
|
||||
|
||||
def to_dict(self, for_user_id=None):
|
||||
return {
|
||||
"id": self.uid,
|
||||
"body": self.body,
|
||||
"author": self.author.to_dict(for_user_id=for_user_id),
|
||||
"createdAt": self.created_at,
|
||||
}
|
||||
|
||||
|
||||
class Region(Base):
|
||||
__tablename__ = "region"
|
||||
|
||||
id = Column(String(24), primary_key=True, index=True)
|
||||
name = Column(Text)
|
||||
geometry = Column(GeometryGeometry)
|
||||
admin_level = Column(Integer, index=True)
|
||||
import_group = Column(String)
|
||||
|
||||
__table_args__ = (
|
||||
# We keep the index name as osm2pgsql created it, way back when.
|
||||
Index(
|
||||
"region_geometry_idx",
|
||||
"geometry",
|
||||
postgresql_using="gist",
|
||||
postgresql_with={"fillfactor": 100},
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
Comment.author = relationship("User", back_populates="authored_comments")
|
||||
User.authored_comments = relationship(
|
||||
"Comment",
|
||||
order_by=Comment.created_at,
|
||||
back_populates="author",
|
||||
passive_deletes=True,
|
||||
)
|
||||
|
||||
Track.author = relationship("User", back_populates="authored_tracks")
|
||||
User.authored_tracks = relationship(
|
||||
"Track", order_by=Track.created_at, back_populates="author", passive_deletes=True
|
||||
)
|
||||
|
||||
Comment.track = relationship("Track", back_populates="comments")
|
||||
Track.comments = relationship(
|
||||
"Comment", order_by=Comment.created_at, back_populates="track", passive_deletes=True
|
||||
)
|
||||
|
||||
OvertakingEvent.track = relationship("Track", back_populates="overtaking_events")
|
||||
Track.overtaking_events = relationship(
|
||||
"OvertakingEvent",
|
||||
order_by=OvertakingEvent.time,
|
||||
back_populates="track",
|
||||
passive_deletes=True,
|
||||
)
|
||||
|
||||
Track.user_device = relationship("UserDevice", back_populates="tracks")
|
||||
UserDevice.tracks = relationship(
|
||||
"Track",
|
||||
order_by=Track.created_at,
|
||||
back_populates="user_device",
|
||||
passive_deletes=False,
|
||||
)
|
||||
|
||||
|
||||
# 0..4 Night, 4..10 Morning, 10..14 Noon, 14..18 Afternoon, 18..22 Evening, 22..00 Night
|
||||
# Two hour intervals
|
||||
_DAYTIMES = [
|
||||
"Night", # 0h - 2h
|
||||
"Night", # 2h - 4h
|
||||
"Morning", # 4h - 6h
|
||||
"Morning", # 6h - 8h
|
||||
"Morning", # 8h - 10h
|
||||
"Noon", # 10h - 12h
|
||||
"Noon", # 12h - 14h
|
||||
"Afternoon", # 14h - 16h
|
||||
"Afternoon", # 16h - 18h
|
||||
"Evening", # 18h - 20h
|
||||
"Evening", # 20h - 22h
|
||||
"Night", # 22h - 24h
|
||||
]
|
||||
|
||||
|
||||
def _get_daytime(d):
|
||||
return _DAYTIMES[math.floor((d.hour % 24) / 2)]
|
||||
|
|
@ -1,377 +0,0 @@
|
|||
import logging
|
||||
import os
|
||||
import json
|
||||
import asyncio
|
||||
import hashlib
|
||||
import struct
|
||||
import pytz
|
||||
from os.path import join
|
||||
from datetime import datetime
|
||||
|
||||
from sqlalchemy import delete, func, select, and_
|
||||
from sqlalchemy.orm import joinedload
|
||||
|
||||
from obs.face.importer import ImportMeasurementsCsv
|
||||
from obs.face.geojson import ExportMeasurements
|
||||
from obs.face.annotate import AnnotateMeasurements
|
||||
from obs.face.filter import (
|
||||
AnonymizationMode,
|
||||
ChainFilter,
|
||||
ConfirmedFilter,
|
||||
DistanceMeasuredFilter,
|
||||
PrivacyFilter,
|
||||
PrivacyZone,
|
||||
PrivacyZonesFilter,
|
||||
RequiredFieldsFilter,
|
||||
)
|
||||
|
||||
from obs.face.osm import DataSource, DatabaseTileSource
|
||||
|
||||
from obs.api.db import OvertakingEvent, RoadUsage, Track, UserDevice, make_session
|
||||
from obs.api.app import app
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def get_data_source():
|
||||
"""
|
||||
Creates a data source based on the configuration of the portal. In *lean*
|
||||
mode, the OverpassTileSource is used to fetch data on demand. In normal
|
||||
mode, the roads database is used.
|
||||
"""
|
||||
return DataSource(DatabaseTileSource())
|
||||
|
||||
|
||||
async def process_tracks_loop(delay):
|
||||
while True:
|
||||
try:
|
||||
async with make_session() as session:
|
||||
track = (
|
||||
await session.execute(
|
||||
select(Track)
|
||||
.where(Track.processing_status == "queued")
|
||||
.order_by(Track.processing_queued_at)
|
||||
.options(joinedload(Track.author))
|
||||
)
|
||||
).scalar()
|
||||
|
||||
if track is None:
|
||||
await asyncio.sleep(delay)
|
||||
continue
|
||||
|
||||
data_source = get_data_source()
|
||||
await process_track(session, track, data_source)
|
||||
except BaseException:
|
||||
log.exception("Failed to process track. Will continue.")
|
||||
await asyncio.sleep(1)
|
||||
continue
|
||||
|
||||
|
||||
async def process_tracks(tracks):
|
||||
"""
|
||||
Processes the tracks and writes event data to the database.
|
||||
|
||||
:param tracks: A list of strings which
|
||||
"""
|
||||
data_source = get_data_source()
|
||||
|
||||
async with make_session() as session:
|
||||
for track_id_or_slug in tracks:
|
||||
track = (
|
||||
await session.execute(
|
||||
select(Track)
|
||||
.where(
|
||||
Track.id == track_id_or_slug
|
||||
if isinstance(track_id_or_slug, int)
|
||||
else Track.slug == track_id_or_slug
|
||||
)
|
||||
.options(joinedload(Track.author))
|
||||
)
|
||||
).scalar()
|
||||
|
||||
if not track:
|
||||
raise ValueError(f"Track {track_id_or_slug!r} not found.")
|
||||
|
||||
await process_track(session, track, data_source)
|
||||
|
||||
|
||||
def to_naive_utc(t):
|
||||
if t is None:
|
||||
return None
|
||||
return t.astimezone(pytz.UTC).replace(tzinfo=None)
|
||||
|
||||
|
||||
async def export_gpx(track, filename, name):
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
gpx = ET.Element("gpx")
|
||||
metadata = ET.SubElement(gpx, "metadata")
|
||||
ET.SubElement(metadata, "name").text = name
|
||||
|
||||
trk = ET.SubElement(gpx, "trk")
|
||||
|
||||
ET.SubElement(trk, "name").text = name
|
||||
ET.SubElement(trk, "type").text = "Cycling"
|
||||
|
||||
trkseg = ET.SubElement(trk, "trkseg")
|
||||
|
||||
for point in track:
|
||||
trkpt = ET.SubElement(
|
||||
trkseg, "trkpt", lat=str(point["latitude"]), lon=str(point["longitude"])
|
||||
)
|
||||
ET.SubElement(trkpt, "time").text = point["time"].isoformat()
|
||||
|
||||
et = ET.ElementTree(gpx)
|
||||
et.write(filename, encoding="utf-8", xml_declaration=True)
|
||||
|
||||
|
||||
async def process_track(session, track, data_source):
|
||||
try:
|
||||
track.processing_status = "complete"
|
||||
track.processed_at = datetime.utcnow()
|
||||
await session.commit()
|
||||
|
||||
original_file_path = track.get_original_file_path(app.config)
|
||||
|
||||
output_dir = join(
|
||||
app.config.PROCESSING_OUTPUT_DIR, track.author.username, track.slug
|
||||
)
|
||||
os.makedirs(output_dir, exist_ok=True)
|
||||
|
||||
log.info("Annotating and filtering CSV file")
|
||||
imported_data, statistics, track_metadata = ImportMeasurementsCsv().read(
|
||||
original_file_path,
|
||||
user_id="dummy", # TODO: user username or id or nothing?
|
||||
dataset_id=Track.slug, # TODO: use track id or slug or nothing?
|
||||
return_metadata=True,
|
||||
)
|
||||
|
||||
annotator = AnnotateMeasurements(
|
||||
data_source,
|
||||
cache_dir=app.config.OBS_FACE_CACHE_DIR,
|
||||
fully_annotate_unconfirmed=True,
|
||||
)
|
||||
input_data = await annotator.annotate(imported_data)
|
||||
|
||||
track_filter = ChainFilter(
|
||||
RequiredFieldsFilter(),
|
||||
PrivacyFilter(
|
||||
user_id_mode=AnonymizationMode.REMOVE,
|
||||
measurement_id_mode=AnonymizationMode.REMOVE,
|
||||
),
|
||||
# TODO: load user privacy zones and create a PrivacyZonesFilter() from them
|
||||
)
|
||||
measurements_filter = DistanceMeasuredFilter()
|
||||
overtaking_events_filter = ConfirmedFilter()
|
||||
|
||||
track_points = track_filter.filter(input_data, log=log)
|
||||
measurements = measurements_filter.filter(track_points, log=log)
|
||||
overtaking_events = overtaking_events_filter.filter(measurements, log=log)
|
||||
|
||||
exporter = ExportMeasurements("measurements.dummy")
|
||||
await exporter.add_measurements(measurements)
|
||||
measurements_json = exporter.get_data()
|
||||
del exporter
|
||||
|
||||
exporter = ExportMeasurements("overtaking_events.dummy")
|
||||
await exporter.add_measurements(overtaking_events)
|
||||
overtaking_events_json = exporter.get_data()
|
||||
del exporter
|
||||
|
||||
track_json = {
|
||||
"type": "Feature",
|
||||
"geometry": {
|
||||
"type": "LineString",
|
||||
"coordinates": [[m["longitude"], m["latitude"]] for m in track_points],
|
||||
},
|
||||
}
|
||||
|
||||
track_raw_json = {
|
||||
"type": "Feature",
|
||||
"geometry": {
|
||||
"type": "LineString",
|
||||
"coordinates": [
|
||||
[m["longitude_GPS"], m["latitude_GPS"]] for m in track_points
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
for output_filename, data in [
|
||||
("measurements.json", measurements_json),
|
||||
("overtakingEvents.json", overtaking_events_json),
|
||||
("track.json", track_json),
|
||||
("trackRaw.json", track_raw_json),
|
||||
]:
|
||||
target = join(output_dir, output_filename)
|
||||
log.debug("Writing file %s", target)
|
||||
with open(target, "w") as fp:
|
||||
json.dump(data, fp, indent=4)
|
||||
|
||||
await export_gpx(track_points, join(output_dir, "track.gpx"), track.slug)
|
||||
|
||||
log.info("Clearing old track data...")
|
||||
await clear_track_data(session, track)
|
||||
await session.commit()
|
||||
|
||||
device_identifier = track_metadata.get("DeviceId")
|
||||
if device_identifier:
|
||||
if isinstance(device_identifier, list):
|
||||
device_identifier = device_identifier[0]
|
||||
|
||||
log.info("Finding or creating device %s", device_identifier)
|
||||
user_device = (
|
||||
await session.execute(
|
||||
select(UserDevice).where(
|
||||
and_(
|
||||
UserDevice.user_id == track.author_id,
|
||||
UserDevice.identifier == device_identifier,
|
||||
)
|
||||
)
|
||||
)
|
||||
).scalar()
|
||||
|
||||
log.debug("user_device is %s", user_device)
|
||||
|
||||
if not user_device:
|
||||
user_device = UserDevice(
|
||||
user_id=track.author_id, identifier=device_identifier
|
||||
)
|
||||
log.debug("Create new device for this user")
|
||||
session.add(user_device)
|
||||
|
||||
track.user_device = user_device
|
||||
else:
|
||||
log.info("No DeviceId in track metadata.")
|
||||
|
||||
log.info("Import events into database...")
|
||||
await import_overtaking_events(session, track, overtaking_events)
|
||||
|
||||
log.info("import road usages...")
|
||||
await import_road_usages(session, track, track_points)
|
||||
|
||||
log.info("Write track statistics and update status...")
|
||||
track.recorded_at = to_naive_utc(statistics["t_min"])
|
||||
track.recorded_until = to_naive_utc(statistics["t_max"])
|
||||
track.duration = statistics["t"]
|
||||
track.length = statistics["d"]
|
||||
track.segments = statistics["n_segments"]
|
||||
track.num_events = statistics["n_confirmed"]
|
||||
track.num_measurements = statistics["n_measurements"]
|
||||
track.num_valid = statistics["n_valid"]
|
||||
track.processing_status = "complete"
|
||||
track.processed_at = datetime.utcnow()
|
||||
await session.commit()
|
||||
|
||||
log.info("Track %s imported.", track.slug)
|
||||
except BaseException as e:
|
||||
await clear_track_data(session, track)
|
||||
track.processing_status = "error"
|
||||
track.processing_log = str(e)
|
||||
track.processed_at = datetime.utcnow()
|
||||
|
||||
await session.commit()
|
||||
raise
|
||||
|
||||
|
||||
async def clear_track_data(session, track):
|
||||
track.recorded_at = None
|
||||
track.recorded_until = None
|
||||
track.duration = None
|
||||
track.length = None
|
||||
track.segments = None
|
||||
track.num_events = None
|
||||
track.num_measurements = None
|
||||
track.num_valid = None
|
||||
|
||||
await session.execute(
|
||||
delete(OvertakingEvent).where(OvertakingEvent.track_id == track.id)
|
||||
)
|
||||
await session.execute(delete(RoadUsage).where(RoadUsage.track_id == track.id))
|
||||
|
||||
|
||||
async def import_overtaking_events(session, track, overtaking_events):
|
||||
# We use a dictionary to prevent per-track hash collisions, ignoring all
|
||||
# but the first event of the same hash
|
||||
event_models = {}
|
||||
|
||||
for m in overtaking_events:
|
||||
hex_hash = hashlib.sha256(
|
||||
struct.pack(
|
||||
"ddQ", m["latitude"], m["longitude"], int(m["time"].timestamp())
|
||||
)
|
||||
).hexdigest()
|
||||
|
||||
event_models[hex_hash] = OvertakingEvent(
|
||||
track_id=track.id,
|
||||
hex_hash=hex_hash,
|
||||
way_id=m.get("OSM_way_id"),
|
||||
direction_reversed=m.get("OSM_way_orientation", 0) < 0,
|
||||
geometry=func.ST_Transform(
|
||||
func.ST_GeomFromGeoJSON(
|
||||
json.dumps(
|
||||
{
|
||||
"type": "Point",
|
||||
"coordinates": [m["longitude"], m["latitude"]],
|
||||
}
|
||||
)
|
||||
),
|
||||
3857,
|
||||
),
|
||||
latitude=m["latitude"],
|
||||
longitude=m["longitude"],
|
||||
time=m["time"].astimezone(pytz.utc).replace(tzinfo=None),
|
||||
distance_overtaker=m["distance_overtaker"],
|
||||
distance_stationary=m["distance_stationary"],
|
||||
course=m["course"],
|
||||
speed=m["speed"],
|
||||
)
|
||||
|
||||
session.add_all(event_models.values())
|
||||
|
||||
|
||||
def get_road_usages(track_points):
|
||||
last_key = None
|
||||
last = None
|
||||
|
||||
for p in track_points:
|
||||
way_id = p.get("OSM_way_id")
|
||||
direction_reversed = p.get("OSM_way_orientation", 0) < 0
|
||||
|
||||
key = (way_id, direction_reversed)
|
||||
|
||||
if last_key is None or last_key[0] is None:
|
||||
last = p
|
||||
last_key = key
|
||||
continue
|
||||
|
||||
if last_key != key:
|
||||
if last_key[0] is not None:
|
||||
yield last
|
||||
last_key = key
|
||||
last = p
|
||||
|
||||
if last is not None and last_key[0] is not None:
|
||||
yield last
|
||||
|
||||
|
||||
async def import_road_usages(session, track, track_points):
|
||||
usages = set()
|
||||
for p in get_road_usages(track_points):
|
||||
direction_reversed = p.get("OSM_way_orientation", 0) < 0
|
||||
way_id = p.get("OSM_way_id")
|
||||
time = p["time"]
|
||||
|
||||
hex_hash = hashlib.sha256(
|
||||
struct.pack("dQ", way_id, int(time.timestamp()))
|
||||
).hexdigest()
|
||||
|
||||
usages.add(
|
||||
RoadUsage(
|
||||
track_id=track.id,
|
||||
hex_hash=hex_hash,
|
||||
way_id=way_id,
|
||||
time=time.astimezone(pytz.utc).replace(tzinfo=None),
|
||||
direction_reversed=direction_reversed,
|
||||
)
|
||||
)
|
||||
session.add_all(usages)
|
||||
|
|
@ -1,261 +0,0 @@
|
|||
import json
|
||||
from enum import Enum
|
||||
from contextlib import contextmanager
|
||||
import zipfile
|
||||
import io
|
||||
import re
|
||||
import math
|
||||
from sqlite3 import connect
|
||||
|
||||
import shapefile
|
||||
from obs.api.db import OvertakingEvent
|
||||
from sqlalchemy import select, func, text
|
||||
from sanic.response import raw
|
||||
from sanic.exceptions import InvalidUsage
|
||||
|
||||
from obs.api.app import api, json as json_response
|
||||
from obs.api.utils import use_request_semaphore
|
||||
|
||||
import logging
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class ExportFormat(str, Enum):
|
||||
SHAPEFILE = "shapefile"
|
||||
GEOJSON = "geojson"
|
||||
|
||||
|
||||
def parse_bounding_box(input_string):
|
||||
left, bottom, right, top = map(float, input_string.split(","))
|
||||
return func.ST_SetSRID(
|
||||
func.ST_MakeBox2D(
|
||||
func.ST_Point(left, bottom),
|
||||
func.ST_Point(right, top),
|
||||
),
|
||||
4326,
|
||||
)
|
||||
|
||||
|
||||
PROJECTION_4326 = (
|
||||
'GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],'
|
||||
'AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],'
|
||||
'UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]]'
|
||||
)
|
||||
|
||||
|
||||
@contextmanager
|
||||
def shapefile_zip(shape_type=shapefile.POINT, basename="events"):
|
||||
zip_buffer = io.BytesIO()
|
||||
shp, shx, dbf = (io.BytesIO() for _ in range(3))
|
||||
writer = shapefile.Writer(
|
||||
shp=shp, shx=shx, dbf=dbf, shapeType=shape_type, encoding="utf8"
|
||||
)
|
||||
|
||||
yield writer, zip_buffer
|
||||
|
||||
writer.balance()
|
||||
writer.close()
|
||||
|
||||
zip_file = zipfile.ZipFile(zip_buffer, "a", zipfile.ZIP_DEFLATED, False)
|
||||
zip_file.writestr(f"{basename}.shp", shp.getbuffer())
|
||||
zip_file.writestr(f"{basename}.shx", shx.getbuffer())
|
||||
zip_file.writestr(f"{basename}.dbf", dbf.getbuffer())
|
||||
zip_file.writestr(f"{basename}.prj", PROJECTION_4326)
|
||||
zip_file.close()
|
||||
|
||||
|
||||
@api.get(r"/export/events")
|
||||
async def export_events(req):
|
||||
async with use_request_semaphore(req, "export_semaphore", timeout=30):
|
||||
bbox = req.ctx.get_single_arg("bbox", default="-180,-90,180,90")
|
||||
assert re.match(r"(-?\d+\.?\d+,?){4}", bbox)
|
||||
bbox = list(map(float, bbox.split(",")))
|
||||
|
||||
fmt = req.ctx.get_single_arg("fmt", convert=ExportFormat)
|
||||
|
||||
events = await req.ctx.db.stream(
|
||||
text(
|
||||
"""
|
||||
SELECT
|
||||
ST_AsGeoJSON(ST_Transform(geometry, 4326)) AS geometry,
|
||||
distance_overtaker,
|
||||
distance_stationary,
|
||||
way_id,
|
||||
direction,
|
||||
speed,
|
||||
time_stamp,
|
||||
course,
|
||||
zone
|
||||
FROM
|
||||
layer_obs_events(
|
||||
ST_Transform(ST_MakeEnvelope(:bbox0, :bbox1, :bbox2, :bbox3, 4326), 3857),
|
||||
19,
|
||||
NULL,
|
||||
'1900-01-01'::timestamp,
|
||||
'2100-01-01'::timestamp
|
||||
)
|
||||
"""
|
||||
).bindparams(bbox0=bbox[0], bbox1=bbox[1], bbox2=bbox[2], bbox3=bbox[3])
|
||||
)
|
||||
|
||||
if fmt == ExportFormat.SHAPEFILE:
|
||||
with shapefile_zip(basename="events") as (writer, zip_buffer):
|
||||
writer.field("distance_overtaker", "N", decimal=4)
|
||||
writer.field("distance_stationary", "N", decimal=4)
|
||||
writer.field("way_id", "N", decimal=0)
|
||||
writer.field("direction", "N", decimal=0)
|
||||
writer.field("course", "N", decimal=4)
|
||||
writer.field("speed", "N", decimal=4)
|
||||
writer.field("zone", "C")
|
||||
|
||||
async for event in events:
|
||||
coords = json.loads(event.geometry)["coordinates"]
|
||||
writer.point(*coords)
|
||||
writer.record(
|
||||
distance_overtaker=event.distance_overtaker,
|
||||
distance_stationary=event.distance_stationary,
|
||||
direction=event.direction,
|
||||
way_id=event.way_id,
|
||||
course=event.course,
|
||||
speed=event.speed,
|
||||
zone=event.zone
|
||||
# "time"=event.time,
|
||||
)
|
||||
|
||||
return raw(zip_buffer.getbuffer())
|
||||
|
||||
if fmt == ExportFormat.GEOJSON:
|
||||
features = []
|
||||
async for event in events:
|
||||
geom = json.loads(event.geometry)
|
||||
features.append(
|
||||
{
|
||||
"type": "Feature",
|
||||
"geometry": geom,
|
||||
"properties": {
|
||||
"distance_overtaker": event.distance_overtaker
|
||||
if event.distance_overtaker is not None
|
||||
and not math.isnan(event.distance_overtaker)
|
||||
else None,
|
||||
"distance_stationary": event.distance_stationary
|
||||
if event.distance_stationary is not None
|
||||
and not math.isnan(event.distance_stationary)
|
||||
else None,
|
||||
"direction": event.direction
|
||||
if event.direction is not None
|
||||
and not math.isnan(event.direction)
|
||||
else None,
|
||||
"way_id": event.way_id,
|
||||
"course": event.course
|
||||
if event.course is not None and not math.isnan(event.course)
|
||||
else None,
|
||||
"speed": event.speed
|
||||
if event.speed is not None and not math.isnan(event.speed)
|
||||
else None,
|
||||
"time": event.time_stamp,
|
||||
"zone": event.zone,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
geojson = {"type": "FeatureCollection", "features": features}
|
||||
return json_response(geojson)
|
||||
|
||||
raise InvalidUsage("unknown export format")
|
||||
|
||||
|
||||
@api.get(r"/export/segments")
|
||||
async def export_segments(req):
|
||||
async with use_request_semaphore(req, "export_semaphore", timeout=30):
|
||||
bbox = req.ctx.get_single_arg("bbox", default="-180,-90,180,90")
|
||||
assert re.match(r"(-?\d+\.?\d+,?){4}", bbox)
|
||||
bbox = list(map(float, bbox.split(",")))
|
||||
|
||||
fmt = req.ctx.get_single_arg("fmt", convert=ExportFormat)
|
||||
segments = await req.ctx.db.stream(
|
||||
text(
|
||||
"""
|
||||
SELECT
|
||||
ST_AsGeoJSON(ST_Transform(geometry, 4326)) AS geometry,
|
||||
way_id,
|
||||
distance_overtaker_mean,
|
||||
distance_overtaker_min,
|
||||
distance_overtaker_max,
|
||||
distance_overtaker_median,
|
||||
overtaking_event_count,
|
||||
usage_count,
|
||||
direction,
|
||||
zone,
|
||||
offset_direction,
|
||||
distance_overtaker_array
|
||||
FROM
|
||||
layer_obs_roads(
|
||||
ST_Transform(ST_MakeEnvelope(:bbox0, :bbox1, :bbox2, :bbox3, 4326), 3857),
|
||||
11,
|
||||
NULL,
|
||||
'1900-01-01'::timestamp,
|
||||
'2100-01-01'::timestamp
|
||||
)
|
||||
WHERE usage_count > 0
|
||||
"""
|
||||
).bindparams(bbox0=bbox[0], bbox1=bbox[1], bbox2=bbox[2], bbox3=bbox[3])
|
||||
)
|
||||
|
||||
if fmt == ExportFormat.SHAPEFILE:
|
||||
with shapefile_zip(shape_type=3, basename="segments") as (
|
||||
writer,
|
||||
zip_buffer,
|
||||
):
|
||||
writer.field("distance_overtaker_mean", "N", decimal=4)
|
||||
writer.field("distance_overtaker_max", "N", decimal=4)
|
||||
writer.field("distance_overtaker_min", "N", decimal=4)
|
||||
writer.field("distance_overtaker_median", "N", decimal=4)
|
||||
writer.field("overtaking_event_count", "N", decimal=4)
|
||||
writer.field("usage_count", "N", decimal=4)
|
||||
writer.field("way_id", "N", decimal=0)
|
||||
writer.field("direction", "N", decimal=0)
|
||||
writer.field("zone", "C")
|
||||
|
||||
async for segment in segments:
|
||||
geom = json.loads(segment.st_asgeojson)
|
||||
writer.line([geom["coordinates"]])
|
||||
writer.record(
|
||||
distance_overtaker_mean=segment.distance_overtaker_mean,
|
||||
distance_overtaker_median=segment.distance_overtaker_median,
|
||||
distance_overtaker_max=segment.distance_overtaker_max,
|
||||
distance_overtaker_min=segment.distance_overtaker_min,
|
||||
usage_count=segment.usage_count,
|
||||
overtaking_event_count=segment.overtaking_event_count,
|
||||
direction=segment.direction,
|
||||
way_id=segment.way_id,
|
||||
zone=segment.zone,
|
||||
)
|
||||
|
||||
return raw(zip_buffer.getbuffer())
|
||||
|
||||
if fmt == ExportFormat.GEOJSON:
|
||||
features = []
|
||||
async for segment in segments:
|
||||
features.append(
|
||||
{
|
||||
"type": "Feature",
|
||||
"geometry": json.loads(segment.geometry),
|
||||
"properties": {
|
||||
"distance_overtaker_mean": segment.distance_overtaker_mean,
|
||||
"distance_overtaker_max": segment.distance_overtaker_max,
|
||||
"distance_overtaker_median": segment.distance_overtaker_median,
|
||||
"overtaking_event_count": segment.overtaking_event_count,
|
||||
"usage_count": segment.usage_count,
|
||||
"distance_overtaker_array": segment.distance_overtaker_array,
|
||||
"direction": segment.direction,
|
||||
"way_id": segment.way_id,
|
||||
"zone": segment.zone,
|
||||
},
|
||||
}
|
||||
)
|
||||
|
||||
geojson = {"type": "FeatureCollection", "features": features}
|
||||
return json_response(geojson)
|
||||
|
||||
raise InvalidUsage("unknown export format")
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
from os.path import join, exists, isfile, abspath
|
||||
|
||||
import sanic.response as response
|
||||
from sanic.exceptions import NotFound
|
||||
|
||||
from obs.api.app import app
|
||||
|
||||
if app.config.FRONTEND_CONFIG:
|
||||
|
||||
@app.get("/config.json")
|
||||
def get_frontend_config(req):
|
||||
result = {
|
||||
"basename": req.ctx.frontend_base_path,
|
||||
**req.app.config.FRONTEND_CONFIG,
|
||||
"apiUrl": f"{req.ctx.api_url}/api",
|
||||
"loginUrl": f"{req.ctx.api_url}/login",
|
||||
"obsMapSource": {
|
||||
"type": "vector",
|
||||
"tiles": [
|
||||
req.ctx.api_url
|
||||
+ req.app.url_for("tiles", zoom="000", x="111", y="222.pbf")
|
||||
.replace("000", "{z}")
|
||||
.replace("111", "{x}")
|
||||
.replace("222", "{y}")
|
||||
],
|
||||
"minzoom": 0,
|
||||
"maxzoom": 14,
|
||||
},
|
||||
}
|
||||
|
||||
return response.json(result)
|
||||
|
||||
|
||||
INDEX_HTML = (
|
||||
join(app.config.FRONTEND_DIR, "index.html")
|
||||
if app.config.get("FRONTEND_DIR")
|
||||
else None
|
||||
)
|
||||
if INDEX_HTML and exists(INDEX_HTML):
|
||||
with open(INDEX_HTML, "rt") as f:
|
||||
index_file_contents = f.read()
|
||||
|
||||
@app.get("/<path:path>")
|
||||
def get_frontend_static(req, path):
|
||||
if path.startswith("api/"):
|
||||
raise NotFound()
|
||||
|
||||
file = join(app.config.FRONTEND_DIR, path)
|
||||
if not abspath(file).startswith(abspath(app.config.FRONTEND_DIR)):
|
||||
raise NotFound()
|
||||
|
||||
if not exists(file) or not path or not isfile(file):
|
||||
return response.html(
|
||||
index_file_contents.replace("__BASE_HREF__", req.ctx.frontend_url + "/")
|
||||
)
|
||||
|
||||
return response.file(file)
|
||||
|
|
@ -1,18 +0,0 @@
|
|||
import logging
|
||||
|
||||
from obs.api.app import api
|
||||
|
||||
from sanic.response import json
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
from obs.api import __version__ as version
|
||||
|
||||
|
||||
@api.route("/info")
|
||||
async def info(req):
|
||||
return json(
|
||||
{
|
||||
"version": version,
|
||||
}
|
||||
)
|
||||
|
|
@ -1,173 +0,0 @@
|
|||
import asyncio
|
||||
import logging
|
||||
import re
|
||||
|
||||
from requests.exceptions import RequestException
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from oic import rndstr
|
||||
from oic.oic import Client
|
||||
from oic.oic.message import AuthorizationResponse, RegistrationResponse
|
||||
from oic.utils.authn.client import CLIENT_AUTHN_METHOD
|
||||
|
||||
from obs.api.app import auth, api
|
||||
from obs.api.db import User
|
||||
|
||||
from sanic.response import json, redirect
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
client = Client(client_authn_method=CLIENT_AUTHN_METHOD)
|
||||
|
||||
# Do not show verbose library output, even when the appliaction is in debug mode
|
||||
logging.getLogger("oic").setLevel(logging.INFO)
|
||||
|
||||
|
||||
@auth.before_server_start
|
||||
async def connect_auth_client(app, loop):
|
||||
client.allow["issuer_mismatch"] = True
|
||||
try:
|
||||
client.provider_config(app.config.KEYCLOAK_URL)
|
||||
client.store_registration_info(
|
||||
RegistrationResponse(
|
||||
client_id=app.config.KEYCLOAK_CLIENT_ID,
|
||||
client_secret=app.config.KEYCLOAK_CLIENT_SECRET,
|
||||
)
|
||||
)
|
||||
except RequestException:
|
||||
log.exception(f"could not connect to {app.config.KEYCLOAK_URL}")
|
||||
log.info("will retry")
|
||||
await asyncio.sleep(2)
|
||||
log.info("retrying")
|
||||
await connect_auth_client(app, loop)
|
||||
|
||||
|
||||
@auth.route("/login")
|
||||
async def login(req):
|
||||
next_url = req.ctx.get_single_arg("next", default=None)
|
||||
|
||||
session = req.ctx.session
|
||||
session["state"] = rndstr()
|
||||
session["nonce"] = rndstr()
|
||||
session["next"] = next_url
|
||||
args = {
|
||||
"client_id": client.client_id,
|
||||
"response_type": "code",
|
||||
"scope": ["openid"],
|
||||
"nonce": session["nonce"],
|
||||
"redirect_uri": req.ctx.api_url + "/login/redirect",
|
||||
"state": session["state"],
|
||||
}
|
||||
|
||||
auth_req = client.construct_AuthorizationRequest(request_args=args)
|
||||
login_url = auth_req.request(client.authorization_endpoint)
|
||||
|
||||
return redirect(login_url)
|
||||
|
||||
|
||||
@auth.route("/login/redirect")
|
||||
async def login_redirect(req):
|
||||
session = req.ctx.session
|
||||
|
||||
auth_response = client.parse_response(
|
||||
AuthorizationResponse, info=dict(req.query_args), sformat="dict"
|
||||
)
|
||||
code = auth_response["code"]
|
||||
state = auth_response["state"]
|
||||
|
||||
assert "state" in session
|
||||
assert state == session["state"]
|
||||
|
||||
client.do_access_token_request(
|
||||
state=state,
|
||||
request_args={"code": code},
|
||||
authn_method="client_secret_basic",
|
||||
)
|
||||
|
||||
userinfo = client.do_user_info_request(state=state)
|
||||
|
||||
# {'sub': '3798e2da-b208-4a1a-98c0-08fecfea1345', 'email_verified': True, 'preferred_username': 'test', 'email': 'test@example.com'}
|
||||
sub = userinfo["sub"]
|
||||
preferred_username = userinfo["preferred_username"]
|
||||
email = userinfo.get("email")
|
||||
|
||||
clean_username = re.sub(r"[^a-zA-Z0-9_.-]", "", preferred_username)
|
||||
if clean_username != preferred_username:
|
||||
log.warning(
|
||||
"Username %r contained invalid characters and was changed to %r",
|
||||
preferred_username,
|
||||
clean_username,
|
||||
)
|
||||
preferred_username = clean_username
|
||||
|
||||
if email is None:
|
||||
raise ValueError(
|
||||
"user has no email set, please configure keycloak to require emails"
|
||||
)
|
||||
|
||||
user = (await req.ctx.db.execute(select(User).where(User.sub == sub))).scalar()
|
||||
|
||||
if user is None:
|
||||
user = (
|
||||
await req.ctx.db.execute(
|
||||
select(User).where(
|
||||
User.email == email
|
||||
and User.username == preferred_username
|
||||
and User.match_by_username_email
|
||||
)
|
||||
)
|
||||
).scalar()
|
||||
|
||||
if user:
|
||||
log.info(
|
||||
"Re-matched existing user %s (sub: %s) based on email and username (%s)",
|
||||
user.id,
|
||||
user.sub,
|
||||
preferred_username,
|
||||
)
|
||||
user.match_by_username_email = False
|
||||
user.sub = sub
|
||||
|
||||
if user is None:
|
||||
log.info(
|
||||
"Registering new user with sub %r (preferred username: %s)",
|
||||
sub,
|
||||
preferred_username,
|
||||
)
|
||||
user = User(sub=sub, username=preferred_username, email=email)
|
||||
req.ctx.db.add(user)
|
||||
else:
|
||||
log.info(
|
||||
"Logged in known user (id: %s, sub: %s, %s).",
|
||||
user.id,
|
||||
user.sub,
|
||||
preferred_username,
|
||||
)
|
||||
|
||||
if email != user.email:
|
||||
log.debug("Updating user (id: %s) email from auth system.", user.id)
|
||||
user.email = email
|
||||
|
||||
if preferred_username != user.username:
|
||||
log.debug("Updating user (id: %s) username from auth system.", user.id)
|
||||
await user.rename(req.app.config, preferred_username)
|
||||
|
||||
await req.ctx.db.commit()
|
||||
|
||||
session["user_id"] = user.id
|
||||
|
||||
next_ = session.pop("next", "/") or "/"
|
||||
return redirect(next_)
|
||||
|
||||
|
||||
@api.route("/logout")
|
||||
async def logout(req):
|
||||
session = req.ctx.session
|
||||
if "user_id" in session:
|
||||
del session["user_id"]
|
||||
|
||||
auth_req = client.construct_EndSessionRequest(state=session["state"])
|
||||
logout_url = auth_req.request(client.end_session_endpoint)
|
||||
|
||||
return redirect(logout_url + f"&post_logout_redirect_uri={req.ctx.api_url}/logout")
|
||||
|
|
@ -1,147 +0,0 @@
|
|||
import json
|
||||
from functools import partial
|
||||
import logging
|
||||
import numpy
|
||||
import math
|
||||
|
||||
from sqlalchemy import select, func, column
|
||||
|
||||
import sanic.response as response
|
||||
from sanic.exceptions import InvalidUsage
|
||||
|
||||
from obs.api.app import api
|
||||
from obs.api.db import Road, OvertakingEvent, Track
|
||||
from obs.api.utils import round_to
|
||||
|
||||
round_distance = partial(round_to, multiples=0.001)
|
||||
round_speed = partial(round_to, multiples=0.1)
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def get_bearing(b, a):
|
||||
# longitude, latitude
|
||||
dL = b[0] - a[0]
|
||||
X = numpy.cos(b[1]) * numpy.sin(dL)
|
||||
Y = numpy.cos(a[1]) * numpy.sin(b[1]) - numpy.sin(a[1]) * numpy.cos(
|
||||
b[1]
|
||||
) * numpy.cos(dL)
|
||||
return numpy.arctan2(Y, X) + 0.5 * math.pi
|
||||
|
||||
|
||||
# Bins for histogram on overtaker distances. 0, 0.25, ... 2.25, infinity
|
||||
DISTANCE_BINS = numpy.arange(0, 2.5, 0.25).tolist() + [float('inf')]
|
||||
|
||||
@api.route("/mapdetails/road", methods=["GET"])
|
||||
async def mapdetails_road(req):
|
||||
longitude = req.ctx.get_single_arg("longitude", convert=float)
|
||||
latitude = req.ctx.get_single_arg("latitude", convert=float)
|
||||
radius = req.ctx.get_single_arg("radius", default=100, convert=float)
|
||||
|
||||
if not (1 <= radius <= 1000):
|
||||
raise InvalidUsage("`radius` parameter must be between 1 and 1000")
|
||||
|
||||
road_geometry = Road.geometry
|
||||
point = func.ST_Transform(
|
||||
func.ST_GeomFromGeoJSON(
|
||||
json.dumps(
|
||||
{
|
||||
"type": "point",
|
||||
"coordinates": [longitude, latitude],
|
||||
}
|
||||
)
|
||||
),
|
||||
3857,
|
||||
)
|
||||
|
||||
road = (
|
||||
await req.ctx.db.execute(
|
||||
select(Road)
|
||||
.where(func.ST_DWithin(road_geometry, point, radius))
|
||||
.order_by(func.ST_Distance(road_geometry, point))
|
||||
.limit(1)
|
||||
)
|
||||
).scalar()
|
||||
|
||||
if road is None:
|
||||
return response.json({})
|
||||
|
||||
arrays = (
|
||||
await req.ctx.db.execute(
|
||||
select(
|
||||
[
|
||||
OvertakingEvent.distance_overtaker,
|
||||
OvertakingEvent.distance_stationary,
|
||||
OvertakingEvent.speed,
|
||||
# Keep this as the last entry always for numpy.partition
|
||||
# below to work.
|
||||
OvertakingEvent.direction_reversed,
|
||||
]
|
||||
).where(OvertakingEvent.way_id == road.way_id)
|
||||
)
|
||||
).all()
|
||||
|
||||
arrays = numpy.array(arrays).T
|
||||
|
||||
if len(arrays) == 0:
|
||||
arrays = numpy.array([[], [], [], []], dtype=float)
|
||||
|
||||
data, mask = arrays[:-1], arrays[-1]
|
||||
data = data.astype(numpy.float64)
|
||||
mask = mask.astype(bool)
|
||||
|
||||
def partition(arr, cond):
|
||||
return arr[:, cond], arr[:, ~cond]
|
||||
|
||||
forwards, backwards = partition(data, ~mask)
|
||||
|
||||
def array_stats(arr, rounder, bins=30):
|
||||
if len(arr):
|
||||
arr = arr[~numpy.isnan(arr)]
|
||||
|
||||
n = len(arr)
|
||||
|
||||
hist, bins = numpy.histogram(arr, bins=bins)
|
||||
|
||||
return {
|
||||
"statistics": {
|
||||
"count": n,
|
||||
"mean": rounder(numpy.mean(arr)) if n else None,
|
||||
"min": rounder(numpy.min(arr)) if n else None,
|
||||
"max": rounder(numpy.max(arr)) if n else None,
|
||||
"median": rounder(numpy.median(arr)) if n else None,
|
||||
},
|
||||
"histogram": {
|
||||
"bins": [None if math.isinf(b) else b for b in bins.tolist()],
|
||||
"counts": hist.tolist(),
|
||||
"zone": road.zone
|
||||
},
|
||||
"values": list(map(rounder, arr.tolist())),
|
||||
}
|
||||
|
||||
bearing = None
|
||||
|
||||
geom = json.loads(road.geometry)
|
||||
if geom["type"] == "LineString":
|
||||
coordinates = geom["coordinates"]
|
||||
bearing = get_bearing(coordinates[0], coordinates[-1])
|
||||
# convert to degrees, as this is more natural to understand for consumers
|
||||
bearing = round_to((bearing / math.pi * 180 + 360) % 360, 1)
|
||||
|
||||
def get_direction_stats(direction_arrays, backwards=False):
|
||||
return {
|
||||
"bearing": ((bearing + 180) % 360 if backwards else bearing)
|
||||
if bearing is not None
|
||||
else None,
|
||||
"distanceOvertaker": array_stats(direction_arrays[0], round_distance, bins=DISTANCE_BINS),
|
||||
"distanceStationary": array_stats(direction_arrays[1], round_distance, bins=DISTANCE_BINS),
|
||||
"speed": array_stats(direction_arrays[2], round_speed),
|
||||
}
|
||||
|
||||
return response.json(
|
||||
{
|
||||
"road": road.to_dict(),
|
||||
"forwards": get_direction_stats(forwards),
|
||||
"backwards": get_direction_stats(backwards, True),
|
||||
}
|
||||
)
|
||||
|
|
@ -1,206 +0,0 @@
|
|||
import logging
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
from operator import and_
|
||||
from functools import reduce
|
||||
|
||||
from sqlalchemy import distinct, select, func, desc
|
||||
|
||||
from sanic.response import json
|
||||
|
||||
from obs.api.app import api
|
||||
from obs.api.db import Track, OvertakingEvent, User, Region, UserDevice
|
||||
from obs.api.utils import round_to
|
||||
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# round to this number of meters for privacy reasons
|
||||
TRACK_LENGTH_ROUNDING = 1000
|
||||
|
||||
# round to this number of seconds for privacy reasons
|
||||
TRACK_DURATION_ROUNDING = 120
|
||||
|
||||
# Everything before this date is probably parsed incorrectly
|
||||
MINUMUM_RECORDING_DATE = datetime(2010, 1, 1)
|
||||
|
||||
|
||||
@api.route("/stats")
|
||||
async def stats(req):
|
||||
user = req.ctx.get_single_arg("user", default=None)
|
||||
start = req.ctx.get_single_arg("start", default=None, convert=datetime)
|
||||
end = req.ctx.get_single_arg("end", default=None, convert=datetime)
|
||||
|
||||
conditions = [
|
||||
Track.recorded_at != None,
|
||||
Track.recorded_at > MINUMUM_RECORDING_DATE,
|
||||
]
|
||||
|
||||
if start is not None:
|
||||
conditions.append(Track.recorded_at >= start)
|
||||
|
||||
if end is not None:
|
||||
conditions.append(Track.recorded_at < end)
|
||||
|
||||
# Only the user can look for their own stats, for now
|
||||
by_user = (
|
||||
user is not None and req.ctx.user is not None and req.ctx.user.id == int(user)
|
||||
)
|
||||
if by_user:
|
||||
conditions.append(Track.author_id == req.ctx.user.id)
|
||||
|
||||
track_condition = reduce(and_, conditions)
|
||||
public_track_condition = Track.public and track_condition
|
||||
|
||||
query = (
|
||||
select(
|
||||
[
|
||||
func.count().label("publicTrackCount"),
|
||||
func.sum(Track.duration).label("trackDuration"),
|
||||
func.sum(Track.length).label("trackLength"),
|
||||
]
|
||||
)
|
||||
.select_from(Track)
|
||||
.where(public_track_condition)
|
||||
)
|
||||
|
||||
public_track_count, track_duration, track_length = (
|
||||
await req.ctx.db.execute(query)
|
||||
).first()
|
||||
|
||||
# This is required because SQL returns NULL when the input set to a
|
||||
# SUM() aggregation is empty.
|
||||
track_duration = track_duration or 0
|
||||
track_length = track_length or 0
|
||||
|
||||
user_count = (
|
||||
1
|
||||
if by_user
|
||||
else (await req.ctx.db.execute(select(func.count()).select_from(User))).scalar()
|
||||
)
|
||||
track_count = (
|
||||
await req.ctx.db.execute(
|
||||
select(func.count()).select_from(Track).where(track_condition)
|
||||
)
|
||||
).scalar()
|
||||
event_count = (
|
||||
await req.ctx.db.execute(
|
||||
select(func.count())
|
||||
.select_from(OvertakingEvent)
|
||||
.join(OvertakingEvent.track)
|
||||
.where(track_condition)
|
||||
)
|
||||
).scalar()
|
||||
device_count = (
|
||||
await req.ctx.db.execute(
|
||||
select(func.count(distinct(UserDevice.id)))
|
||||
.select_from(UserDevice)
|
||||
.join(Track.user_device)
|
||||
.where(track_condition)
|
||||
)
|
||||
).scalar()
|
||||
|
||||
result = {
|
||||
"numEvents": event_count,
|
||||
"userCount": user_count,
|
||||
"trackLength": round_to(track_length or 0, TRACK_LENGTH_ROUNDING),
|
||||
"trackDuration": round_to(track_duration or 0, TRACK_DURATION_ROUNDING),
|
||||
"publicTrackCount": public_track_count,
|
||||
"trackCount": track_count,
|
||||
"deviceCount": device_count,
|
||||
}
|
||||
|
||||
return json(result)
|
||||
|
||||
|
||||
# const trackCount = await Track.find(trackFilter).count();
|
||||
#
|
||||
# const publicTrackCount = await Track.find({
|
||||
# ...trackFilter,
|
||||
# public: true,
|
||||
# }).count();
|
||||
#
|
||||
# const userCount = await User.find({
|
||||
# ...(userFilter
|
||||
# ? { _id: userFilter }
|
||||
# : {
|
||||
# createdAt: dateFilter,
|
||||
# }),
|
||||
# }).count();
|
||||
#
|
||||
# const trackStats = await Track.aggregate([
|
||||
# { $match: trackFilter },
|
||||
# {
|
||||
# $addFields: {
|
||||
# trackLength: {
|
||||
# $cond: [{ $lt: ['$statistics.length', 500000] }, '$statistics.length', 0],
|
||||
# },
|
||||
# numEvents: '$statistics.numEvents',
|
||||
# trackDuration: {
|
||||
# $cond: [
|
||||
# { $and: ['$statistics.recordedUntil', { $gt: ['$statistics.recordedAt', new Date('2010-01-01')] }] },
|
||||
# { $subtract: ['$statistics.recordedUntil', '$statistics.recordedAt'] },
|
||||
# 0,
|
||||
# ],
|
||||
# },
|
||||
# },
|
||||
# },
|
||||
# { $project: { trackLength: true, numEvents: true, trackDuration: true } },
|
||||
# {
|
||||
# $group: {
|
||||
# _id: 'sum',
|
||||
# trackLength: { $sum: '$trackLength' },
|
||||
# numEvents: { $sum: '$numEvents' },
|
||||
# trackDuration: { $sum: '$trackDuration' },
|
||||
# },
|
||||
# },
|
||||
# ]);
|
||||
#
|
||||
# const [trackLength, numEvents, trackDuration] =
|
||||
# trackStats.length > 0
|
||||
# ? [trackStats[0].trackLength, trackStats[0].numEvents, trackStats[0].trackDuration]
|
||||
# : [0, 0, 0];
|
||||
#
|
||||
# const trackLengthPrivatized = Math.floor(trackLength / TRACK_LENGTH_ROUNDING) * TRACK_LENGTH_ROUNDING;
|
||||
# const trackDurationPrivatized =
|
||||
# Math.round(trackDuration / 1000 / TRACK_DURATION_ROUNDING) * TRACK_DURATION_ROUNDING;
|
||||
#
|
||||
# return res.json({
|
||||
# publicTrackCount,
|
||||
# trackLength: trackLengthPrivatized,
|
||||
# trackDuration: trackDurationPrivatized,
|
||||
# numEvents,
|
||||
# trackCount,
|
||||
# userCount,
|
||||
# });
|
||||
# }),
|
||||
# );
|
||||
|
||||
|
||||
@api.route("/stats/regions")
|
||||
async def stats(req):
|
||||
query = (
|
||||
select(
|
||||
[
|
||||
Region.id,
|
||||
Region.name,
|
||||
func.count(OvertakingEvent.id).label("overtaking_event_count"),
|
||||
]
|
||||
)
|
||||
.select_from(Region)
|
||||
.join(
|
||||
OvertakingEvent,
|
||||
func.ST_Within(OvertakingEvent.geometry, Region.geometry),
|
||||
)
|
||||
.group_by(
|
||||
Region.id,
|
||||
Region.name,
|
||||
Region.geometry,
|
||||
)
|
||||
.having(func.count(OvertakingEvent.id) > 0)
|
||||
.order_by(desc("overtaking_event_count"))
|
||||
)
|
||||
|
||||
regions = list(map(dict, (await req.ctx.db.execute(query)).all()))
|
||||
return json(regions)
|
||||
|
|
@ -1,129 +0,0 @@
|
|||
from gzip import decompress
|
||||
from sqlite3 import connect
|
||||
from datetime import datetime, time, timedelta
|
||||
from typing import Optional, Tuple
|
||||
|
||||
import dateutil.parser
|
||||
from sanic.exceptions import Forbidden, InvalidUsage
|
||||
from sanic.response import raw
|
||||
|
||||
from sqlalchemy import text
|
||||
|
||||
from obs.api.app import app
|
||||
from obs.api.utils import use_request_semaphore
|
||||
|
||||
|
||||
def get_tile(filename, zoom, x, y):
|
||||
"""
|
||||
Inspired by:
|
||||
https://github.com/TileStache/TileStache/blob/master/TileStache/MBTiles.py
|
||||
"""
|
||||
|
||||
db = connect(filename)
|
||||
db.text_factory = bytes
|
||||
|
||||
fmt = db.execute("SELECT value FROM metadata WHERE name='format'").fetchone()[0]
|
||||
if fmt != b"pbf":
|
||||
raise ValueError("mbtiles file is in wrong format: %s" % fmt)
|
||||
|
||||
content = db.execute(
|
||||
"SELECT tile_data FROM tiles WHERE zoom_level=? AND tile_column=? AND tile_row=?",
|
||||
(zoom, x, (2**zoom - 1) - y),
|
||||
).fetchone()
|
||||
return content and content[0] or None
|
||||
|
||||
|
||||
def round_date(date, to="weeks", up=False):
|
||||
if to != "weeks":
|
||||
raise ValueError(f"cannot round to {to}")
|
||||
|
||||
midnight = time(0, 0, 0, 0)
|
||||
start_of_day = date.date() # ignore time
|
||||
weekday = date.weekday()
|
||||
|
||||
is_rounded = date.time() == midnight and weekday == 0
|
||||
if is_rounded:
|
||||
return date
|
||||
|
||||
if up:
|
||||
return datetime.combine(start_of_day + timedelta(days=7 - weekday), midnight)
|
||||
else:
|
||||
return datetime.combine(start_of_day - timedelta(days=weekday), midnight)
|
||||
|
||||
|
||||
# regenerate approx. once each day
|
||||
TILE_CACHE_MAX_AGE = 3600 * 24
|
||||
|
||||
|
||||
def get_filter_options(
|
||||
req,
|
||||
) -> Tuple[Optional[str], Optional[datetime], Optional[datetime]]:
|
||||
"""
|
||||
Returns parsed, validated and normalized options for filtering map data, a
|
||||
tuple of
|
||||
|
||||
* user_id (str|None)
|
||||
* start (datetime|None)
|
||||
* end (datetime|None)
|
||||
"""
|
||||
user_id = req.ctx.get_single_arg("user", default=None, convert=int)
|
||||
if user_id is not None and (req.ctx.user is None or req.ctx.user.id != user_id):
|
||||
raise Forbidden()
|
||||
|
||||
parse_date = lambda s: dateutil.parser.parse(s)
|
||||
start = req.ctx.get_single_arg("start", default=None, convert=parse_date)
|
||||
end = req.ctx.get_single_arg("end", default=None, convert=parse_date)
|
||||
|
||||
start = round_date(start, to="weeks", up=False) if start else None
|
||||
end = round_date(end, to="weeks", up=True) if end else None
|
||||
|
||||
if start is not None and end is not None and start >= end:
|
||||
raise InvalidUsage(
|
||||
"end date must be later than start date (note: dates are rounded to weeks)"
|
||||
)
|
||||
|
||||
return user_id, start, end
|
||||
|
||||
|
||||
@app.route(r"/tiles/<zoom:int>/<x:int>/<y:(\d+)\.pbf>")
|
||||
async def tiles(req, zoom: int, x: int, y: str):
|
||||
async with use_request_semaphore(req, "tile_semaphore"):
|
||||
if app.config.get("TILES_FILE"):
|
||||
tile = get_tile(req.app.config.TILES_FILE, int(zoom), int(x), int(y))
|
||||
|
||||
else:
|
||||
user_id, start, end = get_filter_options(req)
|
||||
|
||||
tile = await req.ctx.db.scalar(
|
||||
text(
|
||||
"select data from getmvt(:zoom, :x, :y, :user_id, :min_time, :max_time) as b(data, key);"
|
||||
).bindparams(
|
||||
zoom=int(zoom),
|
||||
x=int(x),
|
||||
y=int(y),
|
||||
user_id=user_id,
|
||||
min_time=start,
|
||||
max_time=end,
|
||||
)
|
||||
)
|
||||
|
||||
gzip = "gzip" in req.headers["accept-encoding"]
|
||||
|
||||
headers = {}
|
||||
headers["Vary"] = "Accept-Encoding"
|
||||
|
||||
if req.app.config.DEBUG:
|
||||
headers["Cache-Control"] = "no-cache"
|
||||
else:
|
||||
headers["Cache-Control"] = f"public, max-age={TILE_CACHE_MAX_AGE}"
|
||||
|
||||
# The tiles in the mbtiles file are gzip-compressed already, so we
|
||||
# serve them actually as-is, and only decompress them if the browser
|
||||
# doesn't accept gzip
|
||||
if gzip:
|
||||
headers["Content-Encoding"] = "gzip"
|
||||
|
||||
if not gzip:
|
||||
tile = decompress(tile)
|
||||
|
||||
return raw(tile, content_type="application/x-protobuf", headers=headers)
|
||||
|
|
@ -1,473 +0,0 @@
|
|||
import logging
|
||||
import re
|
||||
from datetime import date
|
||||
from json import load as jsonload
|
||||
from os.path import join, exists, isfile
|
||||
|
||||
from sanic.exceptions import InvalidUsage, NotFound, Forbidden
|
||||
from sanic.response import file_stream, empty
|
||||
from slugify import slugify
|
||||
from sqlalchemy import select, func, and_
|
||||
from sqlalchemy.orm import joinedload
|
||||
|
||||
from obs.api.app import api, require_auth, read_api_key, json
|
||||
from obs.api.db import Track, Comment, DuplicateTrackFileError
|
||||
from obs.api.utils import tar_of_tracks
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def normalize_user_agent(user_agent):
|
||||
if not user_agent:
|
||||
return None
|
||||
|
||||
m = re.match(r"\bOBS\/[^\s]+", user_agent)
|
||||
return m[0] if m else None
|
||||
|
||||
|
||||
async def _return_tracks(req, extend_query, limit, offset, order_by=None):
|
||||
if limit <= 0 or limit > 1000:
|
||||
raise InvalidUsage("invalid limit")
|
||||
|
||||
if offset < 0:
|
||||
raise InvalidUsage("offset must be positive")
|
||||
|
||||
count_query = extend_query(
|
||||
select(func.count()).select_from(Track).join(Track.author)
|
||||
)
|
||||
track_count = await req.ctx.db.scalar(count_query)
|
||||
|
||||
query = (
|
||||
extend_query(select(Track).options(joinedload(Track.author)))
|
||||
.limit(limit)
|
||||
.offset(offset)
|
||||
.order_by(order_by if order_by is not None else Track.created_at)
|
||||
)
|
||||
|
||||
tracks = (await req.ctx.db.execute(query)).scalars()
|
||||
|
||||
return json(
|
||||
{
|
||||
"trackCount": track_count,
|
||||
"tracks": list(
|
||||
map(
|
||||
lambda t: t.to_dict(
|
||||
for_user_id=req.ctx.user.id if req.ctx.user else None
|
||||
),
|
||||
tracks,
|
||||
)
|
||||
),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@api.get("/tracks")
|
||||
async def get_tracks(req):
|
||||
limit = req.ctx.get_single_arg("limit", default=20, convert=int)
|
||||
offset = req.ctx.get_single_arg("offset", default=0, convert=int)
|
||||
# author = req.ctx.get_single_arg("author", default=None, convert=int)
|
||||
|
||||
def extend_query(q):
|
||||
q = q.where(Track.public)
|
||||
|
||||
# if author is not None:
|
||||
# q = q.where(Track.author_id == author)
|
||||
|
||||
return q
|
||||
|
||||
return await _return_tracks(req, extend_query, limit, offset)
|
||||
|
||||
|
||||
def parse_boolean(s):
|
||||
if s is None:
|
||||
return None
|
||||
|
||||
s = s.lower()
|
||||
if s in ("true", "1", "yes", "y", "t"):
|
||||
return True
|
||||
if s in ("false", "0", "no", "n", "f"):
|
||||
return False
|
||||
|
||||
raise ValueError("invalid value for boolean")
|
||||
|
||||
|
||||
@api.get("/tracks/feed")
|
||||
@require_auth
|
||||
async def get_feed(req):
|
||||
limit = req.ctx.get_single_arg("limit", default=20, convert=int)
|
||||
offset = req.ctx.get_single_arg("offset", default=0, convert=int)
|
||||
user_device_id = req.ctx.get_single_arg("user_device_id", default=None, convert=int)
|
||||
|
||||
order_by_columns = {
|
||||
"recordedAt": Track.recorded_at,
|
||||
"title": Track.title,
|
||||
"visibility": Track.public,
|
||||
"length": Track.length,
|
||||
"duration": Track.duration,
|
||||
"user_device_id": Track.user_device_id,
|
||||
}
|
||||
order_by = req.ctx.get_single_arg(
|
||||
"order_by", default=None, convert=order_by_columns.get
|
||||
)
|
||||
|
||||
reversed_ = req.ctx.get_single_arg("reversed", convert=parse_boolean, default=False)
|
||||
if reversed_:
|
||||
order_by = order_by.desc()
|
||||
|
||||
public = req.ctx.get_single_arg("public", convert=parse_boolean, default=None)
|
||||
|
||||
def extend_query(q):
|
||||
q = q.where(Track.author_id == req.ctx.user.id)
|
||||
|
||||
if user_device_id is not None:
|
||||
q = q.where(Track.user_device_id == user_device_id)
|
||||
|
||||
if public is not None:
|
||||
q = q.where(Track.public == public)
|
||||
|
||||
return q
|
||||
|
||||
return await _return_tracks(req, extend_query, limit, offset, order_by)
|
||||
|
||||
|
||||
@api.post("/tracks/bulk")
|
||||
@require_auth
|
||||
async def tracks_bulk_action(req):
|
||||
body = req.json
|
||||
action = body["action"]
|
||||
track_slugs = body["tracks"]
|
||||
|
||||
if action not in ("delete", "makePublic", "makePrivate", "reprocess", "download"):
|
||||
raise InvalidUsage("invalid action")
|
||||
|
||||
query = select(Track).where(
|
||||
and_(Track.author_id == req.ctx.user.id, Track.slug.in_(track_slugs))
|
||||
)
|
||||
|
||||
files = set()
|
||||
|
||||
for track in (await req.ctx.db.execute(query)).scalars():
|
||||
if action == "delete":
|
||||
await req.ctx.db.delete(track)
|
||||
elif action == "makePublic":
|
||||
if not track.public:
|
||||
track.queue_processing()
|
||||
track.public = True
|
||||
elif action == "makePrivate":
|
||||
if track.public:
|
||||
track.queue_processing()
|
||||
track.public = False
|
||||
elif action == "reprocess":
|
||||
track.queue_processing()
|
||||
elif action == "download":
|
||||
files.add(track.get_original_file_path(req.app.config))
|
||||
|
||||
await req.ctx.db.commit()
|
||||
|
||||
if action == "download":
|
||||
username_slug = slugify(req.ctx.user.username, separator="-")
|
||||
date_str = date.today().isoformat()
|
||||
file_basename = f"tracks_{username_slug}_{date_str}"
|
||||
|
||||
await tar_of_tracks(req, files, file_basename)
|
||||
return
|
||||
|
||||
return empty()
|
||||
|
||||
|
||||
@api.post("/tracks")
|
||||
@read_api_key
|
||||
@require_auth
|
||||
async def post_track(req):
|
||||
try:
|
||||
file = req.files["body"][0]
|
||||
except LookupError as e:
|
||||
raise InvalidUsage(
|
||||
'Track upload needs a single file in "body" multipart field'
|
||||
) from e
|
||||
|
||||
try:
|
||||
body = req.json["track"]
|
||||
except (LookupError, InvalidUsage):
|
||||
body = {}
|
||||
|
||||
title = body.get("title")
|
||||
public = body.get("public")
|
||||
|
||||
track = Track(
|
||||
title=title,
|
||||
customized_title=bool(title),
|
||||
author=req.ctx.user,
|
||||
public=public
|
||||
if public is not None
|
||||
else req.ctx.user.are_tracks_visible_for_all,
|
||||
)
|
||||
track.generate_slug()
|
||||
try:
|
||||
await track.prevent_duplicates(req.ctx.db, file.body)
|
||||
except DuplicateTrackFileError:
|
||||
raise InvalidUsage("Track file is not unique")
|
||||
|
||||
track.uploaded_by_user_agent = normalize_user_agent(req.headers["user-agent"])
|
||||
track.original_file_name = file.name
|
||||
await track.write_to_original_file(req.app.config, file.body)
|
||||
track.queue_processing()
|
||||
track.auto_generate_title()
|
||||
|
||||
req.ctx.db.add(track)
|
||||
await req.ctx.db.commit()
|
||||
|
||||
return await get_track(req, track.slug)
|
||||
|
||||
|
||||
async def _load_track(req, slug, raise_not_found=True):
|
||||
track = (
|
||||
await req.ctx.db.execute(
|
||||
select(Track)
|
||||
.where(Track.slug == slug)
|
||||
.options(joinedload(Track.author))
|
||||
.limit(1)
|
||||
)
|
||||
).scalar()
|
||||
|
||||
if raise_not_found and track is None:
|
||||
raise NotFound()
|
||||
|
||||
if not track.is_visible_to(req.ctx.user):
|
||||
raise Forbidden()
|
||||
|
||||
return track
|
||||
|
||||
|
||||
@api.get("/tracks/<slug:str>")
|
||||
async def get_track(req, slug: str):
|
||||
track = await _load_track(req, slug)
|
||||
return json(
|
||||
{"track": track.to_dict(for_user_id=req.ctx.user.id if req.ctx.user else None)},
|
||||
)
|
||||
|
||||
|
||||
@api.delete("/tracks/<slug:str>")
|
||||
@require_auth
|
||||
async def delete_track(req, slug: str):
|
||||
track = await _load_track(req, slug)
|
||||
if not track.is_visible_to_private(req.ctx.user):
|
||||
raise Forbidden()
|
||||
|
||||
await req.ctx.db.delete(track)
|
||||
await req.ctx.db.commit()
|
||||
|
||||
return empty()
|
||||
|
||||
|
||||
@api.get("/tracks/<slug:str>/data")
|
||||
async def get_track_data(req, slug: str):
|
||||
track = await _load_track(req, slug)
|
||||
|
||||
FILE_BY_KEY = {
|
||||
"measurements": "measurements.json",
|
||||
"overtakingEvents": "overtakingEvents.json",
|
||||
"track": "track.json",
|
||||
"trackRaw": "trackRaw.json",
|
||||
}
|
||||
|
||||
result = {}
|
||||
|
||||
for key, filename in FILE_BY_KEY.items():
|
||||
file_path = join(
|
||||
req.app.config.PROCESSING_OUTPUT_DIR, track.file_path, filename
|
||||
)
|
||||
if not exists(file_path) or not isfile(file_path):
|
||||
continue
|
||||
|
||||
with open(file_path) as f:
|
||||
result[key] = jsonload(f)
|
||||
|
||||
return json(
|
||||
result,
|
||||
)
|
||||
|
||||
|
||||
@api.get("/tracks/<slug:str>/download/original.csv")
|
||||
async def download_original_file(req, slug: str):
|
||||
track = await _load_track(req, slug)
|
||||
|
||||
if not track.is_visible_to_private(req.ctx.user):
|
||||
raise Forbidden()
|
||||
|
||||
return await file_stream(
|
||||
track.get_original_file_path(req.app.config),
|
||||
mime_type="text/csv",
|
||||
filename=f"{slug}.csv",
|
||||
)
|
||||
|
||||
|
||||
@api.get("/tracks/<slug:str>/download/track.gpx")
|
||||
async def download_track_gpx(req, slug: str):
|
||||
track = await _load_track(req, slug)
|
||||
|
||||
if not track.is_visible_to(req.ctx.user):
|
||||
raise Forbidden()
|
||||
|
||||
file_path = join(req.app.config.PROCESSING_OUTPUT_DIR, track.file_path, "track.gpx")
|
||||
if not exists(file_path) or not isfile(file_path):
|
||||
raise NotFound()
|
||||
|
||||
return await file_stream(
|
||||
file_path,
|
||||
mime_type="application/gpx+xml",
|
||||
filename=f"{slug}.gpx",
|
||||
)
|
||||
|
||||
|
||||
@api.put("/tracks/<slug:str>")
|
||||
@require_auth
|
||||
async def put_track(req, slug: str):
|
||||
track = await _load_track(req, slug)
|
||||
|
||||
if track.author_id != req.ctx.user.id:
|
||||
raise Forbidden()
|
||||
|
||||
try:
|
||||
body = req.json["track"]
|
||||
except BaseException:
|
||||
body = {}
|
||||
|
||||
if "title" in body:
|
||||
track.title = (body["title"] or "").strip() or None
|
||||
track.customized_title = track.title is not None
|
||||
|
||||
if "description" in body:
|
||||
track.description = (body["description"] or "").strip() or None
|
||||
|
||||
process = False
|
||||
|
||||
if "public" in body:
|
||||
public = bool(body["public"])
|
||||
process = process or (public != track.public) # if changed
|
||||
track.public = public
|
||||
|
||||
if "body" in req.files:
|
||||
try:
|
||||
file = req.files["body"][0]
|
||||
except LookupError as e:
|
||||
raise InvalidUsage(
|
||||
'Track upload needs a single file in "body" multipart field'
|
||||
) from e
|
||||
|
||||
await track.prevent_duplicates(req.ctx.db, file.body)
|
||||
track.uploaded_by_user_agent = normalize_user_agent(req.headers["user-agent"])
|
||||
track.original_file_name = file.name or (track.slug + ".csv")
|
||||
await track.write_to_original_file(req.app.config, file.body)
|
||||
process = True
|
||||
|
||||
if process:
|
||||
track.queue_processing()
|
||||
|
||||
track.auto_generate_title()
|
||||
await req.ctx.db.commit()
|
||||
|
||||
track = await _load_track(req, track.slug)
|
||||
return json(
|
||||
{"track": track.to_dict(for_user_id=req.ctx.user.id)},
|
||||
)
|
||||
|
||||
|
||||
@api.get("/tracks/<slug:str>/comments")
|
||||
async def get_track_comments(req, slug: str):
|
||||
limit = req.ctx.get_single_arg("limit", default=20, convert=int)
|
||||
offset = req.ctx.get_single_arg("offset", default=0, convert=int)
|
||||
|
||||
track = await _load_track(req, slug)
|
||||
|
||||
comment_count = await req.ctx.db.scalar(
|
||||
select(func.count()).select_from(Comment).where(Comment.track_id == track.id)
|
||||
)
|
||||
|
||||
query = (
|
||||
select(Comment)
|
||||
.options(joinedload(Comment.author))
|
||||
.where(Comment.track_id == track.id)
|
||||
.order_by(Comment.created_at.desc())
|
||||
.limit(limit)
|
||||
.offset(offset)
|
||||
)
|
||||
|
||||
comments = (await req.ctx.db.execute(query)).scalars()
|
||||
|
||||
return json(
|
||||
{
|
||||
"commentCount": comment_count,
|
||||
"comments": list(
|
||||
map(
|
||||
lambda c: c.to_dict(
|
||||
for_user_id=req.ctx.user.id if req.ctx.user else None
|
||||
),
|
||||
comments,
|
||||
)
|
||||
),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@api.post("/tracks/<slug:str>/comments")
|
||||
@require_auth
|
||||
async def post_track_comment(req, slug: str):
|
||||
track = await _load_track(req, slug)
|
||||
|
||||
body = req.json.get("comment", {}).get("body")
|
||||
if not isinstance(body, str):
|
||||
raise InvalidUsage("no comment given")
|
||||
|
||||
# Ensure body is not empty
|
||||
body = body.strip()
|
||||
if not body:
|
||||
raise InvalidUsage("empty comment")
|
||||
|
||||
comment = Comment(
|
||||
body=body,
|
||||
track_id=track.id,
|
||||
author_id=req.ctx.user.id,
|
||||
)
|
||||
|
||||
req.ctx.db.add(comment)
|
||||
await req.ctx.db.commit()
|
||||
|
||||
await req.ctx.db.refresh(comment)
|
||||
|
||||
comment = (
|
||||
await req.ctx.db.execute(
|
||||
select(Comment)
|
||||
.options(joinedload(Comment.author))
|
||||
.where(Comment.id == comment.id)
|
||||
.limit(1)
|
||||
)
|
||||
).scalar()
|
||||
|
||||
return json({"comment": comment.to_dict(for_user_id=req.ctx.user.id)})
|
||||
|
||||
|
||||
@api.delete("/tracks/<slug:str>/comments/<uid:str>")
|
||||
@require_auth
|
||||
async def delete_track_comment(req, slug: str, uid: str):
|
||||
track = await _load_track(req, slug)
|
||||
|
||||
comment = (
|
||||
await req.ctx.db.execute(
|
||||
select(Comment)
|
||||
.options(joinedload(Comment.author))
|
||||
.where(Comment.track_id == track.id and Comment.uid == uid)
|
||||
.limit(1)
|
||||
)
|
||||
).scalar()
|
||||
|
||||
if not comment:
|
||||
raise NotFound()
|
||||
|
||||
if comment.author_id != req.ctx.user.id:
|
||||
raise Forbidden()
|
||||
|
||||
await req.ctx.db.delete(comment)
|
||||
await req.ctx.db.commit()
|
||||
|
||||
return empty()
|
||||
|
|
@ -1,95 +0,0 @@
|
|||
import logging
|
||||
|
||||
from sanic.response import json
|
||||
from sanic.exceptions import InvalidUsage, Forbidden, NotFound
|
||||
from sqlalchemy import and_, select
|
||||
|
||||
from obs.api.app import api, require_auth
|
||||
from obs.api.db import UserDevice
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
from obs.api import __version__ as version
|
||||
|
||||
|
||||
def user_to_json(user):
|
||||
return {
|
||||
"id": user.id,
|
||||
"username": user.username,
|
||||
"displayName": user.display_name,
|
||||
"email": user.email,
|
||||
"bio": user.bio,
|
||||
"image": user.image,
|
||||
"areTracksVisibleForAll": user.are_tracks_visible_for_all,
|
||||
"apiKey": user.api_key,
|
||||
}
|
||||
|
||||
|
||||
@api.get("/user")
|
||||
async def get_user(req):
|
||||
return json(user_to_json(req.ctx.user) if req.ctx.user else None)
|
||||
|
||||
|
||||
@api.get("/user/devices")
|
||||
async def get_user_devices(req):
|
||||
if not req.ctx.user:
|
||||
raise Forbidden()
|
||||
|
||||
query = (
|
||||
select(UserDevice)
|
||||
.where(UserDevice.user_id == req.ctx.user.id)
|
||||
.order_by(UserDevice.id)
|
||||
)
|
||||
|
||||
devices = (await req.ctx.db.execute(query)).scalars()
|
||||
|
||||
return json([device.to_dict(req.ctx.user.id) for device in devices])
|
||||
|
||||
|
||||
@api.put("/user/devices/<device_id:int>")
|
||||
async def put_user_device(req, device_id):
|
||||
if not req.ctx.user:
|
||||
raise Forbidden()
|
||||
|
||||
body = req.json
|
||||
|
||||
query = (
|
||||
select(UserDevice)
|
||||
.where(and_(UserDevice.user_id == req.ctx.user.id, UserDevice.id == device_id))
|
||||
.limit(1)
|
||||
)
|
||||
|
||||
device = (await req.ctx.db.execute(query)).scalar()
|
||||
|
||||
if device is None:
|
||||
raise NotFound()
|
||||
|
||||
new_name = body.get("displayName", "").strip()
|
||||
if new_name and device.display_name != new_name:
|
||||
device.display_name = new_name
|
||||
await req.ctx.db.commit()
|
||||
|
||||
return json(device.to_dict())
|
||||
|
||||
|
||||
@api.put("/user")
|
||||
@require_auth
|
||||
async def put_user(req):
|
||||
user = req.ctx.user
|
||||
data = req.json
|
||||
|
||||
for key in ["email", "bio", "image"]:
|
||||
if key in data and isinstance(data[key], (str, type(None))):
|
||||
setattr(user, key, data[key])
|
||||
|
||||
if "displayName" in data:
|
||||
user.display_name = data["displayName"] or None
|
||||
|
||||
if "areTracksVisibleForAll" in data:
|
||||
user.are_tracks_visible_for_all = bool(data["areTracksVisibleForAll"])
|
||||
|
||||
if data.get("updateApiKey"):
|
||||
user.generate_api_key()
|
||||
|
||||
await req.ctx.db.commit()
|
||||
return json(user_to_json(req.ctx.user))
|
||||
|
|
@ -1,162 +0,0 @@
|
|||
import asyncio
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import datetime
|
||||
import logging
|
||||
from os.path import commonpath, join, relpath
|
||||
import queue
|
||||
import tarfile
|
||||
|
||||
import dateutil.parser
|
||||
from sanic.exceptions import InvalidUsage, ServiceUnavailable
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
RAISE = object()
|
||||
|
||||
|
||||
def get_single_arg(req, name, default=RAISE, convert=None):
|
||||
try:
|
||||
value = req.args[name][0]
|
||||
except LookupError as e:
|
||||
if default is RAISE:
|
||||
raise InvalidUsage(f"missing `{name}`") from e
|
||||
|
||||
value = default
|
||||
|
||||
if convert is not None and value is not None:
|
||||
if convert is datetime or convert in ("date", "datetime"):
|
||||
convert = lambda s: dateutil.parser.parse(s)
|
||||
|
||||
try:
|
||||
value = convert(value)
|
||||
except (ValueError, TypeError) as e:
|
||||
raise InvalidUsage(f"invalid `{name}`: {str(e)}") from e
|
||||
|
||||
return value
|
||||
|
||||
|
||||
def round_to(value: float, multiples: float) -> float:
|
||||
if value is None:
|
||||
return None
|
||||
return round(value / multiples) * multiples
|
||||
|
||||
|
||||
def chunk_list(lst, n):
|
||||
for s in range(0, len(lst), n):
|
||||
yield lst[s : s + n]
|
||||
|
||||
|
||||
class chunk:
|
||||
def __init__(self, iterable, n):
|
||||
self.iterable = iterable
|
||||
self.n = n
|
||||
|
||||
def __iter__(self):
|
||||
if isinstance(self.iterable, list):
|
||||
yield from chunk_list(self.iterable, self.n)
|
||||
return
|
||||
|
||||
it = iter(self.iterable)
|
||||
while True:
|
||||
current = []
|
||||
try:
|
||||
for _ in range(self.n):
|
||||
current.append(next(it))
|
||||
yield current
|
||||
except StopIteration:
|
||||
if current:
|
||||
yield current
|
||||
break
|
||||
|
||||
async def __aiter__(self):
|
||||
if hasattr(self.iterable, "__iter__"):
|
||||
for item in self:
|
||||
yield item
|
||||
return
|
||||
|
||||
it = self.iterable.__aiter__()
|
||||
while True:
|
||||
current = []
|
||||
try:
|
||||
for _ in range(self.n):
|
||||
current.append(await it.__anext__())
|
||||
yield current
|
||||
except StopAsyncIteration:
|
||||
if len(current):
|
||||
yield current
|
||||
break
|
||||
|
||||
|
||||
async def tar_of_tracks(req, files, file_basename="tracks"):
|
||||
response = await req.respond(
|
||||
content_type="application/x-gtar",
|
||||
headers={
|
||||
"content-disposition": f'attachment; filename="{file_basename}.tar.bz2"'
|
||||
},
|
||||
)
|
||||
|
||||
helper = StreamerHelper(response)
|
||||
|
||||
tar = tarfile.open(name=None, fileobj=helper, mode="w|bz2", bufsize=256 * 512)
|
||||
|
||||
root = commonpath(list(files))
|
||||
for fname in files:
|
||||
log.info("Write file to tar: %s", fname)
|
||||
with open(fname, "rb") as fobj:
|
||||
tarinfo = tar.gettarinfo(fname)
|
||||
tarinfo.name = join(file_basename, relpath(fname, root))
|
||||
tar.addfile(tarinfo, fobj)
|
||||
await helper.send_all()
|
||||
tar.close()
|
||||
await helper.send_all()
|
||||
|
||||
await response.eof()
|
||||
|
||||
|
||||
class StreamerHelper:
|
||||
def __init__(self, response):
|
||||
self.response = response
|
||||
self.towrite = queue.Queue()
|
||||
|
||||
def write(self, data):
|
||||
self.towrite.put(data)
|
||||
|
||||
async def send_all(self):
|
||||
while True:
|
||||
try:
|
||||
tosend = self.towrite.get(block=False)
|
||||
await self.response.send(tosend)
|
||||
except queue.Empty:
|
||||
break
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def use_request_semaphore(req, semaphore_name, timeout=10):
|
||||
"""
|
||||
If configured, acquire a semaphore for the map tile request and release it
|
||||
after the context has finished.
|
||||
|
||||
If the semaphore cannot be acquired within the timeout, issue a 503 Service
|
||||
Unavailable error response that describes that the database is overloaded,
|
||||
so users know what the problem is.
|
||||
|
||||
Operates as a noop when the tile semaphore is not enabled.
|
||||
"""
|
||||
semaphore = getattr(req.app.ctx, semaphore_name, None)
|
||||
|
||||
if semaphore is None:
|
||||
yield
|
||||
return
|
||||
|
||||
try:
|
||||
await asyncio.wait_for(semaphore.acquire(), timeout)
|
||||
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
semaphore.release()
|
||||
|
||||
except asyncio.TimeoutError:
|
||||
raise ServiceUnavailable(
|
||||
"Too many requests, database overloaded. Please retry later."
|
||||
)
|
||||
|
|
@ -1 +0,0 @@
|
|||
__path__ = __import__("pkgutil").extend_path(__path__, __name__)
|
||||
|
|
@ -1,66 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
|
||||
import math
|
||||
import sys
|
||||
import os
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
|
||||
import coloredlogs
|
||||
|
||||
from obs.api.app import app
|
||||
from obs.api.db import connect_db
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def format_size(n, b=1024):
|
||||
if n == 0:
|
||||
return "0 B"
|
||||
if n < 0:
|
||||
return "-" + format_size(n, b)
|
||||
e = math.floor(math.log(n, b))
|
||||
prefixes = ["", "Ki", "Mi", "Gi", "Ti"] if b == 1024 else ["", "K", "M", "G", "T"]
|
||||
e = min(e, len(prefixes) - 1)
|
||||
r = n / b**e
|
||||
s = f"{r:0.2f}" if e > 0 else str(n)
|
||||
return f"{s} {prefixes[e]}B"
|
||||
|
||||
|
||||
class AccessLogFilter(logging.Filter):
|
||||
def filter(self, record):
|
||||
if not record.msg:
|
||||
record.msg = (
|
||||
f"{record.request} - {record.status} ({format_size(record.byte)})"
|
||||
)
|
||||
return True
|
||||
|
||||
|
||||
def main():
|
||||
debug = app.config.DEBUG
|
||||
|
||||
coloredlogs.install(
|
||||
level=logging.DEBUG if app.config.get("VERBOSE", debug) else logging.INFO,
|
||||
milliseconds=True,
|
||||
isatty=True,
|
||||
)
|
||||
|
||||
for ln in ["sanic.root", "sanic.error", "sanic.access"]:
|
||||
l = logging.getLogger(ln)
|
||||
for h in list(l.handlers):
|
||||
l.removeHandler(h)
|
||||
|
||||
logging.getLogger("sanic.access").addFilter(AccessLogFilter())
|
||||
|
||||
app.run(
|
||||
host=app.config.HOST,
|
||||
port=app.config.PORT,
|
||||
debug=debug,
|
||||
auto_reload=app.config.get("AUTO_RELOAD", debug),
|
||||
access_log=True,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
|
@ -1,191 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
|
||||
import sys
|
||||
import re
|
||||
import msgpack
|
||||
|
||||
import osmium
|
||||
import shapely.wkb as wkb
|
||||
from shapely.ops import transform
|
||||
|
||||
HIGHWAY_TYPES = {
|
||||
"trunk",
|
||||
"primary",
|
||||
"secondary",
|
||||
"tertiary",
|
||||
"unclassified",
|
||||
"residential",
|
||||
"trunk_link",
|
||||
"primary_link",
|
||||
"secondary_link",
|
||||
"tertiary_link",
|
||||
"living_street",
|
||||
"service",
|
||||
"track",
|
||||
"road",
|
||||
}
|
||||
ZONE_TYPES = {
|
||||
"urban",
|
||||
"rural",
|
||||
"motorway",
|
||||
}
|
||||
URBAN_TYPES = {
|
||||
"residential",
|
||||
"living_street",
|
||||
"road",
|
||||
}
|
||||
MOTORWAY_TYPES = {
|
||||
"motorway",
|
||||
"motorway_link",
|
||||
}
|
||||
|
||||
ADMIN_LEVEL_MIN = 2
|
||||
ADMIN_LEVEL_MAX = 8
|
||||
MINSPEED_RURAL = 60
|
||||
|
||||
ONEWAY_YES = {"yes", "true", "1"}
|
||||
ONEWAY_REVERSE = {"reverse", "-1"}
|
||||
|
||||
|
||||
def parse_number(tag):
|
||||
if not tag:
|
||||
return None
|
||||
|
||||
match = re.search(r"[0-9]+", tag)
|
||||
if not match:
|
||||
return None
|
||||
|
||||
digits = match.group(0)
|
||||
try:
|
||||
return int(digits)
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def determine_zone(tags):
|
||||
highway = tags.get("highway")
|
||||
zone = tags.get("zone:traffic")
|
||||
|
||||
if zone is not None:
|
||||
if "rural" in zone:
|
||||
return "rural"
|
||||
|
||||
if "motorway" in zone:
|
||||
return "motorway"
|
||||
|
||||
return "urban"
|
||||
|
||||
# From here on we are guessing based on other tags
|
||||
|
||||
if highway in URBAN_TYPES:
|
||||
return "urban"
|
||||
|
||||
if highway in MOTORWAY_TYPES:
|
||||
return "motorway"
|
||||
|
||||
maxspeed_source = tags.get("source:maxspeed")
|
||||
if maxspeed_source and "rural" in maxspeed_source:
|
||||
return "rural"
|
||||
if maxspeed_source and "urban" in maxspeed_source:
|
||||
return "urban"
|
||||
|
||||
for key in ["maxspeed", "maxspeed:forward", "maxspeed:backward"]:
|
||||
maxspeed = parse_number(tags.get(key))
|
||||
if maxspeed is not None and maxspeed > MINSPEED_RURAL:
|
||||
return "rural"
|
||||
|
||||
# default to urban if we have no idea
|
||||
return "urban"
|
||||
|
||||
|
||||
def determine_direction(tags, zone):
|
||||
if (
|
||||
tags.get("oneway") in ONEWAY_YES
|
||||
or tags.get("junction") == "roundabout"
|
||||
or zone == "motorway"
|
||||
):
|
||||
return 1, True
|
||||
|
||||
if tags.get("oneway") in ONEWAY_REVERSE:
|
||||
return -1, True
|
||||
|
||||
return 0, False
|
||||
|
||||
|
||||
class StreamPacker:
|
||||
def __init__(self, stream, *args, **kwargs):
|
||||
self.stream = stream
|
||||
self.packer = msgpack.Packer(*args, autoreset=False, **kwargs)
|
||||
|
||||
def _write_out(self):
|
||||
if hasattr(self.packer, "getbuffer"):
|
||||
chunk = self.packer.getbuffer()
|
||||
else:
|
||||
chunk = self.packer.bytes()
|
||||
|
||||
self.stream.write(chunk)
|
||||
self.packer.reset()
|
||||
|
||||
def pack(self, *args, **kwargs):
|
||||
self.packer.pack(*args, **kwargs)
|
||||
self._write_out()
|
||||
|
||||
def pack_array_header(self, *args, **kwargs):
|
||||
self.packer.pack_array_header(*args, **kwargs)
|
||||
self._write_out()
|
||||
|
||||
def pack_map_header(self, *args, **kwargs):
|
||||
self.packer.pack_map_header(*args, **kwargs)
|
||||
self._write_out()
|
||||
|
||||
def pack_map_pairs(self, *args, **kwargs):
|
||||
self.packer.pack_map_pairs(*args, **kwargs)
|
||||
self._write_out()
|
||||
|
||||
|
||||
# A global factory that creates WKB from a osmium geometry
|
||||
wkbfab = osmium.geom.WKBFactory()
|
||||
|
||||
from pyproj import Transformer
|
||||
|
||||
project = Transformer.from_crs("EPSG:4326", "EPSG:3857", always_xy=True).transform
|
||||
|
||||
|
||||
class OSMHandler(osmium.SimpleHandler):
|
||||
def __init__(self, packer):
|
||||
self.packer = packer
|
||||
super().__init__()
|
||||
|
||||
def way(self, way):
|
||||
tags = way.tags
|
||||
|
||||
highway = tags.get("highway")
|
||||
if not highway or highway not in HIGHWAY_TYPES:
|
||||
return
|
||||
|
||||
access = tags.get("access", None)
|
||||
bicycle = tags.get("bicycle", None)
|
||||
if access == "no" and bicycle not in ["designated", "yes", "permissive", "destination"]:
|
||||
return
|
||||
|
||||
zone = determine_zone(tags)
|
||||
directionality, oneway = determine_direction(tags, zone)
|
||||
name = tags.get("name")
|
||||
|
||||
geometry = wkb.loads(wkbfab.create_linestring(way), hex=True)
|
||||
geometry = transform(project, geometry)
|
||||
geometry = wkb.dumps(geometry)
|
||||
self.packer.pack(
|
||||
[b"\x01", way.id, name, zone, directionality, oneway, geometry]
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
with open(sys.argv[2], "wb") as fout:
|
||||
packer = StreamPacker(fout)
|
||||
osmhandler = OSMHandler(packer)
|
||||
osmhandler.apply_file(sys.argv[1], locations=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
9419
api/package-lock.json
generated
Normal file
9419
api/package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load diff
116
api/package.json
Normal file
116
api/package.json
Normal file
|
|
@ -0,0 +1,116 @@
|
|||
{
|
||||
"name": "open-bike-sensor-web-api",
|
||||
"version": "1.0.0",
|
||||
"description": "Backend API for the OpenBikeSensor web app",
|
||||
"main": "app.js",
|
||||
"scripts": {
|
||||
"mongo:start": "docker run --name realworld-mongo -p 27017:27017 mongo & sleep 5",
|
||||
"start": "node src/",
|
||||
"start:worker": "node src/worker.js",
|
||||
"dev": "nodemon src/",
|
||||
"dev:worker": "nodemon -w src/ src/worker.js",
|
||||
"mongo:stop": "docker stop realworld-mongo && docker rm realworld-mongo",
|
||||
"autoformat": "eslint --fix .",
|
||||
"lint": "eslint .",
|
||||
"test": "jest",
|
||||
"migrate": "mongoose-data-migrate -c .migrations.js",
|
||||
"migrate:up": "npm run migrate -- up",
|
||||
"migrate:down": "npm run migrate -- down"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/openbikesensor/obsAPI.git"
|
||||
},
|
||||
"license": "LGPLv3",
|
||||
"dependencies": {
|
||||
"body-parser": "1.19.0",
|
||||
"bull": "^3.22.0",
|
||||
"connect-busboy": "0.0.2",
|
||||
"cors": "2.8.5",
|
||||
"csv-parse": "^4.15.1",
|
||||
"csv-stringify": "^5.6.1",
|
||||
"ejs": "^3.1.6",
|
||||
"errorhandler": "1.5.1",
|
||||
"express": "4.17.1",
|
||||
"express-jwt": "^6.0.0",
|
||||
"express-session": "1.17.1",
|
||||
"jest": "^26.6.3",
|
||||
"joi": "^17.4.0",
|
||||
"jsonwebtoken": "8.5.1",
|
||||
"luxon": "^1.26.0",
|
||||
"method-override": "3.0.0",
|
||||
"methods": "1.1.2",
|
||||
"mongoose": "^6.0.5",
|
||||
"mongoose-data-migrate": "flashstockinc/mongoose-data-migrate",
|
||||
"mongoose-unique-validator": "2.0.3",
|
||||
"morgan": "1.10.0",
|
||||
"nodemailer": "^6.4.18",
|
||||
"oauth2orize": "^1.11.0",
|
||||
"passport": "0.4.1",
|
||||
"passport-custom": "^1.1.1",
|
||||
"passport-http-bearer": "^1.0.1",
|
||||
"passport-jwt": "^4.0.0",
|
||||
"passport-local": "^1.0.0",
|
||||
"pkce": "^1.0.0-beta2",
|
||||
"pug": "^3.0.1",
|
||||
"request": "2.88.2",
|
||||
"sanitize-filename": "^1.6.3",
|
||||
"semantic-ui-css": "^2.4.1",
|
||||
"slug": "^3.5.2",
|
||||
"turf": "^3.0.14",
|
||||
"underscore": "^1.12.1"
|
||||
},
|
||||
"devDependencies": {
|
||||
"eslint": "^7.20.0",
|
||||
"eslint-config-prettier": "^6.15.0",
|
||||
"eslint-config-standard": "^16.0.2",
|
||||
"eslint-plugin-import": "^2.22.1",
|
||||
"eslint-plugin-jest": "^24.1.5",
|
||||
"eslint-plugin-node": "^11.1.0",
|
||||
"eslint-plugin-prettier": "^3.3.1",
|
||||
"eslint-plugin-promise": "^4.3.1",
|
||||
"nodemon": "^2.0.7",
|
||||
"prettier": "^2.2.1"
|
||||
},
|
||||
"jest": {
|
||||
"modulePathIgnorePatterns": [
|
||||
"local"
|
||||
]
|
||||
},
|
||||
"prettier": {
|
||||
"useTabs": false,
|
||||
"trailingComma": "all",
|
||||
"tabWidth": 2,
|
||||
"semi": true,
|
||||
"singleQuote": true,
|
||||
"printWidth": 120
|
||||
},
|
||||
"eslintConfig": {
|
||||
"extends": [
|
||||
"standard",
|
||||
"prettier"
|
||||
],
|
||||
"plugins": [
|
||||
"jest",
|
||||
"prettier"
|
||||
],
|
||||
"env": {
|
||||
"browser": false,
|
||||
"node": true,
|
||||
"jest/globals": true
|
||||
},
|
||||
"rules": {
|
||||
"prettier/prettier": "error",
|
||||
"standard/array-bracket-even-spacing": 0,
|
||||
"standard/computed-property-even-spacing": 0,
|
||||
"standard/object-curly-even-spacing": 0
|
||||
},
|
||||
"root": true,
|
||||
"ignorePatterns": [
|
||||
"postman-examples/**",
|
||||
"public/**",
|
||||
"node_modules",
|
||||
"local"
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
@ -1,22 +0,0 @@
|
|||
coloredlogs~=15.0.1
|
||||
sanic==22.6.2
|
||||
oic~=1.5.0
|
||||
sanic-session~=0.8.0
|
||||
python-slugify~=6.1.2
|
||||
motor~=3.1.1
|
||||
pyyaml~=5.3.1
|
||||
-e git+https://github.com/openmaptiles/openmaptiles-tools#egg=openmaptiles-tools
|
||||
sqlparse~=0.4.3
|
||||
sqlalchemy[asyncio]~=1.4.46
|
||||
asyncpg~=0.27.0
|
||||
pyshp~=2.3.1
|
||||
alembic~=1.9.4
|
||||
stream-zip~=0.0.50
|
||||
msgpack~=1.0.5
|
||||
osmium~=3.6.0
|
||||
psycopg~=3.1.8
|
||||
shapely~=2.0.1
|
||||
pyproj~=3.4.1
|
||||
aiohttp~=3.8.1
|
||||
# sanic requires websocets and chockes on >=10 in 2022.6.2
|
||||
websockets<11
|
||||
|
|
@ -1 +1 @@
|
|||
Subproject commit 664e4d606416417c0651ea1748d32dd36209be6a
|
||||
Subproject commit 6beab2ebfede7e41a1184b5ae1d0be0c83f8f95c
|
||||
34
api/setup.py
34
api/setup.py
|
|
@ -1,34 +0,0 @@
|
|||
from setuptools import setup, find_packages
|
||||
|
||||
setup(
|
||||
name="openbikesensor-api",
|
||||
version="0.0.1",
|
||||
author="OpenBikeSensor Contributors",
|
||||
license="LGPL-3.0",
|
||||
description="OpenBikeSensor Portal API",
|
||||
url="https://github.com/openbikesensor/portal",
|
||||
packages=find_packages(),
|
||||
package_data={},
|
||||
install_requires=[
|
||||
"coloredlogs~=15.0.1",
|
||||
"sanic==22.6.2",
|
||||
"oic>=1.3.0, <2",
|
||||
"sanic-session~=0.8.0",
|
||||
"python-slugify>=5.0.2,<6.2.0",
|
||||
"motor>=2.5.1,<3.1.2",
|
||||
"pyyaml<6",
|
||||
"sqlparse~=0.4.3",
|
||||
"openmaptiles-tools", # install from git
|
||||
"pyshp>=2.2,<2.4",
|
||||
"sqlalchemy[asyncio]~=1.4.46",
|
||||
"asyncpg~=0.27.0",
|
||||
"alembic~=1.9.4",
|
||||
"stream-zip~=0.0.50",
|
||||
],
|
||||
entry_points={
|
||||
"console_scripts": [
|
||||
"openbikesensor-api=obs.bin.openbikesensor_api:main",
|
||||
"openbikesensor-transform-osm=obs.bin.openbikesensor_transform_osm:main",
|
||||
]
|
||||
},
|
||||
)
|
||||
65
api/src/_helpers/generators.js
Normal file
65
api/src/_helpers/generators.js
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
function* pairwise(iter) {
|
||||
let last;
|
||||
let firstLoop = true;
|
||||
for (const it of iter) {
|
||||
if (firstLoop) {
|
||||
firstLoop = false;
|
||||
} else {
|
||||
yield [last, it];
|
||||
}
|
||||
last = it;
|
||||
}
|
||||
}
|
||||
|
||||
function* enumerate(iter) {
|
||||
let i = 0;
|
||||
for (const it of iter) {
|
||||
yield [i, it];
|
||||
i++;
|
||||
}
|
||||
}
|
||||
|
||||
const map = (fn) =>
|
||||
function* (iter) {
|
||||
for (const [i, it] of enumerate(iter)) {
|
||||
yield fn(it, i);
|
||||
}
|
||||
};
|
||||
|
||||
const filter = (fn) =>
|
||||
function* (iter) {
|
||||
for (const it of iter) {
|
||||
if (fn(it)) {
|
||||
yield it;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
const reduce = (fn, init) => (iter) => {
|
||||
let acc = init;
|
||||
for (const it of iter) {
|
||||
acc = fn(acc, it);
|
||||
}
|
||||
return acc;
|
||||
};
|
||||
|
||||
const scan = (fn) =>
|
||||
function* (iter, init) {
|
||||
let acc = init;
|
||||
for (const it of iter) {
|
||||
acc = fn(acc, it);
|
||||
yield acc;
|
||||
}
|
||||
};
|
||||
|
||||
const flow = (...reducers) => (input) => reducers.reduce((c, fn) => fn(c), input);
|
||||
|
||||
module.exports = {
|
||||
filter,
|
||||
map,
|
||||
enumerate,
|
||||
pairwise,
|
||||
flow,
|
||||
reduce,
|
||||
scan,
|
||||
};
|
||||
30
api/src/_helpers/send-email.js
Normal file
30
api/src/_helpers/send-email.js
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
const nodemailer = require('nodemailer');
|
||||
const config = require('../config');
|
||||
|
||||
module.exports = sendEmail;
|
||||
|
||||
async function sendEmail({ to, subject, html }) {
|
||||
if (config.mail) {
|
||||
const from = config.mail.from;
|
||||
|
||||
const transporter = nodemailer.createTransport({
|
||||
host: config.mail.smtp.host,
|
||||
port: config.mail.smtp.port,
|
||||
secure: !config.mail.smtp.starttls,
|
||||
requiretls: config.mail.smtp.starttls,
|
||||
auth: {
|
||||
user: config.mail.smtp.username,
|
||||
pass: config.mail.smtp.password,
|
||||
},
|
||||
});
|
||||
|
||||
await transporter.sendMail({ from, to, subject, html });
|
||||
} else {
|
||||
console.log(`========== E-Mail disabled, see contents below =========
|
||||
To: ${to}
|
||||
Subject: ${subject}
|
||||
|
||||
${html}
|
||||
`)
|
||||
}
|
||||
}
|
||||
9
api/src/_helpers/wrapRoute.js
Normal file
9
api/src/_helpers/wrapRoute.js
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
const wrapRoute = (fn) => async (req, res, next) => {
|
||||
try {
|
||||
return await fn(req, res);
|
||||
} catch (err) {
|
||||
next(err);
|
||||
}
|
||||
};
|
||||
|
||||
module.exports = wrapRoute;
|
||||
23
api/src/_middleware/error-handler.js
Normal file
23
api/src/_middleware/error-handler.js
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
module.exports = errorHandler;
|
||||
|
||||
function errorHandler(err, req, res, next) {
|
||||
switch (true) {
|
||||
case typeof err === 'string': {
|
||||
// custom application error
|
||||
const is404 = err.toLowerCase().endsWith('not found');
|
||||
const statusCode = is404 ? 404 : 400;
|
||||
return res.status(statusCode).json({ message: err });
|
||||
}
|
||||
|
||||
case err.name === 'ValidationError':
|
||||
// mongoose validation error
|
||||
return res.status(400).json({ message: err.message });
|
||||
|
||||
case err.name === 'UnauthorizedError':
|
||||
// jwt authentication error
|
||||
return res.status(401).json({ message: 'Unauthorized' });
|
||||
|
||||
default:
|
||||
return res.status(500).json({ message: err.message });
|
||||
}
|
||||
}
|
||||
19
api/src/_middleware/validate-request.js
Normal file
19
api/src/_middleware/validate-request.js
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
const validateRequest = (schema, property = 'body') => (req, res, next) => {
|
||||
console.log('validateRequest');
|
||||
|
||||
const options = {
|
||||
abortEarly: false, // include all errors
|
||||
allowUnknown: true, // ignore unknown props
|
||||
stripUnknown: true, // remove unknown props
|
||||
};
|
||||
const { error, value } = schema.validate(req[property], options);
|
||||
if (error) {
|
||||
console.log('error: ', error);
|
||||
next(`Validation error: ${error.details.map((x) => x.message).join(', ')}`);
|
||||
} else {
|
||||
req[property] = value;
|
||||
next();
|
||||
}
|
||||
};
|
||||
|
||||
module.exports = validateRequest;
|
||||
146
api/src/accounts/account.service.js
Normal file
146
api/src/accounts/account.service.js
Normal file
|
|
@ -0,0 +1,146 @@
|
|||
const crypto = require('crypto');
|
||||
const sendEmail = require('../_helpers/send-email');
|
||||
const config = require('../config');
|
||||
const { User } = require('../models');
|
||||
|
||||
const baseUrl = config.baseUrl.replace(/\/+$/, '');
|
||||
|
||||
module.exports = {
|
||||
register,
|
||||
verifyEmail,
|
||||
forgotPassword,
|
||||
validateResetToken,
|
||||
resetPassword,
|
||||
};
|
||||
|
||||
async function register(params) {
|
||||
const user = await User.findOne({ email: params.email });
|
||||
|
||||
if (user) {
|
||||
// send already registered error in email to prevent account enumeration
|
||||
return await sendAlreadyRegisteredEmail(params.email);
|
||||
}
|
||||
|
||||
const newUser = new User();
|
||||
|
||||
newUser.username = params.username;
|
||||
newUser.email = params.email;
|
||||
newUser.setPassword(params.password);
|
||||
newUser.verificationToken = randomTokenString();
|
||||
newUser.needsEmailValidation = true;
|
||||
|
||||
await newUser.save();
|
||||
|
||||
// send email
|
||||
await sendVerificationEmail(newUser);
|
||||
}
|
||||
|
||||
async function verifyEmail({ token }) {
|
||||
const account = await User.findOne({ verificationToken: token });
|
||||
|
||||
if (!account) {
|
||||
throw Error('Verification failed');
|
||||
}
|
||||
|
||||
account.needsEmailValidation = false;
|
||||
account.verificationToken = undefined;
|
||||
await account.save();
|
||||
}
|
||||
|
||||
async function forgotPassword({ email }) {
|
||||
const account = await User.findOne({ email });
|
||||
|
||||
console.log('forgotPassword', account, email);
|
||||
|
||||
// always return ok response to prevent email enumeration
|
||||
if (!account) return;
|
||||
|
||||
// create reset token that expires after 24 hours
|
||||
account.resetToken = {
|
||||
token: randomTokenString(),
|
||||
expires: new Date(Date.now() + 24 * 60 * 60 * 1000),
|
||||
};
|
||||
await account.save();
|
||||
|
||||
console.log('forgotPassword account saved', account);
|
||||
|
||||
// send email
|
||||
await sendPasswordResetEmail(account);
|
||||
}
|
||||
|
||||
async function validateResetToken({ token }) {
|
||||
const account = await User.findOne({
|
||||
'resetToken.token': token,
|
||||
'resetToken.expires': { $gt: Date.now() },
|
||||
});
|
||||
|
||||
if (!account) {
|
||||
throw Error('Invalid token');
|
||||
}
|
||||
}
|
||||
|
||||
async function resetPassword({ token, password }) {
|
||||
const account = await User.findOne({
|
||||
'resetToken.token': token,
|
||||
'resetToken.expires': { $gt: Date.now() },
|
||||
});
|
||||
|
||||
if (!account) {
|
||||
throw Error('Invalid token');
|
||||
}
|
||||
|
||||
// update password and remove reset token
|
||||
account.setPassword(password);
|
||||
account.resetToken = undefined;
|
||||
|
||||
// Since password recovery happens through email, we can consider this a
|
||||
// successful verification of the email address.
|
||||
account.needsEmailValidation = false;
|
||||
account.verificationToken = undefined;
|
||||
|
||||
await account.save();
|
||||
}
|
||||
|
||||
function randomTokenString() {
|
||||
return crypto.randomBytes(40).toString('hex');
|
||||
}
|
||||
|
||||
async function sendVerificationEmail(account) {
|
||||
const verifyUrl = `${baseUrl}/verify-email?token=${account.verificationToken}`;
|
||||
const html = [
|
||||
'<h4>Verify Email</h4>',
|
||||
'<p>Thanks for registering!</p>',
|
||||
'<p>Please click the below link to verify your email address:</p>',
|
||||
`<p><a href="${verifyUrl}">${verifyUrl}</a></p>`,
|
||||
].join('\n');
|
||||
|
||||
await sendEmail({
|
||||
to: account.email,
|
||||
subject: 'Sign-up Verification API - Verify Email',
|
||||
html,
|
||||
});
|
||||
}
|
||||
|
||||
async function sendAlreadyRegisteredEmail(email) {
|
||||
const message = `<p>If you don't know your password please visit the <a href="${baseUrl}/forgot-password">forgot password</a> page.</p>`;
|
||||
|
||||
await sendEmail({
|
||||
to: email,
|
||||
subject: 'Sign-up Verification API - Email Already Registered',
|
||||
html: `<h4>Email Already Registered</h4>
|
||||
<p>Your email <strong>${email}</strong> is already registered.</p>
|
||||
${message}`,
|
||||
});
|
||||
}
|
||||
|
||||
async function sendPasswordResetEmail(account) {
|
||||
const resetUrl = `${baseUrl}/reset-password?token=${account.resetToken.token}`;
|
||||
const message = `<p>Please click the below link to reset your password, the link will be valid for 1 day:</p>
|
||||
<p><a href="${resetUrl}">${resetUrl}</a></p>`;
|
||||
await sendEmail({
|
||||
to: account.email,
|
||||
subject: 'Sign-up Verification API - Reset Password',
|
||||
html: `<h4>Reset Password Email</h4>
|
||||
${message}`,
|
||||
});
|
||||
}
|
||||
78
api/src/accounts/accounts.controller.js
Normal file
78
api/src/accounts/accounts.controller.js
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
const express = require('express');
|
||||
const router = express.Router();
|
||||
const Joi = require('joi');
|
||||
const wrapRoute = require('../_helpers/wrapRoute');
|
||||
const validateRequest = require('../_middleware/validate-request');
|
||||
const accountService = require('./account.service');
|
||||
|
||||
router.post(
|
||||
'/register',
|
||||
validateRequest(
|
||||
Joi.object({
|
||||
username: Joi.string().required(),
|
||||
email: Joi.string().email().required(),
|
||||
password: Joi.string().min(6).required(),
|
||||
confirmPassword: Joi.string().valid(Joi.ref('password')).required(),
|
||||
}),
|
||||
),
|
||||
wrapRoute(async (req, res) => {
|
||||
await accountService.register(req.body, req.get('origin'));
|
||||
res.json({ message: 'Registration successful, please check your email for verification instructions' });
|
||||
}),
|
||||
);
|
||||
|
||||
router.post(
|
||||
'/verify-email',
|
||||
validateRequest(
|
||||
Joi.object({
|
||||
token: Joi.string().required(),
|
||||
}),
|
||||
),
|
||||
wrapRoute(async (req, res) => {
|
||||
await accountService.verifyEmail(req.body);
|
||||
res.json({ message: 'Verification successful, you can now login' });
|
||||
}),
|
||||
);
|
||||
|
||||
router.post(
|
||||
'/forgot-password',
|
||||
validateRequest(
|
||||
Joi.object({
|
||||
email: Joi.string().email().required(),
|
||||
}),
|
||||
),
|
||||
wrapRoute(async (req, res) => {
|
||||
await accountService.forgotPassword(req.body, req.get('origin'));
|
||||
res.json({ message: 'Please check your email for password reset instructions' });
|
||||
}),
|
||||
);
|
||||
|
||||
router.post(
|
||||
'/validate-reset-token',
|
||||
validateRequest(
|
||||
Joi.object({
|
||||
token: Joi.string().required(),
|
||||
}),
|
||||
),
|
||||
wrapRoute(async (req, res) => {
|
||||
await accountService.validateResetToken(req.body);
|
||||
res.json({ message: 'Token is valid' });
|
||||
}),
|
||||
);
|
||||
|
||||
router.post(
|
||||
'/reset-password',
|
||||
validateRequest(
|
||||
Joi.object({
|
||||
token: Joi.string().required(),
|
||||
password: Joi.string().min(6).required(),
|
||||
confirmPassword: Joi.string().valid(Joi.ref('password')).required(),
|
||||
}),
|
||||
),
|
||||
wrapRoute(async (req, res) => {
|
||||
await accountService.resetPassword(req.body);
|
||||
res.json({ message: 'Password reset successful, you can now login' });
|
||||
}),
|
||||
);
|
||||
|
||||
module.exports = router;
|
||||
72
api/src/config.js
Normal file
72
api/src/config.js
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
const fs = require('fs');
|
||||
const Joi = require('joi');
|
||||
|
||||
const configSchema = Joi.object({
|
||||
jwtSecret: Joi.string().min(16).max(128).required(),
|
||||
cookieSecret: Joi.string().min(16).max(128).required(),
|
||||
|
||||
imprintUrl: Joi.string(),
|
||||
privacyPolicyUrl: Joi.string(),
|
||||
|
||||
baseUrl: Joi.string().required(),
|
||||
mainFrontendUrl: Joi.string(), // optional
|
||||
|
||||
mail: Joi.alternatives().try(
|
||||
Joi.object({
|
||||
from: Joi.string().required(),
|
||||
smtp: Joi.object({
|
||||
host: Joi.string().required(),
|
||||
port: Joi.number().default(465),
|
||||
starttls: Joi.boolean().default(false),
|
||||
username: Joi.string().required(),
|
||||
password: Joi.string().required(),
|
||||
}).required(),
|
||||
}),
|
||||
Joi.boolean().valid(false),
|
||||
),
|
||||
|
||||
mongodb: Joi.object({
|
||||
url: Joi.string().required(),
|
||||
debug: Joi.boolean().default(process.env.NODE_ENV !== 'production'),
|
||||
}).required(),
|
||||
|
||||
redisUrl: Joi.string().required(),
|
||||
|
||||
oAuth2Clients: Joi.array()
|
||||
.default([])
|
||||
.items(
|
||||
Joi.object({
|
||||
title: Joi.string().required(),
|
||||
clientId: Joi.string().required(),
|
||||
validRedirectUris: Joi.array().required().items(Joi.string()),
|
||||
|
||||
// Set `refreshTokenExpirySeconds` to null to issue no refresh tokens. Set
|
||||
// to a number of seconds to issue refresh tokens with that duration. No
|
||||
// infinite tokens are ever issued, set to big number to simulate that.
|
||||
refreshTokenExpirySeconds: Joi.number()
|
||||
.default(null)
|
||||
.min(1) // 0 would make no sense, use `null` to issue no token
|
||||
.max(1000 * 24 * 60 * 60), // 1000 days, nearly 3 years
|
||||
|
||||
// Set to a scope which cannot be exceeded when requesting client tokens.
|
||||
// Clients must manually request a scope that is smaller or equal to this
|
||||
// scope to get a valid response. Scopes are not automatically truncated.
|
||||
// Leave empty or set to `"*"` for unlimited scopes in this client.
|
||||
maxScope: Joi.string().required(),
|
||||
|
||||
autoAccept: Joi.boolean().optional(),
|
||||
}),
|
||||
),
|
||||
}).required();
|
||||
|
||||
const configFiles = [
|
||||
process.env.CONFIG_FILE,
|
||||
process.env.NODE_ENV === 'production' ? 'config.prod.json' : 'config.dev.json',
|
||||
'config.json',
|
||||
].filter((x) => x && fs.existsSync(x));
|
||||
|
||||
if (!configFiles.length) {
|
||||
throw new Error('No config file found.');
|
||||
}
|
||||
|
||||
module.exports = Joi.attempt(JSON.parse(fs.readFileSync(configFiles[0], 'utf8')), configSchema);
|
||||
13
api/src/db.js
Normal file
13
api/src/db.js
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
const mongoose = require('mongoose');
|
||||
|
||||
const config = require('./config')
|
||||
|
||||
mongoose.connect(config.mongodb.url);
|
||||
mongoose.set('debug', config.mongodb.debug);
|
||||
|
||||
require('./models/User');
|
||||
require('./models/Track');
|
||||
require('./models/Comment');
|
||||
require('./passport');
|
||||
|
||||
module.exports = mongoose;
|
||||
87
api/src/index.js
Normal file
87
api/src/index.js
Normal file
|
|
@ -0,0 +1,87 @@
|
|||
const path = require('path');
|
||||
const express = require('express');
|
||||
const bodyParser = require('body-parser');
|
||||
const session = require('express-session');
|
||||
const cors = require('cors');
|
||||
const errorhandler = require('errorhandler');
|
||||
const passport = require('passport');
|
||||
|
||||
const config = require('./config');
|
||||
require('./passport');
|
||||
|
||||
const isProduction = process.env.NODE_ENV === 'production';
|
||||
|
||||
// Create global app object
|
||||
const app = express();
|
||||
|
||||
app.use(cors());
|
||||
|
||||
// Express configuration
|
||||
app.set('views', path.join(__dirname, '..', 'views'));
|
||||
app.set('view engine', 'pug');
|
||||
|
||||
// Normal express config defaults
|
||||
app.use(require('morgan')('dev'));
|
||||
app.use(bodyParser.json({ limit: '50mb' }));
|
||||
app.use(bodyParser.urlencoded({ limit: '50mb', extended: false }));
|
||||
|
||||
app.use(require('method-override')());
|
||||
app.use(express.static(path.join(__dirname, 'public')));
|
||||
app.use('/semantic-ui', express.static(path.join(__dirname, '..', 'node_modules', 'semantic-ui-css')));
|
||||
|
||||
app.use(session({ secret: config.cookieSecret, cookie: { maxAge: 10 * 60 * 1000 }, resave: false, saveUninitialized: false }));
|
||||
app.use(passport.initialize());
|
||||
app.use(passport.session());
|
||||
|
||||
if (!isProduction) {
|
||||
app.use(errorhandler());
|
||||
}
|
||||
|
||||
require('./db');
|
||||
require('./models');
|
||||
|
||||
app.use(require('./routes'));
|
||||
|
||||
/// catch 404 and forward to error handler
|
||||
app.use(function (req, res, next) {
|
||||
const err = new Error('Not Found');
|
||||
err.status = 404;
|
||||
next(err);
|
||||
});
|
||||
|
||||
/// error handlers
|
||||
|
||||
// development error handler
|
||||
// will print stacktrace
|
||||
if (!isProduction) {
|
||||
app.use(function (err, req, res, next) {
|
||||
console.log(err.stack);
|
||||
|
||||
res.status(err.status || 500);
|
||||
|
||||
res.json({
|
||||
errors: {
|
||||
message: err.message,
|
||||
error: err,
|
||||
},
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
// production error handler
|
||||
// no stacktraces leaked to user
|
||||
app.use(function (err, req, res, next) {
|
||||
res.status(err.status || 500);
|
||||
res.json({
|
||||
errors: {
|
||||
message: err.message,
|
||||
error: {},
|
||||
},
|
||||
});
|
||||
});
|
||||
|
||||
// finally, let's start our server...
|
||||
const port = process.env.PORT || 3000;
|
||||
app.listen(port, () => {
|
||||
console.log('Listening on port ' + port);
|
||||
});
|
||||
357
api/src/logic/_tracks_testdata.js
Normal file
357
api/src/logic/_tracks_testdata.js
Normal file
|
|
@ -0,0 +1,357 @@
|
|||
const TEST_ROWS = [
|
||||
'Date;Time;Latitude;Longitude;Course;Speed;Right;Left;Confirmed;insidePrivacyArea;',
|
||||
'12.07.2020;09:02:59;0.000000;0.000000;0.000;0.0000;255;255;0;0;',
|
||||
'12.07.2020;09:02:59;0.000000;0.000000;0.000;0.0000;255;255;0;0;',
|
||||
'12.07.2020;09:03:00;0.000000;0.000000;0.000;0.0000;255;255;0;0;',
|
||||
'12.07.2020;09:03:01;48.722205;9.270218;0.000;0.4260;255;255;0;0;',
|
||||
'12.07.2020;09:03:02;48.722206;9.270219;0.000;0.5741;255;255;0;0;',
|
||||
'12.07.2020;09:03:03;48.722204;9.270221;0.000;0.5371;255;255;0;0;',
|
||||
'12.07.2020;09:03:04;48.722198;9.270229;0.000;0.7593;255;255;0;0;',
|
||||
'12.07.2020;09:03:05;48.722188;9.270241;0.000;0.5556;255;255;0;0;',
|
||||
'12.07.2020;09:03:06;48.722174;9.270259;0.000;0.4815;255;255;0;0;',
|
||||
'12.07.2020;09:03:07;48.722158;9.270278;0.000;0.3704;255;255;0;0;',
|
||||
'12.07.2020;09:03:08;48.722146;9.270293;0.000;0.5741;255;255;0;0;',
|
||||
'12.07.2020;09:03:09;48.722138;9.270305;0.000;1.2594;255;255;0;0;',
|
||||
'12.07.2020;09:03:10;48.722129;9.270318;0.000;1.5557;255;255;0;0;',
|
||||
'12.07.2020;09:03:11;48.722122;9.270329;0.000;1.5372;255;255;0;0;',
|
||||
'12.07.2020;09:03:12;48.722115;9.270339;0.000;0.4630;255;255;0;0;',
|
||||
'12.07.2020;09:03:13;48.722107;9.270350;0.000;0.2963;255;255;0;0;',
|
||||
'12.07.2020;09:03:14;48.722101;9.270357;0.000;0.2963;255;255;0;0;',
|
||||
'12.07.2020;09:03:15;48.722092;9.270367;0.000;0.8149;255;255;0;0;',
|
||||
'12.07.2020;09:03:16;48.722084;9.270377;0.000;1.2223;255;255;0;0;',
|
||||
'12.07.2020;09:03:17;48.722076;9.270385;0.000;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:03:18;48.722070;9.270391;0.000;1.4816;255;255;0;0;',
|
||||
'12.07.2020;09:03:19;48.722070;9.270392;0.000;1.0927;255;255;0;0;',
|
||||
'12.07.2020;09:03:20;48.722066;9.270395;0.000;1.6668;255;255;0;0;',
|
||||
'12.07.2020;09:03:21;48.722068;9.270391;0.000;2.0742;255;255;0;0;',
|
||||
'12.07.2020;09:03:22;48.722064;9.270396;0.000;1.6853;255;255;0;0;',
|
||||
'12.07.2020;09:03:23;48.722060;9.270401;0.000;1.0927;255;255;0;0;',
|
||||
'12.07.2020;09:03:24;48.722056;9.270406;0.000;0.9445;255;255;0;0;',
|
||||
'12.07.2020;09:03:25;48.722052;9.270411;0.000;0.7964;255;255;0;0;',
|
||||
'12.07.2020;09:03:26;48.722047;9.270416;0.000;0.6482;255;255;0;0;',
|
||||
'12.07.2020;09:03:27;48.722042;9.270419;0.000;1.0556;255;255;0;0;',
|
||||
'12.07.2020;09:03:28;48.722031;9.270433;0.000;2.0372;255;255;0;0;',
|
||||
'12.07.2020;09:03:29;48.722031;9.270432;0.000;2.4261;255;255;0;0;',
|
||||
'12.07.2020;09:03:30;48.722029;9.270433;0.000;0.8704;255;255;0;0;',
|
||||
'12.07.2020;09:03:31;48.722029;9.270433;0.000;1.8150;255;255;0;0;',
|
||||
'12.07.2020;09:03:32;48.722024;9.270439;0.000;1.2223;255;255;0;0;',
|
||||
'12.07.2020;09:03:33;48.722025;9.270439;0.000;0.3889;255;255;0;0;',
|
||||
'12.07.2020;09:03:34;48.722022;9.270440;0.000;0.3519;255;255;0;0;',
|
||||
'12.07.2020;09:03:35;48.722020;9.270445;0.000;0.9445;255;255;0;0;',
|
||||
'12.07.2020;09:03:36;48.722018;9.270447;0.000;0.9260;255;255;0;0;',
|
||||
'12.07.2020;09:03:37;48.722020;9.270444;0.000;0.9075;255;255;0;0;',
|
||||
'12.07.2020;09:03:38;48.722021;9.270443;0.000;1.9261;255;255;0;0;',
|
||||
'12.07.2020;09:03:39;48.722018;9.270447;0.000;0.3334;255;255;0;0;',
|
||||
'12.07.2020;09:03:40;48.722020;9.270445;0.000;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:03:41;48.722023;9.270440;0.000;1.2594;255;255;0;0;',
|
||||
'12.07.2020;09:03:42;48.722023;9.270442;0.000;0.5000;255;255;0;0;',
|
||||
'12.07.2020;09:03:43;48.722025;9.270440;0.000;0.6852;220;255;0;0;',
|
||||
'12.07.2020;09:03:44;48.722023;9.270441;0.000;0.8519;199;255;0;0;',
|
||||
'12.07.2020;09:03:45;48.722026;9.270438;0.000;1.4075;255;255;0;0;',
|
||||
'12.07.2020;09:03:46;48.722029;9.270436;0.000;0.5371;255;255;0;0;',
|
||||
'12.07.2020;09:03:47;48.722028;9.270435;0.000;0.8334;97;255;0;0;',
|
||||
'12.07.2020;09:03:48;48.722029;9.270435;0.000;0.3704;255;255;0;0;',
|
||||
'12.07.2020;09:03:49;48.722029;9.270436;0.000;1.1112;96;255;0;0;',
|
||||
'12.07.2020;09:03:50;48.722029;9.270435;0.000;1.8890;255;255;0;0;',
|
||||
'12.07.2020;09:03:51;48.722034;9.270429;0.000;1.0186;255;255;0;0;',
|
||||
'12.07.2020;09:03:52;48.721942;9.270529;128.450;5.2226;255;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;79;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;178;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;89;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;156;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;168;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;181;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;176;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;186;0;0;',
|
||||
'12.07.2020;09:03:53;48.721929;9.270546;128.450;1.3520;255;255;0;0;',
|
||||
'12.07.2020;09:04:10;48.721896;9.270602;916.230;0.0556;255;255;0;0;',
|
||||
'12.07.2020;09:04:11;48.721894;9.270609;916.230;0.0926;255;192;0;0;',
|
||||
'12.07.2020;09:04:12;48.721892;9.270616;916.230;0.0556;255;255;0;0;',
|
||||
'12.07.2020;09:04:13;48.721890;9.270623;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:04:14;48.721888;9.270629;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:04:15;48.721886;9.270635;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:04:16;48.721883;9.270640;916.230;0.0556;255;255;0;0;',
|
||||
'12.07.2020;09:04:17;48.721881;9.270644;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:04:18;48.721879;9.270649;916.230;0.1111;255;255;0;0;',
|
||||
'12.07.2020;09:04:19;48.721877;9.270653;916.230;0.1296;255;255;0;0;',
|
||||
'12.07.2020;09:04:20;48.721876;9.270657;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:04:21;48.721874;9.270658;916.230;0.3148;255;255;0;0;',
|
||||
'12.07.2020;09:04:22;48.721873;9.270659;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:04:23;48.721872;9.270661;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:04:24;48.721871;9.270661;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:04:25;48.721870;9.270660;916.230;0.3334;255;255;0;0;',
|
||||
'12.07.2020;09:04:26;48.721869;9.270658;916.230;0.5000;255;255;0;0;',
|
||||
'12.07.2020;09:04:27;48.721866;9.270660;916.230;1.6853;255;255;0;0;',
|
||||
'12.07.2020;09:04:28;48.721866;9.270659;916.230;0.8704;255;198;0;0;',
|
||||
'12.07.2020;09:04:29;48.721867;9.270659;916.230;0.5741;255;196;0;0;',
|
||||
'12.07.2020;09:04:30;48.721867;9.270660;916.230;0.3148;255;196;0;0;',
|
||||
'12.07.2020;09:04:31;48.721867;9.270659;916.230;0.1111;255;255;0;0;',
|
||||
'12.07.2020;09:04:32;48.721866;9.270659;916.230;0.0556;255;199;0;0;',
|
||||
'12.07.2020;09:04:33;48.721867;9.270656;916.230;0.1482;255;199;0;0;',
|
||||
'12.07.2020;09:04:34;48.721867;9.270654;916.230;0.0370;255;198;0;0;',
|
||||
'12.07.2020;09:04:35;48.721867;9.270653;916.230;0.1296;255;198;0;0;',
|
||||
'12.07.2020;09:04:36;48.721867;9.270651;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:04:37;48.721867;9.270650;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:04:38;48.721868;9.270650;916.230;0.1852;255;255;0;0;',
|
||||
'12.07.2020;09:04:39;48.721868;9.270649;916.230;0.1667;255;201;0;0;',
|
||||
'12.07.2020;09:04:40;48.721868;9.270647;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:04:41;48.721869;9.270644;916.230;0.0185;255;255;0;0;',
|
||||
'12.07.2020;09:04:42;48.721869;9.270641;916.230;0.0185;255;198;0;0;',
|
||||
'12.07.2020;09:04:43;48.721870;9.270638;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:04:44;48.721870;9.270635;916.230;0.0370;255;199;0;0;',
|
||||
'12.07.2020;09:04:45;48.721871;9.270632;916.230;0.1482;255;204;0;0;',
|
||||
'12.07.2020;09:04:46;48.721871;9.270630;916.230;0.0185;255;201;0;0;',
|
||||
'12.07.2020;09:04:47;48.721873;9.270630;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:04:48;48.721873;9.270629;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:04:49;48.721874;9.270628;916.230;0.4074;255;255;0;0;',
|
||||
'12.07.2020;09:04:50;48.721875;9.270627;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:04:51;48.721876;9.270625;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:04:52;48.721877;9.270623;916.230;0.1111;255;255;0;0;',
|
||||
'12.07.2020;09:04:53;48.721877;9.270622;916.230;0.2037;255;255;0;0;',
|
||||
'12.07.2020;09:04:54;48.721879;9.270621;916.230;0.3148;255;255;0;0;',
|
||||
'12.07.2020;09:04:55;48.721881;9.270618;916.230;0.2408;255;255;0;0;',
|
||||
'12.07.2020;09:04:56;48.721883;9.270615;916.230;0.3148;255;255;0;0;',
|
||||
'12.07.2020;09:04:57;48.721884;9.270612;916.230;0.2778;255;255;0;0;',
|
||||
'12.07.2020;09:04:58;48.721885;9.270609;916.230;0.1296;255;255;0;0;',
|
||||
'12.07.2020;09:04:59;48.721886;9.270606;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:05:00;48.721888;9.270602;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:05:01;48.721889;9.270598;916.230;0.1111;255;191;0;0;',
|
||||
'12.07.2020;09:05:02;48.721890;9.270595;916.230;0.1482;255;193;0;0;',
|
||||
'12.07.2020;09:05:03;48.721891;9.270593;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:05:04;48.721891;9.270591;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:05:05;48.721891;9.270589;916.230;0.1296;255;255;0;0;',
|
||||
'12.07.2020;09:05:06;48.721891;9.270587;916.230;0.3519;255;199;0;0;',
|
||||
'12.07.2020;09:05:07;48.721891;9.270586;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:08;48.721891;9.270588;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:05:09;48.721890;9.270589;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:05:10;48.721889;9.270589;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:11;48.721888;9.270589;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:05:12;48.721887;9.270589;916.230;0.1296;255;255;0;0;',
|
||||
'12.07.2020;09:05:13;48.721886;9.270590;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:05:14;48.721885;9.270591;916.230;0.3148;255;255;0;0;',
|
||||
'12.07.2020;09:05:15;48.721885;9.270592;916.230;0.2037;255;255;0;0;',
|
||||
'12.07.2020;09:05:16;48.721885;9.270596;916.230;0.5556;255;255;0;0;',
|
||||
'12.07.2020;09:05:17;48.721885;9.270598;916.230;0.3519;255;255;0;0;',
|
||||
'12.07.2020;09:05:18;48.721884;9.270600;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:05:19;48.721882;9.270600;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:05:20;48.721881;9.270602;916.230;0.0556;255;255;0;0;',
|
||||
'12.07.2020;09:05:21;48.721879;9.270603;916.230;0.0185;255;206;0;0;',
|
||||
'12.07.2020;09:05:22;48.721878;9.270605;916.230;0.0556;255;203;0;0;',
|
||||
'12.07.2020;09:05:23;48.721876;9.270606;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:05:24;48.721874;9.270605;916.230;0.0185;255;255;0;0;',
|
||||
'12.07.2020;09:05:25;48.721873;9.270605;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:05:26;48.721872;9.270605;916.230;0.1296;255;209;0;0;',
|
||||
'12.07.2020;09:05:27;48.721870;9.270606;916.230;0.0556;255;255;0;0;',
|
||||
'12.07.2020;09:05:28;48.721869;9.270608;916.230;0.1111;255;206;0;0;',
|
||||
'12.07.2020;09:05:29;48.721868;9.270610;916.230;0.3148;255;209;0;0;',
|
||||
'12.07.2020;09:05:30;48.721867;9.270610;916.230;0.2593;255;208;0;0;',
|
||||
'12.07.2020;09:05:31;48.721866;9.270611;916.230;0.0556;255;210;0;0;',
|
||||
'12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:32;48.721866;9.270612;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:44;48.721855;9.270602;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:44;48.721855;9.270602;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:05:46;48.721854;9.270602;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:05:46;48.721854;9.270602;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:05:48;48.721852;9.270606;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:05:49;48.721851;9.270611;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:05:50;48.721851;9.270615;916.230;0.1852;255;255;0;0;',
|
||||
'12.07.2020;09:05:51;48.721851;9.270616;916.230;0.0185;255;255;0;0;',
|
||||
'12.07.2020;09:05:52;48.721851;9.270617;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:05:53;48.721852;9.270617;916.230;0.0185;255;255;0;0;',
|
||||
'12.07.2020;09:05:54;48.721853;9.270616;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:05:55;48.721855;9.270613;916.230;0.0556;255;255;0;0;',
|
||||
'12.07.2020;09:05:56;48.721858;9.270609;916.230;0.6482;255;255;0;0;',
|
||||
'12.07.2020;09:05:57;48.721860;9.270606;916.230;0.4260;255;255;0;0;',
|
||||
'12.07.2020;09:05:58;48.721864;9.270601;916.230;0.6297;255;255;0;0;',
|
||||
'12.07.2020;09:05:59;48.721867;9.270595;916.230;0.4260;255;255;0;0;',
|
||||
'12.07.2020;09:06:00;48.721872;9.270589;916.230;0.5000;255;255;0;0;',
|
||||
'12.07.2020;09:06:01;48.721875;9.270584;916.230;0.2593;255;255;0;0;',
|
||||
'12.07.2020;09:06:02;48.721880;9.270578;916.230;0.5186;255;255;0;0;',
|
||||
'12.07.2020;09:06:03;48.721883;9.270574;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:06:04;48.721886;9.270570;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:06:05;48.721890;9.270565;916.230;0.2408;255;255;0;0;',
|
||||
'12.07.2020;09:06:06;48.721893;9.270562;916.230;0.2593;255;255;0;0;',
|
||||
'12.07.2020;09:06:07;48.721893;9.270560;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:06:08;48.721894;9.270559;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:06:09;48.721894;9.270557;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:06:07;48.721896;9.270556;916.230;0.2778;255;255;0;0;',
|
||||
'12.07.2020;09:06:08;48.721896;9.270556;916.230;0.2408;255;255;0;0;',
|
||||
'12.07.2020;09:06:09;48.721895;9.270557;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:06:10;48.721894;9.270559;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:06:11;48.721892;9.270560;916.230;0.2037;255;255;0;0;',
|
||||
'12.07.2020;09:06:12;48.721891;9.270561;916.230;0.1296;255;255;0;0;',
|
||||
'12.07.2020;09:06:13;48.721892;9.270562;916.230;0.1852;255;255;0;0;',
|
||||
'12.07.2020;09:06:14;48.721891;9.270564;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:06:15;48.721889;9.270566;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:06:16;48.721888;9.270568;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:06:17;48.721888;9.270570;916.230;0.2037;255;255;0;0;',
|
||||
'12.07.2020;09:06:18;48.721888;9.270572;916.230;0.4630;255;255;0;0;',
|
||||
'12.07.2020;09:06:19;48.721887;9.270573;916.230;0.4815;255;255;0;0;',
|
||||
'12.07.2020;09:06:20;48.721886;9.270574;916.230;0.3334;255;255;0;0;',
|
||||
'12.07.2020;09:06:21;48.721885;9.270576;916.230;0.1852;255;255;0;0;',
|
||||
'12.07.2020;09:06:22;48.721884;9.270579;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:06:23;48.721882;9.270581;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:06:24;48.721881;9.270584;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:06:25;48.721880;9.270589;916.230;0.2963;255;255;0;0;',
|
||||
'12.07.2020;09:06:26;48.721879;9.270596;916.230;0.3519;255;255;0;0;',
|
||||
'12.07.2020;09:06:27;48.721878;9.270602;916.230;0.1111;255;255;0;0;',
|
||||
'12.07.2020;09:06:28;48.721876;9.270601;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:06:29;48.721874;9.270603;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:06:30;48.721873;9.270607;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:06:31;48.721872;9.270614;916.230;0.2037;255;255;0;0;',
|
||||
'12.07.2020;09:06:32;48.721870;9.270613;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:06:33;48.721869;9.270614;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:06:34;48.721868;9.270616;916.230;0.1111;255;255;0;0;',
|
||||
'12.07.2020;09:06:35;48.721867;9.270617;916.230;0.2593;255;255;0;0;',
|
||||
'12.07.2020;09:06:36;48.721867;9.270618;916.230;0.1852;255;255;0;0;',
|
||||
'12.07.2020;09:06:37;48.721867;9.270618;916.230;0.1296;255;255;0;0;',
|
||||
'12.07.2020;09:06:38;48.721867;9.270616;916.230;0.2963;255;255;0;0;',
|
||||
'12.07.2020;09:06:39;48.721867;9.270613;916.230;0.1296;255;255;0;0;',
|
||||
'12.07.2020;09:06:40;48.721867;9.270607;916.230;0.0185;255;255;0;0;',
|
||||
'12.07.2020;09:06:41;48.721866;9.270601;916.230;0.5186;255;255;0;0;',
|
||||
'12.07.2020;09:06:42;48.721866;9.270593;916.230;0.2963;255;255;0;0;',
|
||||
'12.07.2020;09:06:43;48.721866;9.270587;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:06:44;48.721866;9.270581;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:06:45;48.721866;9.270576;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:06:46;48.721866;9.270567;916.230;0.1111;255;255;0;0;',
|
||||
'12.07.2020;09:06:47;48.721866;9.270558;916.230;0.4074;255;255;0;0;',
|
||||
'12.07.2020;09:06:48;48.721866;9.270550;916.230;0.4260;255;255;0;0;',
|
||||
'12.07.2020;09:06:49;48.721866;9.270543;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:06:50;48.721867;9.270537;916.230;0.2778;255;255;0;0;',
|
||||
'12.07.2020;09:06:51;48.721867;9.270532;916.230;0.2037;255;255;0;0;',
|
||||
'12.07.2020;09:06:52;48.721867;9.270526;916.230;0.3148;255;255;0;0;',
|
||||
'12.07.2020;09:06:53;48.721868;9.270522;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:06:54;48.721868;9.270517;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:06:55;48.721869;9.270512;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:06:56;48.721869;9.270506;916.230;0.2408;255;255;0;0;',
|
||||
'12.07.2020;09:06:57;48.721869;9.270503;916.230;0.1111;255;255;0;0;',
|
||||
'12.07.2020;09:06:58;48.721870;9.270500;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:06:59;48.721870;9.270497;916.230;0.1296;255;255;0;0;',
|
||||
'12.07.2020;09:07:00;48.721871;9.270494;916.230;0.1296;255;255;0;0;',
|
||||
'12.07.2020;09:07:01;48.721871;9.270493;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:07:02;48.721871;9.270492;916.230;0.2037;255;255;0;0;',
|
||||
'12.07.2020;09:07:03;48.721872;9.270490;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:07:04;48.721873;9.270489;916.230;0.6667;255;255;0;0;',
|
||||
'12.07.2020;09:07:05;48.721873;9.270487;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:07:06;48.721873;9.270486;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:07:07;48.721873;9.270486;916.230;0.2408;255;255;0;0;',
|
||||
'12.07.2020;09:07:08;48.721873;9.270485;916.230;0.1852;255;255;0;0;',
|
||||
'12.07.2020;09:07:09;48.721873;9.270485;916.230;0.2037;255;255;0;0;',
|
||||
'12.07.2020;09:07:10;48.721872;9.270485;916.230;0.1296;255;255;0;0;',
|
||||
'12.07.2020;09:07:11;48.721870;9.270486;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:07:12;48.721869;9.270489;916.230;0.2037;255;255;0;0;',
|
||||
'12.07.2020;09:07:13;48.721867;9.270492;916.230;0.3148;255;255;0;0;',
|
||||
'12.07.2020;09:07:14;48.721865;9.270494;916.230;0.2037;255;255;0;0;',
|
||||
'12.07.2020;09:07:15;48.721863;9.270495;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:07:16;48.721861;9.270497;916.230;0.0000;255;255;0;0;',
|
||||
'12.07.2020;09:07:17;48.721860;9.270496;916.230;0.4074;255;255;0;0;',
|
||||
'12.07.2020;09:07:18;48.721859;9.270495;916.230;0.4445;255;255;0;0;',
|
||||
'12.07.2020;09:07:19;48.721857;9.270496;916.230;0.3889;255;255;0;0;',
|
||||
'12.07.2020;09:07:20;48.721856;9.270496;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:07:21;48.721854;9.270494;916.230;0.7593;255;255;0;0;',
|
||||
'12.07.2020;09:07:22;48.721851;9.270496;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:07:23;48.721850;9.270497;916.230;0.1667;255;255;0;0;',
|
||||
'12.07.2020;09:07:24;48.721848;9.270501;916.230;0.4074;255;255;0;0;',
|
||||
'12.07.2020;09:07:25;48.721847;9.270504;916.230;0.4074;255;255;0;0;',
|
||||
'12.07.2020;09:07:26;48.721846;9.270505;916.230;0.2037;255;255;0;0;',
|
||||
'12.07.2020;09:07:27;48.721844;9.270508;916.230;0.1111;255;255;0;0;',
|
||||
'12.07.2020;09:07:28;48.721843;9.270507;916.230;0.4630;255;255;0;0;',
|
||||
'12.07.2020;09:07:29;48.721842;9.270509;916.230;0.1111;255;255;0;0;',
|
||||
'12.07.2020;09:07:30;48.721841;9.270512;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:07:31;48.721840;9.270515;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:07:32;48.721839;9.270517;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:07:33;48.721838;9.270522;916.230;0.0556;255;255;0;0;',
|
||||
'12.07.2020;09:07:34;48.721838;9.270527;916.230;0.3889;255;255;0;0;',
|
||||
'12.07.2020;09:07:35;48.721837;9.270530;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:07:36;48.721836;9.270532;916.230;0.1111;255;255;0;0;',
|
||||
'12.07.2020;09:07:37;48.721835;9.270536;916.230;0.6112;255;255;0;0;',
|
||||
'12.07.2020;09:07:38;48.721835;9.270541;916.230;1.1668;255;255;0;0;',
|
||||
'12.07.2020;09:07:39;48.721835;9.270543;916.230;0.3889;255;255;0;0;',
|
||||
'12.07.2020;09:07:40;48.721834;9.270545;916.230;0.5000;255;255;0;0;',
|
||||
'12.07.2020;09:07:41;48.721834;9.270544;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:07:42;48.721834;9.270545;916.230;0.7593;255;255;0;0;',
|
||||
'12.07.2020;09:07:43;48.721834;9.270545;916.230;0.8890;255;255;0;0;',
|
||||
'12.07.2020;09:07:44;48.721834;9.270543;916.230;0.4260;255;255;0;0;',
|
||||
'12.07.2020;09:07:45;48.721834;9.270541;916.230;0.2408;255;255;0;0;',
|
||||
'12.07.2020;09:07:46;48.721834;9.270540;916.230;0.3148;255;255;0;0;',
|
||||
'12.07.2020;09:07:47;48.721835;9.270538;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:07:48;48.721835;9.270535;916.230;0.0556;255;255;0;0;',
|
||||
'12.07.2020;09:07:49;48.721835;9.270534;916.230;0.8890;255;255;0;0;',
|
||||
'12.07.2020;09:07:50;48.721835;9.270534;916.230;0.5926;255;255;0;0;',
|
||||
'12.07.2020;09:07:51;48.721835;9.270534;916.230;0.7593;255;255;0;0;',
|
||||
'12.07.2020;09:07:52;48.721836;9.270533;916.230;0.2408;255;255;0;0;',
|
||||
'12.07.2020;09:07:53;48.721836;9.270531;916.230;0.0741;255;255;0;0;',
|
||||
'12.07.2020;09:07:54;48.721836;9.270529;916.230;0.3889;255;255;0;0;',
|
||||
'12.07.2020;09:07:55;48.721836;9.270530;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:07:56;48.721836;9.270530;916.230;0.0185;255;255;0;0;',
|
||||
'12.07.2020;09:07:57;48.721837;9.270531;916.230;0.0185;255;255;0;0;',
|
||||
'12.07.2020;09:07:58;48.721837;9.270530;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:07:59;48.721838;9.270526;916.230;0.3519;255;255;0;0;',
|
||||
'12.07.2020;09:08:00;48.721838;9.270521;916.230;0.4260;255;255;0;0;',
|
||||
'12.07.2020;09:08:01;48.721839;9.270522;916.230;0.5556;255;255;0;0;',
|
||||
'12.07.2020;09:08:02;48.721840;9.270524;916.230;0.3519;255;255;0;0;',
|
||||
'12.07.2020;09:08:03;48.721842;9.270525;916.230;0.2963;255;255;0;0;',
|
||||
'12.07.2020;09:08:04;48.721843;9.270525;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:08:05;48.721844;9.270524;916.230;0.2222;255;255;0;0;',
|
||||
'12.07.2020;09:08:06;48.721846;9.270522;916.230;0.3704;255;255;0;0;',
|
||||
'12.07.2020;09:08:07;48.721847;9.270519;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:08:08;48.721848;9.270516;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:08:09;48.721849;9.270514;916.230;0.1296;255;255;0;0;',
|
||||
'12.07.2020;09:08:10;48.721850;9.270512;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:08:11;48.721851;9.270513;916.230;0.3334;255;255;0;0;',
|
||||
'12.07.2020;09:08:12;48.721851;9.270512;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:08:13;48.721851;9.270512;916.230;0.2593;255;255;0;0;',
|
||||
'12.07.2020;09:08:14;48.721852;9.270511;916.230;0.0926;255;255;0;0;',
|
||||
'12.07.2020;09:08:15;48.721853;9.270512;916.230;0.0370;255;255;0;0;',
|
||||
'12.07.2020;09:08:16;48.721852;9.270515;916.230;0.5371;255;255;0;0;',
|
||||
'12.07.2020;09:08:17;48.721853;9.270517;916.230;0.1482;255;255;0;0;',
|
||||
'12.07.2020;09:08:18;48.721854;9.270519;916.230;0.3148;255;255;0;0;',
|
||||
'12.07.2020;09:08:19;48.721855;9.270520;916.230;0.2408;255;255;0;0;',
|
||||
'12.07.2020;09:08:20;48.721856;9.270523;916.230;0.3704;255;255;0;0;',
|
||||
];
|
||||
|
||||
const test1 = TEST_ROWS.join('$');
|
||||
|
||||
const test2 = `OBSFirmwareVersion=v0.3.999&OBSDataFormat=2&DataPerMeasurement=3&MaximumMeasurementsPerLine=60&OffsetLeft=30&OffsetRight=30&NumberOfDefinedPrivacyAreas=3&PrivacyLevelApplied=AbsolutePrivacy&MaximumValidFlightTimeMicroseconds=18560&DistanceSensorsUsed=HC-SR04/JSN-SR04T&DeviceId=ECEC&OBSUserID=32423432342234
|
||||
Date;Time;Millis;Comment;Latitude;Longitude;Altitude;Course;Speed;HDOP;Satellites;BatteryLevel;Left;Right;Confirmed;Marked;Invalid;InsidePrivacyArea;Factor;Measurements;Tms1;Lus1;Rus1;Tms2;Lus2;Rus2;Tms3;Lus3;Rus3;Tms4;Lus4;Rus4;Tms5;Lus5;Rus5;Tms6;Lus6;Rus6;Tms7;Lus7;Rus7;Tms8;Lus8;Rus8;Tms9;Lus9;Rus9;Tms10;Lus10;Rus10;Tms11;Lus11;Rus11;Tms12;Lus12;Rus12;Tms13;Lus13;Rus13;Tms14;Lus14;Rus14;Tms15;Lus15;Rus15;Tms16;Lus16;Rus16;Tms17;Lus17;Rus17;Tms18;Lus18;Rus18;Tms19;Lus19;Rus19;Tms20;Lus20;Rus20;Tms21;Lus21;Rus21;Tms22;Lus22;Rus22;Tms23;Lus23;Rus23;Tms24;Lus24;Rus24;Tms25;Lus25;Rus25;Tms26;Lus26;Rus26;Tms27;Lus27;Rus27;Tms28;Lus28;Rus28;Tms29;Lus29;Rus29;Tms30;Lus30;Rus30;Tms31;Lus31;Rus31;Tms32;Lus32;Rus32;Tms33;Lus33;Rus33;Tms34;Lus34;Rus34;Tms35;Lus35;Rus35;Tms36;Lus36;Rus36;Tms37;Lus37;Rus37;Tms38;Lus38;Rus38;Tms39;Lus39;Rus39;Tms40;Lus40;Rus40;Tms41;Lus41;Rus41;Tms42;Lus42;Rus42;Tms43;Lus43;Rus43;Tms44;Lus44;Rus44;Tms45;Lus45;Rus45;Tms46;Lus46;Rus46;Tms47;Lus47;Rus47;Tms48;Lus48;Rus48;Tms49;Lus49;Rus49;Tms50;Lus50;Rus50;Tms51;Lus51;Rus51;Tms52;Lus52;Rus52;Tms53;Lus53;Rus53;Tms54;Lus54;Rus54;Tms55;Lus55;Rus55;Tms56;Lus56;Rus56;Tms57;Lus57;Rus57;Tms58;Lus58;Rus58;Tms59;Lus59;Rus59;Tms60;Lus60;Rus60
|
||||
18.11.2020;16:05:59;1265034;;48.723224;9.094103;495.3;189.86;3.2;1.01;7;3.74;770;;0;0;58;54;0;6231;;16;;;36;6350;;52;;;72;6263;;87;;;107;6828;;122;;;143;6836;;158;;;178;6936;;193;;;213;7094;;228;;;248;6822;;263;;;284;7019;;299;;;319;6942;;334;;;354;7110;;370;;;390;7203;;405;;;425;7758;;440;;;461;7266;;476;;;496;7499;;511;;;531;7328;;546;;;567;7354;;582;;;602;7397;;617;;;637;;;664;;;684;16615;;708;;;728;9161;;745;;;765;10238;;783;;;802;8525;;818;;;839;7756;;854;;;875;7580;;890;;;910;7926;;925;;;945;7624;;960;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:00;1266041;DEVELOP: GPSMessages: 2587 GPS crc errors: 0;48.723205;9.0941;495.4;189.86;2.87;1.01;7;3.74;1020;;0;0;58;53;0;8012;;27;;;47;7999;;62;;;83;7660;;98;;;118;7698;;133;;;158;1252;;169;;;194;1146;;204;;;229;1173;;239;;;264;1173;;274;;;300;1147;;310;;;335;7943;;352;;;371;8713;;387;;;407;8005;;423;;;443;8021;;458;;;478;;;505;;;525;8111;;541;;;560;8074;;576;;;596;8254;;612;;;632;8514;;647;;;667;8195;;682;;;703;8094;;718;;;738;8123;;754;;;774;8330;;789;;;810;8966;;826;;;846;9066;;862;;;882;10553;;899;;;920;8345;;935;;;955;9219;;;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:01;1267037;DEVELOP: Mem: 45k Buffer: 4k last write time: 58;48.723197;9.094089;495.7;189.86;2.93;1.01;7;3.74;1090;;0;0;58;53;0;8164;;18;;;39;8184;;53;;;74;16305;;98;;;118;8658;;135;;;155;8198;;170;;;190;8133;;205;;;226;8536;;241;;;261;8676;;276;;;296;8516;;314;;;334;8114;;350;;;370;8294;;385;;;405;8751;;422;;;441;8163;;457;;;478;8062;;493;;;513;8093;;528;;;549;8060;;564;;;584;8085;;599;;;619;8071;;634;;;655;8262;;671;;;690;8746;;707;;;726;9116;;742;;;762;;;789;;;808;8121;;825;;;845;8113;;860;;;881;8129;;896;;;916;8096;;932;;;952;10617;;;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:02;1268027;;48.723185;9.094076;496.1;189.86;3.02;1.01;7;3.74;980;;0;0;58;55;0;8173;;18;;;37;8535;;53;;;73;8435;;88;;;109;8592;;124;;;144;8012;;159;;;180;8037;;195;;;215;7975;;230;;;250;7970;;265;;;286;7850;;301;;;321;7861;;336;;;356;7826;;371;;;392;8097;;407;;;427;8467;;443;;;463;7763;;478;;;498;7687;;513;;;534;7950;;549;;;569;7806;;584;;;604;8253;;620;;;640;7753;;656;;;676;8188;;692;;;711;7533;;727;;;747;7791;;763;;;783;7460;;798;;;825;9827;;843;;;863;7432;;878;;;904;7646;;919;;;939;7538;;955;;;974;7508;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:03;1269096;;48.723177;9.094068;496.2;189.86;3;1.01;7;3.74;920;;0;0;58;51;0;7218;;19;;;38;8144;;54;;;74;7463;;89;;;110;7856;;125;;;145;7869;;161;;;181;7422;;196;;;216;7934;;232;;;252;7363;;267;;;293;7297;;307;;;332;8105;;348;;;367;7468;;383;;;403;7213;;418;;;439;7172;;454;;;478;7184;;489;;;514;7312;;528;;;550;7175;;565;;;585;7180;;600;;;620;7013;;635;;;655;7154;;670;;;691;7240;;706;;;726;7075;;741;;;761;7133;;776;;;801;7511;;815;;;836;7639;;851;;;872;8891;;888;;;908;7070;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:04;1270033;;48.723167;9.094056;496.6;189.86;3.19;1.01;7;3.74;870;;0;0;58;53;0;7617;;19;;;39;6812;;55;;;80;1173;;90;;;116;8173;;133;;;152;7431;;168;;;188;7197;;203;;;223;6984;;238;;;259;7218;;274;;;294;6881;;309;;;329;7111;;344;;;365;7500;;380;;;400;7462;;415;;;435;7094;;450;;;471;6820;;486;;;506;7147;;521;;;541;9156;;558;;;578;6961;;594;;;614;;;641;;;660;7176;;676;;;696;7177;;712;;;732;7199;;747;;;767;7218;;782;;;802;7360;;817;;;838;;;865;;;884;;;904;;;924;;;943;;;962;7252;;;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:05;1271032;;48.723153;9.094046;496.5;189.86;3.48;1.01;7;3.74;940;;0;0;58;54;0;7295;;14;;;35;7183;;50;;;71;7283;;85;;;106;8957;;122;;;142;8178;;158;;;178;7814;;194;;;213;7495;;229;;;249;7713;;265;;;285;7305;;300;;;320;7654;;335;;;356;7687;;371;;;391;7634;;406;;;426;7167;;441;;;461;;;488;;;508;7245;;524;;;544;7283;;559;;;580;7150;;595;;;615;7194;;630;;;650;7410;;665;;;686;7670;;702;;;721;7421;;737;;;757;7588;;772;;;792;7452;;809;;;828;8162;;844;;;865;9078;;881;;;901;7563;;917;;;936;7775;;952;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:06;1272031;;48.723146;9.094036;496.5;189.86;2.44;1.01;7;3.74;1000;;0;0;58;54;0;8193;;18;;;39;7629;;54;;;74;;;102;;;121;7778;;137;;;157;7773;;172;;;193;7922;;208;;;228;7706;;243;;;263;8881;;280;;;299;7776;;315;;;334;7797;;350;;;370;8683;;386;;;406;7863;;422;;;441;7901;;457;;;477;7747;;492;;;513;8246;;529;;;549;7756;;564;;;585;7667;;600;;;620;7657;;635;;;655;;;682;;;702;8193;;719;;;738;7751;;754;;;774;7731;;789;;;809;8109;;825;;;845;7623;;860;;;880;7883;;895;;;916;7579;;931;;;951;7514;;966;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:07;1273044;;48.723134;9.094026;496.7;189.86;3.44;1.01;7;3.74;990;;0;0;58;53;0;7543;;15;;;37;7535;;52;;;72;9628;;90;;;109;8166;;125;;;146;7469;;161;;;181;7923;;197;;;216;7651;;232;;;252;7594;;267;;;288;7796;;303;;;323;7960;;338;;;359;7862;;373;;;394;7633;;409;;;429;7926;;444;;;465;7661;;479;;;500;7546;;515;;;535;7522;;550;;;570;8461;;587;;;606;7520;;622;;;643;;;668;;;688;7495;;704;;;723;7672;;739;;;759;7964;;774;;;795;8725;;811;;;831;7366;;847;;;867;7586;;882;;;902;8634;;919;;;938;;;;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:08;1274030;;48.723126;9.094013;496.7;230.11;3.54;1.01;7;3.74;850;;0;0;58;53;0;7452;;29;;;49;7446;;65;;;91;1147;;102;;;127;7517;;143;;;163;7411;;178;;;204;1148;;214;;;240;7282;;256;;;281;1201;;291;;;316;1144;;326;;;351;1173;;362;;;386;6718;;401;;;422;7303;;437;;;461;7621;;476;;;497;7557;;511;;;532;7451;;547;;;567;7658;;583;;;603;7534;;618;;;638;7306;;653;;;673;7222;;688;;;709;7169;;724;;;744;7115;;759;;;779;7277;;794;;;815;;;841;;;861;;;881;;;900;7403;;916;;;936;7356;;951;;;972;7030;;;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:09;1275038;;48.723121;9.093994;496.9;237.39;3.33;1.01;7;3.74;730;;0;0;58;54;0;7327;;16;;;36;6876;;52;;;72;6953;;87;;;107;7261;;122;;;142;6702;;158;;;178;7286;;193;;;213;6605;;228;;;249;7168;;264;;;284;6641;;299;;;324;7059;;339;;;359;7568;;374;;;394;6476;;409;;;430;6589;;445;;;470;1174;;480;;;505;1173;;515;;;541;1175;;551;;;576;1149;;586;;;611;6222;;626;;;647;6722;;661;;;687;5939;;700;;;723;5989;;735;;;760;6144;;773;;;795;6019;;808;;;830;6306;;844;;;866;6169;;879;;;901;6288;;914;;;936;9882;;954;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:10;1276038;;48.723117;9.093979;497.4;247.62;2.96;1.01;7;3.74;7;69;0;;0;0;58;52;0;;;30;;;50;;;69;;;89;;;109;;6187;124;9730;;144;;6203;160;14558;;182;;6178;195;;;222;;6233;235;;;257;;6323;275;;;295;;6379;311;;;331;;6371;346;8588;;366;;6330;381;2150;;401;;6275;417;1200;;437;;6184;461;;;488;;6033;505;;;525;;5943;543;2550;;561;;5872;579;2563;;596;;5844;614;1225;;631;;5835;650;1173;;667;;5799;685;2243;;702;;5804;720;2275;;737;;5798;759;;;785;;5854;805;;;825;;5984;840;;;860;;5979;875;;;895;;6027;911;7850;;931;;6001;946;13531;;969;;5957;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:11;1277041;;48.723117;9.093965;497.5;247.62;2.74;1.01;7;3.74;143;72;0;;0;0;58;48;0;;;30;;5966;43;;;65;;5940;78;12209;;101;;5923;113;;;140;;7918;155;18175;;182;;6159;199;;;226;;6047;239;;;261;;6283;274;;;297;;;328;;;348;;6015;364;;;384;;;418;1174;;428;;6166;453;;;480;;6265;498;;;518;;6241;536;9449;;553;;6311;571;17498;;597;;6394;611;;;638;;6380;652;;;673;;6408;687;;;708;;7059;722;7897;;744;;7059;759;10810;;779;;6459;794;;;822;;6680;841;15140;;864;;;891;6403;;906;;9053;930;10084;;948;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:12;1278045;;48.723109;9.093963;498;247.62;2.17;1.79;7;3.74;143;76;0;;0;0;58;48;0;;;30;;6485;52;;;71;;6321;93;;;113;;6283;128;;;148;;6319;164;10355;;184;;6232;199;17561;;225;;6259;238;;;265;;;292;16478;;317;;;344;8916;;361;;6160;379;;;406;;6242;426;;;445;;6318;461;;;481;;6172;496;;;516;;6271;534;;;554;;6184;571;10174;;590;;6204;607;14878;;630;;6333;643;;;670;;6332;683;;;705;;6231;718;;;740;;6227;753;;;776;;;803;;;822;;6469;844;;;864;;6215;879;;;899;;;927;;;946;;6326;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:13;1279029;;48.723109;9.093963;498;247.62;0;1.79;7;3.74;116;79;0;;0;0;58;47;0;;;20;;6783;41;;;61;;6343;81;;;101;;6506;121;11871;;142;;6364;162;16368;;185;;6365;197;;;224;;;250;15312;;274;;6698;295;12786;;315;;6428;330;;;357;;6556;375;17429;;401;;6426;418;16587;;444;;6539;462;;;488;;;515;18278;;542;;6507;556;;;584;;6506;593;;;619;;6774;632;;;654;;6775;668;;;690;;;717;;;737;;;764;;;784;;6708;799;;;819;;6908;840;8503;;856;;;882;1202;;893;;6729;917;13601;;939;;;974;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:14;1280049;;48.723109;9.093963;498;247.62;0;1.1;7;3.74;121;92;0;;0;0;58;45;0;;;33;;7087;56;;;75;;7170;94;;;113;;7158;129;;;149;;7575;164;;;184;;7233;204;13424;;226;;7616;248;18289;;282;;;315;;;342;;7353;357;;;377;;7521;400;;;419;;7510;435;;;455;;;481;;;500;;7484;516;12940;;537;;;571;8777;;588;;;615;11659;;634;;;660;1174;;670;;;696;;;723;;;742;;;762;;7640;778;;;797;;7980;819;;;839;;8759;855;;;875;;7752;890;11740;;910;;7612;929;12291;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:15;1281028;;48.723109;9.093963;498;247.62;0;1.01;7;3.74;133;95;0;;0;0;58;48;0;;;30;;7564;49;12590;;70;;7576;84;;;111;;7584;120;;;146;;7794;161;;;182;;7826;196;;;217;;;244;;;264;;8057;286;;;305;;7525;321;8033;;340;;7657;360;11295;;380;;;407;9480;;423;;;451;12842;;472;;7540;487;16790;;512;;;538;1175;;549;;;574;1175;;584;;7788;609;1175;;619;;;647;;;673;;7354;697;;;716;;7287;738;;;758;;7454;773;;;793;;7286;812;9244;;829;;7388;848;17471;;874;;7246;893;;;919;;;952;8724;;968;;7324;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
18.11.2020;16:06:16;1282037;;48.723109;9.093963;498;247.62;0;1.01;7;3.74;5;89;20;;0;0;58;49;0;;;31;;7290;44;;;66;;7277;80;;;101;;;128;;;148;;7213;164;;;183;;6901;202;10902;;223;;7060;242;2257;;258;;7057;277;2124;;293;;7045;313;1201;;328;;;361;2137;;371;;6931;396;2055;;407;;6910;432;1201;;442;;;468;2042;;478;;6961;503;1201;;513;;;548;12669;;568;;6909;590;;;617;;7063;636;;;656;;7148;672;;;691;;6777;707;;;727;;6903;747;11631;;767;;;793;1174;;803;;7283;828;;;856;;;889;9154;;908;;7489;929;9129;;943;;7430;965;14679;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
`;
|
||||
|
||||
const test3 = `Date;Time;Millis;Comment;Latitude;Longitude;Altitude;Course;Speed;HDOP;Satellites;BatteryLevel;Left;Right;Confirmed;Marked;Invalid;InsidePrivacyArea;Factor;Measurements;Tms1;Lus1;Rus1;Tms2;Lus2;Rus2;Tms3;Lus3;Rus3;Tms4;Lus4;Rus4;Tms5;Lus5;Rus5;Tms6;Lus6;Rus6;Tms7;Lus7;Rus7;Tms8;Lus8;Rus8;Tms9;Lus9;Rus9;Tms10;Lus10;Rus10;Tms11;Lus11;Rus11;Tms12;Lus12;Rus12;Tms13;Lus13;Rus13;Tms14;Lus14;Rus14;Tms15;Lus15;Rus15;Tms16;Lus16;Rus16;Tms17;Lus17;Rus17;Tms18;Lus18;Rus18;Tms19;Lus19;Rus19;Tms20;Lus20;Rus20;Tms21;Lus21;Rus21;Tms22;Lus22;Rus22;Tms23;Lus23;Rus23;Tms24;Lus24;Rus24;Tms25;Lus25;Rus25;Tms26;Lus26;Rus26;Tms27;Lus27;Rus27;Tms28;Lus28;Rus28;Tms29;Lus29;Rus29;Tms30;Lus30;Rus30;Tms31;Lus31;Rus31;Tms32;Lus32;Rus32;Tms33;Lus33;Rus33;Tms34;Lus34;Rus34;Tms35;Lus35;Rus35;Tms36;Lus36;Rus36;Tms37;Lus37;Rus37;Tms38;Lus38;Rus38;Tms39;Lus39;Rus39;Tms40;Lus40;Rus40;Tms41;Lus41;Rus41;Tms42;Lus42;Rus42;Tms43;Lus43;Rus43;Tms44;Lus44;Rus44;Tms45;Lus45;Rus45;Tms46;Lus46;Rus46;Tms47;Lus47;Rus47;Tms48;Lus48;Rus48;Tms49;Lus49;Rus49;Tms50;Lus50;Rus50;Tms51;Lus51;Rus51;Tms52;Lus52;Rus52;Tms53;Lus53;Rus53;Tms54;Lus54;Rus54;Tms55;Lus55;Rus55;Tms56;Lus56;Rus56;Tms57;Lus57;Rus57;Tms58;Lus58;Rus58;Tms59;Lus59;Rus59;Tms60;Lus60;Rus60;
|
||||
21.11.2020;14:27:00;66890;;;;;;;3.83;4;3.99;;286;0;;0;0;58;5;0;;;41;;18355;67;;;87;;18374;113;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
`;
|
||||
|
||||
module.exports = { test1, test2, test3 };
|
||||
312
api/src/logic/tracks.js
Normal file
312
api/src/logic/tracks.js
Normal file
|
|
@ -0,0 +1,312 @@
|
|||
const csvParse = require('csv-parse/lib/sync');
|
||||
const csvStringify = require('csv-stringify/lib/sync');
|
||||
|
||||
function _parseFloat(token) {
|
||||
if (typeof token !== 'string') {
|
||||
return null;
|
||||
}
|
||||
|
||||
token = token.trim();
|
||||
|
||||
if (token === '') {
|
||||
return null;
|
||||
}
|
||||
|
||||
if (/^nan$/i.test(token)) {
|
||||
return null;
|
||||
}
|
||||
|
||||
let f = parseFloat(token);
|
||||
|
||||
if (isNaN(f)) {
|
||||
f = parseFloat(token.substring(0, 10));
|
||||
}
|
||||
|
||||
if (isNaN(f)) {
|
||||
f = 0.0;
|
||||
}
|
||||
|
||||
return f;
|
||||
}
|
||||
|
||||
function _parseInt(token) {
|
||||
const asFloat = _parseFloat(token);
|
||||
|
||||
if (asFloat !== null) {
|
||||
return Math.floor(asFloat);
|
||||
} else {
|
||||
return asFloat;
|
||||
}
|
||||
}
|
||||
|
||||
function _parseString(token) {
|
||||
if (typeof token !== 'string') {
|
||||
return null;
|
||||
}
|
||||
// This time we do not trim -- because we assume that the quoting mechanism
|
||||
// from CSV might have kicked in and we actually want the spacing around the
|
||||
// token.
|
||||
|
||||
if (token === '') {
|
||||
return null;
|
||||
}
|
||||
|
||||
return token;
|
||||
}
|
||||
|
||||
function replaceDollarNewlinesHack(body) {
|
||||
// see if we are using the hack with $ as newlines, replace them for the csv parser
|
||||
if (body.endsWith('$') || /insidePrivacyArea;\$/.test(body)) {
|
||||
return body.replace(/\$/g, '\n');
|
||||
}
|
||||
|
||||
return body;
|
||||
}
|
||||
|
||||
function* parseTrackPoints(body, format = null) {
|
||||
if (body instanceof Buffer) {
|
||||
body = body.toString('utf-8')
|
||||
}
|
||||
body = replaceDollarNewlinesHack(body);
|
||||
|
||||
const detectedFormat = format != null ? format : detectFormat(body);
|
||||
|
||||
let parser;
|
||||
switch (detectedFormat) {
|
||||
case 'invalid':
|
||||
throw new Error('track format cannot be detected');
|
||||
|
||||
case 1:
|
||||
parser = parseObsver1;
|
||||
break;
|
||||
|
||||
case 2:
|
||||
parser = parseObsver2;
|
||||
break;
|
||||
}
|
||||
|
||||
yield* parser(body);
|
||||
}
|
||||
|
||||
function detectFormat(body) {
|
||||
body = replaceDollarNewlinesHack(body);
|
||||
|
||||
if (!body.length) {
|
||||
return 'invalid';
|
||||
}
|
||||
|
||||
const firstLinebreakIndex = body.indexOf('\n');
|
||||
|
||||
if (firstLinebreakIndex === -1) {
|
||||
// We need at least one linebreak in the whole file, to separate header and
|
||||
// data. If the file contains no header, it is in valid.
|
||||
return 'invalid';
|
||||
}
|
||||
|
||||
const firstLine = body.substring(0, firstLinebreakIndex);
|
||||
|
||||
const match = firstLine.match(/(^|&)OBSDataFormat=([\d]+)($|&)/);
|
||||
if (match) {
|
||||
return Number(match[2]);
|
||||
}
|
||||
|
||||
// If we have no metadata line, but start immediately with a header, AND it contains
|
||||
// `;Rus`, it is a version 2
|
||||
if (/^Date;Time.*;Rus/.test(firstLine)) {
|
||||
return 2;
|
||||
}
|
||||
|
||||
// If we have no metadata line, but start immediately with a header, it is
|
||||
// format version 1.
|
||||
if (/^Date;Time/.test(firstLine)) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
// If we immediately start with data (a date, formatted as DD.MM.YYYY), then
|
||||
// we have an old OBS not sending the header. It must therefore be old
|
||||
// format, too.
|
||||
if (/^[0-9]{2}\.[0-9]{2}\.[0-9]{4};/.test(firstLine)) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
return 'invalid';
|
||||
}
|
||||
|
||||
function* parseObsver1(body) {
|
||||
for (const record of csvParse(body, {
|
||||
delimiter: ';',
|
||||
encoding: 'utf8',
|
||||
// We specify different column names here, as the order of columns was
|
||||
// always the same, but their naming was different. By enforicing these
|
||||
// column names we don't have to translate between them. Then we just
|
||||
// ignore the first line (or any line that starts with "Date;").
|
||||
// Original header usually is:
|
||||
// Date;Time;Latitude;Longitude;Course;Speed;Right;Left;Confirmed;insidePrivacyArea
|
||||
columns: ['date', 'time', 'latitude', 'longitude', 'course', 'speed', 'd1', 'd2', 'flag', 'private'],
|
||||
relax_column_count: true,
|
||||
cast(value, { column }) {
|
||||
if (['latitude', 'longitude', 'course', 'speed'].includes(column)) {
|
||||
return _parseFloat(value);
|
||||
} else if (['d1', 'd2', 'flag'].includes(column)) {
|
||||
return _parseInt(value);
|
||||
} else if (column === 'private') {
|
||||
return Boolean(_parseInt(value));
|
||||
} else {
|
||||
return _parseString(value);
|
||||
}
|
||||
},
|
||||
})) {
|
||||
if (record.date === 'Date') {
|
||||
// ignore header line
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!record.latitude && !record.longitude) {
|
||||
// invalid record, make sure lat/lng say `null` instead of `0`
|
||||
record.latitude = null;
|
||||
record.longitude = null;
|
||||
}
|
||||
|
||||
// in old format, 255 or 999 means "no measurement"
|
||||
if (record.d1 === 255 || record.d1 === 999) {
|
||||
record.d1 = null;
|
||||
}
|
||||
|
||||
if (record.d2 === 255 || record.d2 === 999) {
|
||||
record.d2 = null;
|
||||
}
|
||||
|
||||
yield record;
|
||||
}
|
||||
}
|
||||
|
||||
function* parseObsver2(body) {
|
||||
for (const record of csvParse(body, {
|
||||
from_line: 2,
|
||||
trim: true,
|
||||
columns: true,
|
||||
skip_empty_lines: true,
|
||||
delimiter: ';',
|
||||
encoding: 'utf8',
|
||||
relax_column_count: true,
|
||||
cast(value, context) {
|
||||
if (value === '') {
|
||||
return null;
|
||||
}
|
||||
|
||||
let type;
|
||||
switch (context.column) {
|
||||
case 'Millis':
|
||||
case 'Left':
|
||||
case 'Right':
|
||||
case 'Confirmed':
|
||||
case 'Invalid':
|
||||
case 'InsidePrivacyArea':
|
||||
case 'Measurements':
|
||||
case 'Satellites':
|
||||
type = 'int';
|
||||
break;
|
||||
|
||||
case 'Date':
|
||||
case 'Time':
|
||||
case 'Comment':
|
||||
case 'Marked':
|
||||
type = 'string';
|
||||
break;
|
||||
|
||||
case 'Latitude':
|
||||
case 'Longitude':
|
||||
case 'Altitude':
|
||||
case 'Course':
|
||||
case 'Speed':
|
||||
case 'HDOP':
|
||||
case 'BatteryLevel':
|
||||
case 'Factor':
|
||||
type = 'float';
|
||||
break;
|
||||
|
||||
default:
|
||||
type = /^(Tms|Lus|Rus)/.test(context.column) ? 'int' : 'string';
|
||||
}
|
||||
|
||||
switch (type) {
|
||||
case 'int':
|
||||
return _parseInt(value);
|
||||
|
||||
case 'float':
|
||||
return _parseFloat(value);
|
||||
|
||||
case 'string':
|
||||
return _parseString(value);
|
||||
}
|
||||
},
|
||||
})) {
|
||||
// We convert the new format back to the old format for storage here, until
|
||||
// we upgrade the storage format as well to include all data. But we'll
|
||||
// have to upgrade the obsApp first.
|
||||
yield {
|
||||
date: record.Date,
|
||||
time: record.Time,
|
||||
latitude: record.Latitude,
|
||||
longitude: record.Longitude,
|
||||
course: record.Course,
|
||||
speed: record.Speed,
|
||||
d1: record.Left,
|
||||
d2: record.Right,
|
||||
flag: Boolean(record.Confirmed),
|
||||
private: Boolean(record.InsidePrivacyArea),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* This function normalizes a User-Agent header for storage in the database. It
|
||||
* make sure that we only store the user-agent if it matches the pattern
|
||||
* `OBS/*`, and extracts that part of the user agent, if it contains more
|
||||
* information. This is the only part we are interested in, the
|
||||
* remainder is too privacy sensitive to keep.
|
||||
*/
|
||||
function normalizeUserAgent(userAgent) {
|
||||
if (!userAgent) {
|
||||
return null;
|
||||
}
|
||||
|
||||
const match = userAgent.match(/\bOBS\/[^\s]+/);
|
||||
if (match) {
|
||||
return match[0];
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
function buildObsver1(points) {
|
||||
return csvStringify(points, {
|
||||
columns: [
|
||||
{ key: 'date', header: 'Date' },
|
||||
{ key: 'time', header: 'Time' },
|
||||
{ key: 'latitude', header: 'Latitude' },
|
||||
{ key: 'longitude', header: 'Longitude' },
|
||||
{ key: 'course', header: 'Course' },
|
||||
{ key: 'speed', header: 'Speed' },
|
||||
{ key: 'd1', header: 'Right' },
|
||||
{ key: 'd2', header: 'Left' },
|
||||
{ key: 'flag', header: 'Confirmed' },
|
||||
{ key: 'private', header: 'insidePrivacyArea' },
|
||||
],
|
||||
cast: {
|
||||
boolean: (v) => (v ? '1' : '0'),
|
||||
},
|
||||
delimiter: ';',
|
||||
header: true,
|
||||
});
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
detectFormat,
|
||||
normalizeUserAgent,
|
||||
parseObsver1,
|
||||
parseObsver2,
|
||||
parseTrackPoints,
|
||||
replaceDollarNewlinesHack,
|
||||
buildObsver1,
|
||||
};
|
||||
171
api/src/logic/tracks.test.js
Normal file
171
api/src/logic/tracks.test.js
Normal file
|
|
@ -0,0 +1,171 @@
|
|||
const {
|
||||
buildObsver1,
|
||||
detectFormat,
|
||||
normalizeUserAgent,
|
||||
parseObsver1,
|
||||
parseObsver2,
|
||||
parseTrackPoints,
|
||||
replaceDollarNewlinesHack,
|
||||
} = require('./tracks');
|
||||
|
||||
const { test1, test2, test3 } = require('./_tracks_testdata');
|
||||
|
||||
describe('parseTrackPoints', () => {
|
||||
it('is a function', () => {
|
||||
expect(typeof parseTrackPoints).toBe('function');
|
||||
});
|
||||
|
||||
it('works on the sample data with an empty track', () => {
|
||||
const points = Array.from(parseTrackPoints(test1));
|
||||
expect(points).toHaveLength(324);
|
||||
expect(points[0]).toEqual({
|
||||
date: '12.07.2020',
|
||||
time: '09:02:59',
|
||||
latitude: null,
|
||||
longitude: null,
|
||||
course: 0,
|
||||
speed: 0,
|
||||
d1: null,
|
||||
d2: null,
|
||||
flag: 0,
|
||||
private: false,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseObsver1', () => {
|
||||
it('can parse sample data', () => {
|
||||
const points = Array.from(parseObsver1(replaceDollarNewlinesHack(test1)));
|
||||
expect(points).toHaveLength(324);
|
||||
expect(points[0]).toEqual({
|
||||
date: '12.07.2020',
|
||||
time: '09:02:59',
|
||||
latitude: null,
|
||||
longitude: null,
|
||||
course: 0,
|
||||
speed: 0,
|
||||
d1: null,
|
||||
d2: null,
|
||||
flag: 0,
|
||||
private: false,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('parseObsver2', () => {
|
||||
it('can parse sample data', () => {
|
||||
const points = Array.from(parseObsver2(test2));
|
||||
expect(points).toHaveLength(18);
|
||||
expect(points[0]).toEqual({
|
||||
date: '18.11.2020',
|
||||
time: '16:05:59',
|
||||
latitude: 48.723224,
|
||||
longitude: 9.094103,
|
||||
course: 189.86,
|
||||
speed: 3.2,
|
||||
d1: 770,
|
||||
d2: null,
|
||||
flag: false,
|
||||
private: true,
|
||||
});
|
||||
|
||||
// this is a non-private, flagged point (i.e. "Confirmed" overtaking)
|
||||
expect(points[17]).toEqual({
|
||||
date: '18.11.2020',
|
||||
time: '16:06:16',
|
||||
latitude: 48.723109,
|
||||
longitude: 9.093963,
|
||||
course: 247.62,
|
||||
speed: 0,
|
||||
d1: 5,
|
||||
d2: 89,
|
||||
flag: true,
|
||||
private: false,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('detectFormat', () => {
|
||||
it('detects format 1', () => {
|
||||
expect(detectFormat(test1)).toBe(1);
|
||||
});
|
||||
|
||||
it('detects format 2', () => {
|
||||
expect(detectFormat(test2)).toBe(2);
|
||||
expect(detectFormat(test3)).toBe(2);
|
||||
});
|
||||
|
||||
it('detects invalid format', () => {
|
||||
expect(detectFormat('foobar\nbaz')).toBe('invalid');
|
||||
expect(detectFormat('')).toBe('invalid');
|
||||
});
|
||||
});
|
||||
|
||||
describe('normalizeUserAgent', () => {
|
||||
it('is a function', () => {
|
||||
expect(typeof normalizeUserAgent).toBe('function');
|
||||
});
|
||||
|
||||
it('ignores falsy values', () => {
|
||||
expect(normalizeUserAgent(null)).toBe(null);
|
||||
expect(normalizeUserAgent('')).toBe(null);
|
||||
});
|
||||
|
||||
it('ignores normal browser agents', () => {
|
||||
const browserAgents = [
|
||||
'Mozilla/5.0 (Linux; Android 6.0.1; Nexus 6P Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.83 Mobile Safari/537.36',
|
||||
'Mozilla/5.0 (Linux; Android 6.0; HTC One M9 Build/MRA58K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.98 Mobile Safari/537.3',
|
||||
'Mozilla/5.0 (Linux; Android 8.0.0; SM-G960F Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.84 Mobile Safari/537.36',
|
||||
'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A5370a Safari/604.1',
|
||||
'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Gecko/20100101 Firefox/15.0.1',
|
||||
];
|
||||
|
||||
for (const browserAgent of browserAgents) {
|
||||
expect(normalizeUserAgent(browserAgent)).toBe(null);
|
||||
}
|
||||
});
|
||||
|
||||
it('detects OBS versions', () => {
|
||||
const agents = ['OBS/123', 'OBS/2', 'OBS/1.2.3.4.5-rc123'];
|
||||
|
||||
for (const agent of agents) {
|
||||
expect(normalizeUserAgent(agent)).toBe(agent);
|
||||
}
|
||||
});
|
||||
|
||||
it('extracts OBS versions from extended formats', () => {
|
||||
const agents = ['foo OBS/123', 'OBS/123 bar', 'foo OBS/123 bar'];
|
||||
|
||||
for (const agent of agents) {
|
||||
expect(normalizeUserAgent(agent)).toBe('OBS/123');
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildObsver1', () => {
|
||||
it('is a function', () => {
|
||||
expect(typeof normalizeUserAgent).toBe('function');
|
||||
});
|
||||
|
||||
it('transforms properly back and forth', () => {
|
||||
const inputString = replaceDollarNewlinesHack(test1);
|
||||
|
||||
const points1 = Array.from(parseObsver1(inputString));
|
||||
const builtString = buildObsver1(points1);
|
||||
const points2 = Array.from(parseObsver1(builtString));
|
||||
|
||||
expect(points2).toEqual(points1);
|
||||
});
|
||||
|
||||
it('produces a header', () => {
|
||||
const builtString = buildObsver1([]);
|
||||
expect(builtString).toBe('Date;Time;Latitude;Longitude;Course;Speed;Right;Left;Confirmed;insidePrivacyArea\n');
|
||||
});
|
||||
|
||||
it('produces empty rows', () => {
|
||||
const builtString = buildObsver1([{}]);
|
||||
expect(builtString).toBe(
|
||||
'Date;Time;Latitude;Longitude;Course;Speed;Right;Left;Confirmed;insidePrivacyArea\n;;;;;;;;;\n',
|
||||
);
|
||||
});
|
||||
});
|
||||
49
api/src/models/AccessToken.js
Normal file
49
api/src/models/AccessToken.js
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
const mongoose = require('mongoose');
|
||||
const uniqueValidator = require('mongoose-unique-validator');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const schema = new mongoose.Schema(
|
||||
{
|
||||
token: { index: true, type: String, required: true, unique: true },
|
||||
user: { type: mongoose.Schema.Types.ObjectId, ref: 'User', required: true },
|
||||
clientId: { type: String, required: true },
|
||||
expiresAt: { type: Date, required: true },
|
||||
scope: { type: String, required: true, defaultValue: '*' },
|
||||
},
|
||||
{ timestamps: true },
|
||||
);
|
||||
|
||||
schema.plugin(uniqueValidator, { message: 'reused token' });
|
||||
|
||||
class AccessTokenClass extends mongoose.Model {
|
||||
toJSON() {
|
||||
return {
|
||||
token: this.token,
|
||||
expires: this.expires,
|
||||
};
|
||||
}
|
||||
|
||||
isValid() {
|
||||
return this.expiresAt < new Date();
|
||||
}
|
||||
|
||||
toHeaderString() {
|
||||
return 'Bearer ' + this.token;
|
||||
}
|
||||
|
||||
static generate(options, expiresInSeconds = 24 * 60 * 60) {
|
||||
const token = crypto.randomBytes(32).toString('hex');
|
||||
|
||||
return new AccessToken({
|
||||
...options,
|
||||
token,
|
||||
expiresAt: new Date(new Date().getTime() + 1000 * expiresInSeconds),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
schema.loadClass(AccessTokenClass);
|
||||
|
||||
const AccessToken = mongoose.model('AccessToken', schema);
|
||||
|
||||
module.exports = AccessToken;
|
||||
32
api/src/models/AuthorizationCode.js
Normal file
32
api/src/models/AuthorizationCode.js
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
const mongoose = require('mongoose');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const schema = new mongoose.Schema(
|
||||
{
|
||||
code: { type: String, unique: true, required: true },
|
||||
user: { type: mongoose.Schema.Types.ObjectId, ref: 'User', required: true },
|
||||
clientId: { type: String, required: true },
|
||||
scope: { type: String, required: true, defaultValue: '*' },
|
||||
redirectUri: { type: String, required: true },
|
||||
expiresAt: { type: Date, required: true },
|
||||
codeChallenge: { type: String, required: true }, // no need to store the method, it is always "S256"
|
||||
},
|
||||
{ timestamps: true },
|
||||
);
|
||||
|
||||
class AuthorizationCodeClass extends mongoose.Model {
|
||||
static generate(options, expiresInSeconds = 60) {
|
||||
const code = crypto.randomBytes(8).toString('hex');
|
||||
|
||||
return new AuthorizationCode({
|
||||
...options,
|
||||
code,
|
||||
expiresAt: new Date(new Date().getTime() + 1000 * expiresInSeconds),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
schema.loadClass(AuthorizationCodeClass)
|
||||
|
||||
const AuthorizationCode = mongoose.model('AuthorizationCode', schema);
|
||||
module.exports = AuthorizationCode
|
||||
24
api/src/models/Comment.js
Normal file
24
api/src/models/Comment.js
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
const mongoose = require('mongoose');
|
||||
|
||||
const schema = new mongoose.Schema(
|
||||
{
|
||||
body: String,
|
||||
author: { type: mongoose.Schema.Types.ObjectId, ref: 'User' },
|
||||
track: { type: mongoose.Schema.Types.ObjectId, ref: 'Track' },
|
||||
},
|
||||
{ timestamps: true },
|
||||
);
|
||||
|
||||
class CommentClass extends mongoose.Model {
|
||||
toJSONFor(user) {
|
||||
return {
|
||||
id: this._id,
|
||||
body: this.body,
|
||||
createdAt: this.createdAt,
|
||||
author: this.author.toProfileJSONFor(user),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
schema.loadClass(CommentClass)
|
||||
module.exports = mongoose.model('Comment', schema);
|
||||
46
api/src/models/RefreshToken.js
Normal file
46
api/src/models/RefreshToken.js
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
const mongoose = require('mongoose');
|
||||
const uniqueValidator = require('mongoose-unique-validator');
|
||||
const crypto = require('crypto');
|
||||
|
||||
const AccessToken = require('./AccessToken')
|
||||
|
||||
const schema = new mongoose.Schema(
|
||||
{
|
||||
token: { index: true, type: String, required: true, unique: true },
|
||||
user: { type: mongoose.Schema.Types.ObjectId, ref: 'User', required: true },
|
||||
clientId: { type: String, required: true },
|
||||
expiresAt: { type: Date, required: false },
|
||||
scope: { type: String, required: true, defaultValue: '*' },
|
||||
},
|
||||
{ timestamps: true },
|
||||
);
|
||||
|
||||
schema.plugin(uniqueValidator, { message: 'reused token' });
|
||||
|
||||
class RefreshTokenClass extends mongoose.Model {
|
||||
toJSON() {
|
||||
return {
|
||||
token: this.token,
|
||||
expires: this.expires,
|
||||
};
|
||||
}
|
||||
|
||||
isValid() {
|
||||
return this.expiresAt == null || this.expiresAt < new Date()
|
||||
}
|
||||
|
||||
static generate(options, expiresInSeconds = 24 * 60 * 60) {
|
||||
const token = crypto.randomBytes(32).toString('hex');
|
||||
|
||||
return new RefreshToken({
|
||||
...options,
|
||||
token,
|
||||
expiresAt: new Date(new Date().getTime() + 1000 * expiresInSeconds),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
schema.loadClass(RefreshTokenClass);
|
||||
|
||||
const RefreshToken = mongoose.model('RefreshToken', schema);
|
||||
module.exports = RefreshToken
|
||||
306
api/src/models/Track.js
Normal file
306
api/src/models/Track.js
Normal file
|
|
@ -0,0 +1,306 @@
|
|||
const crypto = require('crypto');
|
||||
const mongoose = require('mongoose');
|
||||
const _ = require('lodash');
|
||||
const uniqueValidator = require('mongoose-unique-validator');
|
||||
const { DateTime } = require('luxon');
|
||||
const slug = require('slug');
|
||||
const path = require('path');
|
||||
const sanitize = require('sanitize-filename');
|
||||
const fs = require('fs');
|
||||
const uuid = require('uuid/v4');
|
||||
|
||||
const { TRACKS_DIR } = require('../paths');
|
||||
const queue = require('../queue');
|
||||
|
||||
const statisticsSchema = new mongoose.Schema(
|
||||
{
|
||||
recordedAt: Date,
|
||||
recordedUntil: Date,
|
||||
duration: Number,
|
||||
length: Number,
|
||||
segments: Number,
|
||||
numEvents: Number,
|
||||
numMeasurements: Number,
|
||||
numValid: Number,
|
||||
},
|
||||
{ timestamps: false },
|
||||
);
|
||||
|
||||
const schema = new mongoose.Schema(
|
||||
{
|
||||
// A (partially or entirely random generated) string that can be used as a
|
||||
// public identifier
|
||||
slug: { type: String, lowercase: true, unique: true },
|
||||
|
||||
// The title for this track.
|
||||
title: String,
|
||||
|
||||
// The status of this track, whether it is to be processed, is currently
|
||||
// being processed, or has completed or errored.
|
||||
processingStatus: {
|
||||
type: String,
|
||||
enum: ['pending', 'processing', 'complete', 'error'],
|
||||
default: 'pending',
|
||||
},
|
||||
processingJobId: String,
|
||||
|
||||
// Output from the proccessing routines regarding this track. Might be
|
||||
// displayed to the owner or administrators to help in debugging. Should be
|
||||
// set to `null` if no processing has not been finished.
|
||||
processingLog: String,
|
||||
|
||||
// Set to true if the user customized the title. Disables auto-generating
|
||||
// an updated title when the track is (re-)processed.
|
||||
customizedTitle: { type: Boolean, default: false },
|
||||
|
||||
// A user-provided description of the track. May contain markdown.
|
||||
description: String,
|
||||
|
||||
// Whether this track is visible (anonymized) in the public track list or not.
|
||||
public: { type: Boolean, default: false },
|
||||
|
||||
// Whether this track should be exported to the public track database
|
||||
// (after anonymization).
|
||||
includeInPublicDatabase: { type: Boolean, default: false },
|
||||
|
||||
// The user agent string, or a part thereof, that was used to upload this
|
||||
// track. Usually contains only the OBS version, other user agents are
|
||||
// discarded due to being irrelevant.
|
||||
uploadedByUserAgent: String,
|
||||
|
||||
// The name of the original file, as provided during upload. Used for
|
||||
// providing a download with the same name, and for display in the
|
||||
// frontend.
|
||||
originalFileName: {
|
||||
type: String,
|
||||
required: true,
|
||||
validate: {
|
||||
validator: function (v) {
|
||||
// Must be a sane filename, i.e. not change when being sanitized
|
||||
return sanitize(v) === v && v.length > 0 && /.+\.csv$/i.test(v);
|
||||
},
|
||||
message: (props) => `${props.value} is not a valid filename`,
|
||||
},
|
||||
},
|
||||
|
||||
// A hash of the original file's contents. Nobody can upload the same track twice.
|
||||
originalFileHash: {
|
||||
type: String,
|
||||
required: true,
|
||||
},
|
||||
|
||||
// Where the files are stored, relative to a group directory like
|
||||
// TRACKS_DIR or PROCESSING_DIR.
|
||||
filePath: String,
|
||||
|
||||
comments: [{ type: mongoose.Schema.Types.ObjectId, ref: 'Comment' }],
|
||||
author: { type: mongoose.Schema.Types.ObjectId, ref: 'User' },
|
||||
|
||||
statistics: statisticsSchema,
|
||||
},
|
||||
{ timestamps: true },
|
||||
);
|
||||
|
||||
schema.index({ author: 1, originalFileHash: 1 }, { unique: true });
|
||||
|
||||
schema.plugin(uniqueValidator, { message: 'is already taken' });
|
||||
|
||||
schema.pre('validate', async function (next) {
|
||||
try {
|
||||
if (!this.slug) {
|
||||
this.slugify();
|
||||
}
|
||||
|
||||
if (!this.filePath) {
|
||||
await this.generateFilePath();
|
||||
}
|
||||
|
||||
next();
|
||||
} catch (err) {
|
||||
next(err);
|
||||
}
|
||||
});
|
||||
|
||||
// 0..4 Night, 4..10 Morning, 10..14 Noon, 14..18 Afternoon, 18..22 Evening, 22..00 Night
|
||||
// Two hour intervals
|
||||
const DAYTIMES = [
|
||||
'Night', // 0h - 2h
|
||||
'Night', // 2h - 4h
|
||||
'Morning', // 4h - 6h
|
||||
'Morning', // 6h - 8h
|
||||
'Morning', // 8h - 10h
|
||||
'Noon', // 10h - 12h
|
||||
'Noon', // 12h - 14h
|
||||
'Afternoon', // 14h - 16h
|
||||
'Afternoon', // 16h - 18h
|
||||
'Evening', // 18h - 20h
|
||||
'Evening', // 20h - 22h
|
||||
'Night', // 22h - 24h
|
||||
];
|
||||
|
||||
function getDaytime(dateTime) {
|
||||
return DAYTIMES[Math.floor((dateTime.hour % 24) / 2)];
|
||||
}
|
||||
|
||||
class TrackClass extends mongoose.Model {
|
||||
slugify() {
|
||||
this.slug = slug(this.title || 'track') + '-' + ((Math.random() * Math.pow(36, 6)) | 0).toString(36);
|
||||
}
|
||||
|
||||
async generateFilePath() {
|
||||
await this.populate('author');
|
||||
this.filePath = path.join(this.author.username, this.slug);
|
||||
}
|
||||
|
||||
isVisibleTo(user) {
|
||||
if (this.public) {
|
||||
return true;
|
||||
}
|
||||
|
||||
if (!user) {
|
||||
return false;
|
||||
}
|
||||
|
||||
if (user._id.equals(this.author._id)) {
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
isVisibleToPrivate(user) {
|
||||
return user && user._id.equals(this.author._id);
|
||||
}
|
||||
|
||||
async _ensureDirectoryExists() {
|
||||
if (!this.filePath) {
|
||||
await this.generateFilePath();
|
||||
}
|
||||
|
||||
const dir = path.dirname(this.getOriginalFilePath());
|
||||
await fs.promises.mkdir(dir, { recursive: true });
|
||||
}
|
||||
|
||||
getOriginalFilePath() {
|
||||
if (!this.filePath) {
|
||||
throw new Error('Cannot get original file path, `filePath` is not yet set. Call `generateFilePath()` first.');
|
||||
}
|
||||
return path.join(TRACKS_DIR, this.filePath, 'original.csv');
|
||||
}
|
||||
|
||||
async writeToOriginalFile(fileBody) {
|
||||
await this._ensureDirectoryExists();
|
||||
await fs.promises.writeFile(this.getOriginalFilePath(), fileBody);
|
||||
}
|
||||
|
||||
async validateFileBodyUniqueness(fileBody) {
|
||||
// Generate hash
|
||||
const hash = crypto.createHash('sha512').update(fileBody).digest('hex');
|
||||
|
||||
const existingTracks = await Track.find({ originalFileHash: hash, author: this.author });
|
||||
if (existingTracks.length === 0 || (existingTracks.length === 1 && existingTracks[0]._id.equals(this._id))) {
|
||||
this.originalFileHash = hash;
|
||||
return;
|
||||
}
|
||||
|
||||
throw new Error('Track file already uploaded.');
|
||||
}
|
||||
|
||||
/**
|
||||
* Marks this track as needing processing.
|
||||
*
|
||||
* Also deletes all stored information that is derived during processing from
|
||||
* the database, such that it may be filled again with correct information
|
||||
* during the processing operation.
|
||||
*
|
||||
* Saves the track as well, so it is up to date when the worker receives it.
|
||||
*/
|
||||
async queueProcessing() {
|
||||
this.processingStatus = 'pending';
|
||||
this.processingLog = null;
|
||||
this.processingJobId = uuid();
|
||||
this.statistics = null;
|
||||
|
||||
await this.save();
|
||||
|
||||
await queue.add(
|
||||
'processTrack',
|
||||
{
|
||||
trackId: this._id.toString(),
|
||||
},
|
||||
{
|
||||
jobId: this.processingJobId,
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
async readProcessingResults(success = true) {
|
||||
// Copies some information into this object from the outputs of the
|
||||
// processing step. This allows general statistics to be formed, and other
|
||||
// information to be displayed, without having to read individual files
|
||||
// from disk. Each field set here should be unsed in `queueProcessing`.
|
||||
// This routine also moves the `processingStatus` along.
|
||||
}
|
||||
|
||||
async autoGenerateTitle() {
|
||||
if (this.customizedTitle) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Try to figure out when this file was recorded. Either we have it in then
|
||||
// statistics, e.g. after parsing and processing the track, or we can maybe
|
||||
// derive it from the filename.
|
||||
let recordedAt = null;
|
||||
|
||||
if (this.statistics && this.statistics.recordedAt != null) {
|
||||
recordedAt = DateTime.fromJSDate(this.statistics.recordedAt);
|
||||
} else if (this.originalFileName) {
|
||||
const match = this.originalFileName.match(/^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}\.[0-9]{2}\.[0-9]{2}/);
|
||||
if (match) {
|
||||
recordedAt = DateTime.fromFormat(match[0], "yyyy-MM-dd'T'HH.mm.ss");
|
||||
if (!recordedAt.isValid) {
|
||||
recordedAt = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (recordedAt) {
|
||||
const daytime = getDaytime(recordedAt);
|
||||
this.title = `${daytime} ride on ${recordedAt.toLocaleString(recordedAt.DATE_MED)}`;
|
||||
await this.save();
|
||||
return;
|
||||
}
|
||||
|
||||
// Detecting recording date failed, use filename
|
||||
if (this.originalFileName) {
|
||||
this.title = _.upperFirst(_.words(this.originalFileName.replace(/(\.obsdata)?\.csv$/, '')).join(' '));
|
||||
await this.save();
|
||||
}
|
||||
}
|
||||
|
||||
toJSONFor(user) {
|
||||
const includePrivateFields = user && user._id.equals(this.author._id);
|
||||
|
||||
return {
|
||||
slug: this.slug,
|
||||
title: this.title,
|
||||
description: this.description,
|
||||
createdAt: this.createdAt,
|
||||
updatedAt: this.updatedAt,
|
||||
public: this.public,
|
||||
author: this.author.toProfileJSONFor(user),
|
||||
statistics: this.statistics,
|
||||
processingStatus: this.processingStatus,
|
||||
...(includePrivateFields
|
||||
? {
|
||||
uploadedByUserAgent: this.uploadedByUserAgent,
|
||||
originalFileName: this.originalFileName,
|
||||
}
|
||||
: {}),
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
schema.loadClass(TrackClass);
|
||||
const Track = mongoose.model('Track', schema);
|
||||
module.exports = Track;
|
||||
91
api/src/models/User.js
Normal file
91
api/src/models/User.js
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
const mongoose = require('mongoose');
|
||||
const uniqueValidator = require('mongoose-unique-validator');
|
||||
const crypto = require('crypto');
|
||||
const jwt = require('jsonwebtoken');
|
||||
const config = require('../config')
|
||||
|
||||
const schema = new mongoose.Schema(
|
||||
{
|
||||
username: {
|
||||
type: String,
|
||||
lowercase: true,
|
||||
unique: true,
|
||||
required: [true, "can't be blank"],
|
||||
match: [/^[a-zA-Z0-9]+$/, 'is invalid'],
|
||||
index: true,
|
||||
},
|
||||
email: {
|
||||
type: String,
|
||||
lowercase: true,
|
||||
unique: true,
|
||||
required: [true, "can't be blank"],
|
||||
match: [/\S+@\S+\.\S+/, 'is invalid'],
|
||||
index: true,
|
||||
},
|
||||
bio: String,
|
||||
image: String,
|
||||
areTracksVisibleForAll: Boolean,
|
||||
hash: String,
|
||||
salt: String,
|
||||
needsEmailValidation: Boolean,
|
||||
verificationToken: String,
|
||||
resetToken: {
|
||||
token: String,
|
||||
expires: Date,
|
||||
},
|
||||
},
|
||||
{ timestamps: true },
|
||||
);
|
||||
|
||||
schema.plugin(uniqueValidator, { message: 'ist bereits vergeben. Sorry!' });
|
||||
|
||||
class UserClass extends mongoose.Model {
|
||||
validPassword(password) {
|
||||
const hash = crypto.pbkdf2Sync(password, this.salt, 10000, 512, 'sha512').toString('hex');
|
||||
return this.hash === hash;
|
||||
}
|
||||
|
||||
setPassword(password) {
|
||||
this.salt = crypto.randomBytes(16).toString('hex');
|
||||
this.hash = crypto.pbkdf2Sync(password, this.salt, 10000, 512, 'sha512').toString('hex');
|
||||
}
|
||||
|
||||
generateJWT() {
|
||||
const today = new Date();
|
||||
const exp = new Date(today);
|
||||
exp.setDate(today.getDate() + 60);
|
||||
|
||||
return jwt.sign(
|
||||
{
|
||||
id: this._id,
|
||||
username: this.username,
|
||||
exp: parseInt(exp.getTime() / 1000),
|
||||
},
|
||||
config.jwtSecret,
|
||||
);
|
||||
}
|
||||
|
||||
toAuthJSON() {
|
||||
return {
|
||||
username: this.username,
|
||||
email: this.email,
|
||||
token: this.generateJWT(),
|
||||
bio: this.bio,
|
||||
image: this.image,
|
||||
areTracksVisibleForAll: this.areTracksVisibleForAll,
|
||||
apiKey: this._id,
|
||||
};
|
||||
}
|
||||
|
||||
toProfileJSONFor(user) {
|
||||
return {
|
||||
username: this.username,
|
||||
bio: this.bio,
|
||||
image: this.image,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
schema.loadClass(UserClass);
|
||||
const User = mongoose.model('User', schema);
|
||||
module.exports = User
|
||||
6
api/src/models/index.js
Normal file
6
api/src/models/index.js
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
module.exports.AccessToken = require('./AccessToken')
|
||||
module.exports.AuthorizationCode = require('./AuthorizationCode')
|
||||
module.exports.Comment = require('./Comment')
|
||||
module.exports.RefreshToken = require('./RefreshToken')
|
||||
module.exports.Track = require('./Track')
|
||||
module.exports.User = require('./User')
|
||||
215
api/src/passport.js
Normal file
215
api/src/passport.js
Normal file
|
|
@ -0,0 +1,215 @@
|
|||
const passport = require('passport');
|
||||
const { Strategy: LocalStrategy } = require('passport-local');
|
||||
const { Strategy: BearerStrategy } = require('passport-http-bearer');
|
||||
const { Strategy: JwtStrategy } = require('passport-jwt');
|
||||
const { Strategy: CustomStrategy } = require('passport-custom');
|
||||
|
||||
const { User, AccessToken, RefreshToken } = require('./models');
|
||||
|
||||
const config = require('./config');
|
||||
|
||||
// used to serialize the user for the session
|
||||
passport.serializeUser(function (user, done) {
|
||||
done(null, user._id);
|
||||
});
|
||||
|
||||
// used to deserialize the user
|
||||
passport.deserializeUser(function (id, done) {
|
||||
User.findById(id, function (err, user) {
|
||||
done(err, user);
|
||||
});
|
||||
});
|
||||
|
||||
async function loginWithPassword(email, password, done) {
|
||||
try {
|
||||
const user = await User.findOne({ email: email });
|
||||
if (!user || !user.validPassword(password)) {
|
||||
return done(new Error('invalid credentials'), false);
|
||||
}
|
||||
|
||||
// Regardless of whether login is required, if you're logged in as an
|
||||
// unverified user, produce an error.
|
||||
if (user.needsEmailValidation) {
|
||||
return done(new Error('email not verified'), false);
|
||||
}
|
||||
|
||||
return done(null, user);
|
||||
} catch (err) {
|
||||
done(err);
|
||||
}
|
||||
}
|
||||
|
||||
passport.use(
|
||||
'usernameAndPassword',
|
||||
new LocalStrategy(
|
||||
{
|
||||
usernameField: 'user[email]',
|
||||
passwordField: 'user[password]',
|
||||
session: false,
|
||||
},
|
||||
loginWithPassword,
|
||||
),
|
||||
);
|
||||
|
||||
passport.use(
|
||||
'usernameAndPasswordSession',
|
||||
new LocalStrategy(
|
||||
{
|
||||
usernameField: 'email',
|
||||
passwordField: 'password',
|
||||
session: true,
|
||||
},
|
||||
loginWithPassword,
|
||||
),
|
||||
);
|
||||
|
||||
function getRequestToken(req, tokenTypes = ['Token', 'Bearer']) {
|
||||
const authorization = req.headers.authorization;
|
||||
if (typeof authorization !== 'string') {
|
||||
return null;
|
||||
}
|
||||
|
||||
const [tokenType, token] = authorization.split(' ');
|
||||
|
||||
if (tokenTypes.includes(tokenType)) {
|
||||
return token;
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
passport.use(
|
||||
'jwt',
|
||||
new JwtStrategy(
|
||||
{
|
||||
secretOrKey: config.jwtSecret,
|
||||
jwtFromRequest: getRequestToken,
|
||||
algorithms: ['HS256'],
|
||||
},
|
||||
async function (token, done) {
|
||||
try {
|
||||
// we used to put the user ID into the token directly :(
|
||||
const { id } = token;
|
||||
const user = await User.findById(id);
|
||||
return done(null, user || false);
|
||||
} catch (err) {
|
||||
return done(err);
|
||||
}
|
||||
},
|
||||
),
|
||||
);
|
||||
|
||||
passport.use(
|
||||
'accessToken',
|
||||
new BearerStrategy(async function (token, done) {
|
||||
try {
|
||||
const accessToken = await AccessToken.findOne({ token }).populate('user');
|
||||
if (accessToken && accessToken.user) {
|
||||
// TODO: scope
|
||||
return done(null, accessToken.user, { scope: accessToken.scope });
|
||||
} else {
|
||||
return done(null, false);
|
||||
}
|
||||
} catch (err) {
|
||||
return done(err);
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
passport.use(
|
||||
'refreshToken',
|
||||
new BearerStrategy(async function (token, done) {
|
||||
try {
|
||||
const refreshToken = await RefreshToken.findOne({ token }).populate('user');
|
||||
if (refreshToken && refreshToken.user) {
|
||||
// TODO: scope
|
||||
return done(null, refreshToken.user, { scope: 'auth.refresh' });
|
||||
} else {
|
||||
return done(null, false);
|
||||
}
|
||||
} catch (err) {
|
||||
return done(err);
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
passport.use(
|
||||
'userId',
|
||||
new CustomStrategy(async (req, callback) => {
|
||||
try {
|
||||
let userId;
|
||||
|
||||
const headerToken = getRequestToken(req, ['OBSUserId']);
|
||||
if (headerToken && headerToken.length === 24) {
|
||||
userId = headerToken;
|
||||
}
|
||||
|
||||
if (!userId) {
|
||||
const bodyId = req.body && req.body.id;
|
||||
if (bodyId && bodyId.length === 24) {
|
||||
userId = bodyId;
|
||||
}
|
||||
}
|
||||
|
||||
let user;
|
||||
if (userId) {
|
||||
user = await User.findById(userId);
|
||||
}
|
||||
|
||||
callback(null, user || false);
|
||||
} catch (err) {
|
||||
callback(err);
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
/**
|
||||
* This function creates a middleware that does a passport authentication.
|
||||
*/
|
||||
function createMiddleware(strategies, required = true, session = false) {
|
||||
return (req, res, next) => {
|
||||
passport.authenticate(strategies, { session }, (err, user, info) => {
|
||||
// If this authentication produced an error, throw it. In a chain of
|
||||
// multiple strategies, errors are ignored, unless every strategy errors.
|
||||
if (err) {
|
||||
return next(err);
|
||||
}
|
||||
|
||||
// If you *must* be logged in for this action, require a user.
|
||||
if (required && !user) {
|
||||
return res.sendStatus(403);
|
||||
}
|
||||
|
||||
// Regardless of whether login is required, if you're logged in as an
|
||||
// unverified user, produce an error.
|
||||
if (user && user.needsEmailValidation) {
|
||||
return res.status(403).json({ errors: { 'E-Mail-Bestätigung': 'noch nicht erfolgt' } });
|
||||
}
|
||||
|
||||
req.user = user;
|
||||
req.scope = (info && info.scope) || '*';
|
||||
|
||||
return next();
|
||||
})(req, res, next);
|
||||
};
|
||||
}
|
||||
|
||||
module.exports = {
|
||||
// these are the standard authentication mechanisms, for when you want user
|
||||
// information in the route, and either require a login, or don't care
|
||||
optional: createMiddleware(['jwt', 'accessToken'], false),
|
||||
required: createMiddleware(['jwt', 'accessToken'], true),
|
||||
|
||||
// required to check username and passwort for generating a new token, e.g.
|
||||
// on the /users/login route, and later on oauth routes
|
||||
usernameAndPassword: createMiddleware('usernameAndPassword', true),
|
||||
|
||||
usernameAndPasswordSession: createMiddleware('usernameAndPasswordSession', false, true),
|
||||
|
||||
// will be used to verify a refresh token on the route that will exchange the
|
||||
// refresh token for a new access token (not in use yet)
|
||||
refreshToken: createMiddleware('refreshToken', true),
|
||||
|
||||
// for track upload, we still allow "userId" for a while
|
||||
requiredWithUserId: createMiddleware(['jwt', 'accessToken', 'userId'], true),
|
||||
};
|
||||
29
api/src/paths.js
Normal file
29
api/src/paths.js
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
const path = require('path');
|
||||
|
||||
const API_ROOT_DIR = path.resolve(__dirname, '../');
|
||||
|
||||
const DATA_DIR = process.env.DATA_DIR || path.resolve(__dirname, '../../data/');
|
||||
|
||||
// Contains the subtree for processing files
|
||||
const PROCESSING_DIR = path.join(DATA_DIR, 'processing');
|
||||
const PROCESSING_OUTPUT_DIR = path.join(DATA_DIR, 'processing-output');
|
||||
|
||||
// Contains the subtree for processing files, without privatization techniques,
|
||||
// used only for display of tracks to authors
|
||||
const PROCESSING_DIR_PRIVATE = path.join(DATA_DIR, 'private');
|
||||
|
||||
// Contains original track files
|
||||
const TRACKS_DIR = path.join(DATA_DIR, 'tracks');
|
||||
|
||||
// Cache directory for all obs-face calls
|
||||
const OBS_FACE_CACHE_DIR = path.join(DATA_DIR, 'obs-face-cache');
|
||||
|
||||
module.exports = {
|
||||
API_ROOT_DIR,
|
||||
DATA_DIR,
|
||||
PROCESSING_DIR,
|
||||
PROCESSING_OUTPUT_DIR,
|
||||
PROCESSING_DIR_PRIVATE,
|
||||
TRACKS_DIR,
|
||||
OBS_FACE_CACHE_DIR,
|
||||
};
|
||||
188
api/src/process_track.py
Normal file
188
api/src/process_track.py
Normal file
|
|
@ -0,0 +1,188 @@
|
|||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
import json
|
||||
|
||||
from obs.face.importer import ImportMeasurementsCsv
|
||||
from obs.face.annotate import AnnotateMeasurements
|
||||
from obs.face.filter import (
|
||||
AnonymizationMode,
|
||||
ChainFilter,
|
||||
ConfirmedFilter,
|
||||
DistanceMeasuredFilter,
|
||||
PrivacyFilter,
|
||||
PrivacyZone,
|
||||
PrivacyZonesFilter,
|
||||
RequiredFieldsFilter,
|
||||
)
|
||||
from obs.face.osm import DataSource as OSMDataSource
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def main():
|
||||
logging.basicConfig(level=logging.DEBUG, format="%(levelname)s: %(message)s")
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="processes a single track for use in the portal, "
|
||||
"using the obs.face algorithms"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"-i", "--input", required=True, action="store", help="path to input CSV file"
|
||||
)
|
||||
parser.add_argument(
|
||||
"-o", "--output", required=True, action="store", help="path to output directory"
|
||||
)
|
||||
parser.add_argument(
|
||||
"--path-cache",
|
||||
action="store",
|
||||
default=None,
|
||||
dest="cache_dir",
|
||||
help="path where the visualization data will be stored",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--settings",
|
||||
type=argparse.FileType("rt", encoding="utf-8"),
|
||||
default=None,
|
||||
help="path where the visualization data will be stored",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.cache_dir is None:
|
||||
with tempfile.TemporaryDirectory() as cache_dir:
|
||||
args.cache_dir = cache_dir
|
||||
process(args)
|
||||
else:
|
||||
process(args)
|
||||
|
||||
|
||||
def process(args):
|
||||
log.info("Loading OpenStreetMap data")
|
||||
osm = OSMDataSource(cache_dir=args.cache_dir)
|
||||
|
||||
filename_input = os.path.abspath(args.input)
|
||||
dataset_id = os.path.splitext(os.path.basename(args.input))[0]
|
||||
|
||||
os.makedirs(args.output, exist_ok=True)
|
||||
|
||||
log.info("Loading settings")
|
||||
settings = json.load(args.settings)
|
||||
|
||||
log.info("Annotating and filtering CSV file")
|
||||
measurements, statistics = ImportMeasurementsCsv().read(
|
||||
filename_input,
|
||||
user_id="dummy",
|
||||
dataset_id=dataset_id,
|
||||
)
|
||||
|
||||
measurements = AnnotateMeasurements(osm, cache_dir=args.cache_dir).annotate(
|
||||
measurements
|
||||
)
|
||||
|
||||
filters_from_settings = []
|
||||
for filter_description in settings.get("filters", []):
|
||||
filter_type = filter_description.get("type")
|
||||
if filter_type == "PrivacyZonesFilter":
|
||||
privacy_zones = [
|
||||
PrivacyZone(
|
||||
latitude=zone.get("latitude"),
|
||||
longitude=zone.get("longitude"),
|
||||
radius=zone.get("radius"),
|
||||
)
|
||||
for zone in filter_description.get("config", {}).get("privacyZones", [])
|
||||
]
|
||||
filters_from_settings.append(PrivacyZonesFilter(privacy_zones))
|
||||
else:
|
||||
log.warning("Ignoring unknown filter type %r in settings file", filter_type)
|
||||
|
||||
input_filter = ChainFilter(
|
||||
RequiredFieldsFilter(),
|
||||
PrivacyFilter(
|
||||
user_id_mode=AnonymizationMode.REMOVE,
|
||||
measurement_id_mode=AnonymizationMode.REMOVE,
|
||||
),
|
||||
*filters_from_settings,
|
||||
)
|
||||
events_filter = DistanceMeasuredFilter()
|
||||
confirmed_filter = ChainFilter(
|
||||
ConfirmedFilter(),
|
||||
)
|
||||
|
||||
track_measurements = input_filter.filter(measurements, log=log)
|
||||
event_measurements = events_filter.filter(track_measurements , log=log)
|
||||
confirmed_measurements = confirmed_filter.filter(track_measurements, log=log)
|
||||
|
||||
# write out
|
||||
confirmed_measurements_json = {
|
||||
"type": "FeatureCollection",
|
||||
"features": [
|
||||
{
|
||||
"type": "Feature",
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [m["latitude"], m["longitude"]],
|
||||
},
|
||||
"properties": {
|
||||
"distanceOvertaker": m["distance_overtaker"],
|
||||
"distanceStationary": m["distance_stationary"],
|
||||
"confirmed": True,
|
||||
},
|
||||
}
|
||||
for m in confirmed_measurements
|
||||
],
|
||||
}
|
||||
all_measurements_json = {
|
||||
"type": "FeatureCollection",
|
||||
"features": [
|
||||
{
|
||||
"type": "Feature",
|
||||
"geometry": {
|
||||
"type": "Point",
|
||||
"coordinates": [m["latitude"], m["longitude"]],
|
||||
},
|
||||
"properties": {
|
||||
"distanceOvertaker": m["distance_overtaker"],
|
||||
"distanceStationary": m["distance_stationary"],
|
||||
"confirmed": m in confirmed_measurements,
|
||||
},
|
||||
}
|
||||
for m in event_measurements
|
||||
],
|
||||
}
|
||||
|
||||
track_json = {
|
||||
"type": "Feature",
|
||||
"geometry": {
|
||||
"type": "LineString",
|
||||
"coordinates": [
|
||||
[m["latitude"], m["longitude"]] for m in track_measurements
|
||||
],
|
||||
},
|
||||
}
|
||||
|
||||
statistics_json = {
|
||||
"recordedAt": statistics["t_min"].isoformat(),
|
||||
"recordedUntil": statistics["t_max"].isoformat(),
|
||||
"duration": statistics["t"],
|
||||
"length": statistics["d"],
|
||||
"segments": statistics["n_segments"],
|
||||
"numEvents": statistics["n_confirmed"],
|
||||
"numMeasurements": statistics["n_measurements"],
|
||||
"numValid": statistics["n_valid"],
|
||||
}
|
||||
|
||||
for output_filename, data in [
|
||||
("all_measurements.json", all_measurements_json),
|
||||
("confirmed_measurements.json", confirmed_measurements_json),
|
||||
("track.json", track_json),
|
||||
("statistics.json", statistics_json),
|
||||
]:
|
||||
with open(os.path.join(args.output, output_filename), "w") as fp:
|
||||
json.dump(data, fp, indent=4)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
12
api/src/queue.js
Normal file
12
api/src/queue.js
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
const Bull = require('bull');
|
||||
|
||||
const config = require('./config');
|
||||
|
||||
module.exports = new Bull('processQueue', config.redisUrl, {
|
||||
settings: {
|
||||
// if the worker process is killed and restarted, e.g. due to reboot or
|
||||
// upgrade, it is okay to wait for a timeout on the job and restart it
|
||||
maxStalledCount: 3,
|
||||
lockDuration: 120 * 1000,
|
||||
},
|
||||
});
|
||||
25
api/src/routes/api/index.js
Normal file
25
api/src/routes/api/index.js
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
const router = require('express').Router();
|
||||
|
||||
router.use('/', require('./users'));
|
||||
router.use('/profiles', require('./profiles'));
|
||||
router.use('/tracks', require('./tracks'));
|
||||
router.use('/tags', require('./tags'));
|
||||
router.use('/accounts', require('../../accounts/accounts.controller'));
|
||||
router.use('/stats', require('./stats'));
|
||||
router.use('/info', require('./info'));
|
||||
|
||||
router.use(function (err, req, res, next) {
|
||||
if (err.name === 'ValidationError') {
|
||||
return res.status(422).json({
|
||||
errors: Object.keys(err.errors).reduce(function (errors, key) {
|
||||
errors[key] = err.errors[key].message;
|
||||
|
||||
return errors;
|
||||
}, {}),
|
||||
});
|
||||
}
|
||||
|
||||
return next(err);
|
||||
});
|
||||
|
||||
module.exports = router;
|
||||
11
api/src/routes/api/info.js
Normal file
11
api/src/routes/api/info.js
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
const router = require('express').Router();
|
||||
|
||||
const { version } = require('../../../package.json');
|
||||
|
||||
router.route('/').get((req, res) => {
|
||||
res.json({
|
||||
version,
|
||||
});
|
||||
});
|
||||
|
||||
module.exports = router;
|
||||
30
api/src/routes/api/profiles.js
Normal file
30
api/src/routes/api/profiles.js
Normal file
|
|
@ -0,0 +1,30 @@
|
|||
const router = require('express').Router();
|
||||
const wrapRoute = require('../../_helpers/wrapRoute');
|
||||
const auth = require('../../passport');
|
||||
const { User } = require('../../models');
|
||||
|
||||
// Preload user profile on routes with ':username'
|
||||
router.param('username', async function (req, res, next, username) {
|
||||
try {
|
||||
const user = await User.findOne({ username: username });
|
||||
if (!user) {
|
||||
return res.sendStatus(404);
|
||||
}
|
||||
|
||||
req.profile = user;
|
||||
|
||||
return next();
|
||||
} catch (err) {
|
||||
next(err);
|
||||
}
|
||||
});
|
||||
|
||||
router.get(
|
||||
'/:username',
|
||||
auth.optional,
|
||||
wrapRoute(async (req, res) => {
|
||||
return res.json({ profile: req.profile.toProfileJSONFor(req.user) });
|
||||
}),
|
||||
);
|
||||
|
||||
module.exports = router;
|
||||
107
api/src/routes/api/stats.js
Normal file
107
api/src/routes/api/stats.js
Normal file
|
|
@ -0,0 +1,107 @@
|
|||
const router = require('express').Router();
|
||||
const { DateTime } = require('luxon');
|
||||
|
||||
const { Track, User } = require('../../models');
|
||||
const wrapRoute = require('../../_helpers/wrapRoute');
|
||||
const auth = require('../../passport');
|
||||
|
||||
// round to this number of meters for privacy reasons
|
||||
const TRACK_LENGTH_ROUNDING = 1000;
|
||||
|
||||
// round to this number of seconds for privacy reasons
|
||||
const TRACK_DURATION_ROUNDING = 120;
|
||||
|
||||
router.get(
|
||||
'/',
|
||||
auth.optional,
|
||||
wrapRoute(async (req, res) => {
|
||||
const start = DateTime.fromISO(req.query.start);
|
||||
const end = DateTime.fromISO(req.query.end);
|
||||
|
||||
const dateFilter = {
|
||||
$ne: null,
|
||||
...(start.isValid ? { $gte: start.toJSDate() } : {}),
|
||||
...(end.isValid ? { $lt: end.toJSDate() } : {}),
|
||||
};
|
||||
|
||||
let userFilter;
|
||||
if (req.query.user) {
|
||||
const user = await User.findOne({ username: req.query.user });
|
||||
|
||||
// Only the user can look for their own stats, for now
|
||||
if (req.user && req.user._id.equals(user._id)) {
|
||||
userFilter = user._id;
|
||||
} else {
|
||||
userFilter = { $ne: null };
|
||||
}
|
||||
}
|
||||
|
||||
const trackFilter = {
|
||||
'statistics.recordedAt': dateFilter,
|
||||
...(userFilter ? { author: userFilter } : {}),
|
||||
};
|
||||
|
||||
const trackCount = await Track.find(trackFilter).count();
|
||||
|
||||
const publicTrackCount = await Track.find({
|
||||
...trackFilter,
|
||||
public: true,
|
||||
}).count();
|
||||
|
||||
const userCount = await User.find({
|
||||
...(userFilter
|
||||
? { _id: userFilter }
|
||||
: {
|
||||
createdAt: dateFilter,
|
||||
}),
|
||||
}).count();
|
||||
|
||||
const trackStats = await Track.aggregate([
|
||||
{ $match: trackFilter },
|
||||
{
|
||||
$addFields: {
|
||||
trackLength: {
|
||||
$cond: [{ $lt: ['$statistics.length', 500000] }, '$statistics.length', 0],
|
||||
},
|
||||
numEvents: '$statistics.numEvents',
|
||||
trackDuration: {
|
||||
$cond: [
|
||||
{ $and: ['$statistics.recordedUntil', { $gt: ['$statistics.recordedAt', new Date('2010-01-01')] }] },
|
||||
{ $subtract: ['$statistics.recordedUntil', '$statistics.recordedAt'] },
|
||||
0,
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
{ $project: { trackLength: true, numEvents: true, trackDuration: true } },
|
||||
{
|
||||
$group: {
|
||||
_id: 'sum',
|
||||
trackLength: { $sum: '$trackLength' },
|
||||
numEvents: { $sum: '$numEvents' },
|
||||
trackDuration: { $sum: '$trackDuration' },
|
||||
},
|
||||
},
|
||||
]);
|
||||
|
||||
const [trackLength, numEvents, trackDuration] =
|
||||
trackStats.length > 0
|
||||
? [trackStats[0].trackLength, trackStats[0].numEvents, trackStats[0].trackDuration]
|
||||
: [0, 0, 0];
|
||||
|
||||
const trackLengthPrivatized = Math.floor(trackLength / TRACK_LENGTH_ROUNDING) * TRACK_LENGTH_ROUNDING;
|
||||
const trackDurationPrivatized =
|
||||
Math.round(trackDuration / 1000 / TRACK_DURATION_ROUNDING) * TRACK_DURATION_ROUNDING;
|
||||
|
||||
return res.json({
|
||||
publicTrackCount,
|
||||
trackLength: trackLengthPrivatized,
|
||||
trackDuration: trackDurationPrivatized,
|
||||
numEvents,
|
||||
trackCount,
|
||||
userCount,
|
||||
});
|
||||
}),
|
||||
);
|
||||
|
||||
module.exports = router;
|
||||
14
api/src/routes/api/tags.js
Normal file
14
api/src/routes/api/tags.js
Normal file
|
|
@ -0,0 +1,14 @@
|
|||
const router = require('express').Router();
|
||||
const wrapRoute = require('../../_helpers/wrapRoute');
|
||||
const { Track } = require('../../models');
|
||||
|
||||
// return a list of tags
|
||||
router.get(
|
||||
'/',
|
||||
wrapRoute(async (req, res) => {
|
||||
const tags = await Track.find().distinct('tagList');
|
||||
return res.json({ tags: tags });
|
||||
}),
|
||||
);
|
||||
|
||||
module.exports = router;
|
||||
416
api/src/routes/api/tracks.js
Normal file
416
api/src/routes/api/tracks.js
Normal file
|
|
@ -0,0 +1,416 @@
|
|||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const router = require('express').Router();
|
||||
const { Track, User, Comment } = require('../../models');
|
||||
const busboy = require('connect-busboy');
|
||||
const auth = require('../../passport');
|
||||
const { normalizeUserAgent, buildObsver1 } = require('../../logic/tracks');
|
||||
const wrapRoute = require('../../_helpers/wrapRoute');
|
||||
const { PROCESSING_OUTPUT_DIR } = require('../../paths');
|
||||
|
||||
function preloadByParam(target, getValueFromParam) {
|
||||
return async (req, res, next, paramValue) => {
|
||||
try {
|
||||
const value = await getValueFromParam(paramValue);
|
||||
|
||||
if (!value) {
|
||||
return res.sendStatus(404);
|
||||
}
|
||||
|
||||
req[target] = value;
|
||||
return next();
|
||||
} catch (err) {
|
||||
return next(err);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
router.param(
|
||||
'track',
|
||||
preloadByParam('track', (slug) => Track.findOne({ slug }).populate('author')),
|
||||
);
|
||||
router.param(
|
||||
'comment',
|
||||
preloadByParam('comment', (id) => Comment.findById(id)),
|
||||
);
|
||||
|
||||
router.param('comment', async (req, res, next, id) => {
|
||||
try {
|
||||
const comment = await Comment.findById(id);
|
||||
|
||||
if (!comment) {
|
||||
return res.sendStatus(404);
|
||||
}
|
||||
|
||||
req.comment = comment;
|
||||
|
||||
return next();
|
||||
} catch (err) {
|
||||
return next(err);
|
||||
}
|
||||
});
|
||||
|
||||
router.get(
|
||||
'/',
|
||||
auth.optional,
|
||||
wrapRoute(async (req, res) => {
|
||||
const query = { public: true };
|
||||
let limit = 20;
|
||||
let offset = 0;
|
||||
|
||||
if (typeof req.query.limit !== 'undefined') {
|
||||
limit = req.query.limit;
|
||||
}
|
||||
|
||||
if (typeof req.query.offset !== 'undefined') {
|
||||
offset = req.query.offset;
|
||||
}
|
||||
|
||||
if (typeof req.query.tag !== 'undefined') {
|
||||
query.tagList = { $in: [req.query.tag] };
|
||||
}
|
||||
|
||||
const author = req.query.author ? await User.findOne({ username: req.query.author }) : null;
|
||||
|
||||
if (author) {
|
||||
query.author = author._id;
|
||||
}
|
||||
|
||||
const [tracks, tracksCount] = await Promise.all([
|
||||
Track.find(query)
|
||||
.sort('-createdAt')
|
||||
.limit(Number(limit))
|
||||
.skip(Number(offset))
|
||||
.sort({ createdAt: 'desc' })
|
||||
.populate('author')
|
||||
.exec(),
|
||||
Track.countDocuments(query).exec(),
|
||||
]);
|
||||
|
||||
return res.json({
|
||||
tracks: tracks.map((track) => track.toJSONFor(req.user)),
|
||||
tracksCount,
|
||||
});
|
||||
}),
|
||||
);
|
||||
|
||||
router.get(
|
||||
'/feed',
|
||||
auth.required,
|
||||
wrapRoute(async (req, res) => {
|
||||
let limit = 20;
|
||||
let offset = 0;
|
||||
|
||||
if (typeof req.query.limit !== 'undefined') {
|
||||
limit = req.query.limit;
|
||||
}
|
||||
|
||||
if (typeof req.query.offset !== 'undefined') {
|
||||
offset = req.query.offset;
|
||||
}
|
||||
|
||||
const query = { author: req.user.id };
|
||||
const [tracks, tracksCount] = await Promise.all([
|
||||
Track.find(query).sort('-createdAt').limit(Number(limit)).skip(Number(offset)).populate('author').exec(),
|
||||
Track.countDocuments(query),
|
||||
]);
|
||||
|
||||
return res.json({
|
||||
tracks: tracks.map(function (track) {
|
||||
return track.toJSONFor(req.user);
|
||||
}),
|
||||
tracksCount: tracksCount,
|
||||
});
|
||||
}),
|
||||
);
|
||||
|
||||
async function readFile(file) {
|
||||
let fileContent = '';
|
||||
|
||||
file.on('data', function (data) {
|
||||
fileContent += data;
|
||||
});
|
||||
|
||||
await new Promise((resolve, reject) => {
|
||||
file.on('end', resolve);
|
||||
file.on('error', reject);
|
||||
});
|
||||
|
||||
return fileContent;
|
||||
}
|
||||
|
||||
async function getMultipartOrJsonBody(req, mapJsonBody = (x) => x) {
|
||||
const fileInfo = {};
|
||||
let body;
|
||||
|
||||
if (req.busboy) {
|
||||
body = {};
|
||||
|
||||
req.busboy.on('file', async function (fieldname, file, filename, encoding, mimetype) {
|
||||
body[fieldname] = await readFile(file);
|
||||
fileInfo[fieldname] = { filename, encoding, mimetype };
|
||||
});
|
||||
|
||||
req.busboy.on('field', (key, value) => {
|
||||
body[key] = value;
|
||||
});
|
||||
|
||||
req.pipe(req.busboy);
|
||||
|
||||
await new Promise((resolve, reject) => {
|
||||
req.busboy.on('finish', resolve);
|
||||
req.busboy.on('error', reject);
|
||||
});
|
||||
} else if (req.headers['content-type'] === 'application/json') {
|
||||
body = mapJsonBody(req.body);
|
||||
} else {
|
||||
body = { body: await readFile(req), ...req.query };
|
||||
fileInfo.body = {
|
||||
mimetype: req.headers['content-type'],
|
||||
filename: req.headers['content-disposition'],
|
||||
encoding: req.headers['content-encoding'],
|
||||
};
|
||||
}
|
||||
|
||||
return { body, fileInfo };
|
||||
}
|
||||
|
||||
router.post(
|
||||
'/',
|
||||
auth.requiredWithUserId,
|
||||
busboy(), // parse multipart body
|
||||
wrapRoute(async (req, res) => {
|
||||
// Read the whole file into memory. This is not optimal, instead, we should
|
||||
// write the file data directly to the target file. However, we first have
|
||||
// to parse the rest of the track data to know where to place the file.
|
||||
// TODO: Stream into temporary file, then move it later.
|
||||
const { body, fileInfo } = await getMultipartOrJsonBody(req, (body) => body.track);
|
||||
|
||||
const { body: fileBody, public, ...trackBody } = body;
|
||||
|
||||
const track = new Track({
|
||||
...trackBody,
|
||||
author: req.user,
|
||||
public: public == null ? req.user.areTracksVisibleForAll : Boolean(trackBody.public),
|
||||
});
|
||||
track.customizedTitle = track.title != null;
|
||||
track.slugify();
|
||||
|
||||
if (fileBody) {
|
||||
await track.validateFileBodyUniqueness(fileBody);
|
||||
track.uploadedByUserAgent = normalizeUserAgent(req.headers['user-agent']);
|
||||
track.originalFileName = fileInfo.body ? fileInfo.body.filename : track.slug + '.csv';
|
||||
await track.writeToOriginalFile(fileBody);
|
||||
}
|
||||
|
||||
await track.save();
|
||||
|
||||
if (fileBody) {
|
||||
await track.queueProcessing();
|
||||
}
|
||||
|
||||
await track.autoGenerateTitle();
|
||||
|
||||
return res.json({ track: track.toJSONFor(req.user) });
|
||||
}),
|
||||
);
|
||||
|
||||
// return a track
|
||||
router.get(
|
||||
'/:track',
|
||||
auth.optional,
|
||||
wrapRoute(async (req, res) => {
|
||||
if (!req.track.isVisibleTo(req.user)) {
|
||||
return res.sendStatus(403);
|
||||
}
|
||||
|
||||
return res.json({ track: req.track.toJSONFor(req.user) });
|
||||
}),
|
||||
);
|
||||
|
||||
// update track
|
||||
router.put(
|
||||
'/:track',
|
||||
busboy(),
|
||||
auth.required,
|
||||
wrapRoute(async (req, res) => {
|
||||
const track = req.track;
|
||||
|
||||
if (!track.author._id.equals(req.user.id)) {
|
||||
return res.sendStatus(403);
|
||||
}
|
||||
|
||||
const {
|
||||
body: { body: fileBody, ...trackBody },
|
||||
fileInfo,
|
||||
} = await getMultipartOrJsonBody(req, (body) => body.track);
|
||||
|
||||
if (typeof trackBody.title !== 'undefined') {
|
||||
track.title = (trackBody.title || '').trim() || null;
|
||||
track.customizedTitle = track.title != null;
|
||||
}
|
||||
|
||||
if (typeof trackBody.description !== 'undefined') {
|
||||
track.description = (trackBody.description || '').trim() || null;
|
||||
}
|
||||
|
||||
let process = false;
|
||||
|
||||
if (trackBody.public != null) {
|
||||
const public = Boolean(trackBody.public);
|
||||
process |= public !== track.public;
|
||||
track.public = public;
|
||||
}
|
||||
|
||||
if (fileBody) {
|
||||
await track.validateFileBodyUniqueness(fileBody);
|
||||
track.originalFileName = fileInfo.body ? fileInfo.body.filename : track.slug + '.csv';
|
||||
track.uploadedByUserAgent = normalizeUserAgent(req.headers['user-agent']);
|
||||
await track.writeToOriginalFile(fileBody);
|
||||
process = true;
|
||||
}
|
||||
|
||||
await track.save();
|
||||
|
||||
if (process) {
|
||||
await track.queueProcessing();
|
||||
}
|
||||
|
||||
await track.autoGenerateTitle();
|
||||
|
||||
return res.json({ track: track.toJSONFor(req.user) });
|
||||
}),
|
||||
);
|
||||
|
||||
// delete track
|
||||
router.delete(
|
||||
'/:track',
|
||||
auth.required,
|
||||
wrapRoute(async (req, res) => {
|
||||
if (req.track.author._id.equals(req.user.id)) {
|
||||
await req.track.remove();
|
||||
return res.sendStatus(204);
|
||||
} else {
|
||||
return res.sendStatus(403);
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
// return an track's comments
|
||||
router.get(
|
||||
'/:track/comments',
|
||||
auth.optional,
|
||||
wrapRoute(async (req, res) => {
|
||||
if (!req.track.isVisibleTo(req.user)) {
|
||||
return res.sendStatus(403);
|
||||
}
|
||||
|
||||
await req.track.populate({
|
||||
path: 'comments',
|
||||
populate: {
|
||||
path: 'author',
|
||||
},
|
||||
options: {
|
||||
sort: {
|
||||
createdAt: 'asc',
|
||||
},
|
||||
},
|
||||
});
|
||||
|
||||
return res.json({
|
||||
comments: req.track.comments.map(function (comment) {
|
||||
return comment.toJSONFor(req.user);
|
||||
}),
|
||||
});
|
||||
}),
|
||||
);
|
||||
|
||||
// create a new comment
|
||||
router.post(
|
||||
'/:track/comments',
|
||||
auth.required,
|
||||
wrapRoute(async (req, res) => {
|
||||
const comment = new Comment(req.body.comment);
|
||||
comment.track = req.track;
|
||||
comment.author = req.user;
|
||||
|
||||
await comment.save();
|
||||
|
||||
req.track.comments.push(comment);
|
||||
|
||||
await req.track.save();
|
||||
return res.json({ comment: comment.toJSONFor(req.user) });
|
||||
}),
|
||||
);
|
||||
|
||||
router.delete(
|
||||
'/:track/comments/:comment',
|
||||
auth.required,
|
||||
wrapRoute(async (req, res) => {
|
||||
if (req.comment.author.equals(req.user.id)) {
|
||||
req.track.comments.remove(req.comment._id);
|
||||
await req.track.save();
|
||||
await Comment.find({ _id: req.comment._id }).remove();
|
||||
res.sendStatus(204);
|
||||
} else {
|
||||
res.sendStatus(403);
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
// return an track's generated data
|
||||
router.get(
|
||||
'/:track/data',
|
||||
auth.optional,
|
||||
wrapRoute(async (req, res) => {
|
||||
const FILE_BY_KEY = {
|
||||
allMeasurements: 'all_measurements.json',
|
||||
confirmedMeasurements: 'confirmed_measurements.json',
|
||||
track: 'track.json',
|
||||
};
|
||||
|
||||
if (!req.track.isVisibleTo(req.user)) {
|
||||
return res.sendStatus(403);
|
||||
}
|
||||
|
||||
const result = {};
|
||||
for (const [key, filename] of Object.entries(FILE_BY_KEY)) {
|
||||
const filePath = path.join(PROCESSING_OUTPUT_DIR, req.track.filePath, filename);
|
||||
|
||||
let stats;
|
||||
|
||||
try {
|
||||
stats = await fs.promises.stat(filePath);
|
||||
} catch (err) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!stats.isFile()) {
|
||||
// file does not exist (yet)
|
||||
continue;
|
||||
}
|
||||
|
||||
const content = await fs.promises.readFile(filePath);
|
||||
const contentJson = JSON.parse(content);
|
||||
|
||||
result[key] = contentJson;
|
||||
}
|
||||
|
||||
return res.json(result);
|
||||
}),
|
||||
);
|
||||
|
||||
// download the original file
|
||||
router.get(
|
||||
'/:track/download/original.csv',
|
||||
auth.optional,
|
||||
wrapRoute(async (req, res) => {
|
||||
if (!req.track.isVisibleToPrivate(req.user)) {
|
||||
return res.sendStatus(403);
|
||||
}
|
||||
|
||||
return res.download(req.track.getOriginalFilePath(), req.track.originalFileName);
|
||||
}),
|
||||
);
|
||||
|
||||
module.exports = router;
|
||||
52
api/src/routes/api/users.js
Normal file
52
api/src/routes/api/users.js
Normal file
|
|
@ -0,0 +1,52 @@
|
|||
const router = require('express').Router();
|
||||
const wrapRoute = require('../../_helpers/wrapRoute');
|
||||
const auth = require('../../passport');
|
||||
|
||||
router.get(
|
||||
'/user',
|
||||
auth.required,
|
||||
wrapRoute(async (req, res) => {
|
||||
return res.json({ user: req.user.toAuthJSON() });
|
||||
}),
|
||||
);
|
||||
|
||||
router.put(
|
||||
'/user',
|
||||
auth.required,
|
||||
wrapRoute(async (req, res) => {
|
||||
const user = req.user;
|
||||
|
||||
// only update fields that were actually passed...
|
||||
if (typeof req.body.user.username !== 'undefined') {
|
||||
user.username = req.body.user.username;
|
||||
}
|
||||
if (typeof req.body.user.email !== 'undefined') {
|
||||
user.email = req.body.user.email;
|
||||
}
|
||||
if (typeof req.body.user.bio !== 'undefined') {
|
||||
user.bio = req.body.user.bio;
|
||||
}
|
||||
if (typeof req.body.user.image !== 'undefined') {
|
||||
user.image = req.body.user.image;
|
||||
}
|
||||
if (typeof req.body.user.areTracksVisibleForAll !== 'undefined') {
|
||||
user.areTracksVisibleForAll = req.body.user.areTracksVisibleForAll;
|
||||
}
|
||||
if (typeof req.body.user.password === 'string' && req.body.user.password !== '') {
|
||||
user.setPassword(req.body.user.password);
|
||||
}
|
||||
|
||||
await user.save();
|
||||
return res.json({ user: user.toAuthJSON() });
|
||||
}),
|
||||
);
|
||||
|
||||
// Remove this at some point
|
||||
router.post('/users/login',
|
||||
auth.usernameAndPassword,
|
||||
wrapRoute((req, res) => {
|
||||
return res.json({ user: req.user.toAuthJSON() });
|
||||
}),
|
||||
);
|
||||
|
||||
module.exports = router;
|
||||
558
api/src/routes/auth.js
Normal file
558
api/src/routes/auth.js
Normal file
|
|
@ -0,0 +1,558 @@
|
|||
const router = require('express').Router();
|
||||
const passport = require('passport');
|
||||
const { URL } = require('url');
|
||||
const { createChallenge } = require('pkce');
|
||||
|
||||
const { AuthorizationCode, AccessToken, RefreshToken } = require('../models');
|
||||
const auth = require('../passport');
|
||||
const wrapRoute = require('../_helpers/wrapRoute');
|
||||
const config = require('../config');
|
||||
|
||||
const baseUrl = config.baseUrl.replace(/\/+$/, '');
|
||||
|
||||
// Check whether the "bigScope" fully includes the "smallScope".
|
||||
function scopeIncludes(smallScope, bigScope) {
|
||||
const smallScopeParts = smallScope.split(/\s/);
|
||||
const bigScopeParts = bigScope.split(/\s/);
|
||||
return bigScopeParts.includes('*') || smallScopeParts.every((part) => bigScopeParts.includes(part));
|
||||
}
|
||||
|
||||
function returnError(res, error, errorDescription = undefined, status = 400) {
|
||||
return res
|
||||
.status(status)
|
||||
.json({ error, ...(errorDescription != null ? { error_description: errorDescription } : {}) });
|
||||
}
|
||||
|
||||
function redirectWithParams(res, redirectUri, params) {
|
||||
const targetUrl = new URL(redirectUri);
|
||||
for (const [key, value] of Object.entries(params)) {
|
||||
targetUrl.searchParams.append(key, value);
|
||||
}
|
||||
return res.redirect(targetUrl.toString());
|
||||
}
|
||||
|
||||
const ALL_SCOPE_NAMES = `
|
||||
tracks.create
|
||||
tracks.update
|
||||
tracks.list
|
||||
tracks.show
|
||||
tracks.delete
|
||||
users.update
|
||||
users.show
|
||||
tracks.comments.create
|
||||
tracks.comments.update
|
||||
tracks.comments.list
|
||||
tracks.comments.show
|
||||
`.split(/\s/);
|
||||
|
||||
function isValidScope(scope) {
|
||||
return scope === '*' || scopeIncludes(scope, ALL_SCOPE_NAMES.join(' '));
|
||||
}
|
||||
|
||||
router.use((req, res, next) => {
|
||||
res.locals.user = req.user;
|
||||
res.locals.mainFrontendUrl = config.mainFrontendUrl;
|
||||
res.locals.imprintUrl = config.imprintUrl;
|
||||
res.locals.privacyPolicyUrl = config.privacyPolicyUrl;
|
||||
res.locals.baseUrl = baseUrl + '/';
|
||||
next();
|
||||
});
|
||||
|
||||
router.post(
|
||||
'/login',
|
||||
passport.authenticate('usernameAndPasswordSession', { session: true }),
|
||||
(err, req, res, next) => {
|
||||
if (!err) {
|
||||
next();
|
||||
}
|
||||
|
||||
if (err.message === 'invalid credentials') {
|
||||
return res.render('login', { badCredentials: true });
|
||||
}
|
||||
|
||||
let description = 'Unknown error while processing your login.';
|
||||
if (err.message === 'email not verified') {
|
||||
description = 'Your account is not yet verified, please check your email or start the password recovery.';
|
||||
}
|
||||
|
||||
return res.render('message', { type: 'negative', title: 'Login failed', description });
|
||||
},
|
||||
wrapRoute((req, res, next) => {
|
||||
if (!req.user) {
|
||||
return res.redirect(baseUrl + '/login');
|
||||
}
|
||||
|
||||
if (req.session.next) {
|
||||
res.redirect(baseUrl + req.session.next);
|
||||
req.session.next = null;
|
||||
return;
|
||||
}
|
||||
return res.render('message', { type: 'positive', title: 'You are logged in.', showFrontendLink: true });
|
||||
}),
|
||||
);
|
||||
|
||||
router.get(
|
||||
'/login',
|
||||
wrapRoute(async (req, res) => {
|
||||
if (req.user) {
|
||||
return res.render('message', { type: 'positive', title: 'You are already logged in.' });
|
||||
}
|
||||
|
||||
return res.render('login');
|
||||
}),
|
||||
);
|
||||
|
||||
router
|
||||
.route('/logout')
|
||||
.post(
|
||||
auth.usernameAndPasswordSession,
|
||||
wrapRoute(async (req, res) => {
|
||||
req.logout();
|
||||
return res.redirect(baseUrl + '/login');
|
||||
}),
|
||||
)
|
||||
.get((req, res) => {
|
||||
if (req.query.redirectTo) {
|
||||
req.logout();
|
||||
return res.redirect(req.query.redirectTo);
|
||||
}
|
||||
|
||||
return res.render('logout')
|
||||
});
|
||||
|
||||
const isIp = (ip) =>
|
||||
typeof ip === 'string' &&
|
||||
/^([0-9]{1,3}\.)[0-9]{1,3}$/.test(ip) &&
|
||||
ip
|
||||
.split('.')
|
||||
.every(
|
||||
(num, idx) =>
|
||||
!num.startsWith('0') && Number(num) > (idx === 0 || idx === 3 ? 1 : 0) && Number(num) < (idx === 3 ? 254 : 255),
|
||||
);
|
||||
|
||||
const isLocalIp = (ip) => isIp(ip) && (ip.startsWith('10.') || ip.startsWith('172.16.') || ip.startsWith('192.168.'));
|
||||
|
||||
const isValidRedirectUriFor = (redirectUri) => (redirectUriPattern) => {
|
||||
// Here we have an exception to the security requirements demanded by
|
||||
// https://tools.ietf.org/html/draft-ietf-oauth-security-topics-16#section-2.1,
|
||||
// namely, that we do not always require fully specified redirect URIs. This
|
||||
// is because we cannot know beforehand which IP the OBS will be running at.
|
||||
// But since it is usually accessed via local IP, we can allow all local IP
|
||||
// ranges. This special case must only be used in clients that have a very
|
||||
// restricted `maxScope` as well, to prevent misuse should an attack through
|
||||
// this be successful.
|
||||
// This special case does however enforce TLS ("https://"), for it prevents
|
||||
// usage in a non-TLS-secured web server. At least passive sniffing of the
|
||||
// token is not possible then. A self-signed and manually verified
|
||||
// certificate should be used for this (though usually we cannot enforce the
|
||||
// actual verification).
|
||||
if (redirectUriPattern === '__LOCAL__') {
|
||||
const url = new URL(redirectUri);
|
||||
if (url.protocol === 'https:' && isLocalIp(url.host) && !url.search && !url.hash) {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
} else {
|
||||
return redirectUriPattern === redirectUri;
|
||||
}
|
||||
};
|
||||
|
||||
router.get(
|
||||
'/authorize',
|
||||
passport.authenticate('session'),
|
||||
wrapRoute(async (req, res) => {
|
||||
if (!req.user) {
|
||||
req.session.next = req.url;
|
||||
return res.redirect(baseUrl + '/login');
|
||||
}
|
||||
|
||||
try {
|
||||
const {
|
||||
client_id: clientId,
|
||||
redirect_uri: redirectUri,
|
||||
response_type: responseType,
|
||||
scope = '*', // fallback to "all" scope
|
||||
// for PKCE
|
||||
code_challenge: codeChallenge,
|
||||
code_challenge_method: codeChallengeMethod,
|
||||
} = req.query;
|
||||
|
||||
// 1. Find our client and check if it exists
|
||||
if (!clientId) {
|
||||
return returnError(res, 'invalid_request', 'client_id parameter required');
|
||||
}
|
||||
|
||||
const client = config.oAuth2Clients.find((c) => c.clientId === clientId);
|
||||
if (!client) {
|
||||
return returnError(res, 'invalid_client', 'unknown client');
|
||||
}
|
||||
|
||||
// 2. Check that we have a redirect_uri. In addition to [RFC6749] we
|
||||
// *always* require a redirect_uri.
|
||||
if (!redirectUri) {
|
||||
return returnError(res, 'invalid_request', 'redirect_uri parameter required');
|
||||
}
|
||||
|
||||
// We enforce that the redirectUri exactly matches one of the provided URIs
|
||||
const check = isValidRedirectUriFor(redirectUri);
|
||||
if (!client.validRedirectUris.some(check)) {
|
||||
return returnError(res, 'invalid_request', 'invalid redirect_uri');
|
||||
}
|
||||
|
||||
// 3. Find out which type of response to use. [RFC6749] requires one of
|
||||
// "code" or "token", but "token" is implicit grant and we do not support
|
||||
// that.
|
||||
|
||||
if (responseType !== 'code') {
|
||||
return redirectWithParams(res, redirectUri, {
|
||||
error: 'unsupported_grant_type',
|
||||
error_description: 'only authorization code flow with PKCE is supported by this server',
|
||||
});
|
||||
}
|
||||
|
||||
// 4. Verify we're using PKCE with supported (S256) code_challenge_method.
|
||||
if (!codeChallenge) {
|
||||
return redirectWithParams(res, redirectUri, {
|
||||
error: 'invalid_request',
|
||||
error_description: 'a code_challenge for PKCE is required',
|
||||
});
|
||||
}
|
||||
|
||||
if (codeChallengeMethod !== 'S256') {
|
||||
return redirectWithParams(res, redirectUri, {
|
||||
error: 'invalid_request',
|
||||
error_description: 'the code_challenge_method for PKCE must be "S256"',
|
||||
});
|
||||
}
|
||||
|
||||
// 5. Get the scope.
|
||||
if (!isValidScope(scope)) {
|
||||
return redirectWithParams(res, redirectUri, {
|
||||
error: 'invalid_scope',
|
||||
error_description: 'the requested scope is not known',
|
||||
});
|
||||
}
|
||||
|
||||
if (client.maxScope && !scopeIncludes(scope, client.maxScope)) {
|
||||
return redirectWithParams(res, redirectUri, {
|
||||
error: 'access_denied',
|
||||
error_description: 'the requested scope is not valid for this client',
|
||||
});
|
||||
}
|
||||
|
||||
// Ok, let's save all this in the session, and show a dialog for the
|
||||
// decision to the user.
|
||||
//
|
||||
if (client.autoAccept) {
|
||||
const code = AuthorizationCode.generate({
|
||||
clientId,
|
||||
user: req.user,
|
||||
redirectUri,
|
||||
scope,
|
||||
codeChallenge,
|
||||
});
|
||||
await code.save();
|
||||
|
||||
return redirectWithParams(res, redirectUri, { code: code.code, scope });
|
||||
} else {
|
||||
req.session.authorizationTransaction = {
|
||||
responseType,
|
||||
clientId,
|
||||
redirectUri,
|
||||
scope,
|
||||
expiresAt: new Date().getTime() + 1000 * 60 * 2, // 2 minute decision time
|
||||
codeChallenge,
|
||||
};
|
||||
|
||||
res.render('authorize', { clientTitle: client.title, scope, redirectUri });
|
||||
}
|
||||
} catch (err) {
|
||||
res.status(400).json({ error: 'invalid_request', error_description: 'unknown error' });
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
router.post(
|
||||
['/authorize/approve', '/authorize/decline'],
|
||||
passport.authenticate('session'),
|
||||
wrapRoute(async (req, res) => {
|
||||
if (!req.session.authorizationTransaction) {
|
||||
return res.sendStatus(400);
|
||||
}
|
||||
|
||||
if (!req.user) {
|
||||
return res.sendStatus(400);
|
||||
}
|
||||
|
||||
const { clientId, redirectUri, scope, expiresAt, codeChallenge } = req.session.authorizationTransaction;
|
||||
|
||||
if (expiresAt < new Date().getTime()) {
|
||||
return res.status(400).render('message', {
|
||||
type: 'negative',
|
||||
title: 'Expired',
|
||||
description: 'Your authorization has expired. Please go back and retry the process.',
|
||||
});
|
||||
}
|
||||
|
||||
// invalidate the transaction
|
||||
req.session.authorizationTransaction = null;
|
||||
|
||||
if (req.path === '/authorize/approve') {
|
||||
const code = AuthorizationCode.generate({
|
||||
clientId,
|
||||
user: req.user,
|
||||
redirectUri,
|
||||
scope,
|
||||
codeChallenge,
|
||||
});
|
||||
await code.save();
|
||||
|
||||
return redirectWithParams(res, redirectUri, { code: code.code, scope });
|
||||
} else {
|
||||
return redirectWithParams(res, redirectUri, { error: 'access_denied' });
|
||||
}
|
||||
}),
|
||||
);
|
||||
|
||||
/**
|
||||
* This function is called when the client presents an authorization code
|
||||
* (generated above) and wants it turned into an access (and possibly refresh)
|
||||
* token.
|
||||
*/
|
||||
router.get(
|
||||
'/token',
|
||||
wrapRoute(async (req, res) => {
|
||||
const {
|
||||
grant_type: grantType,
|
||||
code,
|
||||
client_id: clientId,
|
||||
redirect_uri: redirectUri,
|
||||
// for PKCE
|
||||
code_verifier: codeVerifier,
|
||||
} = req.query;
|
||||
|
||||
if (!grantType || grantType !== 'authorization_code') {
|
||||
return returnError(
|
||||
res,
|
||||
'unsupported_grant_type',
|
||||
'only authorization code flow with PKCE is supported by this server',
|
||||
);
|
||||
}
|
||||
|
||||
if (!code) {
|
||||
return returnError(res, 'invalid_request', 'code parameter required');
|
||||
}
|
||||
|
||||
// Call this function to destroy the authorization code (if it exists),
|
||||
// invalidating it when a single failed request has been received. The
|
||||
// whole process must be restarted. No trial and error ;)
|
||||
const destroyAuthCode = async () => {
|
||||
await AuthorizationCode.deleteOne({ code });
|
||||
};
|
||||
|
||||
if (!clientId) {
|
||||
await destroyAuthCode();
|
||||
return returnError(res, 'invalid_client', 'client_id parameter required');
|
||||
}
|
||||
|
||||
if (!redirectUri) {
|
||||
await destroyAuthCode();
|
||||
return returnError(res, 'invalid_request', 'redirect_uri parameter required');
|
||||
}
|
||||
|
||||
if (!codeVerifier) {
|
||||
await destroyAuthCode();
|
||||
return returnError(res, 'invalid_request', 'code_verifier parameter required');
|
||||
}
|
||||
|
||||
const client = config.oAuth2Clients.find((c) => c.clientId === clientId);
|
||||
|
||||
if (!client) {
|
||||
await destroyAuthCode();
|
||||
return returnError(res, 'invalid_client', 'invalid client_id');
|
||||
}
|
||||
|
||||
const authorizationCode = await AuthorizationCode.findOne({ code });
|
||||
if (!authorizationCode) {
|
||||
await destroyAuthCode();
|
||||
return returnError(res, 'invalid_grant', 'invalid authorization code');
|
||||
}
|
||||
if (authorizationCode.redirectUri !== redirectUri) {
|
||||
await destroyAuthCode();
|
||||
return returnError(res, 'invalid_grant', 'invalid authorization code');
|
||||
}
|
||||
if (authorizationCode.expiresAt <= new Date().getTime()) {
|
||||
await destroyAuthCode();
|
||||
return returnError(res, 'invalid_grant', 'invalid authorization code');
|
||||
}
|
||||
if (clientId !== authorizationCode.clientId) {
|
||||
await destroyAuthCode();
|
||||
return returnError(res, 'invalid_grant', 'invalid authorization code');
|
||||
}
|
||||
if (createChallenge(codeVerifier) !== authorizationCode.codeChallenge) {
|
||||
await destroyAuthCode();
|
||||
return returnError(res, 'invalid_grant', 'invalid authorization code');
|
||||
}
|
||||
|
||||
// invalidate auth code now, before generating tokens
|
||||
await AuthorizationCode.deleteOne({ _id: authorizationCode._id });
|
||||
|
||||
const accessToken = AccessToken.generate({
|
||||
clientId: authorizationCode.clientId,
|
||||
user: authorizationCode.user,
|
||||
scope: authorizationCode.scope,
|
||||
});
|
||||
await accessToken.save();
|
||||
|
||||
let refreshToken;
|
||||
if (client.refreshTokenExpirySeconds != null) {
|
||||
refreshToken = RefreshToken.generate(
|
||||
{
|
||||
clientId: authorizationCode.clientId,
|
||||
user: authorizationCode.user,
|
||||
scope: authorizationCode.scope,
|
||||
},
|
||||
client.refreshTokenExpirySeconds,
|
||||
);
|
||||
await refreshToken.save();
|
||||
}
|
||||
|
||||
return res.json({
|
||||
access_token: accessToken.token,
|
||||
token_type: 'Bearer',
|
||||
expires_in: Math.round((accessToken.expiresAt - new Date().getTime()) / 1000),
|
||||
scope: accessToken.scope,
|
||||
...(refreshToken != null ? { refresh_token: refreshToken.token } : {}),
|
||||
});
|
||||
}),
|
||||
);
|
||||
|
||||
/**
|
||||
* Metadata endpoint to inform clients about authorization server capabilities,
|
||||
* according to https://tools.ietf.org/html/rfc8414.
|
||||
*/
|
||||
router.get(
|
||||
'/.well-known/oauth-authorization-server',
|
||||
wrapRoute(async (req, res) => {
|
||||
return res.json({
|
||||
issuer: baseUrl,
|
||||
authorization_endpoint: `${baseUrl}/authorize`,
|
||||
token_endpoint: `${baseUrl}/token`,
|
||||
token_endpoint_auth_methods_supported: ['none'], // only public clients
|
||||
userinfo_endpoint: `${baseUrl}/api/user`,
|
||||
// registration_endpoint: `${baseUrl}/register`, // TODO
|
||||
// scopes_supported: ALL_SCOPE_NAMES, // TODO
|
||||
response_types_supported: ['code'], // only auth code, no implicit flow or
|
||||
service_documentation: 'https://github.com/openbikesensor/portal',
|
||||
ui_locales_supported: ['en-US', 'en-GB', 'en-CA', 'fr-FR', 'fr-CA'],
|
||||
code_challenge_methods_supported: ['S256'],
|
||||
});
|
||||
}),
|
||||
);
|
||||
|
||||
module.exports = router;
|
||||
|
||||
const accountService = require('../accounts/account.service');
|
||||
const validateRequest = require('../_middleware/validate-request');
|
||||
const Joi = require('joi');
|
||||
|
||||
router
|
||||
.route('/register')
|
||||
.post(
|
||||
validateRequest(
|
||||
Joi.object({
|
||||
username: Joi.string().required(),
|
||||
email: Joi.string().email().required(),
|
||||
password: Joi.string().min(6).required(),
|
||||
confirmPassword: Joi.string().valid(Joi.ref('password')).required(),
|
||||
|
||||
...(config.privacyPolicyUrl ? {
|
||||
acceptPrivacyPolicy: Joi.boolean().truthy().required(),
|
||||
} : {}),
|
||||
}),
|
||||
),
|
||||
wrapRoute(async (req, res) => {
|
||||
await accountService.register(req.body);
|
||||
|
||||
return res.render('message', {
|
||||
type: 'positive',
|
||||
title: 'Registration successful',
|
||||
description: 'Please check your email for verification instructions.',
|
||||
});
|
||||
}),
|
||||
)
|
||||
.get((req, res) => res.render('register'));
|
||||
|
||||
router.get(
|
||||
'/verify-email',
|
||||
validateRequest(
|
||||
Joi.object({
|
||||
token: Joi.string().required(),
|
||||
}),
|
||||
'query',
|
||||
),
|
||||
wrapRoute(async (req, res) => {
|
||||
await accountService.verifyEmail(req.query);
|
||||
return res.render('message', {
|
||||
type: 'positive',
|
||||
title: 'Verification successful',
|
||||
description: 'You can now log in.',
|
||||
showLoginButton: true,
|
||||
});
|
||||
}),
|
||||
);
|
||||
|
||||
router
|
||||
.route('/forgot-password')
|
||||
.post(
|
||||
validateRequest(
|
||||
Joi.object({
|
||||
email: Joi.string().email().required(),
|
||||
}),
|
||||
),
|
||||
wrapRoute(async (req, res) => {
|
||||
await accountService.forgotPassword(req.body);
|
||||
res.render('message', {
|
||||
type: 'positive',
|
||||
title: 'Recovery mail sent',
|
||||
description: 'Please check your inbox for password recovery instructions.',
|
||||
});
|
||||
}),
|
||||
)
|
||||
.get((req, res) => res.render('forgot-password'));
|
||||
|
||||
router
|
||||
.route('/reset-password')
|
||||
.post(
|
||||
validateRequest(
|
||||
Joi.object({
|
||||
token: Joi.string().required(),
|
||||
password: Joi.string().min(6).required(),
|
||||
confirmPassword: Joi.string().valid(Joi.ref('password')).required(),
|
||||
}),
|
||||
),
|
||||
wrapRoute(async (req, res) => {
|
||||
await accountService.resetPassword(req.body);
|
||||
return res.render('message', {
|
||||
type: 'positive',
|
||||
title: 'Password reset successful',
|
||||
description: 'You can now log in.',
|
||||
showLoginButton: true,
|
||||
});
|
||||
}),
|
||||
)
|
||||
.get(
|
||||
validateRequest(
|
||||
Joi.object({
|
||||
token: Joi.string().required(),
|
||||
}),
|
||||
'query',
|
||||
),
|
||||
wrapRoute(async (req, res) => {
|
||||
const { token } = req.query;
|
||||
await accountService.validateResetToken({ token });
|
||||
res.render('reset-password', { token });
|
||||
}),
|
||||
);
|
||||
|
||||
module.exports = router;
|
||||
8
api/src/routes/index.js
Normal file
8
api/src/routes/index.js
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
const router = require('express').Router();
|
||||
|
||||
router.use('/api', require('./api'));
|
||||
|
||||
// no prefix
|
||||
router.use(require('./auth'));
|
||||
|
||||
module.exports = router;
|
||||
133
api/src/worker.js
Normal file
133
api/src/worker.js
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const { spawn } = require('child_process');
|
||||
|
||||
const queue = require('./queue');
|
||||
require('./db');
|
||||
const { Track } = require('./models');
|
||||
const { API_ROOT_DIR, PROCESSING_DIR, OBS_FACE_CACHE_DIR, PROCESSING_OUTPUT_DIR } = require('./paths');
|
||||
|
||||
queue.process('processTrack', async (job) => {
|
||||
const track = await Track.findById(job.data.trackId);
|
||||
if (!track) {
|
||||
throw new Error('Cannot find track to process');
|
||||
}
|
||||
|
||||
if (track.processingJobId !== job.id) {
|
||||
throw new Error('Track is processed by another job');
|
||||
}
|
||||
|
||||
if (track.processingJobId !== job.id) {
|
||||
throw new Error('Track is processed by another job');
|
||||
}
|
||||
|
||||
if (track.processingStatus !== 'pending') {
|
||||
throw new Error('Track is not pending processing');
|
||||
}
|
||||
|
||||
try {
|
||||
const { filePath } = track;
|
||||
console.log('Will process track', filePath);
|
||||
|
||||
track.processingStatus = 'processing';
|
||||
track.processingLog = '';
|
||||
await track.save();
|
||||
|
||||
// Create input directory
|
||||
const inputDirectory = path.join(PROCESSING_DIR, filePath);
|
||||
await fs.promises.mkdir(inputDirectory, { recursive: true });
|
||||
|
||||
// copy original file to processing dir
|
||||
const inputFilePath = path.join(inputDirectory, 'track.csv');
|
||||
const originalFilePath = track.getOriginalFilePath();
|
||||
console.log(`[${track.slug}] Copy ${originalFilePath} to ${inputFilePath}`);
|
||||
await fs.promises.copyFile(originalFilePath, inputFilePath);
|
||||
|
||||
// create track settings file
|
||||
const settingsFilePath = path.join(inputDirectory, 'track-settings.json');
|
||||
console.log(`[${track.slug}] Create settings at ${settingsFilePath}`);
|
||||
const settings = {
|
||||
settingsGeneratedAt: new Date().getTime(),
|
||||
filters: [
|
||||
// TODO: Add actual privacy zones from user database
|
||||
/* {
|
||||
type: 'PrivacyZonesFilter',
|
||||
config: { privacyZones: [{ longitude: 10, latitude: 10, radius: 250 }] },
|
||||
}, */
|
||||
],
|
||||
};
|
||||
await fs.promises.writeFile(settingsFilePath, JSON.stringify(settings));
|
||||
|
||||
// Create output directory
|
||||
const outputDirectory = path.join(PROCESSING_OUTPUT_DIR, filePath);
|
||||
await fs.promises.mkdir(outputDirectory, { recursive: true });
|
||||
|
||||
const stdoutFile = path.join(outputDirectory, 'stdout.log');
|
||||
const stderrFile = path.join(outputDirectory, 'stderr.log');
|
||||
const stdout = fs.createWriteStream(stdoutFile);
|
||||
const stderr = fs.createWriteStream(stderrFile);
|
||||
|
||||
// TODO: Generate track transformation settings (privacy zones etc)
|
||||
// const settingsFilePath = path.join(inputDirectory, 'track-settings.json');
|
||||
//
|
||||
const child = spawn(
|
||||
'python',
|
||||
[
|
||||
path.join(API_ROOT_DIR, 'src', 'process_track.py'),
|
||||
'--input',
|
||||
inputFilePath,
|
||||
'--output',
|
||||
outputDirectory,
|
||||
'--path-cache',
|
||||
OBS_FACE_CACHE_DIR,
|
||||
'--settings',
|
||||
settingsFilePath,
|
||||
// '--anonymize-user-id', 'remove',
|
||||
// '--anonymize-measurement-id', 'remove',
|
||||
],
|
||||
{
|
||||
cwd: PROCESSING_DIR,
|
||||
},
|
||||
);
|
||||
|
||||
child.stdout.pipe(process.stdout);
|
||||
child.stdout.pipe(stdout);
|
||||
child.stderr.pipe(process.stderr);
|
||||
child.stderr.pipe(stderr);
|
||||
|
||||
const code = await new Promise((resolve) => child.on('close', resolve));
|
||||
|
||||
track.processingLog += (
|
||||
await Promise.all([
|
||||
fs.promises.readFile(stdoutFile),
|
||||
fs.promises.readFile(stderrFile),
|
||||
// split lines
|
||||
])
|
||||
)
|
||||
.join('\n')
|
||||
.trim();
|
||||
|
||||
if (code !== 0) {
|
||||
throw new Error(`Track processing failed with status ${code}`);
|
||||
}
|
||||
|
||||
// Read some results back into the database for quick access and
|
||||
// accumulation
|
||||
const statisticsContent = await fs.promises.readFile(path.join(outputDirectory, 'statistics.json'));
|
||||
track.statistics = JSON.parse(statisticsContent);
|
||||
|
||||
track.processingStatus = 'complete';
|
||||
await track.save();
|
||||
|
||||
// Maybe we have found out the recording date, regenerate the automatic
|
||||
// title (if not yet manually set)
|
||||
await track.autoGenerateTitle();
|
||||
} catch (err) {
|
||||
console.error('Processing failed:', err);
|
||||
track.processingLog += String(err) + '\n' + err.stack + '\n';
|
||||
track.processingStatus = 'error';
|
||||
await track.save();
|
||||
}
|
||||
});
|
||||
|
||||
console.log('Worker started.');
|
||||
|
|
@ -1,188 +0,0 @@
|
|||
#!/usr/bin/env python3
|
||||
import argparse
|
||||
import asyncio
|
||||
import logging
|
||||
import json
|
||||
from datetime import datetime
|
||||
from uuid import uuid4
|
||||
|
||||
from sqlalchemy import select
|
||||
|
||||
from motor.motor_asyncio import AsyncIOMotorClient
|
||||
|
||||
from obs.api.db import make_session, connect_db, User, Track, Comment
|
||||
from obs.api.app import app
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def main():
|
||||
logging.basicConfig(level=logging.DEBUG, format="%(levelname)s: %(message)s")
|
||||
|
||||
parser = argparse.ArgumentParser(
|
||||
description="processes a single track for use in the portal, "
|
||||
"using the obs.face algorithms"
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"mongodb_url",
|
||||
metavar="MONGODB_URL",
|
||||
help="url to the mongodb, in format mongodb://user:pass@host/dbname",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--keycloak-users-file",
|
||||
metavar="FILE",
|
||||
type=argparse.FileType("wt", encoding="utf-8"),
|
||||
help="a file to write a JSON of all old users to, for importing to keycloak",
|
||||
default=None,
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--keep-api-keys",
|
||||
action="store_true",
|
||||
help="keep the old API keys (very insecure!) instead of generating new ones",
|
||||
default=False,
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
if args.keep_api_keys:
|
||||
log.warning(
|
||||
"Importing users with their old API keys. These keys are very insecure and "
|
||||
"could provide access to user data to third parties. Consider to notify "
|
||||
"your users about the need to generate a new API key through their profile pages."
|
||||
)
|
||||
|
||||
async with connect_db(app.config.POSTGRES_URL):
|
||||
async with make_session() as session:
|
||||
mongo = AsyncIOMotorClient(args.mongodb_url).get_default_database()
|
||||
|
||||
log.debug("Connected to mongodb and postgres.")
|
||||
user_id_map = await import_users(
|
||||
mongo, session, args.keycloak_users_file, args.keep_api_keys
|
||||
)
|
||||
|
||||
await import_tracks(mongo, session, user_id_map)
|
||||
|
||||
await session.commit()
|
||||
|
||||
|
||||
async def import_users(mongo, session, keycloak_users_file, keep_api_keys):
|
||||
keycloak_users = []
|
||||
|
||||
old_id_by_email = {}
|
||||
async for user in mongo.users.find({}):
|
||||
old_id_by_email[user["email"]] = user["_id"]
|
||||
|
||||
new_user = User(
|
||||
sub=str(uuid4()),
|
||||
email=user["email"],
|
||||
username=user["username"],
|
||||
bio=user.get("bio"),
|
||||
image=user.get("image"),
|
||||
are_tracks_visible_for_all=user.get("areTracksVisibleForAll") or False,
|
||||
created_at=user.get("createdAt") or datetime.utcnow(),
|
||||
updated_at=user.get("updatedAt") or datetime.utcnow(),
|
||||
match_by_username_email=True,
|
||||
)
|
||||
|
||||
if keep_api_keys:
|
||||
new_user.api_key = str(user["_id"])
|
||||
else:
|
||||
new_user.generate_api_key()
|
||||
|
||||
if keycloak_users_file:
|
||||
needs_email_verification = user.get("needsEmailValidation", True)
|
||||
required_actions = ["UPDATE_PASSWORD"]
|
||||
if needs_email_verification:
|
||||
required_actions.append("VERIFY_EMAIL")
|
||||
|
||||
keycloak_users.append(
|
||||
{
|
||||
"username": new_user.username,
|
||||
"email": new_user.email,
|
||||
"enabled": True,
|
||||
"requiredActions": required_actions,
|
||||
"emailVerified": not needs_email_verification,
|
||||
}
|
||||
)
|
||||
|
||||
session.add(new_user)
|
||||
log.info("Creating user %s", new_user.username)
|
||||
|
||||
await session.commit()
|
||||
|
||||
id_map = {}
|
||||
result = await session.scalars(select(User))
|
||||
for user in result:
|
||||
id_map[old_id_by_email[user.email]] = user.id
|
||||
|
||||
if keycloak_users_file:
|
||||
json.dump({"users": keycloak_users}, keycloak_users_file, indent=4)
|
||||
log.info("Wrote keycloak users file to %s.", keycloak_users_file.name)
|
||||
|
||||
return id_map
|
||||
|
||||
|
||||
def parse_datetime(s):
|
||||
if isinstance(s, str):
|
||||
return datetime.fromisoformat(s)
|
||||
return s
|
||||
|
||||
|
||||
async def import_tracks(mongo, session, user_id_map):
|
||||
track_count = 0
|
||||
|
||||
async for track in mongo.tracks.find({}):
|
||||
stats = track.get("statistics") or {}
|
||||
new_track = Track(
|
||||
created_at=parse_datetime(track.get("createdAt")) or datetime.utcnow(),
|
||||
updated_at=parse_datetime(track.get("updatedAt")) or datetime.utcnow(),
|
||||
slug=track["slug"],
|
||||
title=track.get("title"),
|
||||
processing_status=track.get("processingStatus") or "pending",
|
||||
processing_log=track.get("processingLog"),
|
||||
customized_title=bool(track.get("customizedTitle")),
|
||||
description=track.get("description"),
|
||||
public=track.get("public"),
|
||||
uploaded_by_user_agent=track.get("uploadedByUserAgent"),
|
||||
original_file_name=track.get("originalFileName"),
|
||||
original_file_hash=track.get("originalFileHash"),
|
||||
# statistics
|
||||
recorded_at=parse_datetime(stats.get("recordedAt")),
|
||||
recorded_until=parse_datetime(stats.get("recordedUntil")),
|
||||
duration=stats.get("duration"),
|
||||
length=stats.get("length"),
|
||||
segments=stats.get("segments"),
|
||||
num_events=stats.get("num_events"),
|
||||
num_measurements=stats.get("num_measurements"),
|
||||
num_valid=stats.get("numValid"),
|
||||
author_id=user_id_map[track["author"]],
|
||||
)
|
||||
|
||||
session.add(new_track)
|
||||
|
||||
comment_ids = track.get("comments") or []
|
||||
if comment_ids:
|
||||
async for comment in mongo.comments.find({"_id": {"$in": comment_ids}}):
|
||||
new_comment = Comment(
|
||||
created_at=parse_datetime(comment.get("createdAt"))
|
||||
or datetime.utcnow(),
|
||||
updated_at=parse_datetime(comment.get("updatedAt"))
|
||||
or datetime.utcnow(),
|
||||
body=comment.get("body"),
|
||||
author_id=user_id_map[comment["author"]],
|
||||
)
|
||||
new_track.comments.append(new_comment)
|
||||
session.add(new_comment)
|
||||
|
||||
track_count += 1
|
||||
|
||||
log.info("Created %s tracks", track_count)
|
||||
|
||||
await session.commit()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue