pub-solar/matrix-docker-ansible-deploy
5
0
Fork 0
Code Issues Pull requests Packages Projects Releases Wiki Activity

Add storage-class support to the s3_media_upload command and mention matrix-synapse-s3-storage-provider-migrate script

Browse source
This commit is contained in:
Slavi Pantaleev 2022-10-14 18:29:30 +03:00
parent 6f56b9fb38
commit 30745db6d5
3 changed files with 10 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
docs/configuring-playbook-synapse-s3-storage-provider.md
View file

@ -70,10 +70,16 @@ Then use the following commands (`$` values come from environment variables - th
- `$UPDATE_DB_DURATION` is influenced by the `matrix_synapse_ext_synapse_s3_storage_provider_update_db_day_count` variable (defaults to `0`) - `$UPDATE_DB_DURATION` is influenced by the `matrix_synapse_ext_synapse_s3_storage_provider_update_db_day_count` variable (defaults to `0`)
- `$UPDATE_DB_DURATION` defaults to `0d` (0 days), which means **include files which haven't been accessed for more than 0 days** (that is, **all files will be included**). - `$UPDATE_DB_DURATION` defaults to `0d` (0 days), which means **include files which haven't been accessed for more than 0 days** (that is, **all files will be included**).
- `s3_media_upload check-deleted $MEDIA_PATH` - check whether files in the local cache still exist in the local media repository directory - `s3_media_upload check-deleted $MEDIA_PATH` - check whether files in the local cache still exist in the local media repository directory
- `s3_media_upload upload $MEDIA_PATH $BUCKET --delete --endpoint-url $ENDPOINT` - uploads locally-stored files to S3 and deletes them from the local media repository directory - `s3_media_upload upload $MEDIA_PATH $BUCKET --delete --storage-class $STORAGE_CLASS --endpoint-url $ENDPOINT` - uploads locally-stored files to S3 and deletes them from the local media repository directory
The `upload` command may take a lot of time to complete. The `s3_media_upload upload` command may take a lot of time to complete.
Instead of running the above commands manually in the shell, you can also run the `/usr/local/bin/matrix-synapse-s3-storage-provider-migrate` script which will run the same commands automatically. We demonstrate how to do it manually, because:
- it's what the upstream project demonstrates and it teaches you how to use the `s3_media_upload` tool
- allows you to check and verify the output of each command, to catch mistakes
- includes progress bars and detailed output for each command
- allows you to easily interrupt slow-running commands, etc. (the `/usr/local/bin/matrix-synapse-s3-storage-provider-migrate` starts a container without interactive TTY support, so `Ctrl+C` may not work and you and require killing via `docker kill ..`)
### Using another tool in combination with `s3_media_upload` ### Using another tool in combination with `s3_media_upload`

1
roles/matrix-synapse/templates/synapse/ext/s3-storage-provider/env.j2
View file

@ -4,6 +4,7 @@ AWS_DEFAULT_REGION={{ matrix_synapse_ext_synapse_s3_storage_provider_config_regi
ENDPOINT={{ matrix_synapse_ext_synapse_s3_storage_provider_config_endpoint_url }} ENDPOINT={{ matrix_synapse_ext_synapse_s3_storage_provider_config_endpoint_url }}
BUCKET={{ matrix_synapse_ext_synapse_s3_storage_provider_config_bucket }} BUCKET={{ matrix_synapse_ext_synapse_s3_storage_provider_config_bucket }}
STORAGE_CLASS={{ matrix_synapse_ext_synapse_s3_storage_provider_config_storage_class }}
MEDIA_PATH=/matrix-media-store-parent/{{ matrix_synapse_media_store_directory_name }} MEDIA_PATH=/matrix-media-store-parent/{{ matrix_synapse_media_store_directory_name }}

2
roles/matrix-synapse/templates/synapse/ext/s3-storage-provider/usr-local-bin/matrix-synapse-s3-storage-provider-migrate.j2
View file

@ -10,4 +10,4 @@
--network={{ matrix_docker_network }} \ --network={{ matrix_docker_network }} \
--entrypoint=/bin/bash \ --entrypoint=/bin/bash \
{{ matrix_synapse_docker_image_final }} \ {{ matrix_synapse_docker_image_final }} \
-c 's3_media_upload update-db $UPDATE_DB_DURATION && s3_media_upload --no-progress check-deleted $MEDIA_PATH && s3_media_upload --no-progress upload $MEDIA_PATH $BUCKET --delete --endpoint-url $ENDPOINT' -c 's3_media_upload update-db $UPDATE_DB_DURATION && s3_media_upload --no-progress check-deleted $MEDIA_PATH && s3_media_upload --no-progress upload $MEDIA_PATH $BUCKET --delete --storage-class $STORAGE_CLASS --endpoint-url $ENDPOINT'