platformsh-store-logs-at-s3

Store Platform.sh logs on S3

On Platform.sh all log files are trimmed to 10 MB automatically, but in some cases, it is vital to have complete access logs: to analyze traffic/analytics/etc.

This solution provides a cronjob which uploads logs to AWS S3 (separate file for each day). As result, you will have complete logs. It handles only access logs, but it is relatively easy to adjust it to any other logs.

Installation

1. Set AWS Credenitals: platform variable:create --level=project --name=AWS_ACCESS_KEY_ID --value= --json=false --sensitive=true --prefix=env --visible-build=true --visible-runtime=true
platform variable:create --level=project --name=AWS_SECRET_ACCESS_KEY --value= --json=false --sensitive=true --prefix=env --visible-build=true --visible-runtime=true
2. Set LOGS_S3_BUCKET and LOGS_S3_FOLDER variables. They define where Platform.sh logs will be stored on S3: platform variable:create --level=project --name=LOGS_S3_BUCKET --value=platform.sh-logs --json=false --sensitive=false --prefix=env --visible-build=false --visible-runtime=true
platform variable:create --level=project --name=LOGS_S3_FOLDER --value=my-project --json=false --sensitive=false --prefix=env --visible-build=false --visible-runtime=true
3. Make sure you have a writable mount (/app/var in this example) and set its path as value for LOGS_TMP_PATH variable: platform variable:create --level=project --name=LOGS_TMP_PATH --value=/app/var --json=false --sensitive=false --prefix=env --visible-build=false --visible-runtime=true
4. Install AWS CLI by adding following to build hook: if [ ! -z "$AWS_ACCESS_KEY_ID" ] && [ ! -z "$AWS_SECRET_ACCESS_KEY" ]; then
pip install futures
pip install awscli --upgrade --user 2>/dev/null
fi
5. Add upload_logs_to_s3 cronjob: upload_logs_to_s3:
spec: '0 * * * *'
cmd: |
if [ "$PLATFORM_BRANCH" = master ]; then
bash cron/upload_logs_to_s3.sh
fi
6. Deploy to Platform.sh and enjoy.