Production Server Setup

Steps I took to setup the production server.

cd ~
sudo apt-get update

# Mount bigger EBS as `/data`
# 1. DO ALL THESE THINGS: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html
# 2. Then…
sudo chown ubuntu: /data

# Install Node 12, npm, and yarn
# https://www.digitalocean.com/community/tutorials/how-to-install-node-js-on-ubuntu-18-04
curl -sL https://deb.nodesource.com/setup_12.x -o nodesource_setup.sh
bash nodesource_setup.sh
rm nodesource_setup.sh
sudo apt-get install -y nodejs
curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
sudo apt-get install yarn

# Install Postgres
# Reference: https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-18-04
sudo apt-get install postgresql postgresql-contrib

## Move Postgres data store from local drive to `/data/postgresql`
## Reference: https://www.digitalocean.com/community/tutorials/how-to-move-a-postgresql-data-directory-to-a-new-location-on-ubuntu-18-04
sudo systemctl stop postgresql
sudo rsync -av /var/lib/postgresql /data
sudo mv /var/lib/postgresql/10/main /var/lib/postgresql/10/main.bak
sudo vim /etc/postgresql/10/main/postgresql.conf
### ↑ set `data_directory = '/data/postgresql/10/main'`
### TODO: replace that step ↑ with a sed
sudo systemctl start postgresql
sudo rm -Rf /var/lib/postgresql/10/main.bak

## Setup Postgres users/database
sudo -u postgres createuser --interactive
### ↑ Choose a username; I went with n/n/n for the permissions questions
sudo -u postgres psql
### In all the commands below, replace `{username}` or `{database}` with your actual username/database, no brackets
### postgres=# \password {username}
### ↑ Generate secure password and store username/password in shared vault
sudo -u postgres createdb {database}
sudo -u postgres psql
### postgres=# GRANT ALL PRIVILEGES ON DATABASE {database} TO {username};

# Install/setup Redis
# Reference: https://www.digitalocean.com/community/tutorials/how-to-install-and-secure-redis-on-ubuntu-18-04
sudo apt-get install redis-server
mkdir /data/redis
sudo chown redis: /data/redis
## Give Redis system permission to write to that directory
sudo vim /etc/systemd/system/redis.service
### Config: Add `ReadWriteDirectories=-/data/redis` to `[Service]` section
### Reload daemon
sudo systemctl daemon-reload
## Secure and configure Redis
sudo vim /etc/redis/redis.conf
### Security: Change `supervised no` to `supervised systemd`:
### Config: Change `stop-writes-on-bgsave-error yes` to `stop-writes-on-bgsave-error no`
### Config: Change `dir /var/lib/redis to `dir /data/redis`
### Restart Redis:
sudo systemctl restart redis.service
## Configure system to allow overcommitting memory so Redis persistence doesn't explode
## See https://stackoverflow.com/a/49839193
sudo cp /etc/sysctl.conf /etc/sysctl.conf.bak
sudo vim /etc/sysctl.conf
### Add this block to the bottom and save (minus the `### ` at the beginning of each line):
### ###################################################################
### # Enable overcommitting for Redis persistence
### # See: https://stackoverflow.com/a/49839193
### vm.overcommit_memory = 1

# Install pm2
sudo npm install -g pm2

# Get code
mkdir /data/repos
cd /data/repos
git clone https://github.com/TechAndCheck/tech-and-check-alerts.git

# Copy and fill up .env.template
cd /data/repos/tech-and-check-alerts
cp .env.template .env
vim .env

# Install dependencies
yarn install

# Migrate
yarn migrate

# Setup log directories
## For now, symlink appropriate logs into `/data/logs`
mkdir -p /data/logs/{app,pm2}
ln -s /data/logs/app /data/repos/tech-and-check-alerts/logs
ln -s /data/logs/pm2 ~/.pm2/logs

# Set up logrotate
sudo vi /etc/logrotate.d/alerts
# Populate as follows (uncommented):
# /data/logs/*/*.log {
#         daily
#         missingok
#         rotate 5
#         compress
#         postrotate
#                 kill -s SIGUSR2 `cat /home/ubuntu/.pm2/pm2.pid`
#         endscript
# }