Components 

  1. AWS Elastic Load Balancer directs traffic to a specific PrivX application EC2 instance. The load balancing can be based on sticky session (can be enabled on ELB) or source IP (needs Nginx config changes). The load balancer keeps track of PrivX application server statuses and if it detects an anomaly, requests the autoscaling group to terminate the instance. 

  2. The PrivX EC2 autoscaling group has been configured to retain at least 2 instances of PrivX running. The number of instances can be configured according to load. 

  3. A PrivX application server consists of a Nginx reverse proxy and a number of PrivX microservices. The Nginx reverse proxy also serves the PrivX HTML5 UI static resources for the requesting clients. The PrivX microservices offer REST APIs over HTTPS. The PrivX application servers store all persistent data to AWS RDS - once a PrivX application server has been configured, it is just a matter of taking a snapshot of the server and deploying a new instance of the snapshot to add application nodes. 

  4. The PrivX microservices use AWS Elasticache to sync state between themselves - the cache is used only to trigger updates which are done via REST calls.

  5. The PrivX microservices persist data AWS RDS. The RDS database engine should be PostgreSQL. 

Installation

  1. Configure AWS RDS database for PrivX 
  2. Configure AWS Elasticache for PrivX 
  3. Create an EC2 autoscaling group for PrivX EC2 instances 
  4. Create an AWS Elastic Load Balancer for PrivX 
  5. Create an EC2 instance for PrivX (Amazon Linux, RHEL) 
  6. Install PrivX, configure PrivX to connect to RDS & Elasticache defined in 1 & 2
  7. Attach PrivX EC2 instance to the ELB and ensure that it works
  8. Take a snapshot of the PrivX EC2 instance and configure it to the autoscaling group. Set minimum number of running instances to the autoscaling group. 
  9. Terminate initial EC2 instance and observe the autoscaling group starting a new instance from the snapshot 
  10. Configure ELB to inform autoscaling group of an instance malfunction (ELB health check needs to poll https://<instance_address>:443/role-store/api/v1/status)

For production environments, it is recommended to use CloudFormation or similar template to set up the environment.

Upgrade

  1. Detach instances from the ELB
  2. Set autoscaling group instance count to 1
  3. Transfer PrivX upgrade package to the remaining host
  4. Upgrade host, run postinstall.sh
  5. Attach the instance to ELB, verify that PrivX works 
  6. Take a snapshot of the instance, attach the snapshot to autoscaling group
  7. PrivX is updated 

Backup & restore

PrivX automatically creates full backups (certificates and configuration files) and stores them to /var/backups/privx. Ideally, this directory would be mounted to AWS EFS.

  1. Transfer the backup directory from /var/backups/privx/hostname_yyyy-mm-dd-hhmm to a new PrivX instance 
  2. Install PrivX by running yum install PrivX-xyz.rpm  (do not run postinstall.sh after)
  3. Run /opt/privx/scripts/restore.sh /path/to/backup/directory/from/node/one/hostname_yyyy-mm-dd-hhmm
  4. Ensure that the PrivX service is functional
  5. Take a snapshot of the instance and attach the snapshots to autoscaling group