Components

1. A load balancer directs traffic to a specific PrivX application server. The load balancer keeps track of PrivX application server statuses and removes non-working ones from the pool. The load balancing can be based on either source address or sticky session. Sticky session/session affinity cookie configuration for load balancer is recommended. Sticky load balancer cookies are required, if using PrivX-Extender, PrivX-Carrier and PrivX-Web-Proxy components and there's more than one PrivX node.

2. A PrivX application server consists of an Nginx reverse proxy and a number of PrivX microservices. Nginx reverse proxy also serves the PrivX HTML5 UI static resources for the requesting clients. The PrivX microservices offer REST APIs over HTTPS. The PrivX application servers store all persistent data in a database - once a PrivX application server has been configured, it is just a matter of taking a snapshot of the server and deploying a new instance of the snapshot to add application nodes.

3. The PrivX microservices use Redis in-memory cache to sync state between themselves - Redis is used only to trigger updates which are done via REST calls. To ensure high-availability, Redis should be configured in clustered mode. PrivX will require write access to Redis master node, so it should not be connected directly to replica nodes.

4. The PrivX microservices persist data to PostgreSQL. PostgreSQL should also be highly available, configured in master<>slave mode or as a cluster.  

   PrivX also supports various Hardware Security Modules (optional) for storing keys to separate HSM.

5. The PostgreSQL database is should be configured to clustered mode. if the database master dies, a DNS failover PrivX application servers can be configured to contact the replica via configuration change or a DNS change.

6. The trail storage (by default /var/privx/audit) should be mounted on a secure NAS device and shared between PrivX instances. (Optional: It is also possible to use shared storage for /opt/privx -folder before installing PrivX to make updates easier in the future).

Installation

Note: These high-availability (HA) installation instructions apply to non-cloud environment. For cloud HA deployment, please see for example Amazon Web Services High-Availability Installation for PrivX.

1. Install PostgreSQL master and slave nodes and configure a replication model which best suits your environment (vendor instructions at https://www.postgresql.org/docs/11/different-replication-solutions.html)
2. Install Redis and configure Redis cluster (vendor instructions at https://redis.io/topics/cluster-tutorial).
3. Install PrivX on first node:
   1. Run yum install PrivX-*.x86_64.rpm
      or install from PrivX repository yum install PrivX
   2. Run /opt/privx/scripts/postinstall.sh and instruct PrivX to connect to an external database and Redis configured above
4. Verify that the installation works
5. If using physical servers:
   1. Take backup of the PrivX first node by running /opt/privx/scripts/backup.sh
   2. Transfer the backup directory from /var/backups/privx/hostname_yyyy-mm-dd-hhmm on node1 to node2 
   3. Install PrivX on second node (do not run postinstall.sh):
      1. Set environment variable, run 
         export SKIP_POSTINSTALL=1 
         to prevent running postinstall.
      2. Run yum install PrivX
   4. On the second node, run /opt/privx/scripts/restore.sh /path/to/backup/directory/from/node/one/hostname_yyyy-mm-dd-hhmm 
   5. Run /opt/privx/scripts/postinstall.sh on the second node
   6. Repeat for as many nodes as you wish to deploy.
6. If using virtual machines, after installing PrivX to the first node, you can clone the node and use the cloned instance to launch additional PrivX nodes.
7. PrivX is installed and ready to be configured and used.

Upgrade

Note: These high-availability (HA) upgrade instructions apply to non-cloud environment, but can be in some cases used for upgrading cloud HA setup as well.

1. Upgrade PrivX on first node:
   1. Run
      yum install PrivX-*.x86_64.rpm
      or install from PrivX repository: yum install PrivX
Postinstall script will be run automatically, when running yum install and PrivX is already installed.
2. Verify that the node works. 
3. If using machine specific /opt/privx/ -folders (default):
   
   1. Take backup of the PrivX first node by running /opt/privx/scripts/backup.sh
   2. Transfer the backup directory from /var/backups/privx/hostname_yyyy-mm-dd-hhmm on node1 to node2 
   3.  On the second PrivX node:
      1. Set environment variable, run 
      export SKIP_POSTINSTALL=1 
      to prevent running postinstall.
      2. Run
      yum install PrivX-*.x86_64.rpm
      or install from PrivX repository: yum install PrivX
   4. On the second node, run /opt/privx/scripts/restore.sh /path/to/backup/directory/from/node/one/hostname_yyyy-mm-dd-hhmm
   5. Run /opt/privx/scripts/postinstall.sh
4. If using shared NFS-mounted /opt/privx/ - folder between nodes (optional), step 3 can be ignored. Instead, just run /opt/privx/postinstall.sh on all nodes after step 2.
5. PrivX is installed and ready to be configured and used

Backup & restore

PrivX automatically creates daily full backups (certificates and configuration files) and stores them in /var/backups/privx. Ideally, this directory should be mounted on a secure NAS solution or periodically backed up to a secure location.

Note: PrivX does not automatically backup Postgres databases on external DB. Backing up the external Postgres database is outside of the scope of this document. You should always take a backup of your DB data before upgrading PrivX version.

To restore a backed up set of data to PrivX follow these steps:

1. Transfer the backup directory from /var/backups/privx/hostname_yyyy-mm-dd-hhmm to PrivX node 1 
2. Install PrivX on the first node:
   
   1. Set environment variable, run 
      export SKIP_POSTINSTALL=1
   2. Run yum install PrivX-<version>.x86_64.rpm or install from PrivX repository: yum install PrivX-<version>.x86_64
NOTE: PrivX version to be installed here must match the PrivX version used for the backups you're trying to restore.
3. On the first node, run /opt/privx/scripts/restore.sh /path/to/backup/directory/from/node/one/hostname_yyyy-mm-dd-hhmm

4. Run /opt/privx/postinstall.sh
5. Ensure that the PrivX node is functional
6. Repeat the steps for other nodes