PostgreSQL Replication, High Availability HA and Scalability

PostgreSQL is one of the most powerful and easy-to-use database management systems. It has strong support from the community and is being actively developed with a new release every year.

PostgreSQL supports the most advanced features included in SQL standards. It also provides NoSQL capabilities and very rich data types and extensions. All of this makes PostgreSQL a very attractive solution in software systems.

In this course, we discussed the problem of building scalable solutions based on PostgreSQL utilizing the resources of several servers. There is a natural limitation for such systems—basically, there is always a compromise between performance, reliability, and consistency. It's possible to improve one aspect, but others will suffer. In this course, we'll see how to find the best match for our use-cases so that we know eactly which aspects need scaling, and avoid the common trade-offs of distributed systems.

Scaling PostgreSQL is a journey. You should come out of this course more prepared to assess your scaling needs and understand how to scale reads and how to scale writes.

Each of this solution presented in this course will improve some aspect of the scalability topic, but each of them will add some complexity, and maybe some limitation or constraint.

We have to ask the right questions to get the system requirements, and this why we dedicated an entire lecture, so that we examine what questions we have to put ourself, before starting the Scaling Journey.

After this course, we should come out more prepared and understand how to scale reads.

We have several options for replication, depending on wether we favor performance or flexibility.

Replication can be used as a backup or a standby solution that would take over in case the main server crashes.

Replication can also be used to improve the performance of a software system by making it possible to distribute the load on several database servers.

Then, if we have one sort of replication in place, we could ask ourself if we want to allow several computers to serve the same data.

To achieve this, we should have a mechanism to distribute the requests. We’ll see here two of the most popular options available.

Next, if the number of database connections is great, then we’ll probably want to use a connection pooler. Again, we’ll cover two options here.

We’ll also see, how to scale writes, and how to make your traffic growth more predictable by adding queuing to your architecture.

Then, we’ll check partitioning for those cases when we have to deal with big tables.

Also, we’ll check sharding to scale writes, and all the complex decisions that come with it.

Finally, we’ll see shortly the multi-master solution, which is a relatively new concept that seems to be promising.

If our goal is to achieve only High availability, or the ability to continue working even in the situation where one part of the cluster fails, we can check out only those solutions.

The pre-requirements for HA is to put in place a replication strategy.

Then, we can use tools to allow a second server to take over quickly, if the primary server fails.

Introduction to Scaling PostgreSQL

• Why scale PostgreSQL?

• What is Vertical Scaling?

• What is Horizontal Scaling?

• Read Versus Write Bound Workloads

• Why Statistics are essential?

• How to enable and make us of Statistics? (Hands-on)

• How to scale Postgres for Reads?

• How replication helps to scale out?

• What are the Load-Balancers?

• How to scale Postgres for Writes?

• How to make use of Queues?

• How could Partitioning and Sharding help in scaling out?

• What is the Multi-Master solution about?

Understanding the Limitations of Scaling out PostgreSQL

• CAP Theorem Explained

• PostgreSQL vs. Cassandra

• Use case: CA Systems

• Use case: AP Systems

How to use Streaming Replication?

• What is Streaming Replication?

• Asynchronous vs. Synchronous Replication

• How to Initialise Primary Database? (Hands-on)

• How to Configuring the Primary for Replication?  (Hands-on)

• How to Configuring the Replica Instance?  (Hands-on)

• Testing Replication Setup  (Hands-on)

How to use Logical Replication?

• What is Logical Replication in Postgres?

• Step by step Logical Replication setup

• How to setup the servers for Logical Replication? (Hands-on)

• How to make a selective Copy of the Data? (Hands-on)

• How to Create the Publication? (Hands-on)

• How to Create the Subscription? (Hands-on)

• Postgres Limitations of Logical Replication

• How to Monitoring Logical Replication? (Hands-on)

• Best use-cases for using Logical Replication

How to make use of PgBouncer?

• What is PgBouncer?

• Fundamental concepts of connection pooling

• How to build a PgBouncer Setup? (Hands-on)

• How to install and configure PgBouncer? (Hands-on)

• How to create a basic configuration file for PgBouncer? (Hands-on)

• How to connect to PgBouncer? (Hands-on)

• Explaining Advanced Settings for Performance

• Which are the available Pool Modes?

• Executing a benchmark with PgBouncer (Hands-on)

How to scale PostgreSQL in Google Cloud?

• Introduction

• Key Components on Google Cloud

• Key Characteristics of the Architecture

• How to create PostgreSQL Instances on Google Cloud?  (Hands-on)

• How to create a Google Cloud Engine (GCE) for HAProxy? (Hands-on)

• How to configure HAProxy for Load-Balancing? (Hands-on)

• Testing Load-Balancing

How to make use of PostgreSQL Partitioning?

• What is Partitioning?

• Which Tables Need Partitioning?

• How should the Tables be Partitioned? 

• Declarative vs. Inheritance Partitioning

• How to create a Partitioned Table? (Hands-on)

• Partitioning Methods

How to Shard PostgreSQL?

• What is Sharding?

• Pain-Points of Sharding?

• What is Second Level Sharding?

• What is good Sharding?

• How to query across multiple Shards?

How to setup High Availability (HA) on PostgreSQL?

• Why High Availability?

• Steps to achieve High Availability

• Essential Questions to ask before setting-up High Availability

• Log-Shipping Replication

• Streaming Replication and Logical Replication

• Cascading Replication

• Synchronous vs. Asynchronous Replication

• Automatic Failover and Always-on Strategy

• Simple HA Solution Example

• Better HA Solution Example

How to make use of PgPool II?

• What is PgPool II?

• Pgpool-II Features

• How to Configure Pgpool-II with Streaming Replication? (Hands-on)

• How to setup Streaming Replication? (Hands-on)

• How to Configuring Pgpool-II for Load Balancing ? (Hands-on)

• Testing load-balancing & read/write separation (Hands-on)

• How to Configure Pgpool for PostgreSQL High-Availability? (Hands-on)

• How to Configure PostgreSQL Primary Server? (Hands-on)

• How to Configure Pgpool-II Server? (Hands-on)

• How to Configure PostgreSQL Replica Server? (Hands-on)

• Testing The Failover (Hands-on)

• How to restore failed nodes? (Hands-on)