+1(613)852-9202 [email protected]

A high availability architecture separates the service into independent layers: ingress, stateless frontends, backend APIs, background workers, and a database cluster. Each layer must continue serving its responsibility while an instance is replaced or a component fails.

1. Architecture: Separate Traffic, Processing, and Data

Customers reach the service through a cloud edge and redundant ingress instances. Nginx routes requests to two or more stateless frontend instances. Each frontend serves the interface and calls the backend API; it does not own the business database or scheduled jobs. Place frontend instances on separate hosts or availability zones when host failure must be tolerated.

High availability architecture with redundant ingress, stateless frontends, backend APIs, scheduled workers and a database cluster
Figure 1. Independent frontend, backend, background processing and clustered data tiers.

Backend API instances handle validation, business rules, and data access. A redundant or managed routing layer directs frontend requests to healthy backend instances. Background workers process jobs separately from the APIs, so replacing a frontend does not interrupt a worker.

A database cluster provides a writer and replicas. Applications connect through a supported database endpoint that follows the current writer. The cluster’s failover mechanism must prevent two nodes from accepting writes as primary. Replicas and applications need placement in separate failure domains; a shared host, disk, ingress, or network can otherwise take multiple instances down together. Independent backups protect against deletion and corruption that replication can copy to every replica.

2. Deploy the Frontend and Backend Without Removing Serving Capacity

Start with the database change. Before updating any application instance, review the proposed schema and test it against the currently running application and the version needed for rollback. During this deployment, schema changes are additive only: retain existing tables, columns, constraints, and behavior. Do not drop, rename, or change existing structures in ways that break old code. Adding a column or constraint is not automatically safe: defaults, nullability, locks, and write behavior must also be compatible.

Apply the compatible additions using the database's supported online procedure, then verify the old application still reads and writes correctly. Stop the release if compatibility, data integrity, or database health checks fail. Only after this gate passes should the pipeline update the application.

Build and test one immutable image for each component. Confirm that the remaining instances can handle the workload. Backend and frontend versions may overlap, so maintain compatible API contracts throughout the release.

Schema-first deployment: test backward compatibility, apply additive changes, verify the old application, and roll out instances
Figure 2. Validate and apply backward-compatible schema additions before rolling out application instances.

For a two-instance frontend, update the standby while the primary continues serving. Check readiness and verify a request through the new frontend. Then drain the primary, confirm that ingress sends traffic to the updated standby, and update the primary. After readiness passes, restore the primary preference. If the standby fails, stop the release and keep the primary serving. If the primary update fails after the switch, keep the standby serving and roll back the primary.

Update backend APIs independently using the same capacity and readiness gates. Drain in-flight requests before stopping an instance. Old frontend versions must continue to work with the new backend until the frontend rollout finishes; previously delivered frontend assets must remain available to browsers that still use them.

Ingress also requires a rolling update. Update one ingress instance while a separate entry path continues routing to another healthy ingress. Nginx open-source upstream checks are ordinarily passive: a request can discover a failure before the upstream is marked unavailable. Planned releases therefore need an independent readiness gate. See the NGINX upstream documentation.

3. Keep Scheduled Jobs Running Once

Running a scheduler in every backend replica can launch the same scheduled task multiple times. Running it on only one unprotected instance can miss a task when that instance fails. Use an explicit ownership mechanism, such as a lease or leader election, to decide which scheduler creates each job. Prevent an old owner from continuing after its lease expires when the workload requires fencing.

Record jobs durably and let workers claim them. A worker must be able to resume after another worker is stopped during deployment. A crash can still occur after a side effect completes but before the job is acknowledged, so handlers need durable idempotency keys or uniqueness constraints. For external services, coordinate the idempotency mechanism with the provider. Monitor retries and failed jobs, and provide a recovery path rather than silently discarding work.

4. Update the Database Cluster Safely

Upgrade database replicas one at a time using the database product’s supported version and replication procedure. Check replica health and replication lag before a controlled switchover. After promotion, direct application connections to the new writer, upgrade the former writer, and restore replica redundancy.

The failover controller must fence the former writer to prevent split-brain writes. Do not assume the former database primary should automatically reclaim leadership when it returns. PostgreSQL describes these concerns in its failover documentation.

For this zero-downtime deployment, the database schema change comes first and must remain compatible with the existing application. Only add structures that preserve old read and write behavior; do not remove existing structures in this rollout. Test the migration, old application, new application, and application rollback against the expanded schema. Backfill separately if required, without blocking or changing the old application's behavior. A later removal is a separate, explicitly planned change after every consumer and rollback version no longer needs the old structure; it is not part of this deployment.

A controlled switchover can reduce planned disruption, but connections may still be dropped and in-flight transactions may have uncertain outcomes. Replication mode, acknowledged-data-loss risk, client reconnection, and retry behavior depend on the database and its configuration. A database cluster alone does not guarantee that every transaction experiences zero interruption.

5. Verify Each Layer During a Real Release

Test the architecture by removing an ingress, stopping a frontend, draining a backend, interrupting a worker during a job, and performing a supported database switchover. Verify customer-facing requests, job ownership and deduplication, replication health, client reconnection, and backup restoration.

During deployment, record the running version of each instance, readiness transitions, errors, latency, queue state, and replication lag. Validate the rollback path at each gate. Do not automatically replay a sent non-idempotent request unless the application has a durable idempotency design; an operation may have committed before its response was lost. Nginx documents the relevant proxy retry behavior.

The goal is a continuously usable path through every layer while components change independently. That requires compatible versions, sufficient serving capacity, safe job ownership, a controlled database cluster, and verification of the actual workload.