The aim is to cover all of the key principles:

Security: A validator is subject to severe financial penalties in case of key compromise and duplicate signing of blocks. Ergo, the operational procedures of a validator must guard against these catastrophic events. Ultimately, a validator is like a castle with precious crown jewels to be defended. Our architecture must erect several barriers that prevent external attackers from getting any level of access over the key or signing logic.

High Availability: Many networks impose financial penalties for extended downtime. It is generally advisable for validators to have greater than 99% uptime. We’ve gone the extra mile and are regularly delivering >99.9% uptime in anticipation of future networks that penalize availability to a higher degree.

Minimize maintenance operations conducted under duress: Teams maintaining validators are all too familiar with incidents requiring quick responses such as server failures, configuration issues, network attacks, etc. The hard part about validator incident response is the need to perform critical response operations while financial and reputational damage occurs due to downtime. Imagine yourself as a validator operations personnel conducting security-critical operations, while the validator is losing $10k an hour for its delegators. This is a stressful situation, as you can imagine. Make security critical infrastructure broadly maintainable: Parts of the architecture will be security critical, and validator teams restrict access to these parts to a select number of trusted personnel. This creates risk and dependency on a small number of personnel to maintain validator operations. We are taking steps that will allow an extended circle of team members to maintain our validators without having sensitive access.

There are number of major components to our validation architecture.
These components: validation servers, sentry layer, logging & monitoring and analytics.

It is recommended for validation servers to not hold the validation keys of the validator, as compromise of the servers could lead to theft of key material. Major decisions we faced regarding validation servers were: How many validation servers to use, where to host them and what software to run on them.

We purchase dedicated servers and purpose them as validation servers, in Tier III & IV data centers with 100% network uptime guarantee.

All selected data centers operate monitored closed circuit television, and are manned by both security and technical personnel 24/7/365. Access to racks is limited to the providers’ personnel via biometrics and access card controlled man-traps. The data centers are carrier neutral and connected via diverse routes to multiple tier 1 connectivity providers, and are monitored 24/7/365 by either a remote or on-site Network Operations Center (NOC). Similarly, both are connected to multiple power substations, and are backed by at least N+1 generators. As a result, both vendors offer 100% network connectivity and power SLAs. In addition, in the event of hardware failure, vendors are subject to a one hour hardware replacement SLA.

Sentries are full nodes (for a specific network) placed between the public Internet and the validation servers. Sentry nodes ensure that validation servers are not directly exposed to the public Internet, connect to other full nodes on the network, gossip transactions plus blocks, keep validation servers up in sync with the network, and act as the first line of defense against DDoS attacks. They can also be repurposed for the role of transaction filtering to guard against transaction spam attacks. We’ve deployed sentry nodes on AWS or DO, or Hetzner, as they are less security critical than the validation servers.

Observability of an Infrastructure estate is one of the most basic, but most powerful tools at the disposal of an engineer. It allows one to view, understand, proactively respond, and be automatically alerted to changes in the behaviour or performance of servers.
It can be broken down into three main streams:

Server Metrics: Information about CPU usage, memory usage, disk utilisation, network performance and IO performance allows better predictions of resource usage. These metrics can be used to diagnose when application upgrades begin leaking memory, and can prevent outages from arising when disk utilisation nears 100% or when components begin to fail.

Application Metrics: Metrics produced by applications are collected, aggregated and used for alerting us to potential issues with running applications. In addition to existing metrics endpoints exposed by cosmos-sdk and loom, we also monitor RPC and LCD endpoints to provide supplemental data regarding the networks on which we validate.

We use a combination of Prometheus, Grafana and Node Exporter in a high-availability layout for our internal monitoring, coupled with an external alerting service, which manages 24/7 our on-call rotation, and alert escalation processes.

The onboarding of any new network is not deemed complete until we have sufficient logging and monitoring in place to support the operation of that network.

In order to mitigate the impact on hosts of denial of service attacks (Denial of Service Attacks, Wikipedia) we route inbound packets from the public Internet via cloud-provider DDoS protection.
In addition to our public sentries, we run ’private’ sentries behind NAT (Network Address Translation, Wikipedia) gateways that only permit outbound connections so that in the event of a targeted attack against our public nodes, we will be able to continue receipt and transmission of P2P packets to and from the wider network.
These private sentries are also ‘privately peered’ with carefully selected partners via cloud-provider network peering, to ensure both parties benefit from each others’ connectivity even in the harshest of adversarial climates.

No Public-facing SSH Ports
Operation of our nodes is ultimately conducted via SSH — although we minimise this as much as humanly possible through automation — but as poorly SSH protected SSH endpoints are a primary target for any intruder, we do not expose SSH on any port (exposing on a non-default port is considered security-through-obscurity, and a poor countermeasure), preferring to utilise an encrypted VPN for all control traffic in and out of our infrastructure.

We have invested a significant amount of energy into building a very secure and highly available validator, but no infrastructure is «finished». There will always be the next *improvement to make; the next defense to introduce into one’s setup and the next networks to onboard. We’re open to improvement and continuously working on excellence.

Our operations consist of dedicated servers located in the specialized and highly qualified well-known data centers around the world, using only the highly secure networking and role-model infrastructure setups with the full regular back-up using the N + M scheme, controlled by the external independent watchdog services.

Our team core value is delivering the maximum reliability and security for the various blockchain projects coupled with the maximum transparency for important and respectful delegators.

We are always keen to help out fellow validators with their infrastructure — give us a shout if we could be of help!