Whilst managed Kubernetes services in the cloud offer convenience and reduced operational overhead, regulatory compliance and security requirements often necessitate on-premise deployments. This series examines a distributed Kubernetes architecture designed specifically for on-premise environments, addressing the additional infrastructure components required beyond the core Kubernetes cluster.

The architecture presented here implements a multi-tier deployment model using K3s as the Kubernetes distribution, but this could easily be swapped out for K8s or another distribution if preferred. Let's dive in to the fun stuff!

Architecture Overview

In this architecture, we have several nodes (servers) with dedicated compute resources and functions. All nodes are in a private network which is not discoverable through the public internet, except for the external gateway which sits in a Demilitarised Zone (DMZ). The envisioned benefits are therefore:

• Security: Multi-layered approach with network segmentation, allowing access to only the next necessary node.

• Scalability: Easy horizontal scaling by adding Worker nodes.

• Maintainability: Clear separation of concerns between components thereby simplifying the debugging experience.

Traffic flows from a user and first interacts with the External Gateway node. This forwards the request to an Internal Master node which houses a Kubernetes (K3s) server. Depending on the route, the request is forwarded to a pod which runs on a Worker node (K3s agent). This can interact with the database if needed, and all logs and performance metrics are sent to the monitoring node.

In general, the External Gateway node only has access to the Internal Master node. This Master node only has access to the Worker nodes which it controls. Only the Worker nodes then have access to the Monitoring and Database nodes. The goal is to be strict with access between servers and follow the principle of least privilege.

Architecture Components

External Gateway Node

This is a simple secured Nginx server that acts as a reverse proxy to filter and forward traffic to the Internal Master node. It sits in a DMZ and is publicly accessible. Key characteristics:

• Handles SSL termination and certificate management.

• Implements rate limiting and basic DDoS protection.

• Filters malicious requests before they reach the internal network.

Internal Master Node

This is the brain of our Kubernetes cluster and hosts the Kubernetes control plane as a K3s server. It generally does not run any pods directly (no scheduling), but instead forwards requests to the appropriate pod in the Worker nodes. Key responsibilities:

• Running all Kubernetes commands and managing cluster state.

• Handling API requests and forwarding them to appropriate pods for processing.

• Acting as the internal load balancer to distribute traffic effectively.

• Maintaining cluster configuration and secrets.

Worker Node(s)

These are one or more servers which run pods and other Kubernetes resources in the data plane. They are connected to the Internal Master Node and controlled by it. Characteristics include:

• No direct Kubernetes commands are run here.

• Considered ephemeral - new nodes can be attached to the Internal Master node when more compute is required.

• The Internal Master node determines which Worker nodes to schedule pods onto.

• Horizontal scaling is primarily achieved by adding more Worker nodes (minimum of one required).

Monitoring Node

This hosts our observability and monitoring tools. In our case, we use a multi-node Elasticsearch cluster consisting of:

• Elasticsearch instances for log storage and searching.

• Kibana dashboard for viewing logs and application metrics.

• FileBeat agents deployed on Worker nodes to push logs to this server.

This centralised approach ensures comprehensive visibility across the entire cluster. The actual Elastic stack can be swapped out for a Grafana-based stack or on-premise similar alternative.

Database Node

This is an optional server included in this architecture for completeness. It could run:

• Microsoft SQL Server on Windows.

• PostgreSQL on Linux.

• Other database systems as required.

The database is only accessible from Worker nodes, providing an additional security layer.

Conclusion

In this first part, we've discussed a possible architecture for deploying a Kubernetes-based system powered by K3s in an on-premise environment. This architecture balances security, scalability, and maintainability whilst adhering to some Secure By Design security principles.

In the next parts, we will delve into the details of setting up each component and enabling traffic to flow through the system, including configuration examples and deployment scripts.