mirror of
https://github.com/tailscale/tailscale.git
synced 2026-10-23 03:37:19 +09:00
cmd/k8s-operator,k8s-operator: support dual-stack PeerRelay load balancers (#21663)
The reconciler forced the AWS load balancer address type to ipv4, so on an IPv6 EKS cluster, where pod targets are IPv6, no Network Load Balancer was ever provisioned. Make the address type a default that spec.service.annotations can override, defaulting IPv6 source NAT on when dualstack is requested. Resolve load balancer hostnames for both address families and requeue ready PeerRelays every five minutes, since DNS records for a new load balancer arrive over several minutes. Add spec.service.ipFamilyPolicy and ipFamilies so dual-stack GKE and AKS clusters can request a dual-stack load balancer too. Shorten the PeerRelay field descriptions and link to the PeerRelay docs for the detail. Fixes #21399 Change-Id: Idf22eb02f8ed9a46ff4abea8d4faef4b8c12c111 Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit is contained in:
David Bond authored and GitHub committed
1 parent
87ec06810d
commit
373db59dae
8 files changed
+467
-288
No files matched your search
@@ -60,46 +60,29 @@ spec:
|
||||
properties:
|
||||
aws:
|
||||
description: |-
|
||||
AWS contains configuration for pinning each replica to a specific AWS Elastic IP and subnet. Only meaningful
|
||||
when running on EKS with the AWS Load Balancer Controller. When set, the per-replica values override any
|
||||
aws-load-balancer-eip-allocations or aws-load-balancer-subnets values supplied via spec.service.annotations.
|
||||
|
||||
Leave this unset unless the peer relays must be reachable on addresses you control. Pinning a subnet
|
||||
confines a replica's load balancer to that subnet's availability zone, and an AWS Network Load Balancer
|
||||
only forwards to targets in a zone that is enabled on it, so a replica whose pod is scheduled into any
|
||||
other zone stops receiving traffic. Setting this field therefore also requires pinning the pods to the
|
||||
matching zone with a ProxyClass, as described on ElasticIPs. Without this field the AWS Load Balancer
|
||||
Controller instead provisions each load balancer across every zone it discovers, and the operator turns on
|
||||
cross-zone load balancing so the replica is reachable wherever it happens to be scheduled, with no
|
||||
scheduling constraints needed.
|
||||
AWS pins each replica to an AWS Elastic IP and subnet. It applies only on EKS with the AWS Load Balancer
|
||||
Controller. Leave it unset unless the peer relays must be reachable on addresses you control. When set,
|
||||
you must also pin the pods to one availability zone, as ElasticIPs describes. If unset, each load
|
||||
balancer spans every zone the controller finds, and the replica is reachable in any zone. See
|
||||
https://tailscale.com/docs/kubernetes-operator/peer-relay.
|
||||
type: object
|
||||
required:
|
||||
- elasticIPs
|
||||
properties:
|
||||
elasticIPs:
|
||||
description: |-
|
||||
ElasticIPs pins each replica to a specific AWS EIP allocation and subnet. Only meaningful when Network Load
|
||||
Balancers are provisioned by the AWS Load Balancer Controller. ElasticIPs supplies one allocation-subnet pair
|
||||
per replica: replica N uses ElasticIPs[N]. The list must be at least as long as spec.replicas so every replica
|
||||
has a distinct EIP; extra entries are permitted so that scale-up doesn't immediately trip validation.
|
||||
ElasticIPs lists one Elastic IP allocation and subnet for each replica. Replica N uses entry N. The list
|
||||
must have at least spec.replicas entries. Extra entries are allowed so that a scale-up does not fail
|
||||
validation.
|
||||
|
||||
Pinning a subnet enables only that subnet's availability zone on the replica's load balancer, and a Network
|
||||
Load Balancer only forwards to targets in an enabled zone. Nothing constrains the scheduler to place the
|
||||
replica's pod in that zone, so a pod scheduled elsewhere, including after a reschedule, becomes unreachable
|
||||
on its Elastic IP while still appearing healthy.
|
||||
A load balancer only forwards to pods in the availability zone of its subnet. A pod scheduled into
|
||||
another zone, for example after a reschedule, is unreachable on its Elastic IP but still appears healthy.
|
||||
To prevent this, use subnets in one zone. Then pin the pods to that zone with a ProxyClass that sets
|
||||
spec.statefulSet.pod.nodeSelector to topology.kubernetes.io/zone. A ProxyClass applies to every replica,
|
||||
so this removes the zone redundancy of multiple replicas.
|
||||
|
||||
Every replica of a PeerRelay shares one pod template, so a ProxyClass referenced by spec.proxyClass can
|
||||
confine the pods to a zone but cannot place different replicas in different zones. To use this field
|
||||
safely, name subnets in a single availability zone and pin the pods to that same zone with a ProxyClass
|
||||
setting spec.statefulSet.pod.nodeSelector to topology.kubernetes.io/zone. Note that this trades the zone
|
||||
redundancy that running several replicas would otherwise buy. Spreading replicas across zones with their
|
||||
own Elastic IPs needs a per-replica scheduling constraint that neither PeerRelay nor ProxyClass can
|
||||
currently express.
|
||||
|
||||
When set, the reconciler stamps
|
||||
service.beta.kubernetes.io/aws-load-balancer-eip-allocations and
|
||||
service.beta.kubernetes.io/aws-load-balancer-subnets on each per-replica Service, overriding any values in
|
||||
spec.service.annotations.
|
||||
These values override the service.beta.kubernetes.io/aws-load-balancer-eip-allocations and
|
||||
service.beta.kubernetes.io/aws-load-balancer-subnets annotations in spec.service.annotations.
|
||||
type: array
|
||||
minItems: 1
|
||||
items:
|
||||
@@ -111,76 +94,98 @@ spec:
|
||||
properties:
|
||||
allocationID:
|
||||
description: |-
|
||||
AllocationID is the AWS EIP allocation ID (e.g. eipalloc-0123abcd) whose public IP this replica is reachable
|
||||
on. Stamped as service.beta.kubernetes.io/aws-load-balancer-eip-allocations on the replica's Service.
|
||||
AllocationID is the AWS Elastic IP allocation ID for this replica, for example eipalloc-0123abcd. The
|
||||
operator sets it as the service.beta.kubernetes.io/aws-load-balancer-eip-allocations annotation on the
|
||||
Service of the replica.
|
||||
type: string
|
||||
pattern: ^eipalloc-[0-9a-f]+$
|
||||
subnetID:
|
||||
description: |-
|
||||
SubnetID is the AWS subnet the replica's load balancer is provisioned in (e.g. subnet-0123abcd). It must be
|
||||
a public subnet, and no two replicas may name subnets in the same availability zone, since a load balancer
|
||||
accepts only one Elastic IP per zone. A standard VPC Elastic IP is regional rather than zonal, so it takes
|
||||
the zone of whichever subnet it is paired with here. Stamped as
|
||||
service.beta.kubernetes.io/aws-load-balancer-subnets on the replica's Service.
|
||||
SubnetID is the public AWS subnet for the load balancer of this replica, for example subnet-0123abcd. The
|
||||
Elastic IP takes the availability zone of this subnet. The operator sets it as the
|
||||
service.beta.kubernetes.io/aws-load-balancer-subnets annotation on the Service of the replica.
|
||||
type: string
|
||||
pattern: ^subnet-[0-9a-f]+$
|
||||
x-kubernetes-list-type: atomic
|
||||
hostnamePrefix:
|
||||
description: |-
|
||||
HostnamePrefix specifies the hostname prefix for each
|
||||
replica. Each device will have the integer number
|
||||
from its StatefulSet pod appended to this prefix to form the full hostname.
|
||||
HostnamePrefix can contain lower case letters, numbers and dashes, it
|
||||
must not start with a dash and must be between 1 and 62 characters long.
|
||||
HostnamePrefix is the hostname prefix for each replica. The operator appends the replica index to form
|
||||
the full hostname, for example my-relay-0. If unset, the operator uses the PeerRelay name. The prefix can
|
||||
contain lowercase letters, numbers, and dashes. It must not start with a dash and must be 1 to 62
|
||||
characters long.
|
||||
type: string
|
||||
pattern: ^[a-z0-9][a-z0-9-]{0,61}$
|
||||
proxyClass:
|
||||
description: |-
|
||||
ProxyClass is the name of the ProxyClass custom resource that
|
||||
contains configuration options that should be applied to the
|
||||
resources created for this PeerRelay. If unset, the operator will
|
||||
create resources with the default configuration.
|
||||
ProxyClass is the name of a ProxyClass to apply to the resources the operator creates for this PeerRelay.
|
||||
If unset, the operator uses the default configuration.
|
||||
type: string
|
||||
replicas:
|
||||
description: |-
|
||||
Replicas specifies how many devices to create. Set this to enable
|
||||
high availability for peer relays.
|
||||
https://tailscale.com/kb/1115/high-availability. Defaults to 1.
|
||||
Replicas is the number of peer relay devices to run. Run more than one for high availability. Defaults
|
||||
to 1. See https://tailscale.com/kb/1115/high-availability.
|
||||
type: integer
|
||||
format: int32
|
||||
default: 1
|
||||
minimum: 0
|
||||
service:
|
||||
description: Service contains configuration values to modify the LoadBalancer service used to expose the peer relay.
|
||||
description: Service configures the LoadBalancer Service that the operator creates for each replica.
|
||||
type: object
|
||||
properties:
|
||||
annotations:
|
||||
description: |-
|
||||
Annotations to apply to the LoadBalancer service. Any annotations that conflict with those used by known
|
||||
cloud providers to ensure IP addresses rather than DNS names are ignored.
|
||||
Annotations to apply to the LoadBalancer Service. The operator sets some cloud provider annotations so
|
||||
that the Service gets a public IP address. These annotations override any conflicting value set here.
|
||||
|
||||
The operator sets service.beta.kubernetes.io/aws-load-balancer-ip-address-type to ipv4 unless it is set
|
||||
here. Set it to dualstack on an IPv6 EKS cluster. When it is dualstack, the operator also sets
|
||||
service.beta.kubernetes.io/aws-load-balancer-enable-prefix-for-ipv6-source-nat to on unless it is set here.
|
||||
type: object
|
||||
additionalProperties:
|
||||
type: string
|
||||
ipFamilies:
|
||||
description: |-
|
||||
IPFamilies sets spec.ipFamilies on each LoadBalancer Service. If unset, the cluster default applies. The
|
||||
value must agree with IPFamilyPolicy.
|
||||
type: array
|
||||
maxItems: 2
|
||||
items:
|
||||
description: |-
|
||||
IPFamily represents the IP Family (IPv4 or IPv6). This type is used
|
||||
to express the family of an IP expressed by a type (e.g. service.spec.ipFamilies).
|
||||
type: string
|
||||
enum:
|
||||
- IPv4
|
||||
- IPv6
|
||||
x-kubernetes-list-type: atomic
|
||||
ipFamilyPolicy:
|
||||
description: |-
|
||||
IPFamilyPolicy sets spec.ipFamilyPolicy on each LoadBalancer Service. If unset, the cluster default
|
||||
applies. Set RequireDualStack on a dual-stack cluster to get a load balancer with both IPv4 and IPv6
|
||||
addresses. On EKS, use the service.beta.kubernetes.io/aws-load-balancer-ip-address-type annotation
|
||||
instead.
|
||||
type: string
|
||||
enum:
|
||||
- SingleStack
|
||||
- PreferDualStack
|
||||
- RequireDualStack
|
||||
port:
|
||||
description: |-
|
||||
Port is the UDP port each peer relay replica listens on and that its LoadBalancer Service exposes
|
||||
externally. The two are always equal: the relay advertises address:port to peers, so the load balancer
|
||||
must forward without rewriting the port. Changing the port on an existing PeerRelay briefly interrupts
|
||||
relay traffic while the load balancer updates. Defaults to 41641.
|
||||
Port is the UDP port that each replica listens on and that its LoadBalancer Service exposes. The two
|
||||
ports are always equal because the relay advertises address:port to peers. Changing the port briefly
|
||||
interrupts relay traffic while the load balancer updates. Defaults to 41641.
|
||||
type: integer
|
||||
default: 41641
|
||||
maximum: 65535
|
||||
minimum: 1
|
||||
staticEndpoints:
|
||||
description: |-
|
||||
StaticEndpoints is an optional list of address:port pairs on which every replica of this PeerRelay is
|
||||
reachable from outside the cluster, for example the public side of a NAT or firewall in front of the
|
||||
cluster. These supplement the endpoints discovered from each replica's LoadBalancer Service: they are
|
||||
added to every replica's entries in status.endpoints and advertised to peers alongside them. If an entry
|
||||
names an address a replica's load balancer already provides, the entry's port takes precedence for that
|
||||
address. Each address may appear at most once across all entries.
|
||||
Entries take the form accepted by Go's net/netip.ParseAddrPort, e.g. 203.0.113.1:41641. IPv6
|
||||
addresses must be enclosed in brackets, e.g. [2001:db8::1]:41641.
|
||||
StaticEndpoints lists extra address:port pairs where every replica is reachable from outside the
|
||||
cluster, for example the public side of a NAT or firewall. The operator adds each entry to every replica
|
||||
in status.endpoints and advertises it to peers. If an entry has the same address as a load balancer, the
|
||||
port of the entry replaces the port of the load balancer. An address can appear in only one entry. Write
|
||||
each entry as an IP address and port, for example 203.0.113.1:41641. Put IPv6 addresses in brackets, for
|
||||
example [2001:db8::1]:41641.
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
@@ -188,24 +193,17 @@ spec:
|
||||
x-kubernetes-list-type: set
|
||||
tags:
|
||||
description: |-
|
||||
Tags that the Tailscale node will be tagged with.
|
||||
Defaults to [tag:k8s].
|
||||
To autoapprove the device defined by a PeerRelay,
|
||||
you can configure Tailscale ACLs to give these tags the necessary
|
||||
permissions.
|
||||
See https://tailscale.com/kb/1337/acl-syntax#autoapprovers.
|
||||
If you specify custom tags here, you must also make the operator an owner of these tags.
|
||||
See https://tailscale.com/kb/1236/kubernetes-operator/#setting-up-the-kubernetes-operator.
|
||||
Tags cannot be changed once a PeerRelay node has been created.
|
||||
Tag values must be in form ^tag:[a-zA-Z][a-zA-Z0-9-]*$.
|
||||
Tags to apply to each peer relay device. Defaults to [tag:k8s]. If you set custom tags, the operator must
|
||||
be an owner of each tag. You cannot change the tags after the operator creates the device. Each tag must
|
||||
match ^tag:[a-zA-Z][a-zA-Z0-9-]*$. See https://tailscale.com/docs/kubernetes-operator/peer-relay.
|
||||
type: array
|
||||
items:
|
||||
type: string
|
||||
pattern: ^tag:[a-zA-Z][a-zA-Z0-9-]*$
|
||||
tailnet:
|
||||
description: |-
|
||||
Tailnet specifies the tailnet this PeerRelay should join. If blank, the default tailnet is used. When set, this
|
||||
name must match that of a valid Tailnet resource. This field is immutable and cannot be changed once set.
|
||||
Tailnet is the name of the Tailnet resource for the tailnet this PeerRelay joins. If unset, the operator
|
||||
uses the default tailnet. You cannot change this field after you set it.
|
||||
type: string
|
||||
x-kubernetes-validations:
|
||||
- rule: self == oldSelf
|
||||
@@ -279,11 +277,10 @@ spec:
|
||||
x-kubernetes-list-type: map
|
||||
endpoints:
|
||||
description: |-
|
||||
Endpoints lists the public address:port pairs each peer relay replica is reachable on. Entries appear as the
|
||||
underlying cloud provisions each Service. A replica has one entry per address its LoadBalancer Service was
|
||||
given, which is usually one, but a load balancer spanning several availability zones has an address in each
|
||||
and every one of them is listed. Entries from spec.staticEndpoints are listed for every replica in addition
|
||||
to the load balancer addresses.
|
||||
Endpoints lists the public address:port pairs where each replica is reachable. The operator adds entries
|
||||
as the cloud provisions each Service. A replica has one entry for each address of its load balancer. A
|
||||
load balancer has more than one address when it spans several availability zones or both address
|
||||
families. Every replica also has one entry for each entry in spec.staticEndpoints.
|
||||
type: array
|
||||
items:
|
||||
type: object
|
||||
@@ -294,8 +291,8 @@ spec:
|
||||
properties:
|
||||
address:
|
||||
description: |-
|
||||
Address is the public IP or hostname the cloud has allocated for this replica's LoadBalancer Service, or
|
||||
an address supplied via spec.staticEndpoints. Peers reach this relay by connecting to Address:Port over UDP.
|
||||
Address is a public IP address of the load balancer for this replica, or an address from
|
||||
spec.staticEndpoints. Peers connect to Address:Port over UDP.
|
||||
type: string
|
||||
port:
|
||||
description: Port is the UDP port the peer relay listens on.
|
||||
|
||||
@@ -1553,59 +1553,41 @@ spec:
|
||||
properties:
|
||||
aws:
|
||||
description: |-
|
||||
AWS contains configuration for pinning each replica to a specific AWS Elastic IP and subnet. Only meaningful
|
||||
when running on EKS with the AWS Load Balancer Controller. When set, the per-replica values override any
|
||||
aws-load-balancer-eip-allocations or aws-load-balancer-subnets values supplied via spec.service.annotations.
|
||||
|
||||
Leave this unset unless the peer relays must be reachable on addresses you control. Pinning a subnet
|
||||
confines a replica's load balancer to that subnet's availability zone, and an AWS Network Load Balancer
|
||||
only forwards to targets in a zone that is enabled on it, so a replica whose pod is scheduled into any
|
||||
other zone stops receiving traffic. Setting this field therefore also requires pinning the pods to the
|
||||
matching zone with a ProxyClass, as described on ElasticIPs. Without this field the AWS Load Balancer
|
||||
Controller instead provisions each load balancer across every zone it discovers, and the operator turns on
|
||||
cross-zone load balancing so the replica is reachable wherever it happens to be scheduled, with no
|
||||
scheduling constraints needed.
|
||||
AWS pins each replica to an AWS Elastic IP and subnet. It applies only on EKS with the AWS Load Balancer
|
||||
Controller. Leave it unset unless the peer relays must be reachable on addresses you control. When set,
|
||||
you must also pin the pods to one availability zone, as ElasticIPs describes. If unset, each load
|
||||
balancer spans every zone the controller finds, and the replica is reachable in any zone. See
|
||||
https://tailscale.com/docs/kubernetes-operator/peer-relay.
|
||||
properties:
|
||||
elasticIPs:
|
||||
description: |-
|
||||
ElasticIPs pins each replica to a specific AWS EIP allocation and subnet. Only meaningful when Network Load
|
||||
Balancers are provisioned by the AWS Load Balancer Controller. ElasticIPs supplies one allocation-subnet pair
|
||||
per replica: replica N uses ElasticIPs[N]. The list must be at least as long as spec.replicas so every replica
|
||||
has a distinct EIP; extra entries are permitted so that scale-up doesn't immediately trip validation.
|
||||
ElasticIPs lists one Elastic IP allocation and subnet for each replica. Replica N uses entry N. The list
|
||||
must have at least spec.replicas entries. Extra entries are allowed so that a scale-up does not fail
|
||||
validation.
|
||||
|
||||
Pinning a subnet enables only that subnet's availability zone on the replica's load balancer, and a Network
|
||||
Load Balancer only forwards to targets in an enabled zone. Nothing constrains the scheduler to place the
|
||||
replica's pod in that zone, so a pod scheduled elsewhere, including after a reschedule, becomes unreachable
|
||||
on its Elastic IP while still appearing healthy.
|
||||
A load balancer only forwards to pods in the availability zone of its subnet. A pod scheduled into
|
||||
another zone, for example after a reschedule, is unreachable on its Elastic IP but still appears healthy.
|
||||
To prevent this, use subnets in one zone. Then pin the pods to that zone with a ProxyClass that sets
|
||||
spec.statefulSet.pod.nodeSelector to topology.kubernetes.io/zone. A ProxyClass applies to every replica,
|
||||
so this removes the zone redundancy of multiple replicas.
|
||||
|
||||
Every replica of a PeerRelay shares one pod template, so a ProxyClass referenced by spec.proxyClass can
|
||||
confine the pods to a zone but cannot place different replicas in different zones. To use this field
|
||||
safely, name subnets in a single availability zone and pin the pods to that same zone with a ProxyClass
|
||||
setting spec.statefulSet.pod.nodeSelector to topology.kubernetes.io/zone. Note that this trades the zone
|
||||
redundancy that running several replicas would otherwise buy. Spreading replicas across zones with their
|
||||
own Elastic IPs needs a per-replica scheduling constraint that neither PeerRelay nor ProxyClass can
|
||||
currently express.
|
||||
|
||||
When set, the reconciler stamps
|
||||
service.beta.kubernetes.io/aws-load-balancer-eip-allocations and
|
||||
service.beta.kubernetes.io/aws-load-balancer-subnets on each per-replica Service, overriding any values in
|
||||
spec.service.annotations.
|
||||
These values override the service.beta.kubernetes.io/aws-load-balancer-eip-allocations and
|
||||
service.beta.kubernetes.io/aws-load-balancer-subnets annotations in spec.service.annotations.
|
||||
items:
|
||||
description: PeerRelayAWSElasticIP pairs an EIP allocation with the subnet it is attached to.
|
||||
properties:
|
||||
allocationID:
|
||||
description: |-
|
||||
AllocationID is the AWS EIP allocation ID (e.g. eipalloc-0123abcd) whose public IP this replica is reachable
|
||||
on. Stamped as service.beta.kubernetes.io/aws-load-balancer-eip-allocations on the replica's Service.
|
||||
AllocationID is the AWS Elastic IP allocation ID for this replica, for example eipalloc-0123abcd. The
|
||||
operator sets it as the service.beta.kubernetes.io/aws-load-balancer-eip-allocations annotation on the
|
||||
Service of the replica.
|
||||
pattern: ^eipalloc-[0-9a-f]+$
|
||||
type: string
|
||||
subnetID:
|
||||
description: |-
|
||||
SubnetID is the AWS subnet the replica's load balancer is provisioned in (e.g. subnet-0123abcd). It must be
|
||||
a public subnet, and no two replicas may name subnets in the same availability zone, since a load balancer
|
||||
accepts only one Elastic IP per zone. A standard VPC Elastic IP is regional rather than zonal, so it takes
|
||||
the zone of whichever subnet it is paired with here. Stamped as
|
||||
service.beta.kubernetes.io/aws-load-balancer-subnets on the replica's Service.
|
||||
SubnetID is the public AWS subnet for the load balancer of this replica, for example subnet-0123abcd. The
|
||||
Elastic IP takes the availability zone of this subnet. The operator sets it as the
|
||||
service.beta.kubernetes.io/aws-load-balancer-subnets annotation on the Service of the replica.
|
||||
pattern: ^subnet-[0-9a-f]+$
|
||||
type: string
|
||||
required:
|
||||
@@ -1620,60 +1602,83 @@ spec:
|
||||
type: object
|
||||
hostnamePrefix:
|
||||
description: |-
|
||||
HostnamePrefix specifies the hostname prefix for each
|
||||
replica. Each device will have the integer number
|
||||
from its StatefulSet pod appended to this prefix to form the full hostname.
|
||||
HostnamePrefix can contain lower case letters, numbers and dashes, it
|
||||
must not start with a dash and must be between 1 and 62 characters long.
|
||||
HostnamePrefix is the hostname prefix for each replica. The operator appends the replica index to form
|
||||
the full hostname, for example my-relay-0. If unset, the operator uses the PeerRelay name. The prefix can
|
||||
contain lowercase letters, numbers, and dashes. It must not start with a dash and must be 1 to 62
|
||||
characters long.
|
||||
pattern: ^[a-z0-9][a-z0-9-]{0,61}$
|
||||
type: string
|
||||
proxyClass:
|
||||
description: |-
|
||||
ProxyClass is the name of the ProxyClass custom resource that
|
||||
contains configuration options that should be applied to the
|
||||
resources created for this PeerRelay. If unset, the operator will
|
||||
create resources with the default configuration.
|
||||
ProxyClass is the name of a ProxyClass to apply to the resources the operator creates for this PeerRelay.
|
||||
If unset, the operator uses the default configuration.
|
||||
type: string
|
||||
replicas:
|
||||
default: 1
|
||||
description: |-
|
||||
Replicas specifies how many devices to create. Set this to enable
|
||||
high availability for peer relays.
|
||||
https://tailscale.com/kb/1115/high-availability. Defaults to 1.
|
||||
Replicas is the number of peer relay devices to run. Run more than one for high availability. Defaults
|
||||
to 1. See https://tailscale.com/kb/1115/high-availability.
|
||||
format: int32
|
||||
minimum: 0
|
||||
type: integer
|
||||
service:
|
||||
description: Service contains configuration values to modify the LoadBalancer service used to expose the peer relay.
|
||||
description: Service configures the LoadBalancer Service that the operator creates for each replica.
|
||||
properties:
|
||||
annotations:
|
||||
additionalProperties:
|
||||
type: string
|
||||
description: |-
|
||||
Annotations to apply to the LoadBalancer service. Any annotations that conflict with those used by known
|
||||
cloud providers to ensure IP addresses rather than DNS names are ignored.
|
||||
Annotations to apply to the LoadBalancer Service. The operator sets some cloud provider annotations so
|
||||
that the Service gets a public IP address. These annotations override any conflicting value set here.
|
||||
|
||||
The operator sets service.beta.kubernetes.io/aws-load-balancer-ip-address-type to ipv4 unless it is set
|
||||
here. Set it to dualstack on an IPv6 EKS cluster. When it is dualstack, the operator also sets
|
||||
service.beta.kubernetes.io/aws-load-balancer-enable-prefix-for-ipv6-source-nat to on unless it is set here.
|
||||
type: object
|
||||
ipFamilies:
|
||||
description: |-
|
||||
IPFamilies sets spec.ipFamilies on each LoadBalancer Service. If unset, the cluster default applies. The
|
||||
value must agree with IPFamilyPolicy.
|
||||
items:
|
||||
description: |-
|
||||
IPFamily represents the IP Family (IPv4 or IPv6). This type is used
|
||||
to express the family of an IP expressed by a type (e.g. service.spec.ipFamilies).
|
||||
enum:
|
||||
- IPv4
|
||||
- IPv6
|
||||
type: string
|
||||
maxItems: 2
|
||||
type: array
|
||||
x-kubernetes-list-type: atomic
|
||||
ipFamilyPolicy:
|
||||
description: |-
|
||||
IPFamilyPolicy sets spec.ipFamilyPolicy on each LoadBalancer Service. If unset, the cluster default
|
||||
applies. Set RequireDualStack on a dual-stack cluster to get a load balancer with both IPv4 and IPv6
|
||||
addresses. On EKS, use the service.beta.kubernetes.io/aws-load-balancer-ip-address-type annotation
|
||||
instead.
|
||||
enum:
|
||||
- SingleStack
|
||||
- PreferDualStack
|
||||
- RequireDualStack
|
||||
type: string
|
||||
port:
|
||||
default: 41641
|
||||
description: |-
|
||||
Port is the UDP port each peer relay replica listens on and that its LoadBalancer Service exposes
|
||||
externally. The two are always equal: the relay advertises address:port to peers, so the load balancer
|
||||
must forward without rewriting the port. Changing the port on an existing PeerRelay briefly interrupts
|
||||
relay traffic while the load balancer updates. Defaults to 41641.
|
||||
Port is the UDP port that each replica listens on and that its LoadBalancer Service exposes. The two
|
||||
ports are always equal because the relay advertises address:port to peers. Changing the port briefly
|
||||
interrupts relay traffic while the load balancer updates. Defaults to 41641.
|
||||
maximum: 65535
|
||||
minimum: 1
|
||||
type: integer
|
||||
type: object
|
||||
staticEndpoints:
|
||||
description: |-
|
||||
StaticEndpoints is an optional list of address:port pairs on which every replica of this PeerRelay is
|
||||
reachable from outside the cluster, for example the public side of a NAT or firewall in front of the
|
||||
cluster. These supplement the endpoints discovered from each replica's LoadBalancer Service: they are
|
||||
added to every replica's entries in status.endpoints and advertised to peers alongside them. If an entry
|
||||
names an address a replica's load balancer already provides, the entry's port takes precedence for that
|
||||
address. Each address may appear at most once across all entries.
|
||||
Entries take the form accepted by Go's net/netip.ParseAddrPort, e.g. 203.0.113.1:41641. IPv6
|
||||
addresses must be enclosed in brackets, e.g. [2001:db8::1]:41641.
|
||||
StaticEndpoints lists extra address:port pairs where every replica is reachable from outside the
|
||||
cluster, for example the public side of a NAT or firewall. The operator adds each entry to every replica
|
||||
in status.endpoints and advertises it to peers. If an entry has the same address as a load balancer, the
|
||||
port of the entry replaces the port of the load balancer. An address can appear in only one entry. Write
|
||||
each entry as an IP address and port, for example 203.0.113.1:41641. Put IPv6 addresses in brackets, for
|
||||
example [2001:db8::1]:41641.
|
||||
items:
|
||||
pattern: ^(\d{1,3}(\.\d{1,3}){3}|\[[0-9a-fA-F:.]+\]):\d{1,5}$
|
||||
type: string
|
||||
@@ -1681,24 +1686,17 @@ spec:
|
||||
x-kubernetes-list-type: set
|
||||
tags:
|
||||
description: |-
|
||||
Tags that the Tailscale node will be tagged with.
|
||||
Defaults to [tag:k8s].
|
||||
To autoapprove the device defined by a PeerRelay,
|
||||
you can configure Tailscale ACLs to give these tags the necessary
|
||||
permissions.
|
||||
See https://tailscale.com/kb/1337/acl-syntax#autoapprovers.
|
||||
If you specify custom tags here, you must also make the operator an owner of these tags.
|
||||
See https://tailscale.com/kb/1236/kubernetes-operator/#setting-up-the-kubernetes-operator.
|
||||
Tags cannot be changed once a PeerRelay node has been created.
|
||||
Tag values must be in form ^tag:[a-zA-Z][a-zA-Z0-9-]*$.
|
||||
Tags to apply to each peer relay device. Defaults to [tag:k8s]. If you set custom tags, the operator must
|
||||
be an owner of each tag. You cannot change the tags after the operator creates the device. Each tag must
|
||||
match ^tag:[a-zA-Z][a-zA-Z0-9-]*$. See https://tailscale.com/docs/kubernetes-operator/peer-relay.
|
||||
items:
|
||||
pattern: ^tag:[a-zA-Z][a-zA-Z0-9-]*$
|
||||
type: string
|
||||
type: array
|
||||
tailnet:
|
||||
description: |-
|
||||
Tailnet specifies the tailnet this PeerRelay should join. If blank, the default tailnet is used. When set, this
|
||||
name must match that of a valid Tailnet resource. This field is immutable and cannot be changed once set.
|
||||
Tailnet is the name of the Tailnet resource for the tailnet this PeerRelay joins. If unset, the operator
|
||||
uses the default tailnet. You cannot change this field after you set it.
|
||||
type: string
|
||||
x-kubernetes-validations:
|
||||
- message: PeerRelay tailnet is immutable
|
||||
@@ -1772,17 +1770,16 @@ spec:
|
||||
x-kubernetes-list-type: map
|
||||
endpoints:
|
||||
description: |-
|
||||
Endpoints lists the public address:port pairs each peer relay replica is reachable on. Entries appear as the
|
||||
underlying cloud provisions each Service. A replica has one entry per address its LoadBalancer Service was
|
||||
given, which is usually one, but a load balancer spanning several availability zones has an address in each
|
||||
and every one of them is listed. Entries from spec.staticEndpoints are listed for every replica in addition
|
||||
to the load balancer addresses.
|
||||
Endpoints lists the public address:port pairs where each replica is reachable. The operator adds entries
|
||||
as the cloud provisions each Service. A replica has one entry for each address of its load balancer. A
|
||||
load balancer has more than one address when it spans several availability zones or both address
|
||||
families. Every replica also has one entry for each entry in spec.staticEndpoints.
|
||||
items:
|
||||
properties:
|
||||
address:
|
||||
description: |-
|
||||
Address is the public IP or hostname the cloud has allocated for this replica's LoadBalancer Service, or
|
||||
an address supplied via spec.staticEndpoints. Peers reach this relay by connecting to Address:Port over UDP.
|
||||
Address is a public IP address of the load balancer for this replica, or an address from
|
||||
spec.staticEndpoints. Peers connect to Address:Port over UDP.
|
||||
type: string
|
||||
port:
|
||||
description: Port is the UDP port the peer relay listens on.
|
||||
|
||||
+17
-15
@@ -586,7 +586,7 @@ _Appears in:_
|
||||
|
||||
| Field | Description | Default | Validation |
|
||||
| --- | --- | --- | --- |
|
||||
| `elasticIPs` _[PeerRelayAWSElasticIP](#peerrelayawselasticip) array_ | ElasticIPs pins each replica to a specific AWS EIP allocation and subnet. Only meaningful when Network Load<br />Balancers are provisioned by the AWS Load Balancer Controller. ElasticIPs supplies one allocation-subnet pair<br />per replica: replica N uses ElasticIPs[N]. The list must be at least as long as spec.replicas so every replica<br />has a distinct EIP; extra entries are permitted so that scale-up doesn't immediately trip validation.<br />Pinning a subnet enables only that subnet's availability zone on the replica's load balancer, and a Network<br />Load Balancer only forwards to targets in an enabled zone. Nothing constrains the scheduler to place the<br />replica's pod in that zone, so a pod scheduled elsewhere, including after a reschedule, becomes unreachable<br />on its Elastic IP while still appearing healthy.<br />Every replica of a PeerRelay shares one pod template, so a ProxyClass referenced by spec.proxyClass can<br />confine the pods to a zone but cannot place different replicas in different zones. To use this field<br />safely, name subnets in a single availability zone and pin the pods to that same zone with a ProxyClass<br />setting spec.statefulSet.pod.nodeSelector to topology.kubernetes.io/zone. Note that this trades the zone<br />redundancy that running several replicas would otherwise buy. Spreading replicas across zones with their<br />own Elastic IPs needs a per-replica scheduling constraint that neither PeerRelay nor ProxyClass can<br />currently express.<br />When set, the reconciler stamps<br />service.beta.kubernetes.io/aws-load-balancer-eip-allocations and<br />service.beta.kubernetes.io/aws-load-balancer-subnets on each per-replica Service, overriding any values in<br />spec.service.annotations. | | MinItems: 1 <br /> |
|
||||
| `elasticIPs` _[PeerRelayAWSElasticIP](#peerrelayawselasticip) array_ | ElasticIPs lists one Elastic IP allocation and subnet for each replica. Replica N uses entry N. The list<br />must have at least spec.replicas entries. Extra entries are allowed so that a scale-up does not fail<br />validation.<br />A load balancer only forwards to pods in the availability zone of its subnet. A pod scheduled into<br />another zone, for example after a reschedule, is unreachable on its Elastic IP but still appears healthy.<br />To prevent this, use subnets in one zone. Then pin the pods to that zone with a ProxyClass that sets<br />spec.statefulSet.pod.nodeSelector to topology.kubernetes.io/zone. A ProxyClass applies to every replica,<br />so this removes the zone redundancy of multiple replicas.<br />These values override the service.beta.kubernetes.io/aws-load-balancer-eip-allocations and<br />service.beta.kubernetes.io/aws-load-balancer-subnets annotations in spec.service.annotations. | | MinItems: 1 <br /> |
|
||||
|
||||
|
||||
#### PeerRelayAWSElasticIP
|
||||
@@ -602,8 +602,8 @@ _Appears in:_
|
||||
|
||||
| Field | Description | Default | Validation |
|
||||
| --- | --- | --- | --- |
|
||||
| `allocationID` _string_ | AllocationID is the AWS EIP allocation ID (e.g. eipalloc-0123abcd) whose public IP this replica is reachable<br />on. Stamped as service.beta.kubernetes.io/aws-load-balancer-eip-allocations on the replica's Service. | | Pattern: `^eipalloc-[0-9a-f]+$` <br /> |
|
||||
| `subnetID` _string_ | SubnetID is the AWS subnet the replica's load balancer is provisioned in (e.g. subnet-0123abcd). It must be<br />a public subnet, and no two replicas may name subnets in the same availability zone, since a load balancer<br />accepts only one Elastic IP per zone. A standard VPC Elastic IP is regional rather than zonal, so it takes<br />the zone of whichever subnet it is paired with here. Stamped as<br />service.beta.kubernetes.io/aws-load-balancer-subnets on the replica's Service. | | Pattern: `^subnet-[0-9a-f]+$` <br /> |
|
||||
| `allocationID` _string_ | AllocationID is the AWS Elastic IP allocation ID for this replica, for example eipalloc-0123abcd. The<br />operator sets it as the service.beta.kubernetes.io/aws-load-balancer-eip-allocations annotation on the<br />Service of the replica. | | Pattern: `^eipalloc-[0-9a-f]+$` <br /> |
|
||||
| `subnetID` _string_ | SubnetID is the public AWS subnet for the load balancer of this replica, for example subnet-0123abcd. The<br />Elastic IP takes the availability zone of this subnet. The operator sets it as the<br />service.beta.kubernetes.io/aws-load-balancer-subnets annotation on the Service of the replica. | | Pattern: `^subnet-[0-9a-f]+$` <br /> |
|
||||
|
||||
|
||||
#### PeerRelayEndpoint
|
||||
@@ -620,7 +620,7 @@ _Appears in:_
|
||||
| Field | Description | Default | Validation |
|
||||
| --- | --- | --- | --- |
|
||||
| `replica` _integer_ | Replica is the zero-based index of the peer relay replica this endpoint targets. | | |
|
||||
| `address` _string_ | Address is the public IP or hostname the cloud has allocated for this replica's LoadBalancer Service, or<br />an address supplied via spec.staticEndpoints. Peers reach this relay by connecting to Address:Port over UDP. | | |
|
||||
| `address` _string_ | Address is a public IP address of the load balancer for this replica, or an address from<br />spec.staticEndpoints. Peers connect to Address:Port over UDP. | | |
|
||||
| `port` _integer_ | Port is the UDP port the peer relay listens on. | | |
|
||||
|
||||
|
||||
@@ -657,8 +657,10 @@ _Appears in:_
|
||||
|
||||
| Field | Description | Default | Validation |
|
||||
| --- | --- | --- | --- |
|
||||
| `annotations` _object (keys:string, values:string)_ | Annotations to apply to the LoadBalancer service. Any annotations that conflict with those used by known<br />cloud providers to ensure IP addresses rather than DNS names are ignored. | | Optional: \{\} <br /> |
|
||||
| `port` _integer_ | Port is the UDP port each peer relay replica listens on and that its LoadBalancer Service exposes<br />externally. The two are always equal: the relay advertises address:port to peers, so the load balancer<br />must forward without rewriting the port. Changing the port on an existing PeerRelay briefly interrupts<br />relay traffic while the load balancer updates. Defaults to 41641. | 41641 | Maximum: 65535 <br />Minimum: 1 <br />Optional: \{\} <br /> |
|
||||
| `annotations` _object (keys:string, values:string)_ | Annotations to apply to the LoadBalancer Service. The operator sets some cloud provider annotations so<br />that the Service gets a public IP address. These annotations override any conflicting value set here.<br />The operator sets service.beta.kubernetes.io/aws-load-balancer-ip-address-type to ipv4 unless it is set<br />here. Set it to dualstack on an IPv6 EKS cluster. When it is dualstack, the operator also sets<br />service.beta.kubernetes.io/aws-load-balancer-enable-prefix-for-ipv6-source-nat to on unless it is set here. | | Optional: \{\} <br /> |
|
||||
| `port` _integer_ | Port is the UDP port that each replica listens on and that its LoadBalancer Service exposes. The two<br />ports are always equal because the relay advertises address:port to peers. Changing the port briefly<br />interrupts relay traffic while the load balancer updates. Defaults to 41641. | 41641 | Maximum: 65535 <br />Minimum: 1 <br />Optional: \{\} <br /> |
|
||||
| `ipFamilyPolicy` _[IPFamilyPolicy](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.3/#ipfamilypolicy-v1-core)_ | IPFamilyPolicy sets spec.ipFamilyPolicy on each LoadBalancer Service. If unset, the cluster default<br />applies. Set RequireDualStack on a dual-stack cluster to get a load balancer with both IPv4 and IPv6<br />addresses. On EKS, use the service.beta.kubernetes.io/aws-load-balancer-ip-address-type annotation<br />instead. | | Enum: [SingleStack PreferDualStack RequireDualStack] <br />Optional: \{\} <br /> |
|
||||
| `ipFamilies` _[IPFamily](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.3/#ipfamily-v1-core) array_ | IPFamilies sets spec.ipFamilies on each LoadBalancer Service. If unset, the cluster default applies. The<br />value must agree with IPFamilyPolicy. | | MaxItems: 2 <br />items:Enum: [IPv4 IPv6] <br />Optional: \{\} <br /> |
|
||||
|
||||
|
||||
#### PeerRelaySpec
|
||||
@@ -674,14 +676,14 @@ _Appears in:_
|
||||
|
||||
| Field | Description | Default | Validation |
|
||||
| --- | --- | --- | --- |
|
||||
| `tags` _[Tags](#tags)_ | Tags that the Tailscale node will be tagged with.<br />Defaults to [tag:k8s].<br />To autoapprove the device defined by a PeerRelay,<br />you can configure Tailscale ACLs to give these tags the necessary<br />permissions.<br />See https://tailscale.com/kb/1337/acl-syntax#autoapprovers.<br />If you specify custom tags here, you must also make the operator an owner of these tags.<br />See https://tailscale.com/kb/1236/kubernetes-operator/#setting-up-the-kubernetes-operator.<br />Tags cannot be changed once a PeerRelay node has been created.<br />Tag values must be in form ^tag:[a-zA-Z][a-zA-Z0-9-]*$. | | Pattern: `^tag:[a-zA-Z][a-zA-Z0-9-]*$` <br />Type: string <br />Optional: \{\} <br /> |
|
||||
| `hostnamePrefix` _[HostnamePrefix](#hostnameprefix)_ | HostnamePrefix specifies the hostname prefix for each<br />replica. Each device will have the integer number<br />from its StatefulSet pod appended to this prefix to form the full hostname.<br />HostnamePrefix can contain lower case letters, numbers and dashes, it<br />must not start with a dash and must be between 1 and 62 characters long. | | Pattern: `^[a-z0-9][a-z0-9-]\{0,61\}$` <br />Type: string <br />Optional: \{\} <br /> |
|
||||
| `proxyClass` _string_ | ProxyClass is the name of the ProxyClass custom resource that<br />contains configuration options that should be applied to the<br />resources created for this PeerRelay. If unset, the operator will<br />create resources with the default configuration. | | Optional: \{\} <br /> |
|
||||
| `replicas` _integer_ | Replicas specifies how many devices to create. Set this to enable<br />high availability for peer relays.<br />https://tailscale.com/kb/1115/high-availability. Defaults to 1. | 1 | Minimum: 0 <br />Optional: \{\} <br /> |
|
||||
| `tailnet` _string_ | Tailnet specifies the tailnet this PeerRelay should join. If blank, the default tailnet is used. When set, this<br />name must match that of a valid Tailnet resource. This field is immutable and cannot be changed once set. | | Optional: \{\} <br /> |
|
||||
| `service` _[PeerRelayService](#peerrelayservice)_ | Service contains configuration values to modify the LoadBalancer service used to expose the peer relay. | | Optional: \{\} <br /> |
|
||||
| `staticEndpoints` _string array_ | StaticEndpoints is an optional list of address:port pairs on which every replica of this PeerRelay is<br />reachable from outside the cluster, for example the public side of a NAT or firewall in front of the<br />cluster. These supplement the endpoints discovered from each replica's LoadBalancer Service: they are<br />added to every replica's entries in status.endpoints and advertised to peers alongside them. If an entry<br />names an address a replica's load balancer already provides, the entry's port takes precedence for that<br />address. Each address may appear at most once across all entries.<br />Entries take the form accepted by Go's net/netip.ParseAddrPort, e.g. 203.0.113.1:41641. IPv6<br />addresses must be enclosed in brackets, e.g. [2001:db8::1]:41641. | | items:Pattern: `^(\d\{1,3\}(\.\d\{1,3\})\{3\}\|\[[0-9a-fA-F:.]+\]):\d\{1,5\}$` <br />Optional: \{\} <br /> |
|
||||
| `aws` _[PeerRelayAWS](#peerrelayaws)_ | AWS contains configuration for pinning each replica to a specific AWS Elastic IP and subnet. Only meaningful<br />when running on EKS with the AWS Load Balancer Controller. When set, the per-replica values override any<br />aws-load-balancer-eip-allocations or aws-load-balancer-subnets values supplied via spec.service.annotations.<br />Leave this unset unless the peer relays must be reachable on addresses you control. Pinning a subnet<br />confines a replica's load balancer to that subnet's availability zone, and an AWS Network Load Balancer<br />only forwards to targets in a zone that is enabled on it, so a replica whose pod is scheduled into any<br />other zone stops receiving traffic. Setting this field therefore also requires pinning the pods to the<br />matching zone with a ProxyClass, as described on ElasticIPs. Without this field the AWS Load Balancer<br />Controller instead provisions each load balancer across every zone it discovers, and the operator turns on<br />cross-zone load balancing so the replica is reachable wherever it happens to be scheduled, with no<br />scheduling constraints needed. | | Optional: \{\} <br /> |
|
||||
| `tags` _[Tags](#tags)_ | Tags to apply to each peer relay device. Defaults to [tag:k8s]. If you set custom tags, the operator must<br />be an owner of each tag. You cannot change the tags after the operator creates the device. Each tag must<br />match ^tag:[a-zA-Z][a-zA-Z0-9-]*$. See https://tailscale.com/docs/kubernetes-operator/peer-relay. | | Pattern: `^tag:[a-zA-Z][a-zA-Z0-9-]*$` <br />Type: string <br />Optional: \{\} <br /> |
|
||||
| `hostnamePrefix` _[HostnamePrefix](#hostnameprefix)_ | HostnamePrefix is the hostname prefix for each replica. The operator appends the replica index to form<br />the full hostname, for example my-relay-0. If unset, the operator uses the PeerRelay name. The prefix can<br />contain lowercase letters, numbers, and dashes. It must not start with a dash and must be 1 to 62<br />characters long. | | Pattern: `^[a-z0-9][a-z0-9-]\{0,61\}$` <br />Type: string <br />Optional: \{\} <br /> |
|
||||
| `proxyClass` _string_ | ProxyClass is the name of a ProxyClass to apply to the resources the operator creates for this PeerRelay.<br />If unset, the operator uses the default configuration. | | Optional: \{\} <br /> |
|
||||
| `replicas` _integer_ | Replicas is the number of peer relay devices to run. Run more than one for high availability. Defaults<br />to 1. See https://tailscale.com/kb/1115/high-availability. | 1 | Minimum: 0 <br />Optional: \{\} <br /> |
|
||||
| `tailnet` _string_ | Tailnet is the name of the Tailnet resource for the tailnet this PeerRelay joins. If unset, the operator<br />uses the default tailnet. You cannot change this field after you set it. | | Optional: \{\} <br /> |
|
||||
| `service` _[PeerRelayService](#peerrelayservice)_ | Service configures the LoadBalancer Service that the operator creates for each replica. | | Optional: \{\} <br /> |
|
||||
| `staticEndpoints` _string array_ | StaticEndpoints lists extra address:port pairs where every replica is reachable from outside the<br />cluster, for example the public side of a NAT or firewall. The operator adds each entry to every replica<br />in status.endpoints and advertises it to peers. If an entry has the same address as a load balancer, the<br />port of the entry replaces the port of the load balancer. An address can appear in only one entry. Write<br />each entry as an IP address and port, for example 203.0.113.1:41641. Put IPv6 addresses in brackets, for<br />example [2001:db8::1]:41641. | | items:Pattern: `^(\d\{1,3\}(\.\d\{1,3\})\{3\}\|\[[0-9a-fA-F:.]+\]):\d\{1,5\}$` <br />Optional: \{\} <br /> |
|
||||
| `aws` _[PeerRelayAWS](#peerrelayaws)_ | AWS pins each replica to an AWS Elastic IP and subnet. It applies only on EKS with the AWS Load Balancer<br />Controller. Leave it unset unless the peer relays must be reachable on addresses you control. When set,<br />you must also pin the pods to one availability zone, as ElasticIPs describes. If unset, each load<br />balancer spans every zone the controller finds, and the replica is reachable in any zone. See<br />https://tailscale.com/docs/kubernetes-operator/peer-relay. | | Optional: \{\} <br /> |
|
||||
|
||||
|
||||
#### PeerRelayStatus
|
||||
@@ -698,7 +700,7 @@ _Appears in:_
|
||||
| Field | Description | Default | Validation |
|
||||
| --- | --- | --- | --- |
|
||||
| `conditions` _[Condition](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.3/#condition-v1-meta) array_ | | | Optional: \{\} <br /> |
|
||||
| `endpoints` _[PeerRelayEndpoint](#peerrelayendpoint) array_ | Endpoints lists the public address:port pairs each peer relay replica is reachable on. Entries appear as the<br />underlying cloud provisions each Service. A replica has one entry per address its LoadBalancer Service was<br />given, which is usually one, but a load balancer spanning several availability zones has an address in each<br />and every one of them is listed. Entries from spec.staticEndpoints are listed for every replica in addition<br />to the load balancer addresses. | | Optional: \{\} <br /> |
|
||||
| `endpoints` _[PeerRelayEndpoint](#peerrelayendpoint) array_ | Endpoints lists the public address:port pairs where each replica is reachable. The operator adds entries<br />as the cloud provisions each Service. A replica has one entry for each address of its load balancer. A<br />load balancer has more than one address when it spans several availability zones or both address<br />families. Every replica also has one entry for each entry in spec.staticEndpoints. | | Optional: \{\} <br /> |
|
||||
|
||||
|
||||
#### Pod
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
package v1alpha1
|
||||
|
||||
import (
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
)
|
||||
|
||||
@@ -47,122 +48,111 @@ type PeerRelayList struct {
|
||||
|
||||
// +kubebuilder:validation:XValidation:rule="!has(self.aws) || !has(self.aws.elasticIPs) || self.aws.elasticIPs.size() >= self.replicas",message="spec.aws.elasticIPs must contain at least one entry per replica"
|
||||
type PeerRelaySpec struct {
|
||||
// Tags that the Tailscale node will be tagged with.
|
||||
// Defaults to [tag:k8s].
|
||||
// To autoapprove the device defined by a PeerRelay,
|
||||
// you can configure Tailscale ACLs to give these tags the necessary
|
||||
// permissions.
|
||||
// See https://tailscale.com/kb/1337/acl-syntax#autoapprovers.
|
||||
// If you specify custom tags here, you must also make the operator an owner of these tags.
|
||||
// See https://tailscale.com/kb/1236/kubernetes-operator/#setting-up-the-kubernetes-operator.
|
||||
// Tags cannot be changed once a PeerRelay node has been created.
|
||||
// Tag values must be in form ^tag:[a-zA-Z][a-zA-Z0-9-]*$.
|
||||
// Tags to apply to each peer relay device. Defaults to [tag:k8s]. If you set custom tags, the operator must
|
||||
// be an owner of each tag. You cannot change the tags after the operator creates the device. Each tag must
|
||||
// match ^tag:[a-zA-Z][a-zA-Z0-9-]*$. See https://tailscale.com/docs/kubernetes-operator/peer-relay.
|
||||
// +optional
|
||||
Tags Tags `json:"tags,omitempty"`
|
||||
|
||||
// HostnamePrefix specifies the hostname prefix for each
|
||||
// replica. Each device will have the integer number
|
||||
// from its StatefulSet pod appended to this prefix to form the full hostname.
|
||||
// HostnamePrefix can contain lower case letters, numbers and dashes, it
|
||||
// must not start with a dash and must be between 1 and 62 characters long.
|
||||
// HostnamePrefix is the hostname prefix for each replica. The operator appends the replica index to form
|
||||
// the full hostname, for example my-relay-0. If unset, the operator uses the PeerRelay name. The prefix can
|
||||
// contain lowercase letters, numbers, and dashes. It must not start with a dash and must be 1 to 62
|
||||
// characters long.
|
||||
// +optional
|
||||
HostnamePrefix HostnamePrefix `json:"hostnamePrefix,omitzero"`
|
||||
|
||||
// ProxyClass is the name of the ProxyClass custom resource that
|
||||
// contains configuration options that should be applied to the
|
||||
// resources created for this PeerRelay. If unset, the operator will
|
||||
// create resources with the default configuration.
|
||||
// ProxyClass is the name of a ProxyClass to apply to the resources the operator creates for this PeerRelay.
|
||||
// If unset, the operator uses the default configuration.
|
||||
// +optional
|
||||
ProxyClass string `json:"proxyClass,omitempty"`
|
||||
|
||||
// Replicas specifies how many devices to create. Set this to enable
|
||||
// high availability for peer relays.
|
||||
// https://tailscale.com/kb/1115/high-availability. Defaults to 1.
|
||||
// Replicas is the number of peer relay devices to run. Run more than one for high availability. Defaults
|
||||
// to 1. See https://tailscale.com/kb/1115/high-availability.
|
||||
// +optional
|
||||
// +kubebuilder:validation:Minimum=0
|
||||
// +kubebuilder:default=1
|
||||
Replicas *int32 `json:"replicas,omitzero"`
|
||||
|
||||
// Tailnet specifies the tailnet this PeerRelay should join. If blank, the default tailnet is used. When set, this
|
||||
// name must match that of a valid Tailnet resource. This field is immutable and cannot be changed once set.
|
||||
// Tailnet is the name of the Tailnet resource for the tailnet this PeerRelay joins. If unset, the operator
|
||||
// uses the default tailnet. You cannot change this field after you set it.
|
||||
// +optional
|
||||
// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="PeerRelay tailnet is immutable"
|
||||
Tailnet string `json:"tailnet,omitempty"`
|
||||
|
||||
// Service contains configuration values to modify the LoadBalancer service used to expose the peer relay.
|
||||
// Service configures the LoadBalancer Service that the operator creates for each replica.
|
||||
// +optional
|
||||
Service *PeerRelayService `json:"service,omitzero"`
|
||||
|
||||
// StaticEndpoints is an optional list of address:port pairs on which every replica of this PeerRelay is
|
||||
// reachable from outside the cluster, for example the public side of a NAT or firewall in front of the
|
||||
// cluster. These supplement the endpoints discovered from each replica's LoadBalancer Service: they are
|
||||
// added to every replica's entries in status.endpoints and advertised to peers alongside them. If an entry
|
||||
// names an address a replica's load balancer already provides, the entry's port takes precedence for that
|
||||
// address. Each address may appear at most once across all entries.
|
||||
// Entries take the form accepted by Go's net/netip.ParseAddrPort, e.g. 203.0.113.1:41641. IPv6
|
||||
// addresses must be enclosed in brackets, e.g. [2001:db8::1]:41641.
|
||||
// StaticEndpoints lists extra address:port pairs where every replica is reachable from outside the
|
||||
// cluster, for example the public side of a NAT or firewall. The operator adds each entry to every replica
|
||||
// in status.endpoints and advertises it to peers. If an entry has the same address as a load balancer, the
|
||||
// port of the entry replaces the port of the load balancer. An address can appear in only one entry. Write
|
||||
// each entry as an IP address and port, for example 203.0.113.1:41641. Put IPv6 addresses in brackets, for
|
||||
// example [2001:db8::1]:41641.
|
||||
// +optional
|
||||
// +listType=set
|
||||
// +kubebuilder:validation:items:Pattern=`^(\d{1,3}(\.\d{1,3}){3}|\[[0-9a-fA-F:.]+\]):\d{1,5}$`
|
||||
StaticEndpoints []string `json:"staticEndpoints,omitempty"`
|
||||
|
||||
// AWS contains configuration for pinning each replica to a specific AWS Elastic IP and subnet. Only meaningful
|
||||
// when running on EKS with the AWS Load Balancer Controller. When set, the per-replica values override any
|
||||
// aws-load-balancer-eip-allocations or aws-load-balancer-subnets values supplied via spec.service.annotations.
|
||||
//
|
||||
// Leave this unset unless the peer relays must be reachable on addresses you control. Pinning a subnet
|
||||
// confines a replica's load balancer to that subnet's availability zone, and an AWS Network Load Balancer
|
||||
// only forwards to targets in a zone that is enabled on it, so a replica whose pod is scheduled into any
|
||||
// other zone stops receiving traffic. Setting this field therefore also requires pinning the pods to the
|
||||
// matching zone with a ProxyClass, as described on ElasticIPs. Without this field the AWS Load Balancer
|
||||
// Controller instead provisions each load balancer across every zone it discovers, and the operator turns on
|
||||
// cross-zone load balancing so the replica is reachable wherever it happens to be scheduled, with no
|
||||
// scheduling constraints needed.
|
||||
// AWS pins each replica to an AWS Elastic IP and subnet. It applies only on EKS with the AWS Load Balancer
|
||||
// Controller. Leave it unset unless the peer relays must be reachable on addresses you control. When set,
|
||||
// you must also pin the pods to one availability zone, as ElasticIPs describes. If unset, each load
|
||||
// balancer spans every zone the controller finds, and the replica is reachable in any zone. See
|
||||
// https://tailscale.com/docs/kubernetes-operator/peer-relay.
|
||||
// +optional
|
||||
AWS *PeerRelayAWS `json:"aws,omitzero"`
|
||||
}
|
||||
|
||||
type PeerRelayService struct {
|
||||
// Annotations to apply to the LoadBalancer service. Any annotations that conflict with those used by known
|
||||
// cloud providers to ensure IP addresses rather than DNS names are ignored.
|
||||
// Annotations to apply to the LoadBalancer Service. The operator sets some cloud provider annotations so
|
||||
// that the Service gets a public IP address. These annotations override any conflicting value set here.
|
||||
//
|
||||
// The operator sets service.beta.kubernetes.io/aws-load-balancer-ip-address-type to ipv4 unless it is set
|
||||
// here. Set it to dualstack on an IPv6 EKS cluster. When it is dualstack, the operator also sets
|
||||
// service.beta.kubernetes.io/aws-load-balancer-enable-prefix-for-ipv6-source-nat to on unless it is set here.
|
||||
// +optional
|
||||
Annotations map[string]string `json:"annotations,omitempty"`
|
||||
|
||||
// Port is the UDP port each peer relay replica listens on and that its LoadBalancer Service exposes
|
||||
// externally. The two are always equal: the relay advertises address:port to peers, so the load balancer
|
||||
// must forward without rewriting the port. Changing the port on an existing PeerRelay briefly interrupts
|
||||
// relay traffic while the load balancer updates. Defaults to 41641.
|
||||
// Port is the UDP port that each replica listens on and that its LoadBalancer Service exposes. The two
|
||||
// ports are always equal because the relay advertises address:port to peers. Changing the port briefly
|
||||
// interrupts relay traffic while the load balancer updates. Defaults to 41641.
|
||||
// +optional
|
||||
// +kubebuilder:default=41641
|
||||
// +kubebuilder:validation:Minimum=1
|
||||
// +kubebuilder:validation:Maximum=65535
|
||||
Port *uint16 `json:"port,omitzero"`
|
||||
|
||||
// IPFamilyPolicy sets spec.ipFamilyPolicy on each LoadBalancer Service. If unset, the cluster default
|
||||
// applies. Set RequireDualStack on a dual-stack cluster to get a load balancer with both IPv4 and IPv6
|
||||
// addresses. On EKS, use the service.beta.kubernetes.io/aws-load-balancer-ip-address-type annotation
|
||||
// instead.
|
||||
// +optional
|
||||
// +kubebuilder:validation:Enum=SingleStack;PreferDualStack;RequireDualStack
|
||||
IPFamilyPolicy *corev1.IPFamilyPolicy `json:"ipFamilyPolicy,omitzero"`
|
||||
|
||||
// IPFamilies sets spec.ipFamilies on each LoadBalancer Service. If unset, the cluster default applies. The
|
||||
// value must agree with IPFamilyPolicy.
|
||||
// +optional
|
||||
// +listType=atomic
|
||||
// +kubebuilder:validation:MaxItems=2
|
||||
// +kubebuilder:validation:items:Enum=IPv4;IPv6
|
||||
IPFamilies []corev1.IPFamily `json:"ipFamilies,omitempty"`
|
||||
}
|
||||
|
||||
// PeerRelayAWS contains AWS-specific configuration for a PeerRelay.
|
||||
type PeerRelayAWS struct {
|
||||
// ElasticIPs pins each replica to a specific AWS EIP allocation and subnet. Only meaningful when Network Load
|
||||
// Balancers are provisioned by the AWS Load Balancer Controller. ElasticIPs supplies one allocation-subnet pair
|
||||
// per replica: replica N uses ElasticIPs[N]. The list must be at least as long as spec.replicas so every replica
|
||||
// has a distinct EIP; extra entries are permitted so that scale-up doesn't immediately trip validation.
|
||||
// ElasticIPs lists one Elastic IP allocation and subnet for each replica. Replica N uses entry N. The list
|
||||
// must have at least spec.replicas entries. Extra entries are allowed so that a scale-up does not fail
|
||||
// validation.
|
||||
//
|
||||
// Pinning a subnet enables only that subnet's availability zone on the replica's load balancer, and a Network
|
||||
// Load Balancer only forwards to targets in an enabled zone. Nothing constrains the scheduler to place the
|
||||
// replica's pod in that zone, so a pod scheduled elsewhere, including after a reschedule, becomes unreachable
|
||||
// on its Elastic IP while still appearing healthy.
|
||||
// A load balancer only forwards to pods in the availability zone of its subnet. A pod scheduled into
|
||||
// another zone, for example after a reschedule, is unreachable on its Elastic IP but still appears healthy.
|
||||
// To prevent this, use subnets in one zone. Then pin the pods to that zone with a ProxyClass that sets
|
||||
// spec.statefulSet.pod.nodeSelector to topology.kubernetes.io/zone. A ProxyClass applies to every replica,
|
||||
// so this removes the zone redundancy of multiple replicas.
|
||||
//
|
||||
// Every replica of a PeerRelay shares one pod template, so a ProxyClass referenced by spec.proxyClass can
|
||||
// confine the pods to a zone but cannot place different replicas in different zones. To use this field
|
||||
// safely, name subnets in a single availability zone and pin the pods to that same zone with a ProxyClass
|
||||
// setting spec.statefulSet.pod.nodeSelector to topology.kubernetes.io/zone. Note that this trades the zone
|
||||
// redundancy that running several replicas would otherwise buy. Spreading replicas across zones with their
|
||||
// own Elastic IPs needs a per-replica scheduling constraint that neither PeerRelay nor ProxyClass can
|
||||
// currently express.
|
||||
//
|
||||
// When set, the reconciler stamps
|
||||
// service.beta.kubernetes.io/aws-load-balancer-eip-allocations and
|
||||
// service.beta.kubernetes.io/aws-load-balancer-subnets on each per-replica Service, overriding any values in
|
||||
// spec.service.annotations.
|
||||
// These values override the service.beta.kubernetes.io/aws-load-balancer-eip-allocations and
|
||||
// service.beta.kubernetes.io/aws-load-balancer-subnets annotations in spec.service.annotations.
|
||||
// +listType=atomic
|
||||
// +kubebuilder:validation:MinItems=1
|
||||
ElasticIPs []PeerRelayAWSElasticIP `json:"elasticIPs"`
|
||||
@@ -170,16 +160,15 @@ type PeerRelayAWS struct {
|
||||
|
||||
// PeerRelayAWSElasticIP pairs an EIP allocation with the subnet it is attached to.
|
||||
type PeerRelayAWSElasticIP struct {
|
||||
// AllocationID is the AWS EIP allocation ID (e.g. eipalloc-0123abcd) whose public IP this replica is reachable
|
||||
// on. Stamped as service.beta.kubernetes.io/aws-load-balancer-eip-allocations on the replica's Service.
|
||||
// AllocationID is the AWS Elastic IP allocation ID for this replica, for example eipalloc-0123abcd. The
|
||||
// operator sets it as the service.beta.kubernetes.io/aws-load-balancer-eip-allocations annotation on the
|
||||
// Service of the replica.
|
||||
// +kubebuilder:validation:Pattern=`^eipalloc-[0-9a-f]+$`
|
||||
AllocationID string `json:"allocationID"`
|
||||
|
||||
// SubnetID is the AWS subnet the replica's load balancer is provisioned in (e.g. subnet-0123abcd). It must be
|
||||
// a public subnet, and no two replicas may name subnets in the same availability zone, since a load balancer
|
||||
// accepts only one Elastic IP per zone. A standard VPC Elastic IP is regional rather than zonal, so it takes
|
||||
// the zone of whichever subnet it is paired with here. Stamped as
|
||||
// service.beta.kubernetes.io/aws-load-balancer-subnets on the replica's Service.
|
||||
// SubnetID is the public AWS subnet for the load balancer of this replica, for example subnet-0123abcd. The
|
||||
// Elastic IP takes the availability zone of this subnet. The operator sets it as the
|
||||
// service.beta.kubernetes.io/aws-load-balancer-subnets annotation on the Service of the replica.
|
||||
// +kubebuilder:validation:Pattern=`^subnet-[0-9a-f]+$`
|
||||
SubnetID string `json:"subnetID"`
|
||||
}
|
||||
@@ -190,11 +179,10 @@ type PeerRelayStatus struct {
|
||||
// +optional
|
||||
Conditions []metav1.Condition `json:"conditions"`
|
||||
|
||||
// Endpoints lists the public address:port pairs each peer relay replica is reachable on. Entries appear as the
|
||||
// underlying cloud provisions each Service. A replica has one entry per address its LoadBalancer Service was
|
||||
// given, which is usually one, but a load balancer spanning several availability zones has an address in each
|
||||
// and every one of them is listed. Entries from spec.staticEndpoints are listed for every replica in addition
|
||||
// to the load balancer addresses.
|
||||
// Endpoints lists the public address:port pairs where each replica is reachable. The operator adds entries
|
||||
// as the cloud provisions each Service. A replica has one entry for each address of its load balancer. A
|
||||
// load balancer has more than one address when it spans several availability zones or both address
|
||||
// families. Every replica also has one entry for each entry in spec.staticEndpoints.
|
||||
// +listType=map
|
||||
// +listMapKey=replica
|
||||
// +listMapKey=address
|
||||
@@ -206,8 +194,8 @@ type PeerRelayEndpoint struct {
|
||||
// Replica is the zero-based index of the peer relay replica this endpoint targets.
|
||||
Replica int32 `json:"replica"`
|
||||
|
||||
// Address is the public IP or hostname the cloud has allocated for this replica's LoadBalancer Service, or
|
||||
// an address supplied via spec.staticEndpoints. Peers reach this relay by connecting to Address:Port over UDP.
|
||||
// Address is a public IP address of the load balancer for this replica, or an address from
|
||||
// spec.staticEndpoints. Peers connect to Address:Port over UDP.
|
||||
Address string `json:"address"`
|
||||
|
||||
// Port is the UDP port the peer relay listens on.
|
||||
|
||||
@@ -684,6 +684,16 @@ func (in *PeerRelayService) DeepCopyInto(out *PeerRelayService) {
|
||||
*out = new(uint16)
|
||||
**out = **in
|
||||
}
|
||||
if in.IPFamilyPolicy != nil {
|
||||
in, out := &in.IPFamilyPolicy, &out.IPFamilyPolicy
|
||||
*out = new(corev1.IPFamilyPolicy)
|
||||
**out = **in
|
||||
}
|
||||
if in.IPFamilies != nil {
|
||||
in, out := &in.IPFamilies, &out.IPFamilies
|
||||
*out = make([]corev1.IPFamily, len(*in))
|
||||
copy(*out, *in)
|
||||
}
|
||||
}
|
||||
|
||||
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PeerRelayService.
|
||||
|
||||
@@ -304,9 +304,16 @@ func (r *Reconciler) createOrUpdate(ctx context.Context, logger *zap.SugaredLogg
|
||||
return reconcile.Result{RequeueAfter: 30 * time.Second}, nil
|
||||
}
|
||||
|
||||
return reconcile.Result{}, nil
|
||||
// Keep re-resolving once ready. A cloud publishes DNS records for a new load balancer over several minutes,
|
||||
// one address family or zone at a time, and nothing in the cluster changes when a record lands, so without
|
||||
// a periodic pass whatever subset the first lookup returned would be advertised for good.
|
||||
return reconcile.Result{RequeueAfter: ReadyResyncInterval}, nil
|
||||
}
|
||||
|
||||
// ReadyResyncInterval is how often a ready PeerRelay is reconciled to pick up load balancer addresses that DNS
|
||||
// started returning after the endpoints were last published.
|
||||
const ReadyResyncInterval = 5 * time.Minute
|
||||
|
||||
func peerRelayReady(pr *tsapi.PeerRelay) bool {
|
||||
for _, c := range pr.Status.Conditions {
|
||||
if c.Type == string(tsapi.PeerRelayReady) {
|
||||
|
||||
@@ -52,6 +52,11 @@ func testResolver(_ context.Context, _ string, host string) ([]netip.Addr, error
|
||||
netip.MustParseAddr("203.0.113.30"),
|
||||
netip.MustParseAddr("203.0.113.20"),
|
||||
},
|
||||
// A dualstack load balancer answers on both families.
|
||||
"dualstack.elb.amazonaws.com": {
|
||||
netip.MustParseAddr("2001:db8::10"),
|
||||
netip.MustParseAddr("203.0.113.40"),
|
||||
},
|
||||
}
|
||||
|
||||
if addrs, ok := r[host]; ok {
|
||||
@@ -72,6 +77,8 @@ type expectedService struct {
|
||||
Annotations map[string]string
|
||||
AbsentLabels []string
|
||||
AbsentAnnotations []string
|
||||
IPFamilyPolicy *corev1.IPFamilyPolicy // asserted only when non-nil
|
||||
IPFamilies []corev1.IPFamily // asserted only when non-nil
|
||||
}
|
||||
|
||||
type statefulSetSpec struct {
|
||||
@@ -266,6 +273,93 @@ func TestReconciler_Reconcile(t *testing.T) {
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
// The AWS address type is a default rather than a forced value. On an IPv6 EKS cluster the pods are
|
||||
// IPv6 and the AWS Load Balancer Controller refuses to put them behind an IPv4 load balancer, so the
|
||||
// user must be able to ask for dualstack. A dualstack UDP load balancer also needs IPv6 source NAT
|
||||
// or the controller rejects it, so that is defaulted on alongside.
|
||||
Name: "dualstack-address-type-is-honoured",
|
||||
Request: reconcile.Request{NamespacedName: types.NamespacedName{Name: "test"}},
|
||||
PeerRelay: &tsapi.PeerRelay{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test"},
|
||||
Spec: tsapi.PeerRelaySpec{
|
||||
Service: &tsapi.PeerRelayService{Annotations: map[string]string{
|
||||
ipAddressTypeAnnotation: "dualstack",
|
||||
}},
|
||||
},
|
||||
},
|
||||
ExpectedServices: []expectedService{
|
||||
{
|
||||
Name: "peerrelay-test-0",
|
||||
Annotations: map[string]string{
|
||||
ipAddressTypeAnnotation: "dualstack",
|
||||
ipv6SourceNATAnnotation: "on",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
// A user who sets the source NAT annotation themselves keeps their value, even when it disagrees
|
||||
// with what the operator would have chosen.
|
||||
Name: "dualstack-respects-user-source-nat",
|
||||
Request: reconcile.Request{NamespacedName: types.NamespacedName{Name: "test"}},
|
||||
PeerRelay: &tsapi.PeerRelay{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test"},
|
||||
Spec: tsapi.PeerRelaySpec{
|
||||
Service: &tsapi.PeerRelayService{Annotations: map[string]string{
|
||||
ipAddressTypeAnnotation: "dualstack",
|
||||
ipv6SourceNATAnnotation: "off",
|
||||
}},
|
||||
},
|
||||
},
|
||||
ExpectedServices: []expectedService{
|
||||
{
|
||||
Name: "peerrelay-test-0",
|
||||
Annotations: map[string]string{
|
||||
ipAddressTypeAnnotation: "dualstack",
|
||||
ipv6SourceNATAnnotation: "off",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
// Without a user value the address type defaults to ipv4 and source NAT is left alone, since an
|
||||
// IPv4 load balancer has no use for it and the controller rejects it on one.
|
||||
Name: "ipv4-address-type-by-default",
|
||||
Request: reconcile.Request{NamespacedName: types.NamespacedName{Name: "test"}},
|
||||
PeerRelay: &tsapi.PeerRelay{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test"},
|
||||
},
|
||||
ExpectedServices: []expectedService{
|
||||
{
|
||||
Name: "peerrelay-test-0",
|
||||
Annotations: map[string]string{ipAddressTypeAnnotation: "ipv4"},
|
||||
AbsentAnnotations: []string{ipv6SourceNATAnnotation},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
// spec.service.ipFamilyPolicy and ipFamilies pass straight through to the Service so a dual-stack
|
||||
// cluster can be asked for a load balancer that answers on both families.
|
||||
Name: "ip-families-pass-through",
|
||||
Request: reconcile.Request{NamespacedName: types.NamespacedName{Name: "test"}},
|
||||
PeerRelay: &tsapi.PeerRelay{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test"},
|
||||
Spec: tsapi.PeerRelaySpec{
|
||||
Service: &tsapi.PeerRelayService{
|
||||
IPFamilyPolicy: new(corev1.IPFamilyPolicyRequireDualStack),
|
||||
IPFamilies: []corev1.IPFamily{corev1.IPv6Protocol, corev1.IPv4Protocol},
|
||||
},
|
||||
},
|
||||
},
|
||||
ExpectedServices: []expectedService{
|
||||
{
|
||||
Name: "peerrelay-test-0",
|
||||
IPFamilyPolicy: new(corev1.IPFamilyPolicyRequireDualStack),
|
||||
IPFamilies: []corev1.IPFamily{corev1.IPv6Protocol, corev1.IPv4Protocol},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
// The reconciler applies via server-side apply, so a drifted Service (wrong Spec.Type, wrong Ports)
|
||||
// is restored on reconcile. Kubernetes owns the merge with fields belonging to other managers
|
||||
@@ -389,6 +483,40 @@ func TestReconciler_Reconcile(t *testing.T) {
|
||||
ExpectedReadyStatus: metav1.ConditionTrue,
|
||||
ExpectedReadyReason: peerrelay.ReasonReady,
|
||||
},
|
||||
{
|
||||
// A dualstack load balancer's hostname resolves to addresses in both families, and every one of them
|
||||
// is advertised so IPv6-only peers get a direct path while IPv4 peers keep theirs.
|
||||
Name: "dualstack-hostname-advertises-both-families",
|
||||
Request: reconcile.Request{NamespacedName: types.NamespacedName{Name: "test"}},
|
||||
PeerRelay: &tsapi.PeerRelay{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "test"},
|
||||
Spec: tsapi.PeerRelaySpec{
|
||||
Service: &tsapi.PeerRelayService{
|
||||
Annotations: map[string]string{ipAddressTypeAnnotation: "dualstack"},
|
||||
},
|
||||
},
|
||||
},
|
||||
ExistingResources: []client.Object{
|
||||
managedServiceWithLB("test", 0, "", "dualstack.elb.amazonaws.com"),
|
||||
managedStatefulSet("test", 1, 1),
|
||||
},
|
||||
ExpectedServices: []expectedService{{Name: "peerrelay-test-0"}},
|
||||
ExpectedEndpoints: []tsapi.PeerRelayEndpoint{
|
||||
{Replica: 0, Address: "2001:db8::10", Port: 41641},
|
||||
{Replica: 0, Address: "203.0.113.40", Port: 41641},
|
||||
},
|
||||
ExpectedConfigs: map[string]expectedConfig{
|
||||
"peerrelay-test-0-config": {
|
||||
RelayServerPort: 41641,
|
||||
RelayServerStaticEndpoints: []netip.AddrPort{
|
||||
netip.MustParseAddrPort("[2001:db8::10]:41641"),
|
||||
netip.MustParseAddrPort("203.0.113.40:41641"),
|
||||
},
|
||||
},
|
||||
},
|
||||
ExpectedReadyStatus: metav1.ConditionTrue,
|
||||
ExpectedReadyReason: peerrelay.ReasonReady,
|
||||
},
|
||||
{
|
||||
// Unresolvable hostname (e.g. NXDOMAIN during LB provisioning) stays in Pending — no hard error,
|
||||
// reconciler will retry when the hostname comes online. Service still carries the eip-allocations
|
||||
@@ -955,7 +1083,7 @@ func TestReconciler_Reconcile(t *testing.T) {
|
||||
Logger: logger.Sugar(),
|
||||
})
|
||||
|
||||
_, err = r.Reconcile(t.Context(), tc.Request)
|
||||
res, err := r.Reconcile(t.Context(), tc.Request)
|
||||
if tc.ExpectsError && err == nil {
|
||||
t.Fatalf("expected error, got none")
|
||||
}
|
||||
@@ -963,6 +1091,12 @@ func TestReconciler_Reconcile(t *testing.T) {
|
||||
t.Fatalf("expected no error, got %v", err)
|
||||
}
|
||||
|
||||
// A ready PeerRelay is still revisited so load balancer addresses DNS publishes late are picked up,
|
||||
// while one that is not yet ready polls more eagerly.
|
||||
if tc.ExpectedReadyStatus == metav1.ConditionTrue && res.RequeueAfter != peerrelay.ReadyResyncInterval {
|
||||
t.Errorf("expected ready PeerRelay to requeue after %v, got %v", peerrelay.ReadyResyncInterval, res.RequeueAfter)
|
||||
}
|
||||
|
||||
var svcs corev1.ServiceList
|
||||
if err = fc.List(t.Context(), &svcs, client.InNamespace(tailscaleNamespace)); err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -1164,6 +1298,15 @@ func assertService(t *testing.T, want expectedService, got *corev1.Service) {
|
||||
t.Errorf("Service %q: expected type %q, got %q", want.Name, want.Type, got.Spec.Type)
|
||||
}
|
||||
|
||||
if want.IPFamilyPolicy != nil {
|
||||
if got.Spec.IPFamilyPolicy == nil || *got.Spec.IPFamilyPolicy != *want.IPFamilyPolicy {
|
||||
t.Errorf("Service %q: expected ipFamilyPolicy %v, got %v", want.Name, *want.IPFamilyPolicy, got.Spec.IPFamilyPolicy)
|
||||
}
|
||||
}
|
||||
if want.IPFamilies != nil && !slices.Equal(got.Spec.IPFamilies, want.IPFamilies) {
|
||||
t.Errorf("Service %q: expected ipFamilies %v, got %v", want.Name, want.IPFamilies, got.Spec.IPFamilies)
|
||||
}
|
||||
|
||||
if want.Port != 0 || want.Protocol != "" || want.NodePort != 0 {
|
||||
if len(got.Spec.Ports) != 1 {
|
||||
t.Fatalf("Service %q: expected exactly one port, got %d", want.Name, len(got.Spec.Ports))
|
||||
@@ -1241,6 +1384,8 @@ const (
|
||||
eipAllocationsAnnotation = "service.beta.kubernetes.io/aws-load-balancer-eip-allocations"
|
||||
subnetsAnnotation = "service.beta.kubernetes.io/aws-load-balancer-subnets"
|
||||
lbAttributesAnnotation = "service.beta.kubernetes.io/aws-load-balancer-attributes"
|
||||
ipAddressTypeAnnotation = "service.beta.kubernetes.io/aws-load-balancer-ip-address-type"
|
||||
ipv6SourceNATAnnotation = "service.beta.kubernetes.io/aws-load-balancer-enable-prefix-for-ipv6-source-nat"
|
||||
)
|
||||
|
||||
func managedStatefulSet(prName string, replicas, ready int32) *appsv1.StatefulSet {
|
||||
|
||||
@@ -16,7 +16,6 @@ import (
|
||||
|
||||
"go.uber.org/zap"
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/apimachinery/pkg/util/intstr"
|
||||
|
||||
tsapi "tailscale.com/k8s-operator/apis/v1alpha1"
|
||||
@@ -43,16 +42,24 @@ const (
|
||||
annotationEIPAllocations = "service.beta.kubernetes.io/aws-load-balancer-eip-allocations"
|
||||
annotationSubnets = "service.beta.kubernetes.io/aws-load-balancer-subnets"
|
||||
annotationLBAttributes = "service.beta.kubernetes.io/aws-load-balancer-attributes"
|
||||
annotationIPAddressType = "service.beta.kubernetes.io/aws-load-balancer-ip-address-type"
|
||||
annotationIPv6SourceNAT = "service.beta.kubernetes.io/aws-load-balancer-enable-prefix-for-ipv6-source-nat"
|
||||
|
||||
// ipAddressTypeDualStack is the aws-load-balancer-ip-address-type value that gives a Network Load Balancer
|
||||
// both an IPv4 and an IPv6 address. It is the only value that can front IPv6 targets, which is what the pods
|
||||
// are on an IPv6 EKS cluster.
|
||||
ipAddressTypeDualStack = "dualstack"
|
||||
)
|
||||
|
||||
// cloudAnnotations are the cloud-provider-specific annotations applied to every generated LoadBalancer Service to
|
||||
// ensure the Service is provisioned with a publicly addressable IP rather than a DNS name.
|
||||
// ensure the Service is provisioned with a publicly addressable IP rather than a DNS name. They override anything
|
||||
// the user supplies in spec.service.annotations. Annotations the user may override are defaulted separately in
|
||||
// peerRelayServiceAnnotations.
|
||||
var cloudAnnotations = map[string]string{
|
||||
// AWS: provision an internet-facing NLB in IP target mode via the AWS Load Balancer Controller.
|
||||
"service.beta.kubernetes.io/aws-load-balancer-type": "external",
|
||||
"service.beta.kubernetes.io/aws-load-balancer-nlb-target-type": "ip",
|
||||
"service.beta.kubernetes.io/aws-load-balancer-scheme": "internet-facing",
|
||||
"service.beta.kubernetes.io/aws-load-balancer-ip-address-type": "ipv4",
|
||||
|
||||
// AWS: health check the pod over HTTP against containerboot's /healthz rather than the port the Service
|
||||
// forwards. A peer relay listens only on UDP, so the default TCP check against the traffic port can never
|
||||
@@ -102,6 +109,23 @@ func peerRelayServiceAnnotations(pr *tsapi.PeerRelay, idx int32) map[string]stri
|
||||
|
||||
maps.Copy(annotations, cloudAnnotations)
|
||||
|
||||
// Default the AWS load balancer to IPv4 only, but let the user ask for dualstack. On an IPv6 EKS cluster the
|
||||
// Service and pod targets are IPv6, and the AWS Load Balancer Controller refuses to put IPv6 targets behind
|
||||
// an IPv4 load balancer, so dualstack is the only address type that provisions there. A dualstack load
|
||||
// balancer still answers on IPv4, so peers without IPv6 reach the relay as before.
|
||||
if _, ok := annotations[annotationIPAddressType]; !ok {
|
||||
annotations[annotationIPAddressType] = "ipv4"
|
||||
}
|
||||
|
||||
// A dualstack Network Load Balancer forwarding UDP to IPv6 targets has to source NAT the client's address,
|
||||
// since the target cannot answer an IPv4 client directly, and the AWS Load Balancer Controller rejects the
|
||||
// Service outright when that isn't enabled. Turn it on unless the user has set it themselves.
|
||||
if annotations[annotationIPAddressType] == ipAddressTypeDualStack {
|
||||
if _, ok := annotations[annotationIPv6SourceNAT]; !ok {
|
||||
annotations[annotationIPv6SourceNAT] = "on"
|
||||
}
|
||||
}
|
||||
|
||||
// Unless the user has taken control of the load balancer attributes themselves, default cross-zone on. When no
|
||||
// subnet is pinned the AWS Load Balancer Controller spreads the load balancer over every zone it discovers,
|
||||
// and cross-zone is what lets all of those addresses reach the replica's pod regardless of the zone the
|
||||
@@ -126,17 +150,13 @@ func (r *Reconciler) peerRelayService(pr *tsapi.PeerRelay, idx int32) *corev1.Se
|
||||
name := replicaName(pr.Name, idx)
|
||||
port := int32(peerRelayPort(pr))
|
||||
|
||||
return &corev1.Service{
|
||||
TypeMeta: metav1.TypeMeta{
|
||||
APIVersion: "v1",
|
||||
Kind: "Service",
|
||||
},
|
||||
ObjectMeta: metav1.ObjectMeta{
|
||||
Name: name,
|
||||
Namespace: r.tailscaleNamespace,
|
||||
Labels: peerRelayServiceLabels(pr.Name, idx),
|
||||
Annotations: peerRelayServiceAnnotations(pr, idx),
|
||||
},
|
||||
svc := &corev1.Service{
|
||||
APIVersion: "v1",
|
||||
Kind: "Service",
|
||||
Name: name,
|
||||
Namespace: r.tailscaleNamespace,
|
||||
Labels: peerRelayServiceLabels(pr.Name, idx),
|
||||
Annotations: peerRelayServiceAnnotations(pr, idx),
|
||||
Spec: corev1.ServiceSpec{
|
||||
Type: corev1.ServiceTypeLoadBalancer,
|
||||
// The Service targets the specific StatefulSet pod for this replica. The StatefulSet controller
|
||||
@@ -154,6 +174,16 @@ func (r *Reconciler) peerRelayService(pr *tsapi.PeerRelay, idx int32) *corev1.Se
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
// The address families are only set when the user asked for them. Left unset, the API server defaults them
|
||||
// from the cluster's configuration and, since this is a server-side apply, the operator never takes ownership
|
||||
// of the fields, so the defaults stand.
|
||||
if pr.Spec.Service != nil {
|
||||
svc.Spec.IPFamilyPolicy = pr.Spec.Service.IPFamilyPolicy
|
||||
svc.Spec.IPFamilies = pr.Spec.Service.IPFamilies
|
||||
}
|
||||
|
||||
return svc
|
||||
}
|
||||
|
||||
func replicaIndexFromLabels(labels map[string]string) (int32, bool) {
|
||||
@@ -196,7 +226,10 @@ func (r *Reconciler) peerRelayEndpoints(ctx context.Context, logger *zap.Sugared
|
||||
// No IP was assigned, so fall back to resolving the hostname. This is primarily for AWS, where NLB-backed
|
||||
// Service resources are only ever given hostnames. The addresses behind such a hostname belong to the load
|
||||
// balancer and are fixed for its lifetime, whether they came from spec.aws.elasticIPs or were assigned by
|
||||
// AWS, so we resolve it here rather than making the user supply the address themselves.
|
||||
// AWS, so we resolve it here rather than making the user supply the address themselves. Both address
|
||||
// families are looked up so a dualstack load balancer's IPv6 addresses are advertised alongside its IPv4
|
||||
// ones, giving IPv6-only peers a direct path; an IPv4 load balancer has no AAAA records so gets the same
|
||||
// endpoints as before.
|
||||
for _, ing := range svc.Status.LoadBalancer.Ingress {
|
||||
if ing.Hostname == "" {
|
||||
continue
|
||||
@@ -205,7 +238,7 @@ func (r *Reconciler) peerRelayEndpoints(ctx context.Context, logger *zap.Sugared
|
||||
resolveCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
addrs, err := r.resolver(resolveCtx, "ip4", ing.Hostname)
|
||||
addrs, err := r.resolver(resolveCtx, "ip", ing.Hostname)
|
||||
if err != nil || len(addrs) == 0 {
|
||||
logger.Debugf("failed to resolve LoadBalancer hostname %q for Service %q: %v", ing.Hostname, svc.Name, err)
|
||||
// Preserve the previously-known endpoints (if any) so that a failure here doesn't erase
|
||||
|
||||
Reference in new issue
Block a user