net/sockstats uses both on darwin, ios, and android, so those builds
failed with ts_omit_clientmetrics.
Updates #22283
Change-Id: Ib242334dbcba639603fbf57a3b016efdecaea3d5
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
flowRemoved identified an address assignment by transit IP alone, but a
transit IP does not identify an assignment for the lifetime of a flow.
An assignment can be dropped while a flow created against it is still in
the client flow table, and the pools then hand the same transit IP back
out to an unrelated assignment. The stale removal was charged to that
stranger, dropping its activeFlowCount while its own flows were still
forwarding, so popExpired reclaimed both of its addresses back to the
pools mid-connection. If the stranger had no flows of its own the count
went to -1, which can never again satisfy popExpired's zero-flow test,
pinning it in the expiry heap and leaking both addresses for the life of
the process.
flowCreated now returns a function, which the datapath calls on removal
to clean up the flow.
Fixes https://github.com/tailscale/corp/issues/50760
Change-Id: I3693c487b1031737763b5e49f8f8400c72848e8e
Signed-off-by: Mazdak Nasab <mazdak.nasab@gmail.com>
extension.Shutdown closed client.addrsCh, but the hooks that feed that
channel are still live at that point. LocalBackend.shutdown shuts the
extension host down before closing the engine, and nothing unregisters
what installHooks registered on the DNS manager and the tun device, so a
DNS response arriving during the shutdown window still reaches
mapDNSResponse -> rewriteDNSResponse -> reserveAddresses ->
enqueueAddressAssignment, and sends on the closed channel. That's a
panic, not a dropped assignment.
Every send already happens under c.mu, so the client can shut its own
queue down: client.shutdown takes the lock, sets a closed flag and
drains the queue, and enqueueAddressAssignment refuses with
errClientShutdown from then on.
A late DNS response now fails the reservation instead of queueing it for
a sendLoop that is gone. reserveAddresses rolls back, returning the
magic and transit IPs to their pools, and the query servfails rather
than being answered with a magic IP that no connector will ever hear
about.
Fixes https://github.com/tailscale/corp/issues/51235
Change-Id: I712ba8493e77b8176e133decc6097efb52b1b4ab
Signed-off-by: Mazdak Nasab <mazdak.nasab@gmail.com>
Apple and Android both implement their own login orchestration, coordinating preference updates, backend startup, and interative login. This requires the clients to understand details about the backend lifecycle and has led to duplicated logic and ordering issues.
Android needs to avoid setting WantRunning=true via EditPrefs before calling Start, because doing so triggers Login on the existing control client, which Start then tears down, potentially canceling an in-flight registration before an authentication URL is returned (see https://github.com/tailscale/tailscale-android/commit/83057865e13b8f116a0871a3c6d6adf46f5c3d32). Apple has similar orchestration, including separate handling for auth keys, custom control servers, and reauth.
This adds a LocalBackend helper that accepts optional masked prefs and an auth key, prepares the desired startup prefs, and initiates or resumes auth. This avoids unnecessary control client resets, preserves the existing control client and auth url when the login configuration hasn't changed, and ensures WantRunning is applied as part of Start when restarting the backend. This can be used for:
-first time login with a custom control url or auth key
-resuming an existing interactive login without losing auth url
-retrying login without unnecessarily resetting the control client
-reauth without restarting the backend
A followup will expose this through localapi.
Updates tailscale/tailscale#18304
Signed-off-by: kari-ts <kari@tailscale.com>
When a Serve or Funnel handler serves a path, serveFileOrDirectory
used os.Stat and os.Open on it, and http.FileServer(http.Dir(dir)) for
a directory. All of these follow symlinks, so a symlink in place of a
served file or directory, or inside a served directory, could make
tailscaled serve any file it can read from elsewhere on the host.
Those directories may be writable by local users who can't configure
Serve, and a symlink placed there is followed at request time.
Resolve the configured path through an os.Root of its parent
directory, and serve a directory through an os.Root of its own.
os.Root refuses symlinks whose target leaves the root, and absolute
symlinks, and still follows relative symlinks that stay inside it. A
refused path inside a served directory is reported as not found, like
a missing file. A refused configured path is logged and reported as a
server error, like other errors reading it.
This changes behavior for setups that serve such symlinks on purpose,
including absolute symlinks and Windows junctions: a configured file
or directory that is one, or that points out of its parent directory,
now fails, and such links inside a served directory now return 404.
Symlinks higher up the configured path are still followed, and a
filesystem root can still be served. Opening the parent directory
needs read permission on it, where os.Stat only needed search
permission. Hardlinks are not covered, since os.Root can't tell them
apart from regular files.
RELNOTE: tailscale serve stops following symlinks out of served directories
Updates tailscale/corp#51116
Signed-off-by: Aditya Merchant <adityamerchant08@gmail.com>
A customer noticed that UDP reads returned EADDRNOTAVAIL during a link
change while the interface had no IPv4 address. magicsock returned the
error to wireguard-go, which treated it as permanent and shut down the
receive loop.
magicsock now retries the read when EADDRNOTAVAIL is returned on Unix.
Each error is counted in the magicsock_recv_udp_error_unknown metric.
EADDRNOTAVAIL isn't a documented result of a UDP recv. We suspect that
certain macOS network extensions return it while the network interface
is mid-change and has no address. We expect the error to stop once the
interface has an address again. A short delay between retries prevents
magicsock from spinning when the address doesn't come back right away.
Updates #22097
Change-Id: I70790f78d9488999ce00f9685aec3f0fabc15699
Signed-off-by: Nick Beirne <nickb@tailscale.com>
This includes vmtests to test that the tailscale client
renews PCP and NAT-PMP portmaps correctly, and unit
tests for the natlab server implementation. UPnP will
require a different approach since the client doesn't
currently have a way to notice that its requested
lifetime was capped.
It also makes TestPortmapCreate parallel. Both
portmapping tests run within 2.5min on my laptop.
TestPortmapRenew is gated behind `--run-slow-tests`.
Updates #21713
Change-Id: Ie2a5f5e906c1f4eda8d10f5d4eff28b159b455cc
Signed-off-by: Francois Marier <francois@tailscale.com>
Move Type and its constants to an unconditional file so syspolicy, and
its dependents such as tsnet, compile with ts_omit_clientmetrics.
Fixes#22252
Change-Id: Ib9ca1c192e3e1c570aa32d41650d72fbff1aff0c
Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net>
Avoid a startup panic when no log directory is available and the
sockstat logger is disabled by ts_omit_logtail.
Fixes#22246
Change-Id: I06efea76cdaf6cda4b454dbede8cf2e69bbd588d
Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net>
Avoid creating or reading tailscaled.log.conf with ts_omit_logtail.
Fixes#22244
Change-Id: If1ac073501aa6a42e4b7d9757980d2bb79fbabac
Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net>
Gate certificate fetching on ACME support and return an error when omitted.
Fixes#22250
Change-Id: I709928f1fe11b7f20cc854d299175ca3048c037e
Signed-off-by: Fredrik Fornwall <fredrik@fornwall.net>
GetWindowsDefault called GetAdaptersAddresses to build a map of every
adapter before walking the routing table. Enumerating adapters is
expensive on Windows and, since the call doesn't skip DNS info, it also
goes through the DNS Client service, which on recent Windows builds
enumerates every interface for each DNS server on each adapter.
In this PR, we query only what we need: the routing table, the metric
of each interface with a default route, and the interface row for routes
that beat the current best. We also identify the Tailscale interface by
its Wintun GUID rather than by its description, and move the GUID to
tsconst so that net/tstun and net/netmon share it.
Updates #17772
Updates #21885
Change-Id: I4c6a9d9ed791788fb0567154ee1dbe0f6a2c52a2
Signed-off-by: Nick Khyl <nickk@tailscale.com>
The test resumed a live session, so notifications already
on the wire (notably, spurious health change notifications)
could arrive ahead of the initial status.
In this PR, we add LocalBackend.SuspendWatchNotifications,
which suspends a session as if it had fallen behind,
and update the test to suspend the session and drain
pending notifications before resuming it.
Updates tailscale/corp#51067
Updates tailscale/corp#50256
Updates tailscale/corp#50907
Signed-off-by: Nick Khyl <nickk@tailscale.com>
Since edcc2c94, tailscaled closes the watch when a subscriber falls
behind. That doesn't work well for GUIs where closing the watch has side
effects. On Windows, the last client disconnecting can reset the backend
state unless unattended or always-on mode is enabled. On Android and
Apple platforms, nothing restarts a watch that ended this way, so the
UI silently stops updating.
Add ipn.NotifySuspendWhenBehind. A watcher that sets it stays connected
when it falls behind, but gets a Notify with Suspended set and nothing
else until the client resumes the session via the new resume-ipn-bus
LocalAPI endpoint or LocalBackend.ResumeWatchNotifications. Resuming
replays the initial Notify and new notifications follow from there.
Updates tailscale/corp#51067
Updates tailscale/corp#50256
Updates tailscale/corp#50907
Signed-off-by: Nick Khyl <nickk@tailscale.com>
openresolv before 3.13 exits 1 from "resolvconf -i" when its interface
directory holds no snippets. FreeBSD ships openresolv 3.9.2 in its base
system, so on FreeBSD hosts with static network configuration that status
aborted every DNS reconfiguration. A listing that prints nothing on stdout
or stderr now yields an empty base config whatever its exit status, which
also covers the exit status 2 that openresolv 3.17 returns.
The natlab DNSOpenresolv mode now provisions FreeBSD guests as well, by
deregistering the snippets dhclient added, and the openresolv VM tests run
on FreeBSD too. Without the net/dns change, the new FreeBSD test fails with
the health error reported in #11907.
Updates #11907
Updates #11129
Signed-off-by: Brendan Creane <bcreane@gmail.com>
Give the client a generation counter, bump it in reset, and stamp each
assignment with it as it is queued. insertTransitConnMapping rejects
assignments from an older generation; because that check happens under
c.mu it is exclusive with reset, which is what actually closes the
window. handleAddressAssignment also checks before sending, so an
assignment invalidated while it was still queued doesn't reach a
connector at all.
Fixes https://github.com/tailscale/corp/issues/51125
Change-Id: Ib524324b6e8a196fdcb5b4cd307854f61c7a0f04
Signed-off-by: Mazdak Nasab <mazdak.nasab@gmail.com>
Since aa5da2e5f, a peer that control adds to a running client arrives as a
netmap delta rather than a full netmap. The delta path never re-resolved
the exit node prefs. An exit node selected by IP before it was in the
netmap stayed unresolved, and an auto exit node stayed on the blackhole
route, until tailscaled restarted or a full netmap arrived.
Re-resolve the exit node in UpdateNetmapDelta when the delta can change
the result. Any upserted peer qualifies while an ExitNodeIP is set. An
exit node being upserted or coming online qualifies while an auto exit
node is unresolved. The selected auto exit node going offline, losing its
exit routes, or being removed qualifies once one is selected. The last
case replaces the separate offline check, so the exit node is resolved
once per delta. The auto exit node check walks every peer, so it only
runs in those cases.
On the delta path, the resolved prefs ride in the same Notify as the peer
changes, sent while b.mu is held. resolveExitNodeLocked no longer sends a
Notify; its callers do. The full-netmap path still sends the prefs in a
separate Notify after the netmap.
Fixes#22012
Updates #22206
Signed-off-by: Brendan Creane <bcreane@gmail.com>
client.reset clears the IP pools, assignments and byConnKey, but leaves
addrsCh alone. The channel is buffered, so assignments queued before the
reset are still delivered by sendLoop afterwards: each one POSTs to a
connector from the profile we just left, registering a transit IP this
client no longer knows about, and then fails in addTransitIPForConnector
because the assignment is gone from c.assignments.
Drain addrsCh at the end of reset. The drain is non-blocking, so holding
c.mu can't deadlock against reserveAddresses, which enqueues under the
same lock, or against sendLoop, which receives without it.
Fixes https://github.com/tailscale/tailscale/issues/22185
Change-Id: I579131bc54e82d2c797b4741ccaf6cbe7abfdc4a
Signed-off-by: Mazdak Nasab <mazdak.nasab@gmail.com>
ipn/ipnlocal: resolve serve file redirects against the request path
http.FileServer sends a Location relative to the request URL, such as
"nested/" or "../file". fixLocationHeaderResponseWriter prefixed that
value with the mount point, which dropped every directory between the
mount point and the target. A request for /doc/subdir/nested was
redirected to /doc/nested/.
Resolve the relative Location against the original request path
instead. Collapse leading slashes in the result so a request path like
//host/dir cannot produce a protocol-relative Location.
Fixes#22049
Signed-off-by: Alex Castillo González <alex.castillog33@gmail.com>
This adds a test that walks a PeerRelay through the ProxyClass spec.metrics settings and checks the metrics Service, ServiceMonitor and TS_ENABLE_METRICS at each step, and a case for a PeerRelay waiting on a ProxyClass that is not ready. It also checks that ApplyProxyClass leaves one entry per env name when the ProxyClass sets an env var the operator also sets.
Updates #22034
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
The PeerRelay reconciler ignored a ProxyClass's spec.metrics, so enabling metrics created no metrics Service or ServiceMonitor and the relay Pods never served /metrics. This fixes this by setting TS_ENABLE_METRICS in ApplyProxyClass and calling tailscaled.EnsureMetricsResources for PeerRelay. PeerRelay now also waits for its ProxyClass to be ready before creating anything, as the other proxies do, so the existing ProxyClass test fixture is marked ready.
ApplyProxyClass now replaces an env var that the ProxyClass also sets instead of appending a second one. The PeerRelay StatefulSet is applied with server-side apply, which rejects duplicate env names, so a ProxyClass that sets an env var the operator also sets, such as TS_ENABLE_METRICS, would stop the StatefulSet from being applied.
Updates #22034
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
The metrics Service and ServiceMonitor handling lived in cmd/k8s-operator,
and the ProxyClass reconciler kept its own copy of the ServiceMonitor CRD
check, so reconcilers in k8s-operator/reconciler could not share it. This
moves it into the tailscaled package as EnsureMetricsResources and
EnsureMetricsResourcesDeleted, which take the proxy type and parent, and
switches cmd/k8s-operator and the ProxyClass reconciler over to it. No
behaviour change.
Updates #22034
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Serve looked up its config with the name exactly as the client sent
it. Config keys are lowercase with no trailing dot, so a plain HTTP
request for "svelte.example.ts.net." or "Svelte.Example.TS.net" got a
404. HTTPS had the same gap for case. TLS rejects a trailing dot in
SNI but does not lowercase it, so a client that sent mixed-case SNI
failed the handshake with "no webserver configured". Browsers and curl
lowercase the hostname before connecting, which hid that path.
Lowercase the name and trim the trailing dot in both branches of
getServeHandler, and lowercase the SNI in the cert callback.
Fixes#21538Fixes#21986
Signed-off-by: Brendan Creane <bcreane@gmail.com>
Each send operation (sendPacket, sendPeerGone, sendPong,
sendKeepAlive, etc) calls SetWriteDeadline on the
underlying connection. This re-arms the netpoll timer
on each packet frame write to the intermidiate lazy
buffer for a socket write that happens later. Not the
actual socket write done by runWriter.
Move the SetWriteDeadline down to the buffer that is pulled
by runWriter for each actual socket write.
Updates tailscale/corp#50628
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
Adds tests that a conflict while provisioning a Service, a Service on a
ProxyGroup, a ProxyGroup or a Connector proxy results in a timed
requeue, and that the Connector and ProxyGroup status is set to
creating rather than failed. The test for a Service on a ProxyGroup
fails the update to the shared ingress config ConfigMap, as no watch
maps that ConfigMap back to the Service, so without the requeue the
conflict was never retried.
Updates #22074
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Reconcilers logged "optimistic lock error, retrying" with the full
conflict error at Info, and Connector, Recorder and ProxyGroup also put
it in their status. That reads like a failure, but a conflict between a
cached read and an update is expected and is retried. The reconcilers
also returned an empty result, so the retry only happened if a watch
event arrived. This fixes this by logging the conflict at Debug as an
update conflict, using a neutral status message and requeueing after a
short delay.
Updates #22074
Updates #18922
Signed-off-by: chaosinthecrd <tom@tmlabs.co.uk>
Move the Recorder reconciler and its object specs into
k8s-operator/reconciler/recorder, following the earlier ProxyClass,
DNSConfig and nameserver moves, and switch it to the shared finalizer,
resource tracker and CreateOrUpdate helpers.
Hoist two duplicated helpers into tailscaled.PrefsFromStateSecret and
reconciler.IsOptimisticLockError, and add
k8s-operator/reconciler/reconcilertest for the test helpers each package
had copied, including one fake Tailscale API client in place of four.
No behaviour change; the Recorder's watches and labels are untouched.
Fixestailscale/corp#37086
Signed-off-by: David Bond <davidsbond93@gmail.com>
The PreferredDERP selection loop only considers RegionLatency, which
STUN probes populate for STUN-only regions too. When such a region is
the only one with a latency sample (e.g. self-hosted DERP behind a
TCP-only tunnel while a public STUN server is reachable), it gets
selected as the preferred DERP and all relay traffic blackholes, since
the region has no DERP server to relay through.
Filter STUN-only regions in the selection loop by reusing the existing
regionHasDERPNode predicate (renamed to CanServeDERP), which was
already applied to the HTTPS/ICMP fallback probing but not here.
Regions missing from the map, or present without any nodes, keep the
old behavior, as some tests use latency-only DERP maps.
Fixes#21975
Signed-off-by: Luo Chen <luochen1990@gmail.com>
reserveAddresses inserts the assignment into c.assignments before
enqueueing it, and leaves it there when the enqueue fails. The connector
never learns about such a mapping, and nothing tell it later.
Add addrAssignments.remove and use it to undo the insert and return both
addresses to their pools when enqueueAddressAssignment fails. The caller
still gets a SERVFAIL, but the next DNS query starts from a clean slate.
Fixes https://github.com/tailscale/corp/issues/50917
Change-Id: I5a387bdcf5a5e87e2dca44867e88ed6149678729
Signed-off-by: Mazdak Nasab <mazdak.nasab@gmail.com>
This commit changes the egress ProxyGroup readiness gate to mark a Pod
ready only when, for each egress Service, the Pod is an endpoint in the
EndpointSlice of every IP family that the proxy reports in its egress
status. Before, the gate accepted a Pod that was in either family's
EndpointSlice. On a dual-stack cluster, the StatefulSet rollout could
then move to the next Pod before the other family routed to the new Pod.
The gate reads the families from the proxy's state Secret, because a
proxy older than v1.102 does not report an IPv6 address. Such a proxy
never joins the IPv6 EndpointSlice, so its Pod must not wait for it.
Fixes#21077
Signed-off-by: David Bond <davidsbond93@gmail.com>
This commit removes the owner reference from ProxyGroup state Secrets,
also from Secrets that an older operator created. On a Foreground
delete, the garbage collector deleted these Secrets before the
finalizers ran. The finalizers read them to find devices, so cleanup
failed and devices stayed in the tailnet.
The ProxyGroup cleanup now deletes the Secrets after the devices, and
the reconciler watches them by label. The kube-apiserver cleanup finds
its cert resources by label, so it does not need the state Secrets. A
new e2e test covers Foreground deletes.
Fixes#22072
Signed-off-by: David Bond <davidsbond93@gmail.com>
bufioWriterPool was sized at 2KiB, this limited the number
of packets per TLS record during a push, limiting throughput
and increasing syscalls
Increase the buffer to 16KiB to pass extra DERP frames per
flush and TLS record write.
Update the websocket path to also use the pooled buffer
Updates tailscale/corp#50628
Signed-off-by: Mike O'Driscoll <mikeo@tailscale.com>
In singleton egress mode (TS_TAILNET_TARGET_FQDN), a failed resolution
of the target FQDN logged the error but then broke out of the main
runLoop instead of continuing to the next netmap update. From then on
containerboot blocked in wg.Wait() while the pod still looked healthy,
and the egress proxy kept forwarding to stale addresses until restart.
A missing target is typically transient (peer not yet in the netmap,
ACL change in flight), so log and continue to let the next netmap
update retry the resolution and reinstall the forwarding rules.
Fixes#22102
Signed-off-by: Santhi Prakash <b.santhiprakash@gmail.com>
This is a follow-up to #21777, which added this functionality but didn't
test it yet.
Updates tailscale/corp#39984.
Signed-off-by: Naman Sood <mail@nsood.in>
When tailscaled denies a serve config, the CLI advises configuring an
operator to avoid root. That advice is wrong for path and Unix socket
targets, which require root regardless. Replace the access-denied
error for those targets with one that says to use sudo.
Updates #21204
Signed-off-by: Brendan Creane <bcreane@gmail.com>
The operator user could serve a path or Unix socket if tailscaled
found a sudo rule for them, but the check used the GECOS display name
and so failed for most users with a 401. The check cannot be repaired
safely: listing another user's sudo rules never prompts, so it proves
a rule exists, not that the user has authenticated. An operator who
had not entered their sudo password could serve any root-readable file
or root-only socket. Require root instead and say so in the error.
Fixes#21204Fixes#21985Fixes#21988
Signed-off-by: Brendan Creane <bcreane@gmail.com>