* feat: expose RRDtool 1.11 graph axis controls
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix: constrain axis template inheritance checkboxes
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test(rrd): exercise native aggregate validation and enforce coverage
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* ci: share changed-code coverage contracts and enforce shell policy
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(ci): run request coverage with portable path matching
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* ci: retain bootstrap coverage and validate request dispatch
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test: synchronize native coverage fixture dispatch
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test: measure coverage at the exact PR revision
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
---------
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* feature: user-defined filter layouts with administrator publishing
Add a render_layouts toggle to the CactiTableFilter class so list pages
offer Save / New / Rename / Delete controls for saved filter layouts,
modelled on the monitor plugin. Layouts persist to a new user_layouts
table (id, user_id, page, name, url). The toggle defaults on for every
page except the Console > Presets pages (CDEFs, VDEFs, Colors, GPRINTs,
Data Profiles, SNMP), where it is off.
Each user manages their own layouts from any filter. Administrators who
hold the Settings/Utilities realm can publish a layout to all users
(moving it to user_id = 0) from the filter, and from a new
Console > Presets > Filters management page (layouts.php) where they can
view and manage every user's layouts. Published layouts appear for all
users on that page.
Layout create/rename/delete/publish requests are handled centrally by
filter_layouts_handle_request(), invoked from include/auth.php after the
user has been authorized for the current page, so the controls work on any
filter page while publishing stays gated on the realm. layouts.php itself
is mapped to the Settings/Utilities realm so the Presets > Filters menu
pick is administrator-only.
Schema added to cacti.sql, install/upgrades/1_3_0.php, and
docs/audit_schema.sql. Adds tests/Unit/FilterLayoutsTest.php covering the
URL-safety validation, preset-page defaults, and the request/permission
wiring.
* changelog: 8185.feature fragment for filter layouts
* chore: enforce LF line endings via .gitattributes and normalize new files
* fix: address review - attribute-escape layout options, honor failed/non-editable layout ops, regenerate cacti.pot, move test to Unit/Ui/Html
* chore: renormalize pre-existing CRLF files to LF per .gitattributes
* chore: keep vendored JS (big.js, tablesorter) byte-identical to upstream and exclude from LF normalization
* style: apply php-cs-fixer formatting and resolve PHPStan level 8 findings in filter layouts
* ci: re-trigger checks on current head
* refactor: store filter layouts as a versioned JSON document in a TEXT data column
Replace the user_layouts.url varchar(1024) column with a data TEXT column holding a versioned {version,page,filter} document. The navigable url is rebuilt server-side from the document (page basename + rebuilt query) plus the filter_layout marker, removing the stored-url redirect/XSS surface and making the backend forward-compatible with a structured query builder without a further schema change.
* feature: add Modern/Legacy page filter format with an Edit dialog for filters
Modern (default) collapses the filter to a Layouts selector plus an Edit dialog holding the Filter Name and all filter fields; Save and Publish live in the dialog, with Edit/Rename/Delete/Save As on the bar and time controls/refresh kept on the bar. Legacy keeps the classic inline filter. The format is a per-user General setting (page_filter_format). Field rendering is factored into emit_field() so both layouts share one source.
* fix: render page-action buttons (import, export, sort) on the modern filter bar
Import, Export, Purge, Save, and sort ascending/descending are page actions, not filter inputs, so they render on the bar beside the Layouts selector instead of inside the Edit dialog. Only go/clear are dropped from the bar; the dialog keeps the Search/Clear footer.
* feature: render the modern Edit dialog one filter variable per row
The dialog no longer mirrors the inline filter's multi-field row packing; each filter variable renders on its own row (label + control) for a clean, responsive form. Hidden/validate fields stay in the form without an empty visible row. Rows carry a cactiFilterEditRow hook for label-column alignment styling.
* feature: align the modern edit dialog one-per-row and skin it per theme
Render all dialog fields in a single table so the label column aligns via the existing table-cell layout. Tag the dialog widget with cactiFilterDialog and add a scoped skin to the modern theme (slate titlebar, styled button pane, accented primary action). Add a bar separator between layout and page-action buttons.
* feature: skin the modern filter Edit dialog across the classic themes
Add the scoped .cactiFilterDialog skin (accent titlebar, white title/close, styled button pane with an accented primary action) to cacti, carrot, dark, hollyberry, paper-plane, paw, raspberry, and sunrise, each using its own brand accent. The skin is accent-only and inherits each theme's dialog surface, so it works on light and dark themes. MidWinter is intentionally skipped (its variable-based CSS is handled in a separate Epic).
* fix: delete a user's saved filter layouts when the user is removed
user_remove() now deletes the user's own user_layouts rows alongside their other per-user data. Scoped to user_id so published (user_id = 0) layouts remain available to everyone.
* fix: address PR review — PHPStan return type, escaping, callbacks, SQL columns, mb truncation, contrast, POT
- create_filter(): cast both ob_get_clean() returns to string and drop the now-unused PHPStan baseline entry (fixes Level 6 failure).
- Edit dialog title now uses html_escape_attr() (attribute-context double-encoding).
- Modern Search/Clear honor the page's configured go/clear callbacks instead of hardcoding applyFilter()/clearFilter().
- filter_layouts_get() and layouts.php select explicit columns instead of *.
- Layout name truncation uses mb_substr() to avoid cutting multibyte codepoints.
- layouts.php management callback checks result.ok before reloading.
- carrot/paper-plane dialog accents darkened to meet WCAG AA (>=4.5:1) with white text.
- Added the new translatable msgids (Page Filter Format, Filters, Edit Filter, Filter Name, Save As, ...) to cacti.pot.
* fix: resolve Semgrep reflected-xss by reading filter_layout inside layout_select
The request var is no longer passed into layout_select() as a parameter (which Semgrep tracked parameter-to-return to the print sink); it is read internally like create_filter(), and the option markup remains escaped. No behavior change.
---------
Co-authored-by: TheWitness <copilot@users.noreply.github.com>
Co-authored-by: TheWitness <TheWitness@users.noreply.github.com>
* feat(auth): automatic User Group assignment from Login Provider groups
Add per-Login-Provider "Auto Assignment" rules to the User Group editor so a
user is added to (or removed from) a Cacti User Group at every login based on
the groups their Login Provider reports.
User Group editor (user_group_admin.php):
- Add General / Auto Assignment / Rights spacers to the General tab and rename
"General Group Options" to "Group Options".
- Render one text field per eligible Login Provider under Auto Assignment
(label = provider name). The admin enters a plain Group Name (e.g.
"Cacti Admins"), never a DN.
- Store the rules as JSON in the new user_auth_group.auto_assignments column,
keyed by login_providers.id.
Group resolution:
- SAML2 and OpenID match the configured name against the group attribute /
group claim values.
- LDAP/AD resolve membership live against the directory using a raw group name
plus the Search Base, trying AD (memberOf + nested LDAP_MATCHING_RULE_IN_CHAIN),
FreeIPA / 389 DS (groupOfNames/groupOfUniqueNames) and RFC2307 posixGroup
shapes so one plain name works everywhere. This requires a directory-read
bind, so the field is only offered for Anonymous Searching or Specific
Searching with a stored service account.
- A directory that cannot be reached returns "undetermined" so existing
memberships are left untouched rather than revoked on a transient outage.
Lifecycle:
- auto_assignments is reconciled on every login (lib/auth.php credential flow
and login_sso.php redirect flow).
- Deleting a Login Provider strips its key from every group's auto_assignments.
Schema: add user_auth_group.auto_assignments (cacti.sql + 1_3_0 upgrade).
* ci: document free-text group-name validateInput for constraint ratchet
* fix(auth): address PR review on automatic User Group assignment
- LDAP: add an Active Directory bare-name query (objectClass=group with the
member:1.2.840.113556.1.4.1941 nested-membership rule) so a plain group
name resolves on AD, which neither the posixGroup nor groupOfNames probe
could match.
- LDAP: make single-group membership tri-state and propagate an "undetermined"
result, so a search that fails after binding (timeout/dropped connection) no
longer looks like confirmed non-membership and revokes assignments.
- SAML2/OpenID: supportsAutoAssignment() now requires a configured group
claim, and applyAutoAssignments() skips any provider that is not currently
capable, so a blank claim can never strip users from their groups.
- Only reset the permission cache when a membership row actually changed
(accumulate db_affected_rows) rather than on every login that has a rule.
- Add unit tests for the reconciliation boundaries (add / remove / no-op /
incapable / undetermined) and for the LDAP group-query construction
(AD plain + nested, posixGroup, groupOfNames, DN mode).
- Regenerate locales/po/cacti.pot for the new UI strings.
* fix(phpstan): resolve level-8 errors from auto group assignment
- Guard supportsAutoAssignment() behind an instanceof AbstractLoginProvider
check so it is not called on the base LoginProviderInterface.
- Assign the injected field list to a variable (inject_form_variables takes it
by reference) and pass an is_array-guarded group row.
- Bump the lib/ldap.php ldap_get_entries baseline count 3->6 for the three new
group-membership query helpers.
* feat(auth): rename User Domains to Login Providers
Collapses user_domains/user_domains_ldap into a single login_providers table with a JSON parameters column, migrated in place during the 1.3.0 upgrade. Renames the admin page, menu, breadcrumbs, and help links to match.
* feat(auth): rework login providers as Cacti\Auth classes
Introduces LoginProviderInterface with credential (LDAP/AD) and redirect (SAML2/OpenID) flavors, replacing the procedural domains_login_process()/domains_ldap_*() functions. Adds blank-field group membership gating shared by every provider type.
* feat(auth): add SAML2 and OpenID Connect login providers
Adds onelogin/php-saml and firebase/php-jwt, a new cacti_http() SSRF-hardened outbound request helper for every OpenID discovery/token/JWKS call, and login_sso.php to handle the redirect-based SSO flow. Login page now shows a button per enabled SAML2/OpenID provider, and per-provider auth cookie opt-out is enforced on both the password and SSO login paths.
* chore(i18n): regenerate cacti.pot
* feat(auth): add Local and Basic auth as Cacti\Auth providers
LocalAuthLoginProvider (CredentialLoginProviderInterface) and BasicAuthLoginProvider (new PreAuthenticatedLoginProviderInterface, since Basic Auth has no password for Cacti to check) are thin adapters over the existing local_auth_login_process()/basic_auth_login_process() functions, which are left untouched to avoid disturbing their regression-tested lockout/timing/rehash behavior.
* feat(auth): move local password verification into LocalAuthLoginProvider
secpass_login_process() and local_auth_login_process() are retired; their lockout, constant-time verification, forced-password-change, and rehash-on-login logic now live end to end in the class (verifyCredential/enforcePasswordPolicy/rehashIfNeeded). secpass_check_pass() stays a shared lib/auth.php function since password change/reset/user_admin also call it outside any login flow. Local auth still has no login_providers row: isEnabled() is always true and it cannot be disabled or deleted.
* feat(auth): move Login Provider CRUD into Cacti\Auth classes
AbstractLoginProvider gains static deleteById()/enableById()/disableById()/makeDefaultById()/persist(); each concrete provider owns its own collectParameters(), dispatched via LoginProviderFactory. login_providers.php's form_save()/form_actions() call these instead of doing raw SQL, and the retired provider_remove()/provider_disable()/provider_enable()/provider_default()/provider_collect_parameters() procedural functions are gone. Local/Basic auth are unaffected: they never extend AbstractLoginProvider, so they structurally cannot expose delete/disable.
* fix(auth): address Copilot review findings on SSO/SAML/OIDC security
Fixes callback/ACS/SLS/metadata URLs that used the raw provider id instead of the login realm (broke every SSO login), adds InResponseTo correlation for SAML and sub/azp validation for OIDC, requires HTTPS for OIDC discovery/endpoints, pins cacti_http() connections against DNS rebinding via CURLOPT_RESOLVE, exempts login_sso.php from the authenticated-page redirect guard and gates it on auth_method, wraps provider dispatch in a exception boundary, adds a dedicated SAML SLO path instead of routing it through login completion, makes remember-me an explicit opt-in for SSO, fixes disabled/deleted providers not revoking existing auth cookies, restricts the default-realm action to LDAP/AD, and fixes the 1.3.0 migration silently enabling group enforcement for installs that had it off. Also updates the E2E seed script for the login_providers schema.
* chore(auth): refresh CI security baselines and document no-op validation
Regenerate the sink inventory baseline for the new/renamed lib/Auth/* and login_providers.php files, and add '// no-validation:' comments on the intentionally-unconstrained admin/free-text/secret fields (LDAP DNs/filters, SAML certs/keys, OIDC client credentials, provider display fields) flagged by the validator constraint ratchet.
Refreshes the validator baseline to drop the deleted user_domains.php entries.
* refactor(ldap): remove dead DB-lookup path from Ldap::__construct
The domain_id>0 branch duplicated the JSON-parameter-to-property mapping already owned by LdapLoginProvider::buildLdap(), and nothing constructs Ldap with a real id anymore (all call sites pass 0). Drop the parameter and the branch so Ldap stays a pure connection/protocol class with no DB coupling.
Also fixes a latent bug: group_require is a typed bool property, but LdapLoginProvider::authenticate() assigned it an int (1/0) from a strict_types=1 file, which throws a TypeError on every LDAP/AD login attempt. Assign a real bool instead.
* fix(auth): fix LDAP login, upgrade ordering, and hardening guard failures
LdapLoginProvider compared LdapError::GetErrorDetails()'s int error_num against the string '0' with strict equality, so it never matched and every LDAP/AD login was rejected regardless of valid credentials. Compare against int 0 instead.
install/upgrades/1_3_0.php called login_providers_convert_1_3_0() (which reads and then drops user_domains/user_domains_ldap) before the pre-existing legacy ALTER TABLE statements that add the debug/network_timeout/bind_timeout columns those reads depend on, so upgrades failed once the tables were already gone. Moved the call to run after those ALTERs.
OpenIdLoginProvider::complete() read state/code from raw \ instead of gnrv(), tripping the hardening pattern guard; switched to the request helper.
* style(auth): apply php-cs-fixer formatting and normalize line endings to LF
Runs php-cs-fixer across the new lib/Auth/* classes, login_providers.php, login_sso.php, and the other touched files, fixing indentation/alignment/docblock style to match project convention. Also normalizes several files that had picked up CRLF line endings back to the LF the project's .php-cs-fixer.php config (setLineEnding) and the rest of the tracked tree actually use.
* fix(auth): address second Copilot review round (9 high/medium findings)
- Make login_providers_convert_1_3_0() resumable: CREATE TABLE IF NOT EXISTS and INSERT ... ON DUPLICATE KEY UPDATE so an interrupted upgrade can be re-run instead of leaving a half-migrated, unrecoverable table.
- SamlLoginProvider::complete() now fails closed when no saved AuthnRequest ID exists instead of passing null to processResponse(), which skipped InResponseTo validation and accepted unsolicited IdP-initiated assertions.
- LoginResult gains an isCredentialFailure flag; only a genuine wrong-password LDAP bind (LdapError::Failure) counts toward the lockout counter now, not connection errors or group-membership denials.
- Fixed button_label/debug being read from the JSON parameters column in OpenIdLoginProvider/SamlLoginProvider/LdapLoginProvider when they are actually top-level login_providers columns; AbstractLoginProvider now hydrates and exposes both.
- Removed SamlLoginProvider's wantMessagesSigned, which silently required the outer SAML message (not just the assertion) to be signed, rejecting IdPs that only sign the assertion as the UI promises.
- Added getTemplateUserId() to CredentialLoginProviderInterface and implemented it on LocalAuthLoginProvider so the contract (not just incidental AbstractLoginProvider inheritance) guarantees the method lib/auth.php calls.
- cacti_http()'s CURLOPT_RESOLVE entries now bracket IPv6 literals correctly, and safe-IP validation strips parse_url()'s brackets before checking, fixing IPv6 target pinning.
- login_providers.php's type dropdown now swaps the Full Name attribute default (cn/displayName) on an actual user-driven type change, not on page load, so a new Active Directory provider does not silently save the LDAP default.
- Extracted auth_apply_group_login_opts() and call it from both auth_login.php and login_sso.php, so SSO logins honor a user's group-derived login_opts the same way password logins already do.
* fix(auth): fix PHPStan level 8 errors and pest harness stub drift
Test harness fix: the LoginProvidersLoginRealmValidationTest stub LoginResult class was missing the isCredentialFailure property added to the real class, causing 'Undefined property' warnings that broke the harness's JSON output. Added the property and set it on the simulated bad-password failure.
PHPStan (48+ errors, all newly introduced by this branch's rewrite, never actually verified until now): narrowed db_fetch_row_prepared()/db_fetch_assoc()'s array|bool (including bare true, which survives a '?: []' short-circuit - fixed with explicit is_array() checks instead) at every call site across auth_login.php, login_sso.php, lib/auth.php, lib/Auth/LocalAuthLoginProvider.php, lib/Auth/LoginProviderFactory.php, and install/upgrades/1_3_0.php.
Fixed OpenIdLoginProvider::discover()'s return shape to match its declared docblock instead of returning the raw decoded JSON. Fixed lib/functions.php's cacti_http() to narrow curl_exec()'s string|bool result with is_string() (not ===false, which still leaves 'true' as a residual type) and guard / as non-empty for curl_setopt_array()'s stubs.
Fixed a real bug in SamlLoginProvider::processLogout(): processSLO(true) was passing 'true' as \ (the first parameter) instead of \ (the fifth), so it always redirected/exited internally with the default \=false, making Cacti's own logout.php never regain control. Now calls processSLO(keepLocalSession: true, stay: true) and is typed ': never' to match; login_sso.php's now-genuinely-unreachable break statements after processLogout()/initiate() are removed.
Fixed login_providers.php passing a expression (not a variable) to inject_form_variables()'s by-reference parameter.
Verified: PHPStan analyse clean (0 errors), full Pest Unit/Security suite passes (128 tests), hardening guard/sink inventory/validator ratchet all still clean.
* fix(auth): fix ambiguous column and array+bool crash in user_log filter
The user filter dropdown query selected a bare 'id' column after joining user_auth and login_providers, which is ambiguous now that login_providers also has an id column (unlike the old user_domains.domain_id) - MySQL rejected the query outright. Qualified as ua.id.
Also narrow db_fetch_assoc()'s array|bool result before the \ + \ + \ merge, which was throwing a TypeError whenever the query failed instead of surfacing the real SQL error.
* fix(auth): address third Copilot review round (5 findings + tests)
- AbstractLoginProvider::persist() now disables the template account for every provider type; SAML2/OpenID also copy it via user_copy() in login_sso_complete() and were left directly usable as a local login.
- cacti_http_resolve_safe_ips() now detects IPv4-mapped IPv6 literals (::ffff:0:0/96) via inet_pton() and validates the embedded IPv4 address, closing an SSRF bypass via e.g. ::ffff:169.254.169.254.
- login_providers_convert_1_3_0() now tracks db_install_execute()'s status for every row and only drops the legacy user_domains/user_domains_ldap tables once every row (and the final row count) is confirmed migrated.
- Ldap::Getcn() now looks up ldap_get_entries()'s always-lowercased attribute keys case-insensitively while still storing the result under the originally-requested key, fixing AD's displayName (and any other mixed-case configured attribute) always resolving to a blank full name.
- SamlLoginProvider::processLogout() now redirects to the LogoutResponse URL processSLO() returns for a LogoutRequest (stay:true), instead of discarding it and always redirecting to Cacti's own logout page - federated logout was never actually acknowledged to the IdP.
- Added tests/Unit/Security/Auth/OpenIdLoginProviderTest.php (13 tests) covering verifyIdToken() signature/issuer/audience/azp/nonce validation, discover()'s HTTPS enforcement, and complete()'s callback-state and UserInfo-subject-matching checks, against a locally-generated RSA keypair/JWKS.
- Regenerated the sink inventory baseline for the new header_redirect call in SamlLoginProvider::processLogout().
Verified: PHPStan clean, full Pest Security/Auth suite passes (830 tests including the 13 new ones), php-cs-fixer/hardening-guard/sink-inventory/validator-ratchet all clean.
* fix(boost): harden data handoffs and cache writes
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* Define $updates_ok when there is nothing to flush
PHPStan level 6 also flags the poller ownership check, which is the POLLER_ID
constant case the file already annotates above.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* Use a line comment to match the fixer
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* Record the boost process columns in the audit schema
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test: resolve includes through Cacti path constants
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* Fix Boost handoff review concerns
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test: seed settings in Boost database fixture
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test: refresh SQL interpolation baseline
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* docs: record Boost handoff hardening
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix: stop boost discarding a cycle and wedging a shard
Three defects in the handoff hardening.
A rejected ownership handoff returned false, which the poller reads as boost
having consumed the data, so it skipped the RRD write. The poller_output rows
were already deleted by then, so a stale poller_item copy on a remote poller
silently lost the whole cycle. Return true instead and let the poller write
directly, matching the database failure path.
Any non-OK RRDtool response retained the shard, and the retry then replayed rows
already written, which RRDtool rejects as an illegal update time. That failed
again on every pass, so the archive tables accumulated and boost never drained.
Treat an at-or-behind timestamp as applied, since the sample is in the RRD;
genuine errors still retain the shard.
The two new poller_output_boost_processes columns passed 'null' where
db_install_update_table reads 'NULL', so NOT NULL was never emitted and upgraded
installs diverged from cacti.sql.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix: spell the NULL column key so upgrades emit NOT NULL
db_add_column() reads $column['NULL'] and PHP array keys are case sensitive, so
the thirteen definitions in the 1.3.0 upgrade that passed a lowercase 'null' key
had the flag discarded and created nullable columns. Fresh installs built from
cacti.sql got NOT NULL, upgraded installs did not, and Cacti's own schema audit
reports the difference.
The same file already spells the key in upper case 144 times, and the wrapper's
own docblock example uses the upper case form.
Guard the class with a test over every upgrade file rather than only the one
that was wrong.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test: assert the NULL key plainly
toContain() takes needles rather than a failure message.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): close remaining handoff review gaps
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): guard nullable graph output
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): keep in-flight completion rows through the 1.3.0 upgrade
The truncate only existed so the run_child UNIQUE key could build over rows
that predate run_id; deleting just those rows leaves a running boost its
completion records.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* refactor(boost): drop the poller ownership gate
Its boost_poller_on_demand() call sits behind the boost_redirect early return
that poller.php forces on for every multi-collector install, and its recovery
call turns a device deletion on main into a recovery that never brings the
collector back online.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): bound atomic cache write allocations
Guard upgrade source reads so failures report clearly, write graph cache data in fixed-size chunks, and exercise a multi-chunk payload.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): support Windows cache replacement
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test(boost): expand handoff failure coverage
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test(boost): support ANSI_QUOTES upgrade checks
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): stop staging after chunk failure
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(recovery): use numeric poller status
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(sql): support ANSI_QUOTES in recovery paths
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): preserve cache on replacement failure
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): retain readable cache permissions
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): repair completion schema at runtime
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(recovery): bound timestamp batch fetches
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): satisfy runtime schema analysis
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(recovery): restore status after transfer failure
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test(boost): rely on unit bootstrap helpers
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): refresh generated cache secret
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): normalize cache path separators
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test(boost): fail clearly on unreadable fixtures
* fix(boost): name ignored duplicate sample keys
The staging warning only said rows were dropped, so a retry could not be told from a colliding writer.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* Lack of pre-commit hook execution allowed lint failure
---------
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Co-authored-by: TheWitness <thewitness@cacti.net>
* Serialize process registry admission with Symfony Lock
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* Fix PDO process lock mode handling
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* ci: run process lock coverage on PHP 8.2
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix: confirm a pid is still ours before signalling it
#7574 added the still-running guard to dsstats but left six byte-identical
siblings signalling a pid straight out of the processes table. After a crash or
a power loss the row survives, the pid is recycled by an unrelated daemon, and
the next start SIGTERMs a stranger. That is issue #7425, still live in rrdcheck,
boost, poller_commands, float_rrdfiles, poller_reindex_hosts and
rebuild_poller_cache.
Wrap each kill in cacti_process_still_running(), matching the dsstats shape, and
leave unregister_process() outside the guard so a stale row is still cleaned up.
Cover all seven with a test that walks every kill site, so a new variant cannot
reintroduce it.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test: assert the guard plainly
toContain() takes needles rather than a failure message, so the file name was
being asserted as a second required substring and the test always failed.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* docs(poller): describe required lock parameters
---------
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): forward still-open-round archive rows instead of dropping them
boost_process_poller_output() correctly excluded rows from an
in-progress poll round when building the RRD write, but the
archive-table cleanup right after deleted every row for the
local_data_id anyway, with no time filter. Those rows now get
INSERT IGNORE'd into the live poller_output_boost table before the
archive table is cleared, so a still-open round's data survives.
Also fixes the duplicate-record guard in the same function: $last_item
was initialized once before the results loop and never reassigned, so
the comparison against it could never be true. It is now updated after
each processed row.
Fixes#7519Fixes#7523
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): gate archive-table drop on shard completeness
poller_boost.php decided whether to drop a run's archive tables using
SELECT SUM(status) FROM poller_output_boost_processes, an aggregate
over whatever completion rows landed. If one child crashed before
writing its completion row, the sum from the surviving shards was
still positive and every archive table for the run got dropped,
including the crashed shard's unprocessed data.
The drain loop already computes boost_completed_children() vs
$expected_children to detect a crashed child; the drop now uses that
same completeness check instead, and leaves the archive tables in
place with a warning when it is not met.
Fixes#7520
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(dsstats,rrdcheck): apply Boost's startup barrier to launch loops
dsstats_launch_children()/rrdcheck_launch_children() fork children via
exec_background() (non-blocking) and the callers then wait with a bare
while ($running = ..._processes_running($type)) { sleep(2); }. If the
parent evaluates that before any child has registered in the processes
table, $running reads 0 and the loop body never runs, treating "not
started yet" as "already done" -- the same race Boost's
boost_all_children_registered() barrier was added to close.
dsstats_launch_children()/rrdcheck_launch_children() now return the
launched count, and dsstats_wait_for_children()/
rrdcheck_wait_for_children() wait for the running count to reach it
(bounded by a deadline) before trusting the drain loop's running-count
poll.
Fixes#7526
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* refactor(poller): consolidate poller_item deletes behind a shared helper
Extract poller_item_delete_for_data_source() and poller_item_delete_for_host()
in lib/poller.php and migrate the existing local+remote delete pairs in
lib/api_device.php and lib/api_data_source.php onto them, so the
local-delete-plus-remote-mirror pattern lives in one place instead of eight.
Refs #7535
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(poller): mirror poller_item deletes to the remote poller on handoff
api_data_source_change_host(), data_query_remove_disabled_items(), and
data_source_to_data_template() each moved or removed a data source's
poller_item row locally but never on the remote Data Collector that
still owned it, so that Collector kept polling an item it no longer
should. All three now resolve the owning remote poller before the
local change and purge the row there too via the shared helper.
Fixes#7521Fixes#7524
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(poller): correct poller_item join semantics and cron reset in cmd.php
poller_item and host share ~13 identically-named snmp_*/hostname columns;
SELECT * across the join let PDO FETCH_ASSOC silently collapse them to
host's live values instead of poller_item's cached copies. Select pi.*
plus explicitly aliased host/site disabled columns instead. The now
already-fetched $poller_items also makes the separate script-server-call
COUNT(*) re-join redundant, replaced with an in-PHP count. Also apply the
same cron_interval-aware rrd_next_step reset used by the ranged branch to
the --allhost branch, which previously ignored it.
Fixes#7522, #7525, #7528.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(poller): thread known poller_id through update_poller_cache() callers
update_poller_cache() re-derived poller_id via a host/data_local JOIN on
every call, even when the caller already had it: repopulate_poller_cache()
and push_out_data_input_method() both already select h.poller_id and
discarded it, push_out_host() has the host row cached per data source, and
api_device_enable_devices()/update_poller_cache_from_query() compute it
once outside their loops. Add an optional $poller_id argument (and honor
an already-present 'poller_id' array key) so these callers skip the
redundant query.
Fixes#7527.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(poller): scope global aggregate scans and make process-leveling sargable
Three poller.php queries (total_snmp_ports and the mutually-exclusive
active_profiles pair) had no poller_id filter, unlike the rest of the
file's central-only blocks, so every remote poller redundantly recomputed
and overwrote the same global config options; the snmp_port scan is also
an unindexed full table scan. Gate them behind poller_id == 1, with remote
pollers reading back the value the central poller already stored.
Also rewrite the process-leveling $sql_where's rrd_next_step predicate
from the non-sargable `rrd_next_step - $poller_interval <= 0` to the
algebraically equivalent `rrd_next_step <= $poller_interval`, restoring
use of the poller_id_rrd_next_step index.
Fixes#7529, #7530.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(poller): batch update_data_query_cache()'s poller cache writes
Each changed data source called update_poller_cache($data_source, true),
which flushes immediately via poller_update_poller_cache_from_buffer() --
its own present=0 UPDATE, INSERT ON DUPLICATE KEY UPDATE, and stale-row
DELETE -- on every iteration. Re-indexing a host where every port changed
issued up to 3x as many statements as ports instead of one batched flush.
Accumulate changed rows across the loop and call update_poller_cache()
with commit=false, then flush once via poller_update_poller_cache_from_buffer()
after the loop, matching the batched pattern already used by
push_out_data_input_method().
Fixes#7531.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): add USING BTREE to poller_output_boost_local_data_ids indexes
MEMORY-engine indexes default to HASH unless USING BTREE is specified,
unlike the sibling poller_output/poller_output_boost tables in the same
schema file. boost_output_rrd_data()'s ORDER BY/LIMIT query and
boost_process_local_data_ids()'s ranged DELETE both need to scan
local_data_id in order, which a HASH index can't serve, forcing a
filesort on every boost worker pass.
Verified against a real MariaDB 11 instance: EXPLAIN on the ORDER BY
query shows "Using filesort" with the old HASH schema and no filesort
with USING BTREE, using the primary key for ordering instead.
Also aligns boost_prepare_process_table()'s dead CREATE TABLE IF NOT
EXISTS fallback with the corrected schema, and adds an upgrade step
that rebuilds both indexes as BTREE on existing installs (checked via
information_schema.STATISTICS, since db_install_add_key() only compares
index columns, not algorithm, and would otherwise skip an existing HASH
index with the right column list).
Closes#7532
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): skip the UNION ALL wrapper for the single-archive-table case
boost_process_local_data_ids() always wrapped its flush query in
"SELECT * FROM (... UNION ALL ...) t ORDER BY ...", even when there was
only one archive table -- the steady-state case, since old archive
tables are dropped at the end of each cycle. UNION ALL results wrapped
in a derived table are one of the documented cases MySQL/MariaDB's
derived-table merging doesn't apply to, so every pass paid for a
filesort over an indexless derived table regardless of how well-ordered
the underlying archive table's rows already were.
Extracts the per-table SELECT into boost_archive_select_sql() and uses
it directly, without the wrapper, when there's exactly one archive
table. The UNION ALL/derived-table shape is kept as the fallback for
the genuinely rare case where a prior crashed run left more than one
archive table behind.
Verified against a real MariaDB 11 instance and via a SQLite-backed
integration test that both shapes return correctly filtered and sorted
rows.
Closes#7533
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): batch-prefetch RRD field name lookups instead of per-row queries
Two N+1 patterns in the Boost hot paths:
boost_process_local_data_ids() (poller_boost.php) re-ran the
unused_data_source_names/nt_rrd_field_names lookups on every
local_data_id boundary in the sorted flush result set.
process_poller_output() (lib/poller.php) had the equivalent lookup, but
without even a boundary check -- it re-ran per matching row whenever
the output contained a ':', including consecutive rows for the same
data source.
Adds poller_prefetch_rrd_field_names() (lib/poller.php, shared by both
callers since poller_boost.php already requires lib/poller.php) to
batch-fetch these mappings once per invocation via IN()-batched queries
keyed by the full set of local_data_ids in the batch, chunked at 1000
ids to match the array_chunk(..., 1000) convention used elsewhere
(lib/api_data_source.php, lib/api_graph.php). Both call sites now look
up the prefetched maps in-memory inside their loops.
While consolidating the lookups, fixes a latent bug the old
process_poller_output() code had in its non-templated fallback branch:
a query referencing gti.task_item_id without joining
graph_templates_item, which throws an unknown-column error whenever
that path is hit. The batched query always joins correctly.
Integration test proves both correctness (against a real SQLite-backed
schema) and the query-count reduction: 5 local_data_ids that would
previously cost up to 10 queries for these two lookups now cost exactly
3, regardless of how many ids are in the batch.
Closes#7534
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* refactor(boost): consolidate poller_output_boost inserts into one helper
The INSERT INTO poller_output_boost (...) VALUES ... statement was
hand-written four times: three in cmd.php (behind the same
boost_redirect == 'on' && boost_rrd_update_enable == 'on' guard) using
INSERT IGNORE, and once in boost_poller_on_demand() (lib/boost.php)
using ON DUPLICATE KEY UPDATE. Beyond the duplication, that was a real
data-integrity inconsistency between two paths writing the same shape
of data: silently-drop vs last-write-wins on the same key.
Since boost_redirect gates cmd.php's writes and boost_poller_on_demand()
returns early when boost_redirect is on, the two paths never run
concurrently for the same install; they're alternate write paths
selected by that one setting, not competing writers. A genuine
collision on (local_data_id, rrd_name, time) in either path represents
a retry of an already-collected sample -- a batch resent after a
partial failure, or two writers racing on the same rounded second --
not two independently meaningful readings, since RRD only stores one
value per timestamp regardless. Standardizes on INSERT IGNORE
(first write wins), matching the sibling poller_output insert this
table shadows, which has always used INSERT IGNORE, and 3 of the 4
existing call sites.
Extracts boost_flush_output_batch() (lib/boost.php) as the one shared
helper, handling both the batched INSERT and the max_allowed_packet-safe
chunking cmd.php's three sites previously didn't do at all (they only
chunked by item count).
Integration test exercises a genuine duplicate-key collision against a
real (SQLite-backed) table and asserts the chosen first-write-wins
behavior.
Closes#7536
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(tests): resolve cross-branch test interactions found by full-suite verification
Two test-only bugs surfaced only when all four fix branches for the
poller_item/Boost hardening pass ran together, not when each ran in
isolation:
- BoostArchiveHandoffIntegrationTest extracted the archive-forward INSERT
IGNORE by searching lib/boost.php for the first occurrence of "INSERT
IGNORE INTO poller_output_boost". The Boost SQL-optimization branch adds
a second, earlier occurrence (the new boost_flush_output_batch() helper's
single-line prefix), so the naive search now grabbed the wrong snippet.
Anchor on the multi-line "...poller_output_boost\n\tSELECT *" shape that
only the archive-forward statement has.
- PollerItemRemotePurgeHandOffTest calls push_out_host() (via
api_data_source_change_host()), which needs CACTI_PATH_LIBRARY. The test
never requires include/global.php (which normally defines it), and was
relying on some other file in a full multi-thousand-file suite run having
defined it first as a process-wide side effect. Define it directly
instead of depending on unguaranteed load order.
No production code changed.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* style(poller): satisfy php-cs-fixer strict_param in the script-server count
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): order the single-archive-table flush by the table's PK columns
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(boost): key the max_allowed_packet cache per connection
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* fix(tests): restore the default db_* session after swapping it for a fake handle
Left swapped, the fake handle answered every later read_config_option() in the
run; on PHP 8.4 the extra deprecation traffic routed through CactiErrorHandler
turned that into an uncaught PDOException and killed the pest process.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
---------
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
Co-authored-by: TheWitness <thewitness@cacti.net>
* Fix#6539 - Allow Standard Cacti Log Files to be Rotated on Demand from the Cacti UI
* QA: php-cs-fixer updates and documentation for lib/database.php
* QA: Update .gitignore for composer files
* QA: Update .gitignore for composer files
* QA: Back off .gitignore change its duplicated
* QA: Fix typing of db_has_permissions
* QA: Fixing typing issue in db_format_index_create
* Update lib/clog_webapi.php
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* QA: Fix additional code issues
* QA: Applying more defaults to add device
* QA: Fix at the database layer
* QA: Fixes based upon automated testing
* QA: Fix at the database layer
* QA: After new SQL checking
* QA: Skip MacTrack oui database download warnings
* QA: Fixing type issue
* QA: Recommendation from Copilot
* QA: Recommendation from Copilot
* QA: Recommendation from Copilot
* QA: Recommendation from Copilot
* QA: Some minor changes to avoid errors in testing
* QA: Unused variable
* QA: Recommendation from Copilot
* QA: Recommendation from Copilot
* QA: Recommendation from Copilot
* QA: Minor introduced code defect
* QA: Adding some unifying space around concatination
* QA: Consistent comments and new function alias'
* QA: Multi-line comment corrections
* QA: Fix SQL Style
* QA: Missed one function alias
* QA: PHPStan Level 2 and PHPDoc fixes
* QA: PHPStan Level 2 and PHPDoc Fixes
* QA: PHPStan Level 6 Fixes
* QA: Fix lib/xml.php to PHPStan Level 6
* QA: Wrong grep -v
* PHPStan Level 6 for lib/vdef.php
* PHPStan Level 8 for lib/time.php
* QA: Raise PHPStan to Level 6 for lib/database.php
* QA: Remove @deprecated tag from grv()
* QA: Switch to htmle() alias over html_escape()
* QA: More addressing PHPStan issues at Level 6
* QA: Change recommended by Copilot for Readability
* QA: More PHPStan fixes
* QA: More PHPStan fixes
* QA: More PHPStan fixes for lib/utility.php
* QA: More PHPStan fixes at Level 6
* QA: Raise PHPStan to Level 6 More Functions
* QA: Raise PHPStan to Level 6 More Functions
* QA: Raise PHPStan to Level 6 More Functions
* QA: Raise PHPStan to Level 6 More Functions
* QA: Raise PHPStan to Level 6 More Functions
* QA: More PHPStan fixes at Level 6
* QA: Raise PHPStan to Level 6 More Functions
* QA: Raise PHPStan to Level 6 More Functions
* QA: More PHPStan fixes at Level 6
* QA: More PHPStan fixes at Level 6
* QA: Raise PHPStan to Level 6 More Functions
* QA: Raise PHPStan to Level 6 More Functions
* QA: More PHPStan fixes at Level 6
* QA: Raise PHPStan to Level 6 More Functions
* QA: Raise PHPStan to Level 6 More Functions
* QA: More PHPStan fixes at Level 6
* QA: Raise PHPStan to Level 6 More Functions
* QA: Raise PHPStan to Level 6 More Functions
* QA: More PHPStan fixes at Level 6
* QA: More PHPStan fixes at Level 6
* QA: More PHPStan fixes at Level 6
* QA: Raise PHPStan to Level 6 More Functions
* QA: More PHPStan fixes at Level 6
* QA: Some phpseclib3 fixups to update to latest
* QA: More fixups for graphing for magic strings
* QA: More PHPStan fixes at Level 6
* QA: Fixing order string type on get_allowed_graphs()
* QA: Fixing improper use of print
* QA: Optimization of color_templates
* QA: Copilot recommendation
* QA: Copilot recommendation
* QA: Copilot recommendation
* QA: Copilot recommendation
* QA: Copilot recommendation
* QA: Copilot recommendation
* QA: Copilot recommendation
* QA: Copilot recommendation
* QA: Raise PHPStan to Level 6 More Functions
* QA: Issues insalling Cacti
* QA: Issues insalling Cacti
* QA: Issues insalling Cacti
* QA: Issues insalling Cacti
* QA: Issues insalling Cacti
* QA: Issues testing Cacti
* QA: Copilot recommendation
* QA: More PHPStan fixes at Level 6
* QA: More PHPStan fixes at Level 6
* QA: More PHPStan fixes at Level 6
* QA: More PHPStan fixes at Level 6
* QA: More PHPStan fixes at Level 6
* QA: Installer failing
* Undo ownership changes to remove tar errors
* QA: Updates after php-cs-fixer
* QA: Adding code checks to syntax checking
* QA Change syntax flow language
* QA: Reduce the number of cures
* QA: Use sequentional option
* Change runner to self-hosted for check job
* Fix typo in OS version in workflow file
* Add Git installation step in workflow
Added step to install Git if not already installed.
* Add '-y' flag to apt-get install and update commands
Updated apt-get commands to include '-y' for automatic confirmation.
* Change workflow OS version to 'ubuntu-latest'
Updated the workflow to use 'ubuntu-latest' instead of 'ubuntu:latest' and modified the 'runs-on' configuration.
* Remove sequential option from phpcsfixer script
* Remove PHPCS fixer step from workflow
Removed PHP coding standards check from workflow.
* Move code checks to before plugins
Moving these core checks to before the plugins are installed as we really want to check only the base Cacti here.
* Add Composer validation and caching to start of workflow
* QA: Addressing PHPStan additional
* QA: Restore composer lock file
* QA: Addressing PHPStan additional
* QA: Addressing PHPStan additional
* QA: Complete LDAP Project
* QA: php-cs-fixer doing it's thing
* QA: Syntax is already happening in PHPStan
* QA: The md5 function no longer exists
* QA: Update after ldap changes
* QA: Undo that last password hack
* QA: Attempting to switch to MariaDB 11.8
* QA: Attempting to switch to MariaDB 11.8
* QA: Resolving some issues with PHP8.4
* QA: Resolving some issues with PHP8.4
* QA: Add PHP8.5 to the matrix
* QA: Removing PHP8.5 due to dependency issues
* QA: Add the DB version to the testing output
* QA: Switch back to MySQL for the moment
* QA: More fidelity in reporting
* QA: The pipe must be mixed
* QA: Moving audit_schema.php changes from 1.2.x
* QA: Moving audit_schema.php changes from 1.2.x
* QA: Restore mysql monitoring
* QA: php-cs-fixer issues
* QA: Attempt to roll back to MariaDB 10.6
* QA: Attempt to fix check_all_pages.sh
* QA: Make installer functional again
* QA: Fix installer php-cs-fixer issues
* QA: Type to a string
* QA: Get the database version correctly
* QA: Fix issue with reports in check_all_pages.sh
* QA: Check the server version in the right place
* Feature: Support SHA256 as a new hash for passwords
* QA: Function definition and return typing audit
* QA: Refinement of form types
* QA: Refinement of form types
* QA: Consistency of comments
* QA: Issues with php-cs-fixer and more comments
* QA: More typing issues
* QA: php-cs-fixer on merge conflict resolution
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Create a Reporting Queue for Cacti and Plugins to schedule reports
* Create a Reporting log table to hold various data from Cacti Reports for base Cacti and from Plugins
This first pass does the following:
* Moves archiving into the API
* Adds an archive function for Device Templates
* Moves package.php functions into lib/package.php
* Moves several create table syntax to the db_update_table use
* Creates a new table to hold Package Archives
* Enhances filter_value() to present a tooltip
* Add the homepage to the host_template table required for archiving.
* Fixes a PHP8 deprecation warning in sqltable_to_php.php
ToDo: Includes:
* Restoral UI
* Export UI
* This change adds two example repos where users can get packages
* Fixes a JavaScript issue package_import.php
* Creates a default manifest for our base packages
* In support of the packaging plugin, we need to add some columns to the host_template table including:
- version
- author
- email
- copyright
- tags
- installation
We also create new General Settings for Default Author information.
Thomas Vincent