mirror of
https://github.com/Cacti/cacti.git
synced 2026-11-09 22:13:05 +09:00
fix/graph-wrapper-date-range-develop
8
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 TheWitness
|
46908e350b |
fix(ci): pin htmx.org back to 2.0.10 to match vendored asset and SRI (#8091)
* fix(ci): pin htmx.org back to 2.0.10 to match vendored asset and SRI The npm group bump #8085 raised htmx.org to 4.0.0, but the vendored include/js/htmx.js, the HTMX_2_0_10_SRI integrity constant, and the htmx.js.version file were not regenerated. CI's 'Build managed JavaScript' step (build/sync-js.mjs) copies node_modules/htmx.org/dist/htmx.js, so it shipped 4.0.0 while htmx_script_tag() still emitted the 2.0.10 SRI, failing tests/Unit/Ui/Html/HtmxTest. Revert htmx.org to the vendored/pinned 2.0.10; a real htmx 4.x upgrade should be a separate, tested PR. * fix(htmx): sync version file, tests and docs to vendored 2.0.10 |
||
|
Thomas VincentandTheWitness
|
3c20f4ce7e |
Harden Boost data handoffs and graph cache writes (#7729)
* fix(boost): harden data handoffs and cache writes Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Define $updates_ok when there is nothing to flush PHPStan level 6 also flags the poller ownership check, which is the POLLER_ID constant case the file already annotates above. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Use a line comment to match the fixer Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Record the boost process columns in the audit schema Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: resolve includes through Cacti path constants Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Fix Boost handoff review concerns Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: seed settings in Boost database fixture Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: refresh SQL interpolation baseline Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * docs: record Boost handoff hardening Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix: stop boost discarding a cycle and wedging a shard Three defects in the handoff hardening. A rejected ownership handoff returned false, which the poller reads as boost having consumed the data, so it skipped the RRD write. The poller_output rows were already deleted by then, so a stale poller_item copy on a remote poller silently lost the whole cycle. Return true instead and let the poller write directly, matching the database failure path. Any non-OK RRDtool response retained the shard, and the retry then replayed rows already written, which RRDtool rejects as an illegal update time. That failed again on every pass, so the archive tables accumulated and boost never drained. Treat an at-or-behind timestamp as applied, since the sample is in the RRD; genuine errors still retain the shard. The two new poller_output_boost_processes columns passed 'null' where db_install_update_table reads 'NULL', so NOT NULL was never emitted and upgraded installs diverged from cacti.sql. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix: spell the NULL column key so upgrades emit NOT NULL db_add_column() reads $column['NULL'] and PHP array keys are case sensitive, so the thirteen definitions in the 1.3.0 upgrade that passed a lowercase 'null' key had the flag discarded and created nullable columns. Fresh installs built from cacti.sql got NOT NULL, upgraded installs did not, and Cacti's own schema audit reports the difference. The same file already spells the key in upper case 144 times, and the wrapper's own docblock example uses the upper case form. Guard the class with a test over every upgrade file rather than only the one that was wrong. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: assert the NULL key plainly toContain() takes needles rather than a failure message. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): close remaining handoff review gaps Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): guard nullable graph output Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): keep in-flight completion rows through the 1.3.0 upgrade The truncate only existed so the run_child UNIQUE key could build over rows that predate run_id; deleting just those rows leaves a running boost its completion records. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * refactor(boost): drop the poller ownership gate Its boost_poller_on_demand() call sits behind the boost_redirect early return that poller.php forces on for every multi-collector install, and its recovery call turns a device deletion on main into a recovery that never brings the collector back online. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): bound atomic cache write allocations Guard upgrade source reads so failures report clearly, write graph cache data in fixed-size chunks, and exercise a multi-chunk payload. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): support Windows cache replacement Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): expand handoff failure coverage Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): support ANSI_QUOTES upgrade checks Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): stop staging after chunk failure Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(recovery): use numeric poller status Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(sql): support ANSI_QUOTES in recovery paths Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): preserve cache on replacement failure Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): retain readable cache permissions Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): repair completion schema at runtime Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(recovery): bound timestamp batch fetches Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): satisfy runtime schema analysis Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(recovery): restore status after transfer failure Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): rely on unit bootstrap helpers Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): refresh generated cache secret Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): normalize cache path separators Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): fail clearly on unreadable fixtures * fix(boost): name ignored duplicate sample keys The staging warning only said rows were dropped, so a retry could not be told from a colliding writer. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Lack of pre-commit hook execution allowed lint failure --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: TheWitness <thewitness@cacti.net> |
||
|
dependabot[bot]andThomas Vincent
|
c54a5595de |
deps(deps): bump the npm group with 8 updates (#7919)
* deps(deps): bump the npm group with 8 updates Bumps the npm group with 8 updates: | Package | From | To | | --- | --- | --- | | [big.js](https://github.com/MikeMcl/big.js) | `6.2.2` | `7.0.1` | | [billboard.js](https://github.com/naver/billboard.js) | `3.18.0` | `4.0.3` | | [d3](https://github.com/d3/d3) | `7.8.2` | `7.9.0` | | [htmx.org](https://github.com/bigskysoftware/htmx) | `2.0.6` | `2.0.10` | | [jquery](https://github.com/jquery/jquery) | `3.7.1` | `4.0.0` | | [jquery-validation](https://github.com/jquery-validation/jquery-validation) | `1.21.0` | `1.22.1` | | [screenfull](https://github.com/sindresorhus/screenfull) | `3.3.3` | `6.0.2` | | [tablesorter](https://github.com/Mottie/tablesorter) | `2.31.3` | `2.32.0` | Updates `big.js` from 6.2.2 to 7.0.1 - [Changelog](https://github.com/MikeMcl/big.js/blob/main/CHANGELOG.md) - [Commits](https://github.com/MikeMcl/big.js/compare/v6.2.2...v7.0.1) Updates `billboard.js` from 3.18.0 to 4.0.3 - [Release notes](https://github.com/naver/billboard.js/releases) - [Changelog](https://github.com/naver/billboard.js/blob/master/CHANGELOG.md) - [Commits](https://github.com/naver/billboard.js/compare/3.18.0...4.0.3) Updates `d3` from 7.8.2 to 7.9.0 - [Release notes](https://github.com/d3/d3/releases) - [Changelog](https://github.com/d3/d3/blob/main/CHANGES.md) - [Commits](https://github.com/d3/d3/compare/v7.8.2...v7.9.0) Updates `htmx.org` from 2.0.6 to 2.0.10 - [Release notes](https://github.com/bigskysoftware/htmx/releases) - [Changelog](https://github.com/bigskysoftware/htmx/blob/master/CHANGELOG.md) - [Commits](https://github.com/bigskysoftware/htmx/compare/v2.0.6...v2.0.10) Updates `jquery` from 3.7.1 to 4.0.0 - [Release notes](https://github.com/jquery/jquery/releases) - [Changelog](https://github.com/jquery/jquery/blob/main/changelog.md) - [Commits](https://github.com/jquery/jquery/compare/3.7.1...4.0.0) Updates `jquery-validation` from 1.21.0 to 1.22.1 - [Release notes](https://github.com/jquery-validation/jquery-validation/releases) - [Changelog](https://github.com/jquery-validation/jquery-validation/blob/master/changelog.md) - [Commits](https://github.com/jquery-validation/jquery-validation/compare/1.21.0...1.22.1) Updates `screenfull` from 3.3.3 to 6.0.2 - [Release notes](https://github.com/sindresorhus/screenfull/releases) - [Commits](https://github.com/sindresorhus/screenfull/compare/v3.3.3...v6.0.2) Updates `tablesorter` from 2.31.3 to 2.32.0 - [Changelog](https://github.com/Mottie/tablesorter/blob/master/CHANGELOG.md) - [Commits](https://github.com/Mottie/tablesorter/compare/v2.31.3...v2.32.0) --- updated-dependencies: - dependency-name: big.js dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-major dependency-group: npm - dependency-name: billboard.js dependency-version: 4.0.3 dependency-type: direct:production update-type: version-update:semver-major dependency-group: npm - dependency-name: d3 dependency-version: 7.9.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm - dependency-name: htmx.org dependency-version: 2.0.10 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: npm - dependency-name: jquery dependency-version: 4.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: npm - dependency-name: jquery-validation dependency-version: 1.22.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm - dependency-name: screenfull dependency-version: 6.0.2 dependency-type: direct:production update-type: version-update:semver-major dependency-group: npm - dependency-name: tablesorter dependency-version: 2.32.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: npm ... Signed-off-by: dependabot[bot] <support@github.com> * Keep screenfull on browser-compatible release Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Refresh htmx integrity metadata Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Thomas Vincent <thomasvincent@gmail.com> |
||
|
Sean Mancini
|
96033267cb |
refactor(tests): reorganize unit tests into domain category folders (#7733)
* refactor(tests): reorganize unit tests into domain category folders - Consolidate per-advisory GHSA regression tests into Security/ subfolders by vuln class - Group all 186 test files into domain folders: DataCollection/, Ui/, Core/, Database/, Installer/, Scripts/, Security/, Plugin/ - Strip GitHub issue/GHSA numbers from filenames; keep IDs in test descriptions - Fix all __DIR__ relative paths to match new folder nesting depth - Update CI workflow path triggers and phpunit XML configs for new file locations - Update .github/copilot-instructions.md with unit test organization guidelines - Add tests/vendor/ to .gitignore * add gaurds * fix up guard for unit tests * fix redeclaration of cacti_log in tests * fix updates to unit tests * fix test patjhs * fix: update test __DIR__ paths for reorganized test structure * fix: conditionally define read_user_setting to prevent redeclaration in tests * fix: conditionally define functions to prevent redeclaration in tests - Wrap clean_up_lines, raise_message, cacti_debug_backtrace, and get_debug_prefix with function_exists() checks to prevent conflicts when test stubs are loaded before lib/functions.php in the same PHP process * fix: add unit test bootstrap to prevent function redeclaration errors - Create tests/bootstrap-unit.php to load core libraries before tests - Ensures lib/functions.php loads before test stubs define stub functions - Test stubs use function_exists() checks, so they detect pre-loaded functions - Update phpunit.xml to use bootstrap-unit.php for Unit testsuite - Fixes 'Cannot redeclare read_user_setting()' errors when tests run in same process * fix: load test stubs before global.php in unit test bootstrap - Reorder bootstrap to load CactiStubs.php and UnitStubs.php first - Ensures stub functions like __() are available when global.php loads - global.php calls get_cacti_version_text() which needs __() for translations - Fixes 'Call to undefined function __()' error during bootstrap * fix: remove function conflicts from test stubs - Remove stub definitions for functions already in lib/functions.php: cacti_sizeof, cacti_log, cacti_strtolower, read_config_option, read_user_setting, cacti_count - Keep translation function stubs (__(), __esc()) needed by global.php - Update bootstrap to load lib/functions.php after stubs - This prevents function redeclaration errors while maintaining test isolation * fix: prevent constant and variable redeclaration in bootstrap - Remove explicit PHP_TESTING and CACTI_TEST_BOOTSTRAP defines from bootstrap (CactiStubs.php now handles this with guards) - Add guards to CactiStubs.php to prevent redefinition of PHP_TESTING - Only set CACTI_TEST_BOOTSTRAP env vars if not already set - Fixes PHP warnings about 'Constant X already defined' * fix: remove redundant lib/functions.php load from bootstrap include/global.php already loads lib/functions.php at lines 287-288. The bootstrap was trying to load it again after global.php, but global.php needs format_cacti_version() to be available on lines 293-294 to define CACTI_VERSION and CACTI_VERSION_FULL constants. Solution: Let global.php handle loading lib/functions.php itself, and don't load it separately in the bootstrap. * fix: remove more conflicting function stubs from test helpers - Remove stubs for functions also defined in lib/functions.php: raise_message, clean_up_lines, cacti_debug_backtrace, get_debug_prefix - Remove POLLER_ID constant stub (defined unconditionally by global.php) - Keep only translation functions and test-specific helpers - This prevents function/constant redeclaration errors * fix: minimize test stubs to only translation functions Remove ALL helper functions and constants from UnitStubs.php that are: - Defined in lib/functions.php (raise_message, clean_up_lines, etc.) - Defined in lib/html_utility.php (srv, set_request_var) - Defined in global_constants.php (POLLER_VERBOSITY_*, MESSAGE_LEVEL_*) - Defined in global.php (POLLER_ID) Keep ONLY translation functions (__(), __esc()) that are: - Called by global.php during initialization - Not defined elsewhere - Required for proper bootstrap sequence This eliminates all constant and function redeclaration errors. * fix: remove __esc stub - already defined in global_languages.php Keep only __() translation function stub, which is needed by global.php and not defined elsewhere in Cacti core. * fix: handle no-arg case in __() translation stub The __() function may be called with just a string and no format arguments. Previously, vsprintf() would fail with 'arguments array must contain 2 items'. Now, if no args are provided, return the text as-is. |
||
|
Thomas VincentandTheWitness
|
a75bf823bb |
feat: consolidate htmx loader and renderer abstraction global.php includes (#7517)
* feat(rendering): add view renderer abstraction Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * isolate renderer traversal test file * Standardize renderer headers and endpoint guard * style(renderer): use standard unit-test header in CactiRendererTest Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(renderer): reject null-byte paths before realpath Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci: refresh sink and hotspot baselines for line shifts Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(renderer): guard root base path and template-dir redirect Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(renderer): canonicalize both paths in index.php direct-access guard Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * hardening: normalize renderer base path to one separator Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(renderer): reject filesystem root as template path A /-rooted renderer collapsed the containment prefix to /, which str_starts_with() matches for every absolute path, so any file was includable. Fail closed by refusing a root template path. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * hardening: reject windows drive-root template path Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * refactor: adopt PHP 8.4 idioms in view renderer Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: cover renderer abstraction to full coverage Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * hardening(renderer): reject UNC template names and fix sink baselines Refuse a leading-backslash (UNC) template name the same way absolute POSIX paths are refused, skip the direct-request redirect when SCRIPT_FILENAME carries a null byte, and correct the renderer include and views redirect line numbers in the security baselines. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * style: apply php-cs-fixer Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(renderer): use PHP 8.1-compatible readonly property in CactiRenderer Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * hardening(rendering): drop template context array before include Unset the internal context array after extract so an included template only sees the extracted variables, not the full context. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci: refresh security baselines for renderer line shift Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(renderer): simplify containment check and drain nested output buffers The redundant real_path !== template_path comparison in validateTemplateFile() can never be false once is_file()/is_dir() have run, so drop it and keep the str_starts_with() prefix check as the actual containment guard. renderFile() also only unwound a single output buffer, so a template that opens (and leaves) its own buffer could leak an ob level or drop its content; unwind every buffer opened during rendering and reassemble them in the order they were written. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci: refresh security baselines for renderer buffer/containment fix The dynamic_include line for lib/renderer.php's include shifted again after adding drainOutputBuffers(). Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * feat(htmx): vendor htmx 2.0.6 and add opt-in filter loader Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: cover htmx filter loader Add missing-vendored-file case for htmx_script_tag and mark the pre-bootstrap read_config_option guard as uncoverable. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: harden htmx loader harness and tighten enable check Skip anything but an exact 'on' for htmx_enabled, reject 4xx/5xx fixture responses, fail fast when the fixture base path is unresolved, pin setup-php to a commit SHA, and reindent the e2e specs to tabs. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * style: apply php-cs-fixer to htmx test files Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(htmx): drop duplicate script-tag src assertion The subdir-install test repeated the preceding assertion; CACTI_PATH_URL is fixed at bootstrap so it could not exercise a distinct scenario. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(e2e): drop sqlite3 from docker-php-ext-install php:8.2-apache already ships sqlite3 built in, and its ext/sqlite3 source only carries config0.m4 (not config.m4), so building it standalone always fails with "Cannot find config.m4", breaking the e2e image build before Playwright ever runs. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(htmx): clear stat cache around version-file rename Earlier tests already stat the version file via file_exists()/ file_get_contents(), so PHP's stat cache can report it present right after these tests rename it away, making the assertions flaky. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(htmx): recompute script-tag cache-buster per call The md5 cache-buster was cached in a static, so FPM/script_server processes kept serving a stale ?v=... hash after an htmx.min.js upgrade. Recompute it per call, matching get_md5_include_js() and get_theme_paths() in lib/functions.php, and drop the 'v=' parameter name to match their path + '?' + hash shape. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(e2e): drop obsolete MariaDB options * ci(htmx): run the htmx integration test by path instead of a testsuite filter Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * build: vendor htmx unminified Debian rejects minified sources. The file is the htmx 2.0.6 dist source; its minified sibling hashes identically to what was committed, and the pinned SRI is recomputed over the unminified file. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix: address htmx review feedback * test(e2e): resolve Cacti version through PHP * test(htmx): harden portable server cleanup * ci(actions): update checkout action pins --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: TheWitness <thewitness@cacti.net> |
||
|
Thomas VincentandTheWitness
|
db39178735 |
fix: harden the poller_item cache and Boost RRD staging subsystems (#7538)
* fix(boost): forward still-open-round archive rows instead of dropping them boost_process_poller_output() correctly excluded rows from an in-progress poll round when building the RRD write, but the archive-table cleanup right after deleted every row for the local_data_id anyway, with no time filter. Those rows now get INSERT IGNORE'd into the live poller_output_boost table before the archive table is cleared, so a still-open round's data survives. Also fixes the duplicate-record guard in the same function: $last_item was initialized once before the results loop and never reassigned, so the comparison against it could never be true. It is now updated after each processed row. Fixes #7519 Fixes #7523 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): gate archive-table drop on shard completeness poller_boost.php decided whether to drop a run's archive tables using SELECT SUM(status) FROM poller_output_boost_processes, an aggregate over whatever completion rows landed. If one child crashed before writing its completion row, the sum from the surviving shards was still positive and every archive table for the run got dropped, including the crashed shard's unprocessed data. The drain loop already computes boost_completed_children() vs $expected_children to detect a crashed child; the drop now uses that same completeness check instead, and leaves the archive tables in place with a warning when it is not met. Fixes #7520 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(dsstats,rrdcheck): apply Boost's startup barrier to launch loops dsstats_launch_children()/rrdcheck_launch_children() fork children via exec_background() (non-blocking) and the callers then wait with a bare while ($running = ..._processes_running($type)) { sleep(2); }. If the parent evaluates that before any child has registered in the processes table, $running reads 0 and the loop body never runs, treating "not started yet" as "already done" -- the same race Boost's boost_all_children_registered() barrier was added to close. dsstats_launch_children()/rrdcheck_launch_children() now return the launched count, and dsstats_wait_for_children()/ rrdcheck_wait_for_children() wait for the running count to reach it (bounded by a deadline) before trusting the drain loop's running-count poll. Fixes #7526 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * refactor(poller): consolidate poller_item deletes behind a shared helper Extract poller_item_delete_for_data_source() and poller_item_delete_for_host() in lib/poller.php and migrate the existing local+remote delete pairs in lib/api_device.php and lib/api_data_source.php onto them, so the local-delete-plus-remote-mirror pattern lives in one place instead of eight. Refs #7535 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): mirror poller_item deletes to the remote poller on handoff api_data_source_change_host(), data_query_remove_disabled_items(), and data_source_to_data_template() each moved or removed a data source's poller_item row locally but never on the remote Data Collector that still owned it, so that Collector kept polling an item it no longer should. All three now resolve the owning remote poller before the local change and purge the row there too via the shared helper. Fixes #7521 Fixes #7524 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): correct poller_item join semantics and cron reset in cmd.php poller_item and host share ~13 identically-named snmp_*/hostname columns; SELECT * across the join let PDO FETCH_ASSOC silently collapse them to host's live values instead of poller_item's cached copies. Select pi.* plus explicitly aliased host/site disabled columns instead. The now already-fetched $poller_items also makes the separate script-server-call COUNT(*) re-join redundant, replaced with an in-PHP count. Also apply the same cron_interval-aware rrd_next_step reset used by the ranged branch to the --allhost branch, which previously ignored it. Fixes #7522, #7525, #7528. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): thread known poller_id through update_poller_cache() callers update_poller_cache() re-derived poller_id via a host/data_local JOIN on every call, even when the caller already had it: repopulate_poller_cache() and push_out_data_input_method() both already select h.poller_id and discarded it, push_out_host() has the host row cached per data source, and api_device_enable_devices()/update_poller_cache_from_query() compute it once outside their loops. Add an optional $poller_id argument (and honor an already-present 'poller_id' array key) so these callers skip the redundant query. Fixes #7527. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): scope global aggregate scans and make process-leveling sargable Three poller.php queries (total_snmp_ports and the mutually-exclusive active_profiles pair) had no poller_id filter, unlike the rest of the file's central-only blocks, so every remote poller redundantly recomputed and overwrote the same global config options; the snmp_port scan is also an unindexed full table scan. Gate them behind poller_id == 1, with remote pollers reading back the value the central poller already stored. Also rewrite the process-leveling $sql_where's rrd_next_step predicate from the non-sargable `rrd_next_step - $poller_interval <= 0` to the algebraically equivalent `rrd_next_step <= $poller_interval`, restoring use of the poller_id_rrd_next_step index. Fixes #7529, #7530. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): batch update_data_query_cache()'s poller cache writes Each changed data source called update_poller_cache($data_source, true), which flushes immediately via poller_update_poller_cache_from_buffer() -- its own present=0 UPDATE, INSERT ON DUPLICATE KEY UPDATE, and stale-row DELETE -- on every iteration. Re-indexing a host where every port changed issued up to 3x as many statements as ports instead of one batched flush. Accumulate changed rows across the loop and call update_poller_cache() with commit=false, then flush once via poller_update_poller_cache_from_buffer() after the loop, matching the batched pattern already used by push_out_data_input_method(). Fixes #7531. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): add USING BTREE to poller_output_boost_local_data_ids indexes MEMORY-engine indexes default to HASH unless USING BTREE is specified, unlike the sibling poller_output/poller_output_boost tables in the same schema file. boost_output_rrd_data()'s ORDER BY/LIMIT query and boost_process_local_data_ids()'s ranged DELETE both need to scan local_data_id in order, which a HASH index can't serve, forcing a filesort on every boost worker pass. Verified against a real MariaDB 11 instance: EXPLAIN on the ORDER BY query shows "Using filesort" with the old HASH schema and no filesort with USING BTREE, using the primary key for ordering instead. Also aligns boost_prepare_process_table()'s dead CREATE TABLE IF NOT EXISTS fallback with the corrected schema, and adds an upgrade step that rebuilds both indexes as BTREE on existing installs (checked via information_schema.STATISTICS, since db_install_add_key() only compares index columns, not algorithm, and would otherwise skip an existing HASH index with the right column list). Closes #7532 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): skip the UNION ALL wrapper for the single-archive-table case boost_process_local_data_ids() always wrapped its flush query in "SELECT * FROM (... UNION ALL ...) t ORDER BY ...", even when there was only one archive table -- the steady-state case, since old archive tables are dropped at the end of each cycle. UNION ALL results wrapped in a derived table are one of the documented cases MySQL/MariaDB's derived-table merging doesn't apply to, so every pass paid for a filesort over an indexless derived table regardless of how well-ordered the underlying archive table's rows already were. Extracts the per-table SELECT into boost_archive_select_sql() and uses it directly, without the wrapper, when there's exactly one archive table. The UNION ALL/derived-table shape is kept as the fallback for the genuinely rare case where a prior crashed run left more than one archive table behind. Verified against a real MariaDB 11 instance and via a SQLite-backed integration test that both shapes return correctly filtered and sorted rows. Closes #7533 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): batch-prefetch RRD field name lookups instead of per-row queries Two N+1 patterns in the Boost hot paths: boost_process_local_data_ids() (poller_boost.php) re-ran the unused_data_source_names/nt_rrd_field_names lookups on every local_data_id boundary in the sorted flush result set. process_poller_output() (lib/poller.php) had the equivalent lookup, but without even a boundary check -- it re-ran per matching row whenever the output contained a ':', including consecutive rows for the same data source. Adds poller_prefetch_rrd_field_names() (lib/poller.php, shared by both callers since poller_boost.php already requires lib/poller.php) to batch-fetch these mappings once per invocation via IN()-batched queries keyed by the full set of local_data_ids in the batch, chunked at 1000 ids to match the array_chunk(..., 1000) convention used elsewhere (lib/api_data_source.php, lib/api_graph.php). Both call sites now look up the prefetched maps in-memory inside their loops. While consolidating the lookups, fixes a latent bug the old process_poller_output() code had in its non-templated fallback branch: a query referencing gti.task_item_id without joining graph_templates_item, which throws an unknown-column error whenever that path is hit. The batched query always joins correctly. Integration test proves both correctness (against a real SQLite-backed schema) and the query-count reduction: 5 local_data_ids that would previously cost up to 10 queries for these two lookups now cost exactly 3, regardless of how many ids are in the batch. Closes #7534 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * refactor(boost): consolidate poller_output_boost inserts into one helper The INSERT INTO poller_output_boost (...) VALUES ... statement was hand-written four times: three in cmd.php (behind the same boost_redirect == 'on' && boost_rrd_update_enable == 'on' guard) using INSERT IGNORE, and once in boost_poller_on_demand() (lib/boost.php) using ON DUPLICATE KEY UPDATE. Beyond the duplication, that was a real data-integrity inconsistency between two paths writing the same shape of data: silently-drop vs last-write-wins on the same key. Since boost_redirect gates cmd.php's writes and boost_poller_on_demand() returns early when boost_redirect is on, the two paths never run concurrently for the same install; they're alternate write paths selected by that one setting, not competing writers. A genuine collision on (local_data_id, rrd_name, time) in either path represents a retry of an already-collected sample -- a batch resent after a partial failure, or two writers racing on the same rounded second -- not two independently meaningful readings, since RRD only stores one value per timestamp regardless. Standardizes on INSERT IGNORE (first write wins), matching the sibling poller_output insert this table shadows, which has always used INSERT IGNORE, and 3 of the 4 existing call sites. Extracts boost_flush_output_batch() (lib/boost.php) as the one shared helper, handling both the batched INSERT and the max_allowed_packet-safe chunking cmd.php's three sites previously didn't do at all (they only chunked by item count). Integration test exercises a genuine duplicate-key collision against a real (SQLite-backed) table and asserts the chosen first-write-wins behavior. Closes #7536 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(tests): resolve cross-branch test interactions found by full-suite verification Two test-only bugs surfaced only when all four fix branches for the poller_item/Boost hardening pass ran together, not when each ran in isolation: - BoostArchiveHandoffIntegrationTest extracted the archive-forward INSERT IGNORE by searching lib/boost.php for the first occurrence of "INSERT IGNORE INTO poller_output_boost". The Boost SQL-optimization branch adds a second, earlier occurrence (the new boost_flush_output_batch() helper's single-line prefix), so the naive search now grabbed the wrong snippet. Anchor on the multi-line "...poller_output_boost\n\tSELECT *" shape that only the archive-forward statement has. - PollerItemRemotePurgeHandOffTest calls push_out_host() (via api_data_source_change_host()), which needs CACTI_PATH_LIBRARY. The test never requires include/global.php (which normally defines it), and was relying on some other file in a full multi-thousand-file suite run having defined it first as a process-wide side effect. Define it directly instead of depending on unguaranteed load order. No production code changed. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * style(poller): satisfy php-cs-fixer strict_param in the script-server count Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): order the single-archive-table flush by the table's PK columns Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): key the max_allowed_packet cache per connection Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(tests): restore the default db_* session after swapping it for a fake handle Left swapped, the fake handle answered every later read_config_option() in the run; on PHP 8.4 the extra deprecation traffic routed through CactiErrorHandler turned that into an uncaught PDOException and killed the pest process. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: TheWitness <thewitness@cacti.net> |
||
|
Thomas Vincent
|
cbd711869c | test: run the Integration and HandOff suites in isolated processes (#7550) | ||
|
Thomas VincentandTheWitness
|
ae4c147f04 |
chore(test): unified Pest test infrastructure with HandOff support (#7124)
* test: shared hand-off + mutation infrastructure for feature PRs Consolidates infection/infection dev dep, allow-plugins entry, baseline infection.json5, tests/HandOff/HandOffHelpers.php with reusable stubs (cacti_log capture, temp-file fixtures, minimal-zip builder), HandOff testsuite registration, and the documented pattern. Feature PRs that add hand-off coverage rebase onto this and contribute only ONE HandOffTest file plus an optional infection.json5 filter override. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: consolidate per-feature hand-off suites into shared infra Each test guards on its feature file and skips when not present on develop. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(test): replace eval stub, unify log buffer ref, guard ZipArchive add Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(ci): correct infection vendor path in infection.json5 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci(phpstan): baseline upstream undefined-variable and isset errors Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: add HandOff and Integration testsuites; add regression guards - phpunit.xml: add Integration testsuite alongside existing Unit/HandOff - tests/Integration/: DbDumpIntegrationTest, PingIntegrationTest (from develop), CactiProcessIntegrationTest, SqlScriptsIntegrationTest - tests/HandOff/RegressionGuardTest: guards against backsliding on shell_exec, password-in-argv, RLIKE concat, and open-redirect patterns Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * chore(test): pest test infrastructure and PSR-4 autoload Add Pest 2 dev dependency, regenerate composer.lock under PHP 8.1, declare PSR-4 autoload for the Cacti namespace, and seed the test bootstrap with three unit tests plus an orb integration check. Pest stays on the v2 line because v3 transitively requires PHP 8.2 and breaks the 8.1 matrix entry. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * chore(phpstan): catch up baseline with 11 upstream-detected entries Cacti Commit Audit fails on develop with 11 PHPStan errors that are not yet in phpstan-baseline.neon. Append the matching entries so the PR-A branch passes Run PHPStan at Level 6. Same set of entries already exists on PR #7077; this is a transient catch-up that upstream will absorb when those entries land on develop. Files: aggregate_graphs.php, color_templates.php, graph_templates.php, graphs.php, lib/html.php (3 entries). Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * chore(test): drop composer.lock; let CI resolve per PHP version The lock file pinned brianium/paratest v7.3.1 which only supports PHP 8.1-8.3, so the 8.4 matrix entry rejected the lock with "Your lock file does not contain a compatible set of packages". A single lock cannot satisfy 8.1 (paratest 7.3.x) and 8.4 (paratest 7.4.x) at the same time because pestphp/pest 2.36.0 is the last 2.x release that supports 8.1 and the 8.1-compatible paratest tag predates 8.4 support. Match upstream develop's behaviour: no committed lock; let `composer install` in CI resolve per matrix entry. Pest stays on the v2 line in composer.json so 8.1 still gets a compatible Pest. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(test): apply Copilot review feedback for test infra Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * chore(test): drop psr-4 autoload and align deps with canonical PRs Match the flat un-namespaced lib/CactiX.php convention used by #7088, #7073, #7077, #7075. Drop symfony/process (added by #7073) and symfony/validator (added by #7077) to avoid composer.json conflicts at merge time. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: skip integration suites and regression guards pending feature PRs Guards integration tests and regression guards from #7083 against absence of CactiProcess and other feature-PR hardening, so the consolidated foundation PR is green on develop. Suites activate automatically once their feature PRs merge. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(ci): drop infection/infection to unbreak PHP 8.4 matrix infection/infection ^0.27 transitively pins thecodingmachine/safe ^2.1.2, resolving to v2.5.0. Its generated stubs use implicit-nullable parameter declarations, which PHP 8.4 emits as deprecations into cacti.log and trips the log-quiet assertion. infection has no config or test wiring in this repo, so removing the dev dep clears the transitive pin without losing functionality. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: TheWitness <thewitness@cacti.net> |