mirror of
https://github.com/Cacti/cacti.git
synced 2026-10-30 08:37:01 +09:00
fix/graph-wrapper-date-range-develop
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 TheWitness
|
d57431371b |
perf(poller/boost): cached per-data-source field-name lookups, remove batched prefetch (#8078)
* perf(poller/boost): replace batched field-name prefetch with cached per-data-source lookups poller_prefetch_rrd_field_names() (issue#7534) batched the unused/nt field-name lookups into large IN() lists and rebuilt them on every process_poller_output()/boost_process_local_data_ids() call, so the completion-wait loop re-ran them each pass (DB thrash) and the IN() lists risk crashing some MariaDB/MySQL releases and do not scale to multi-million-data-source installs. Option B (full): remove poller_prefetch_rrd_field_names() and replace it with two small per-data-source helpers - poller_get_unused_data_source_names() and poller_get_nt_rrd_field_names() - each a single indexed WHERE local_data_id = ? query cached by local_data_id in a caller-held static, so every data source is queried at most once per poller process. Templated field names resolve from the static poller_data_template_field_mappings cache (the Data Template dictates); only non-templated (data_template_id == 0) manual sources hit the per-instance query - a rare exception. Both process_poller_output() (lib/poller.php) and boost_process_local_data_ids() (poller_boost.php) now use the helpers. Added a WARNING documentation block to both bulk hot-path functions spelling out the query-discipline rules for future maintainers/AI (minimal queries, no query in the per-row loop, avoid large IN() lists and full-table scans, single pass, static template-dictated caching). Updated BoostProcessBugsTest assertions to the new helpers and removed the now-obsolete BoostPrefetchRrdFieldNamesTest integration file. * fix(poller/boost): split GROUP_CONCAT'd field mappings; guard orphan-name query to templated sources; add coverage * boost: normalize orphan-name guard to ternary at third call site The reset_template branch in boost_process_local_data_ids() guarded the poller_get_unused_data_source_names() call with an if/else block rather than the ternary used at the other two call sites. Normalize it so all three sites read identically and the template-id guard is uniform. |
||
|
Thomas VincentandTheWitness
|
182b21dc53 |
fix(csv): align export and graph sample windows (#7987)
* fix(csv): align export and graph sample windows Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci(csv): satisfy release policy checks * test(csv): align legacy capacity contract * test(csv): add standard license headers * fix(rrd): infer step from trailing fetch row Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci(csv): trigger on resolution query coverage Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(rrd): mirror fetch helper arguments Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(rrd): isolate percentile globals Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(rrd): annotate intentional eval Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(rrd): restore integration config Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(rrd): enable strict types Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(rrd): restore font config Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci: align CSV coverage with supported PHP versions Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: TheWitness <thewitness@cacti.net> |
||
|
Thomas VincentandTheWitness
|
3c20f4ce7e |
Harden Boost data handoffs and graph cache writes (#7729)
* fix(boost): harden data handoffs and cache writes Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Define $updates_ok when there is nothing to flush PHPStan level 6 also flags the poller ownership check, which is the POLLER_ID constant case the file already annotates above. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Use a line comment to match the fixer Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Record the boost process columns in the audit schema Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: resolve includes through Cacti path constants Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Fix Boost handoff review concerns Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: seed settings in Boost database fixture Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: refresh SQL interpolation baseline Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * docs: record Boost handoff hardening Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix: stop boost discarding a cycle and wedging a shard Three defects in the handoff hardening. A rejected ownership handoff returned false, which the poller reads as boost having consumed the data, so it skipped the RRD write. The poller_output rows were already deleted by then, so a stale poller_item copy on a remote poller silently lost the whole cycle. Return true instead and let the poller write directly, matching the database failure path. Any non-OK RRDtool response retained the shard, and the retry then replayed rows already written, which RRDtool rejects as an illegal update time. That failed again on every pass, so the archive tables accumulated and boost never drained. Treat an at-or-behind timestamp as applied, since the sample is in the RRD; genuine errors still retain the shard. The two new poller_output_boost_processes columns passed 'null' where db_install_update_table reads 'NULL', so NOT NULL was never emitted and upgraded installs diverged from cacti.sql. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix: spell the NULL column key so upgrades emit NOT NULL db_add_column() reads $column['NULL'] and PHP array keys are case sensitive, so the thirteen definitions in the 1.3.0 upgrade that passed a lowercase 'null' key had the flag discarded and created nullable columns. Fresh installs built from cacti.sql got NOT NULL, upgraded installs did not, and Cacti's own schema audit reports the difference. The same file already spells the key in upper case 144 times, and the wrapper's own docblock example uses the upper case form. Guard the class with a test over every upgrade file rather than only the one that was wrong. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: assert the NULL key plainly toContain() takes needles rather than a failure message. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): close remaining handoff review gaps Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): guard nullable graph output Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): keep in-flight completion rows through the 1.3.0 upgrade The truncate only existed so the run_child UNIQUE key could build over rows that predate run_id; deleting just those rows leaves a running boost its completion records. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * refactor(boost): drop the poller ownership gate Its boost_poller_on_demand() call sits behind the boost_redirect early return that poller.php forces on for every multi-collector install, and its recovery call turns a device deletion on main into a recovery that never brings the collector back online. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): bound atomic cache write allocations Guard upgrade source reads so failures report clearly, write graph cache data in fixed-size chunks, and exercise a multi-chunk payload. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): support Windows cache replacement Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): expand handoff failure coverage Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): support ANSI_QUOTES upgrade checks Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): stop staging after chunk failure Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(recovery): use numeric poller status Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(sql): support ANSI_QUOTES in recovery paths Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): preserve cache on replacement failure Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): retain readable cache permissions Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): repair completion schema at runtime Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(recovery): bound timestamp batch fetches Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): satisfy runtime schema analysis Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(recovery): restore status after transfer failure Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): rely on unit bootstrap helpers Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): refresh generated cache secret Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): normalize cache path separators Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): fail clearly on unreadable fixtures * fix(boost): name ignored duplicate sample keys The staging warning only said rows were dropped, so a retry could not be told from a colliding writer. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Lack of pre-commit hook execution allowed lint failure --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: TheWitness <thewitness@cacti.net> |
||
|
Thomas Vincent
|
1ae479d4f5 |
feat(cli): add unified command frontend and normalize legacy failures (#7903)
* feat(cli): add unified command frontend and normalize legacy failures Add the Symfony-backed bin/cacti entry point while preserving legacy argument, terminal, signal, and exit-status behavior. Fold in the focused legacy CLI correctness fixes from #7906 without adding its unused parser abstraction. Keep path containment and the PHP 8.3 floor out of this review; the latter remains in #7939. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(cli): enforce PHP floor and deny web dispatch --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> |
||
|
Thomas Vincent
|
95060c49ba | Close test-suite discovery and coverage gaps (#7970) | ||
|
Thomas Vincent
|
a155419ff3 |
Serialize process registry admission with portable locks (#7771)
* Serialize process registry admission with Symfony Lock Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Fix PDO process lock mode handling Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci: run process lock coverage on PHP 8.2 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix: confirm a pid is still ours before signalling it #7574 added the still-running guard to dsstats but left six byte-identical siblings signalling a pid straight out of the processes table. After a crash or a power loss the row survives, the pid is recycled by an unrelated daemon, and the next start SIGTERMs a stranger. That is issue #7425, still live in rrdcheck, boost, poller_commands, float_rrdfiles, poller_reindex_hosts and rebuild_poller_cache. Wrap each kill in cacti_process_still_running(), matching the dsstats shape, and leave unregister_process() outside the guard so a stale row is still cleaned up. Cover all seven with a test that walks every kill site, so a new variant cannot reintroduce it. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: assert the guard plainly toContain() takes needles rather than a failure message, so the file name was being asserted as a second required substring and the test always failed. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * docs(poller): describe required lock parameters --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> |
||
|
|
da7015e497 |
Break out of the --bulk_walk case in change_device.php (#7904)
* Break out of the --bulk_walk case in change_device.php * Add CHANGELOG entry for #7904 * Fix CHANGELOG entry ordering Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(cli): pin bulk walk option control flow Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: Sean Mancini <sean@seanmancini.com> |
||
|
Thomas Vincent
|
66a2808cc9 |
Let one notification address several recipients (#7782)
* feat(notify): let one notification address several recipients Symfony calls a channel once per recipient, so a caller that needs a single message sent to several people could not express it. The email 'to' option folds the extra addresses into the same mailer() call and drops any that match the recipient. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(notify): compare the local part of an address exactly RFC 5321 leaves local part case to the receiving host, so folding the whole address deduplicated OPS@example.com and ops@example.com into one mailbox. Only the domain folds now. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> |
||
|
Thomas Vincent
|
964c1fb1f4 |
Harden RRDtool execution and data handoff (#7615)
* fix(rrdtool): quote RRDfile paths passed to the command line __rrd_execute() hands the assembled line to shell_exec(), and its docblock already required callers to quote each variable argument. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(tests): widen bare RRD path sweep Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(rrdtool): quote binary path as one argument Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(rrd): preserve caller logging context Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Make RRD XML mutations transactional Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Harden RRDtool proxy transport handling Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(rrdproxy): reject invalid command argument lists Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(rrdproxy): preserve binary graph payloads Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Keep RRDtool proxy sessions encrypted Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Harden local RRDtool command and output handoff Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Eliminate graph rendering metadata lookups Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Batch DSStats metadata and last-value reads Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Batch interface speed metadata reads Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Update architectural hotspot baseline for RRD execution Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Batch RRD resolution profile lookups Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Batch DSStats summary aggregates Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Narrow phpseclib key types for the proxy encrypt and decrypt calls Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(rrd): bound the forked proxy peers so a stall cannot hang the suite The children accepted and read with no deadline while the parent used a 2s socket timeout, so any mismatch left the child blocked in read and the parent blocked in a bare pcntl_waitpid(). Note that accept() ignores SO_RCVTIMEO, so readiness has to be waited on with socket_select() rather than a socket option. Also corrects the array-command expectation, which still assumed develop's escape-everything behaviour rather than rrdtool_build_command() leaving the operation unquoted. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(rrd): seed proxy settings where the real read_config_option looks UnitStubs only declares its stub when lib/functions.php has not been loaded yet, so during a full run the real database-backed function is the one in effect and $unit_config_options was being ignored. The parent then dialled an address the child was not listening on, which is what stalled the handshake. Seeding the option cache keeps the settings visible to either implementation. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(rrd): assert the shell-free RRDtool launch instead of the old quoting __rrd_init now hands proc_open an argv array with bypass_shell, so escaping the binary path would put the quotes inside the filename. The assertion still expected the pre-argv line and had been failing unnoticed behind the suite hang. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(dsstats): derive the expected stamps instead of assuming UTC dsstats_poller_output() formats with date(), so the hardcoded UTC strings only matched on a UTC runner. The failure was masked by the suite hang. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(rrd): keep RRDtool control codes intact in stdin commands Escaping the backslash when converting a shell-quoted argument to an RRDtool token left legends such as " IO Wait\:" ending in a dangling control code, so RRDtool refused every graph carrying an escaped colon. Verified against RRDtool 1.7.2 in pipe mode; values with no unambiguous quoted form are now refused rather than mangled. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: use Cacti test bootstrap and config Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: resolve includes through Cacti path constants Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: match the escaping helpers by token, not by text lib/rrd.php documents its escaping behaviour in a comment, and the regex over raw source counted that as a call site. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci: refresh sink inventory after develop rebase Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(rrd): prove argv isolation and concurrent pipe draining Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(security): refresh sink inventory baseline after rebase onto develop Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> |
||
|
Thomas Vincent
|
fe5d824d13 | ci: restore coverage and workflow policy gates (#7791) | ||
|
Thomas Vincent
|
619b33142e | feat(rendering): add HTML, URL, and inline-data output helpers (#7252) | ||
|
Thomas Vincent
|
4d021a93e3 | Add plugin-extensible notification API (#7775) | ||
|
Thomas Vincent
|
49f5212181 |
fix(tests): restore paths and isolate test doubles (#7764)
* fix(tests): correct path depths after the domain reorganisation The category move left nine files resolving paths relative to their old depth, so the bootstrap could not read tests/Helpers/UnitStubs.php and no test ran at all. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix tests without replacing production functions * remove unrelated failing integration tests * remove unrelated broken test workflows --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> |
||
|
Sean Mancini
|
96033267cb |
refactor(tests): reorganize unit tests into domain category folders (#7733)
* refactor(tests): reorganize unit tests into domain category folders - Consolidate per-advisory GHSA regression tests into Security/ subfolders by vuln class - Group all 186 test files into domain folders: DataCollection/, Ui/, Core/, Database/, Installer/, Scripts/, Security/, Plugin/ - Strip GitHub issue/GHSA numbers from filenames; keep IDs in test descriptions - Fix all __DIR__ relative paths to match new folder nesting depth - Update CI workflow path triggers and phpunit XML configs for new file locations - Update .github/copilot-instructions.md with unit test organization guidelines - Add tests/vendor/ to .gitignore * add gaurds * fix up guard for unit tests * fix redeclaration of cacti_log in tests * fix updates to unit tests * fix test patjhs * fix: update test __DIR__ paths for reorganized test structure * fix: conditionally define read_user_setting to prevent redeclaration in tests * fix: conditionally define functions to prevent redeclaration in tests - Wrap clean_up_lines, raise_message, cacti_debug_backtrace, and get_debug_prefix with function_exists() checks to prevent conflicts when test stubs are loaded before lib/functions.php in the same PHP process * fix: add unit test bootstrap to prevent function redeclaration errors - Create tests/bootstrap-unit.php to load core libraries before tests - Ensures lib/functions.php loads before test stubs define stub functions - Test stubs use function_exists() checks, so they detect pre-loaded functions - Update phpunit.xml to use bootstrap-unit.php for Unit testsuite - Fixes 'Cannot redeclare read_user_setting()' errors when tests run in same process * fix: load test stubs before global.php in unit test bootstrap - Reorder bootstrap to load CactiStubs.php and UnitStubs.php first - Ensures stub functions like __() are available when global.php loads - global.php calls get_cacti_version_text() which needs __() for translations - Fixes 'Call to undefined function __()' error during bootstrap * fix: remove function conflicts from test stubs - Remove stub definitions for functions already in lib/functions.php: cacti_sizeof, cacti_log, cacti_strtolower, read_config_option, read_user_setting, cacti_count - Keep translation function stubs (__(), __esc()) needed by global.php - Update bootstrap to load lib/functions.php after stubs - This prevents function redeclaration errors while maintaining test isolation * fix: prevent constant and variable redeclaration in bootstrap - Remove explicit PHP_TESTING and CACTI_TEST_BOOTSTRAP defines from bootstrap (CactiStubs.php now handles this with guards) - Add guards to CactiStubs.php to prevent redefinition of PHP_TESTING - Only set CACTI_TEST_BOOTSTRAP env vars if not already set - Fixes PHP warnings about 'Constant X already defined' * fix: remove redundant lib/functions.php load from bootstrap include/global.php already loads lib/functions.php at lines 287-288. The bootstrap was trying to load it again after global.php, but global.php needs format_cacti_version() to be available on lines 293-294 to define CACTI_VERSION and CACTI_VERSION_FULL constants. Solution: Let global.php handle loading lib/functions.php itself, and don't load it separately in the bootstrap. * fix: remove more conflicting function stubs from test helpers - Remove stubs for functions also defined in lib/functions.php: raise_message, clean_up_lines, cacti_debug_backtrace, get_debug_prefix - Remove POLLER_ID constant stub (defined unconditionally by global.php) - Keep only translation functions and test-specific helpers - This prevents function/constant redeclaration errors * fix: minimize test stubs to only translation functions Remove ALL helper functions and constants from UnitStubs.php that are: - Defined in lib/functions.php (raise_message, clean_up_lines, etc.) - Defined in lib/html_utility.php (srv, set_request_var) - Defined in global_constants.php (POLLER_VERBOSITY_*, MESSAGE_LEVEL_*) - Defined in global.php (POLLER_ID) Keep ONLY translation functions (__(), __esc()) that are: - Called by global.php during initialization - Not defined elsewhere - Required for proper bootstrap sequence This eliminates all constant and function redeclaration errors. * fix: remove __esc stub - already defined in global_languages.php Keep only __() translation function stub, which is needed by global.php and not defined elsewhere in Cacti core. * fix: handle no-arg case in __() translation stub The __() function may be called with just a string and no format arguments. Previously, vsprintf() would fail with 'arguments array must contain 2 items'. Now, if no args are provided, return the text as-is. |