mirror of
https://github.com/Cacti/cacti.git
synced 2026-11-02 10:07:22 +09:00
develop
17
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Thomas Vincent
|
a12fe59627 |
fix(poller): [develop] retain output until bounded storage batches succeed (#8332)
* fix(poller): retain output until bounded storage batches succeed Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(poller): refresh SQL audit after exact-key acknowledgements Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): preserve handoff contracts with explicit acknowledgements Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): retain failed timestamp ordering across batches Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): acknowledge complete timestamp groups together Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): retry failed source metadata reads on develop Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): preload standalone script exclusions once per process Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): bound metadata failures and validate acknowledgement queries Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> |
||
|
Thomas Vincent
|
af9d1591a3 |
fix(aggregate): honor skipped item sequences in totals and percentiles (#8300)
* fix(aggregate): honor skipped item sequences in totals and percentiles Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: tighten SQL baseline for aggregate prepared queries Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * docs(changelog): defer release entries to final backlog PR Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> |
||
|
Thomas Vincent
|
06c32ab13f |
fix(installer): preserve upgrade state and propagate failures (#8266)
* fix(installer): preserve upgrade state and propagate failures Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * docs(changelog): record installer recovery fixes for #8266 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(security): refresh installer sink inventories Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(installer): retain merge precedence for browser parameters Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(installer): defer debug settings until the process lock is held Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(installer): enforce table conversion failures and postconditions Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(installer): refresh SQL audit baseline after clause refactor Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> |
||
|
TheWitnessandci
|
9facd9b500 |
security: batch 7 develop - consolidate remaining develop-only GHSA advisories (12 fixes + 8 confirmations) (#8149)
* security: batch 7 develop - consolidate remaining develop-only GHSA advisories (12 fixes + 8 regression confirmations) Fixes: f7jw (automation preview XSS), rp5g (db_column_exists LIKE), vx2m (automation pivot CASE WHEN), h5wg (plugin DDL identifiers), 929h (aggregate LIKE filter), mfw8 (add_graphs REGEXP), m49v (user_admin sort_column allowlist), mpfm (rrd error-image theme allowlist), m3fh (SNMPv3 control chars), cg45 (remember-me row revoke), xq26 (remember-me retention window), m67r (import xml_path containment). Confirmed already fixed on develop, pinned with regression tests: 4gph, 977w, mgcq, 5p79, cfhh, wfcf, pjq5, wp33. Excluded: 227p (1.2.x-only), mx5c (maintainer private branch). * security: address batch-7 review feedback (lexical-first xml_path containment, session-wide sort allowlist, connection-aware db_column_exists quoting, reject-not-normalize plugin DDL identifiers, behavior-based test names) * style: apply php-cs-fixer single-quote fix to api_automation pivot builders; add WSL dev-environment note to copilot instructions * fix phpstan string typing * test: refresh SQL interpolation baseline for db_column_exists db_qstr change (database.php 9->8) --------- Co-authored-by: ci <ci@local> |
||
|
github-actions[bot]
|
fe45229820 |
ci: refresh security/SQL-interpolation baselines (#8086)
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Thomas VincentandTheWitness
|
3c20f4ce7e |
Harden Boost data handoffs and graph cache writes (#7729)
* fix(boost): harden data handoffs and cache writes Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Define $updates_ok when there is nothing to flush PHPStan level 6 also flags the poller ownership check, which is the POLLER_ID constant case the file already annotates above. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Use a line comment to match the fixer Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Record the boost process columns in the audit schema Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: resolve includes through Cacti path constants Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Fix Boost handoff review concerns Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: seed settings in Boost database fixture Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: refresh SQL interpolation baseline Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * docs: record Boost handoff hardening Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix: stop boost discarding a cycle and wedging a shard Three defects in the handoff hardening. A rejected ownership handoff returned false, which the poller reads as boost having consumed the data, so it skipped the RRD write. The poller_output rows were already deleted by then, so a stale poller_item copy on a remote poller silently lost the whole cycle. Return true instead and let the poller write directly, matching the database failure path. Any non-OK RRDtool response retained the shard, and the retry then replayed rows already written, which RRDtool rejects as an illegal update time. That failed again on every pass, so the archive tables accumulated and boost never drained. Treat an at-or-behind timestamp as applied, since the sample is in the RRD; genuine errors still retain the shard. The two new poller_output_boost_processes columns passed 'null' where db_install_update_table reads 'NULL', so NOT NULL was never emitted and upgraded installs diverged from cacti.sql. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix: spell the NULL column key so upgrades emit NOT NULL db_add_column() reads $column['NULL'] and PHP array keys are case sensitive, so the thirteen definitions in the 1.3.0 upgrade that passed a lowercase 'null' key had the flag discarded and created nullable columns. Fresh installs built from cacti.sql got NOT NULL, upgraded installs did not, and Cacti's own schema audit reports the difference. The same file already spells the key in upper case 144 times, and the wrapper's own docblock example uses the upper case form. Guard the class with a test over every upgrade file rather than only the one that was wrong. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: assert the NULL key plainly toContain() takes needles rather than a failure message. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): close remaining handoff review gaps Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): guard nullable graph output Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): keep in-flight completion rows through the 1.3.0 upgrade The truncate only existed so the run_child UNIQUE key could build over rows that predate run_id; deleting just those rows leaves a running boost its completion records. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * refactor(boost): drop the poller ownership gate Its boost_poller_on_demand() call sits behind the boost_redirect early return that poller.php forces on for every multi-collector install, and its recovery call turns a device deletion on main into a recovery that never brings the collector back online. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): bound atomic cache write allocations Guard upgrade source reads so failures report clearly, write graph cache data in fixed-size chunks, and exercise a multi-chunk payload. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): support Windows cache replacement Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): expand handoff failure coverage Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): support ANSI_QUOTES upgrade checks Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): stop staging after chunk failure Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(recovery): use numeric poller status Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(sql): support ANSI_QUOTES in recovery paths Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): preserve cache on replacement failure Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): retain readable cache permissions Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): repair completion schema at runtime Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(recovery): bound timestamp batch fetches Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): satisfy runtime schema analysis Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(recovery): restore status after transfer failure Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): rely on unit bootstrap helpers Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): refresh generated cache secret Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): normalize cache path separators Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(boost): fail clearly on unreadable fixtures * fix(boost): name ignored duplicate sample keys The staging warning only said rows were dropped, so a retry could not be told from a colliding writer. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Lack of pre-commit hook execution allowed lint failure --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: TheWitness <thewitness@cacti.net> |
||
|
Thomas Vincent
|
ead882172e |
Make installer completion fail closed and add test coverage (#7626)
* Make installer completion fail closed Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Add installer unit and Docker integration gates Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Require installer coverage evidence Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Stabilize installer failure scenarios Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Address installer review feedback Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Reject incomplete installer selection payloads Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Match table validation to rendered controls Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Protect long-running installer migrations Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Load installer table metadata in one query Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Optimize installer completion queries Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Batch installer default settings writes Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Batch installer color upserts Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Batch installer template and poller lookups Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Refresh SQL interpolation baseline after develop rebase Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci: use Cacti Composer test setup Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test: resolve includes through Cacti path constants Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci: align installer workflows with policy Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci: refresh security baselines after develop rebase Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci: reuse TCP database credentials for installer checks Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(security): refresh sink inventory baseline after rebase onto develop Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci: run installer coverage on PHP 8.2 The 8.1 pin is below the ^8.2 composer requirement, so pest died in platform_check before any test ran. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> |
||
|
Thomas Vincent
|
5da1a2d5e8 | security: restrict graph input fields across handoffs (CVE-2024-31458) (#7669) | ||
|
Thomas VincentandTheWitness
|
ae54a2a552 |
security: bind device IDs and escape graph and device output (#7732)
* security: bind device IDs and escape graph and device output Device removal waits for graph cleanup inside a transaction before deleting the host row. Clone and template-sync ID lists go through db_in_clause, SNMP and ping strings are escaped at HTML sinks, and graph page size and column settings use bounded validators. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix: skip cloned RRD task item remap when no matching data_template_rrd row exists Guards the outer loop in api_clone_device_template() the same way the inner loop already guards $new_graph_template_id: db_fetch_cell_prepared() returns false when the cloned Data Template has no matching data_template_rrd row, and binding that into task_item_id would corrupt the cloned graph template's wiring instead of leaving it alone. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * Number the CHANGELOG entry so the format check passes Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: TheWitness <thewitness@cacti.net> |
||
|
github-actions[bot]
|
085bee630d |
ci: refresh security/SQL-interpolation baselines (#7608)
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Thomas Vincent
|
ec05b4f816 | Expire auth tokens when permissions change (#7601) | ||
|
github-actions[bot]
|
967da9443d |
ci: refresh security/SQL-interpolation baselines (#7565)
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> |
||
|
Thomas VincentandTheWitness
|
db39178735 |
fix: harden the poller_item cache and Boost RRD staging subsystems (#7538)
* fix(boost): forward still-open-round archive rows instead of dropping them boost_process_poller_output() correctly excluded rows from an in-progress poll round when building the RRD write, but the archive-table cleanup right after deleted every row for the local_data_id anyway, with no time filter. Those rows now get INSERT IGNORE'd into the live poller_output_boost table before the archive table is cleared, so a still-open round's data survives. Also fixes the duplicate-record guard in the same function: $last_item was initialized once before the results loop and never reassigned, so the comparison against it could never be true. It is now updated after each processed row. Fixes #7519 Fixes #7523 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): gate archive-table drop on shard completeness poller_boost.php decided whether to drop a run's archive tables using SELECT SUM(status) FROM poller_output_boost_processes, an aggregate over whatever completion rows landed. If one child crashed before writing its completion row, the sum from the surviving shards was still positive and every archive table for the run got dropped, including the crashed shard's unprocessed data. The drain loop already computes boost_completed_children() vs $expected_children to detect a crashed child; the drop now uses that same completeness check instead, and leaves the archive tables in place with a warning when it is not met. Fixes #7520 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(dsstats,rrdcheck): apply Boost's startup barrier to launch loops dsstats_launch_children()/rrdcheck_launch_children() fork children via exec_background() (non-blocking) and the callers then wait with a bare while ($running = ..._processes_running($type)) { sleep(2); }. If the parent evaluates that before any child has registered in the processes table, $running reads 0 and the loop body never runs, treating "not started yet" as "already done" -- the same race Boost's boost_all_children_registered() barrier was added to close. dsstats_launch_children()/rrdcheck_launch_children() now return the launched count, and dsstats_wait_for_children()/ rrdcheck_wait_for_children() wait for the running count to reach it (bounded by a deadline) before trusting the drain loop's running-count poll. Fixes #7526 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * refactor(poller): consolidate poller_item deletes behind a shared helper Extract poller_item_delete_for_data_source() and poller_item_delete_for_host() in lib/poller.php and migrate the existing local+remote delete pairs in lib/api_device.php and lib/api_data_source.php onto them, so the local-delete-plus-remote-mirror pattern lives in one place instead of eight. Refs #7535 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): mirror poller_item deletes to the remote poller on handoff api_data_source_change_host(), data_query_remove_disabled_items(), and data_source_to_data_template() each moved or removed a data source's poller_item row locally but never on the remote Data Collector that still owned it, so that Collector kept polling an item it no longer should. All three now resolve the owning remote poller before the local change and purge the row there too via the shared helper. Fixes #7521 Fixes #7524 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): correct poller_item join semantics and cron reset in cmd.php poller_item and host share ~13 identically-named snmp_*/hostname columns; SELECT * across the join let PDO FETCH_ASSOC silently collapse them to host's live values instead of poller_item's cached copies. Select pi.* plus explicitly aliased host/site disabled columns instead. The now already-fetched $poller_items also makes the separate script-server-call COUNT(*) re-join redundant, replaced with an in-PHP count. Also apply the same cron_interval-aware rrd_next_step reset used by the ranged branch to the --allhost branch, which previously ignored it. Fixes #7522, #7525, #7528. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): thread known poller_id through update_poller_cache() callers update_poller_cache() re-derived poller_id via a host/data_local JOIN on every call, even when the caller already had it: repopulate_poller_cache() and push_out_data_input_method() both already select h.poller_id and discarded it, push_out_host() has the host row cached per data source, and api_device_enable_devices()/update_poller_cache_from_query() compute it once outside their loops. Add an optional $poller_id argument (and honor an already-present 'poller_id' array key) so these callers skip the redundant query. Fixes #7527. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): scope global aggregate scans and make process-leveling sargable Three poller.php queries (total_snmp_ports and the mutually-exclusive active_profiles pair) had no poller_id filter, unlike the rest of the file's central-only blocks, so every remote poller redundantly recomputed and overwrote the same global config options; the snmp_port scan is also an unindexed full table scan. Gate them behind poller_id == 1, with remote pollers reading back the value the central poller already stored. Also rewrite the process-leveling $sql_where's rrd_next_step predicate from the non-sargable `rrd_next_step - $poller_interval <= 0` to the algebraically equivalent `rrd_next_step <= $poller_interval`, restoring use of the poller_id_rrd_next_step index. Fixes #7529, #7530. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(poller): batch update_data_query_cache()'s poller cache writes Each changed data source called update_poller_cache($data_source, true), which flushes immediately via poller_update_poller_cache_from_buffer() -- its own present=0 UPDATE, INSERT ON DUPLICATE KEY UPDATE, and stale-row DELETE -- on every iteration. Re-indexing a host where every port changed issued up to 3x as many statements as ports instead of one batched flush. Accumulate changed rows across the loop and call update_poller_cache() with commit=false, then flush once via poller_update_poller_cache_from_buffer() after the loop, matching the batched pattern already used by push_out_data_input_method(). Fixes #7531. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): add USING BTREE to poller_output_boost_local_data_ids indexes MEMORY-engine indexes default to HASH unless USING BTREE is specified, unlike the sibling poller_output/poller_output_boost tables in the same schema file. boost_output_rrd_data()'s ORDER BY/LIMIT query and boost_process_local_data_ids()'s ranged DELETE both need to scan local_data_id in order, which a HASH index can't serve, forcing a filesort on every boost worker pass. Verified against a real MariaDB 11 instance: EXPLAIN on the ORDER BY query shows "Using filesort" with the old HASH schema and no filesort with USING BTREE, using the primary key for ordering instead. Also aligns boost_prepare_process_table()'s dead CREATE TABLE IF NOT EXISTS fallback with the corrected schema, and adds an upgrade step that rebuilds both indexes as BTREE on existing installs (checked via information_schema.STATISTICS, since db_install_add_key() only compares index columns, not algorithm, and would otherwise skip an existing HASH index with the right column list). Closes #7532 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): skip the UNION ALL wrapper for the single-archive-table case boost_process_local_data_ids() always wrapped its flush query in "SELECT * FROM (... UNION ALL ...) t ORDER BY ...", even when there was only one archive table -- the steady-state case, since old archive tables are dropped at the end of each cycle. UNION ALL results wrapped in a derived table are one of the documented cases MySQL/MariaDB's derived-table merging doesn't apply to, so every pass paid for a filesort over an indexless derived table regardless of how well-ordered the underlying archive table's rows already were. Extracts the per-table SELECT into boost_archive_select_sql() and uses it directly, without the wrapper, when there's exactly one archive table. The UNION ALL/derived-table shape is kept as the fallback for the genuinely rare case where a prior crashed run left more than one archive table behind. Verified against a real MariaDB 11 instance and via a SQLite-backed integration test that both shapes return correctly filtered and sorted rows. Closes #7533 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): batch-prefetch RRD field name lookups instead of per-row queries Two N+1 patterns in the Boost hot paths: boost_process_local_data_ids() (poller_boost.php) re-ran the unused_data_source_names/nt_rrd_field_names lookups on every local_data_id boundary in the sorted flush result set. process_poller_output() (lib/poller.php) had the equivalent lookup, but without even a boundary check -- it re-ran per matching row whenever the output contained a ':', including consecutive rows for the same data source. Adds poller_prefetch_rrd_field_names() (lib/poller.php, shared by both callers since poller_boost.php already requires lib/poller.php) to batch-fetch these mappings once per invocation via IN()-batched queries keyed by the full set of local_data_ids in the batch, chunked at 1000 ids to match the array_chunk(..., 1000) convention used elsewhere (lib/api_data_source.php, lib/api_graph.php). Both call sites now look up the prefetched maps in-memory inside their loops. While consolidating the lookups, fixes a latent bug the old process_poller_output() code had in its non-templated fallback branch: a query referencing gti.task_item_id without joining graph_templates_item, which throws an unknown-column error whenever that path is hit. The batched query always joins correctly. Integration test proves both correctness (against a real SQLite-backed schema) and the query-count reduction: 5 local_data_ids that would previously cost up to 10 queries for these two lookups now cost exactly 3, regardless of how many ids are in the batch. Closes #7534 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * refactor(boost): consolidate poller_output_boost inserts into one helper The INSERT INTO poller_output_boost (...) VALUES ... statement was hand-written four times: three in cmd.php (behind the same boost_redirect == 'on' && boost_rrd_update_enable == 'on' guard) using INSERT IGNORE, and once in boost_poller_on_demand() (lib/boost.php) using ON DUPLICATE KEY UPDATE. Beyond the duplication, that was a real data-integrity inconsistency between two paths writing the same shape of data: silently-drop vs last-write-wins on the same key. Since boost_redirect gates cmd.php's writes and boost_poller_on_demand() returns early when boost_redirect is on, the two paths never run concurrently for the same install; they're alternate write paths selected by that one setting, not competing writers. A genuine collision on (local_data_id, rrd_name, time) in either path represents a retry of an already-collected sample -- a batch resent after a partial failure, or two writers racing on the same rounded second -- not two independently meaningful readings, since RRD only stores one value per timestamp regardless. Standardizes on INSERT IGNORE (first write wins), matching the sibling poller_output insert this table shadows, which has always used INSERT IGNORE, and 3 of the 4 existing call sites. Extracts boost_flush_output_batch() (lib/boost.php) as the one shared helper, handling both the batched INSERT and the max_allowed_packet-safe chunking cmd.php's three sites previously didn't do at all (they only chunked by item count). Integration test exercises a genuine duplicate-key collision against a real (SQLite-backed) table and asserts the chosen first-write-wins behavior. Closes #7536 Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(tests): resolve cross-branch test interactions found by full-suite verification Two test-only bugs surfaced only when all four fix branches for the poller_item/Boost hardening pass ran together, not when each ran in isolation: - BoostArchiveHandoffIntegrationTest extracted the archive-forward INSERT IGNORE by searching lib/boost.php for the first occurrence of "INSERT IGNORE INTO poller_output_boost". The Boost SQL-optimization branch adds a second, earlier occurrence (the new boost_flush_output_batch() helper's single-line prefix), so the naive search now grabbed the wrong snippet. Anchor on the multi-line "...poller_output_boost\n\tSELECT *" shape that only the archive-forward statement has. - PollerItemRemotePurgeHandOffTest calls push_out_host() (via api_data_source_change_host()), which needs CACTI_PATH_LIBRARY. The test never requires include/global.php (which normally defines it), and was relying on some other file in a full multi-thousand-file suite run having defined it first as a process-wide side effect. Define it directly instead of depending on unguaranteed load order. No production code changed. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * style(poller): satisfy php-cs-fixer strict_param in the script-server count Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): order the single-archive-table flush by the table's PK columns Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(boost): key the max_allowed_packet cache per connection Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(tests): restore the default db_* session after swapping it for a fake handle Left swapped, the fake handle answered every later read_config_option() in the run; on PHP 8.4 the extra deprecation traffic routed through CactiErrorHandler turned that into an uncaught PDOException and killed the pest process. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: TheWitness <thewitness@cacti.net> |
||
|
Thomas VincentandTheWitness
|
66c00774c6 |
feat: batch the graph-create task_item_id wiring (#7380) (#7383)
* feat(template): batch the graph-create task_item_id wiring (#7380) Pre-fetch the graph-item and data-source id maps in bulk and extract the wiring into graph_template_connect_task_items(), replacing the per-item SELECT pair plus UPDATE with one UPDATE per item. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * perf(template): build task_item_id IN() via db_in_clause Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(7380): assert bulk rrd fetch via db_in_clause marker Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix: order bulk graph-item prefetch by id Add ORDER BY id to the new-item prefetch queries so the first-match per key is deterministic and matches the previous db_fetch_cell() behavior. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(7380): add integration coverage for real ORDER BY dedupe behavior Exercises graph_template_connect_task_items() from lib/template.php directly (not an eval'd extract) against a real SQLite-backed PDO connection, proving the ORDER BY id clauses -- not insertion order -- decide which row wins the first-match dedupe. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * docs: add changelog entry * fix(template): revert local_data_id IN() to implode idiom TheWitness asked to revert the bulk rrd prefetch to the previous implode/intval style used elsewhere in this file instead of db_in_clause(); update the matching test assertion accordingly. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * style(template): hoist bulk graph-create fetches out of the foreach headers Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: TheWitness <thewitness@cacti.net> |
||
|
Thomas Vincent
|
2d623c66f8 |
Parameterise and harden aggregate_graphs.php (#7261)
* security(aggregate): parameterise form_actions and purge_old_graphs IN() lists
The template-migration queries in form_actions() and the purge_old_graphs()
DELETE queries built IN() lists with implode(','); they now use db_*_prepared()
with placeholder arrays. aggregate_items() rfilter and aggregate_graph_id
parameterisation already landed on develop via the #7254/#7260 filter refactor,
so this keeps only the remaining unique fixes plus the gfrv()/rawurlencode() URL
hardening.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test: use whitespace-tolerant regex for grv/gfrv assertion
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test(security): verify purge_old_graphs' parameterized deletes against real SQL
tests/Unit/InClausePlaceholderTest.php only checks the placeholder-count
arithmetic; it never runs a prepared statement, so it can't catch a
mismatched id list feeding the wrong query. This extracts the real
purge_old_graphs() from aggregate_graphs.php and runs it against a
sqlite-backed connection across five joined tables.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* ci: refresh SQL interpolation ratchet baseline for aggregate_graphs parameterization
Rebasing onto develop pulled in prior parameterization work (#7262,
with; aggregate_graphs.php now drops out entirely after this branch's
own db_* parameterization.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* test(aggregate): restore db globals after purge_old_graphs integration test
issue7261_wire_default_connection() overwrote $database_hostname/
$database_port/$database_default and added a fake PDO to
$database_sessions without ever resetting them, so any later test in the
same Pest process that hit a db_* helper resolved against the stale
sqlite handle instead of its own fixture -- which crashed the whole
suite before the HandOff tests ran.
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
* docs: add changelog entry
* docs: fix changelog entry format
---------
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com>
|
||
|
Thomas Vincent
|
12ac06d99c |
ci: refresh SQL interpolation ratchet baseline for develop (#7501)
Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> |
||
|
Thomas VincentandTheWitness
|
37fe6eab9d |
Gate new unparameterised SQL value interpolation (#7260)
* test(security): add SQL interpolation classifier and ratchet baseline Tokenizes each db_fetch/db_execute call and classifies its SQL argument as prepared, static, qstr-escaped, opaque, identifier or value. The value bucket is the migration target: a raw variable interpolated into value position. The baseline records the current per-file value count so CI can block new ones. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * ci(security): gate new SQL value interpolation in quality workflow Runs the interpolation audit in --check mode so a pull request that adds an unparameterised value interpolation fails until it uses db_*_prepared(). Pins setup-php to a commit SHA. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * refactor(security): split prebuilt SQL fragments from scalar interpolation Cacti names query fragments consistently (sql_where, sql_join, ...). A variable with such a name is a SQL snippet, not a bindable scalar, so it now lands in a separate fragment category instead of inflating the value count. Extracts scan_source() and a library-only include guard so the classifier can be unit tested. Baseline value count drops from 373 to 300. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * test(security): cover the SQL interpolation classifier Unit tests exercise each category (prepared, static, qstr, value, identifier, fragment, opaque) against snippets, including IF EXISTS identifiers and mixed scalar/fragment statements. An integration test runs the CLI against the tree and asserts the committed baseline stays in sync. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix(tooling): gate chdir to CLI runs and align checkout version The audit is require'd by the unit tests in library mode; running chdir() at load time leaked a working-directory change into other tests. Gate it behind the library sentinel. Also aligns the ratchet job's checkout to the version the rest of the workflow uses. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * style(tooling): use PHP 8 string helpers and short array syntax in audit Switches the classifier to str_starts_with/str_ends_with/str_contains and short array and list syntax, matching the idiom the rest of the codebase uses. Output is unchanged. Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> * fix installer failure handling * style(lib): align installer assignment per cs-fixer Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> --------- Signed-off-by: Thomas Vincent <thomasvincent@gmail.com> Co-authored-by: TheWitness <thewitness@cacti.net> |