Add DISK_IDS as part of the VM backup configuration so backups can target a
subset of eligible VM disks. The configuration parser now normalizes and
validates the list against the VM and resets the incremental chain when the
selected disk set changes.
co-authored-by: Pavel Czerny <pczerny@opennebula.io>
(cherry picked from commit 59c2cac474de545eddd642e03e3caac8fb47c1cc)
Add support for per-VM migration flags (--auto-converge and --compressed)
controlled by VM template attributes:
- MIGRATE_AUTO_CONVERGE: Initial CPU throttling rate/increment (ranges: 0-100).
Format: <initial>[,<increment>]
- MIGRATE_COMPRESSED: Enable migration compression (YES/NO). Default: NO
This commit also adds a FEATURES parser and use it during VM creation and updateconf.
The parser normalizes boolean FEATURES values, validates GIC and MIGRATE_AUTO_CONVERG.
Since FEATURES are normalized in oned, the commit removes some redundant checks.
updateconf allow updates GIC, RAS, MIGRATE_* options
See also https://libvirt.org/manpages/virsh.html#migrate
Author: Guillermo Ramos <gramos@opennebula.io>
co-authored-by: Guillermo Ramos <gramos@opennebula.io>
co-authored-by: Ruben S. Montero <rsmontero@opennebula.org>
Signed-off-by: Guillermo Ramos <gramos@opennebula.io>
NAME = test_ds
DS_MAD = virtiofs
TM_MAD = local
TYPE = IMAGE_DS
DISK_TYPE = FILESYSTEM
oneadmin@ubuntu2204-kvm-ssh-7-0-f4a9-0:~$ cat img.txt
NAME="test_fs"
TYPE="FILESYSTEM"
PATH="/var/tmp/sharedfs_test"
PERSISTENT="YES"
PERSISTENT_TYPE="SHAREABLE"
DISK=[
IMAGE_ID="21" ]
Notes:
* Same FS image cannot be used twice in the same VirtualMachine, nor the
same tag twice
* OpenNebula will generate a MOUNT_POINT = /mnt/one-fs-<image_id> and a
MOUNT_TAG = one-tag-<image_id>. Both can be overwritten at DISK level.
* When using Hugepages in the VM that will be used for virtiofs and
memory access will default to shared. Do not use
MEMORY_ACCESS='private'
* If not using hugepages the VM will use memfd as memoryBacking source
if it includes a filesystem.
* OpenNebula will map FS access (UID/GID) to a given user that can be
configured per host, cluster or globally in vmm_exec_kvm.conf. By
default all access is mapped to oneadmin account.
* Datastore monitorization will report 0 free/used/total size. Images,
representing the FS folder, reports size as the result of a du -sb
command.
* Most of the disk operations do not apply to filesystem disks: resize, snapshots,
save-as, backup.
co-authored-by: MarioRobres <mrobres@opennebula.io>
co-authored-by: Marco Mancini <mmancini@opennebula.io>
Signed-off-by: MarioRobres <mrobres@opennebula.io>
Signed-off-by: Marco Mancini <mmancini@opennebula.io>
The following TM's now support storage live migrations:
- File system based (local and shared variants)
- LVM based (pure LVM, and LVM thin)
The LVM TM's does not support migrating a VM with disk snapshots.
co-authored-by: Guillermo Ramos <gramos@opennebula.io>
co-authored-by: josepselga <jselga@opennebula.io>
Author: Guillermo Ramos <gramos@opennebula.io>
* Support for "empty" NUMA nodes. Each GB200 GPU requires eight dedicated NUMA
nodes for MIG support. Nodes cannot be shared with other resources (i.e.
memory, CPU or other GPUs)
* FEATURES. New features attributes:
- GIC = General Interrupt Controller, aarch64 architectures uses GIC
through APIC interface
- RAS = Report host memory errors to a guest using ACPI and guest external abort exceptions
- PCIHOLE64 defines MMIO space for GPU support, GB200 needs aprox 4T
(4294967296 KiB)
* New PCIe topology for dedicated devices. It also removes the PCI
switches. Each numa node includes a pci-expander-bus with 8 pci-root
ports.
* PCI. New attributes for PCI passthrough devices:
- ACPI_NODESET = <libvirt nodeset> NUMA nodes for ACPI
- ROOT = <dedicated|shared> to attach the PCI to its own expander bus
or the common one
- IOMMUFD= <Yes|No> enables iommufd in the hostdev driver to propagate DMA
mappings to the kernel for VM-assigned host device.
- IOMMU = <Yes|No> add a dedicated iommu device.
* IOMMU. New attribute to add a iommu device, includes:
- MODE = <single|device> Add a single iommu device or one for each PCI
passthrough device that request it
- MODEL = <smmuv3|intel|amd|virtio> for the iommu device
- OPTIONS = <whitespace-separated list of options> for the driver
element
* NUMA_NODE. New attribute, uses the node-set syntax with distance
e.g. 0:254,3-15:20,23:10
- DISTANCE to set cross NUMA node distance
co-author: Pavel Czerny <pczerny@opennebula.io>
This commit adds support for memory encryption. It includes:
* A new VM template attribute to set memory encryption
* A host probe to gather the memory extensions available (SEV,
SEV-ES...)
* When selected AUTOMATIC_REQUIREMENTS include memory encryption as a
requirement
* Domain XML generation has been updated to include memory encryption
attributes
* A kvm.conf attribute MEMORY_TUNE_OFFSET to add an additional memory
offset for encryption.
Author: MarioRobres <mrobres@opennebula.io>
Author: Daniel Clavijo Coca <dclavijo@opennebula.io>
This commit adds support for remote execution of commands in a VM through qemu Guest Agent (GA). The feature includes:
* New API calls so create, cancel and retry command execution
* Command results are included in VM information
* Command line tool exposing this functionality
* Monitor probe to reap execution of commands
This commits adds support for generating a NUMA PCIe hierarchy by adding
a pcie-expander-bus in each virtual node. When a PCI passthrough or
SR-IOV device is added is attached to the same virtual node as the
physical server. Changes include:
- Generate VM_* device parameters when the allocation is performed
- Generate CONTEXT PCIi_ADDRESS parameters at deployment time
- Refactor of the HostSharePCI to simplify argument passing.
The hierarchy created is as follows (example 2 NUMA nodes):
+-----------+
| |
| PCIe Root |
| |
+--+-----+--+
| ... |
+-----+ +-----+
NUMA node 0 | | NUMA node 1
+----+-----+ +----+-----+
| expander | | expander |
| bus | | bus |
+----+-----+ +----+-----+
| |
| ...
|
+------> port (0)
| ...
+------> port (3)
|
| +-----------+
| up | PCIe +-----> downstream port (0)
+----+ Switch | ...
| +-----> downstream port (8)
+-----------+
This commits introduces a new framework for OpenNebula, it is based on
standard the OpenNebula middleware access driver (mad) model. Benefits
compared to the current implementation:
* Less overhead on API server and streamlined communications
* Ease integration with third party schedulers
* New "optimize" action to balance the workload of a cluster
* Embedded Plan Manager to apply optimization and deployment plans with
custom parameters for the scheduling window
* Custom optimization policies per cluster
This commits introduces 2 schedulers:
* Rank scheduler based on the match-making algorithm present in 6.x
series
* OpenNebula DRS an optimization algorithm based on an ILP solver for
placement and cluster optimization
Also new API calls has been added for onecluster as well as the CLI
updates to expose them.
co-authored-by: MarioRobres <mrobres@opennebula.io>
co-authored-by: Pavel Czerny <pczerny@opennebula.io>
co-authored-by: Mirko Stojiljkovic <mstojiljkovic@opennebula.io>
co-authored-by: Valentyn Bohdan <vbohdan@opennebula.io>
This feature adds support for VM quotas at cluster level. CLUSTER_IDS
can be set for a VM quota so the admin can limit the number of resources (e.g. VMs) a
user/group can create in a given cluster.
This commit also fixes:
- B #2226
- B #6823
Co-authored-by: Pavel Czerny <pczerny@opennebula.io>
* Remove vm_import table from DB
* Remove imported vms actions
* Fix fsck for image and network
* onedb fsck fix running_vms only for non-backup images
- oned load only last 2 history records (not the full list)
- Dump all history records only if needed in VirtualMachine::to_xml.
- Dump conforms XML schecam and removes VM template from history records.
Speed up of onevm show command:
- for small SQLite DB is for VM with 500 histories: 130 ms down to 5 ms
- for big MySQL DB VM with 687 histories: 1000 ms down to 200 ms
* Refactor release vnc port
* Fix VNC port for recover --recreate in stopped state
* Improve error messages
* Update recover --recreate description
(cherry picked from commit 9ed1fba8b95c1ea5d3984f136e638b973577ad69)
This commit implements the in-place restore of VM backups. Selected VM disks will
be replaced with the specified backup:
* A new API call has been added to the XML-RPC API (`one.vm.restore`) with
the following arguments:
- VM ID to be restored, needs to be in **poweroff** state
- IMAGE ID of the backup to restore
- INCREMENT ID, only for incremental backups, the increment to use
(defults to -1 to use the last increment available)
- DISK ID of the disk to restore (defaults to -1 to restore all VM
disks)
* Datastore drivers needs to implemente a new operation `ls`. This new
operation takes the VM, image information of the backup and datastore
information and returns the restore URL for the disks in the backup.
* This commit includes the implementation for qcow2 and ssh drivers,
ceph will be implemented in a separated PR. The new driver action is
`restore host:vm_dir vm_id img_id inc_id disk_id`
* The restore operation is performed in a new state `PROLOG_RESTORE`
rendered as `RESTORE` and `rest` in short form. State in in RSuntone.
TODO:
- Remove any existing VM snapshot (system/disk) in the VM. Note that
snapshots are not included in a backup.
- Ceph drivers
- JAVA, GO Lang API bindings
- Sunstone interface, new state and new operation. Review new state in
RSuntone.
co-authored-by: Pavel Czerny <pczerny@opennebula.io>
When a disk with snapshots is resized and a subsequent revert operation
is performed, the following actions are now done:
* DISK/SIZE is updated to reflect the active snapshot size
* Quotas are updated to account the space freed after the resize
operation
(cherry picked from commit 178142105dea30c7deffd1842d0f0ad23d7dc93b)
(cherry picked from commit d2aaf2c8823d4172b1be673f959c4c922266534f)
If q35 machine type is detected the slot of the pci device is set to 0
and the bus to pci_id + 1.
Q35 models uses pcie-root-ports to attach PCI devices. Each PCI port is
selected by the bus parameter of the PCI address and it that does not accept a
slot number greater than 0.
Example:
A VM with 2 X710 VFs is defined OpenNebula as:
PCI=[
ADDRESS="0000:44:0a:0",
BUS="44",
CLASS="0200",
DEVICE="154c",
DOMAIN="0000",
FUNCTION="0",
NUMA_NODE="0",
PCI_ID="0",
SHORT_ADDRESS="44:0a.0",
SLOT="0a",
VENDOR="8086",
VM_ADDRESS="01:00.0",
VM_BUS="0x01",
VM_DOMAIN="0x0000",
VM_FUNCTION="0",
VM_SLOT="0000" ]
PCI=[
ADDRESS="0000:44:0a:1",
BUS="44",
CLASS="0200",
DEVICE="154c",
DOMAIN="0000",
FUNCTION="1",
NUMA_NODE="0",
PCI_ID="1",
SHORT_ADDRESS="44:0a.1",
SLOT="0a",
VENDOR="8086",
VM_ADDRESS="02:00.0",
VM_BUS="0x02",
VM_DOMAIN="0x0000",
VM_FUNCTION="0",
VM_SLOT="0000" ]
Each PCI VFs is attached to different pcie-root-port, selected with the
VM_BUS parameter:
00:02.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
The PCI topology is:
-[0000:00]-+-00.0 Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
+-01.0 Cirrus Logic GD 5446
+-02.0-[01]----00.0 Intel Corporation Ethernet Virtual Function 700 Series
+-02.1-[02]----00.0 Intel Corporation Ethernet Virtual Function 700 Series
+-02.2-[03-04]----00.0-[04]--
+-02.3-[05]----00.0 Red Hat, Inc. Virtio network device
+-02.4-[06]----00.0 Red Hat, Inc. Virtio SCSI
+-02.5-[07]----00.0 Red Hat, Inc. QEMU XHCI Host Controller
+-02.6-[08]----00.0 Red Hat, Inc. Virtio console
+-02.7-[09]----00.0 Red Hat, Inc. Virtio memory balloon
+-03.0-[0a]--
+-03.1-[0b]--
+-03.2-[0c]--
+-03.3-[0d]--
+-03.4-[0e]--
+-03.5-[0f]--
+-03.6-[10]--
+-03.7-[11]--
+-1f.0 Intel Corporation 82801IB (ICH9) LPC Interface Controller
+-1f.2 Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
\-1f.3 Intel Corporation 82801I (ICH9 Family) SMBus Controller
* Adds 2 new API calls one.vm.attachpci and one.vm.detachpci
* The operation is only for POWEROFF VMs
* Adds 2 new actions to register the event in the VM history pci-detach/pci-attach
* Ruby OCA bindings
* onevm cli commands
TODO:
- JAVA and GO bindings
- Sunstone implementation
(cherry picked from commit 5c4331a4dd)
- Generic quotas are defined and read from oned.conf (QUOTA_VM_ATTRIBUTE)
- Generic quotas react to the running mode (i.e. account only when the VM is
running). For each quota oned also adds a RUNNIN_* metric
- one.system.config API call has been updated so every user can read
quota configuration:
. Sensitive information is hidden (***)
. Oneadmin group has no longer access to sensitive information
through the API
- CLI has been updated to render generic quotas
- onedb fsck fixes generic quotas reading the configuration from
oned.conf
Other changes in this PR:
- Refactor Quota metrics to use std::vector instead of C array
Squashed commit of the following:
- New methods to Template and Attribute classes to render hidden
attributes as "***"
- Update Quota calls to not include unneeded quotas
co-authored-by:Ruben S. Montero <rsmontero@opennebula.org>
- New attribute VIDEO, that includes
- TYPE: Custom video device (none, vga, cirrus, virtio)
 - IOMMU: Enable IOMMU for the video device
 - ATS: Enable ATS (Address Translation Services) for the video device
 - VRAM: VRAM allocated to the video device (Megabytes)
 - RESOLUTION: Video resolution, in format like: 1280x720 or 1920x1080
- Support for onevm CLI command
- Updated VM xsd file.
- Add suport for delete snapshots in the middle of the chain in qcow2
(ALLOW_ORPHANS=NO). Example, snapshot 2 below:
0 <- 1 <- 2 <- 3
- It supports removing the last snapshot and active (blockpull) and
first one (snapshot 0). The first snapshot cannot be deleted in
persistent images because of the underlying link setting.
- Snapshots are deleted by blockcommit. For example to delete 2, 3 is
committed into 2:
0 <- 1 <- 2: (2+3)
A new file 3.current is created in the filesystem to look for the
actual file name supporting snapshot 3.
- qcow2/shared operations snap_delete, snap_revert and cpds
(disk-saveas) has been updated to react to the .current files
- The kvm/deploy action has been updated to resolve links in source file
attributes. This is needed so the <backingStore> of the libvirt domain
does not contain the sysmlinks that changes on the revert/create
operations.
- The kvm/migrate action also includes a minor modification to resolv
the VM disk
- Snapshots with more than 1 child (or their relatives) cannot be deleted to not
break backing chains.
- It has been found that the snap-delete operation in suspend state is
insecure as the guest caches may not be updated accordingly. The
operation is now not allowed in this state.
- The associated state has been removed from oned. It needs to be
removed in CLI/Sunstone and APIs
- The changes are for VM disk-snapshots. A similar update can be made
for images in the datastore/fs snap_delete action. This operation is
not implemented in this commit.
Note: Output of xmllint may or may not include new lines when multiple matches
of an xpath expression occurs. Parsing of xmllint output adds additional new lines
to deal with different versions.
Includes the following changes:
- xml-schema for Backup Job and Scheduled Actions
- GO, Java api
- Deprecate onevm update-chart, delete-chart
* The commands are replaced by sched-update and sched-delete
* Refactor method deprecate_command, it's still possible to run the
command
* Delete 'shutdown' and 'delete' commands deprecated years ago
* Fix --verbose option for sched-update and sched-delete
- Re-implementation of scheduled actions, now are managed and executed
by oned
- Backup Job objects, API, and CLI commands
- Qemu will terminate VMs with system snapshots and checkpoints, example:
libvirtd[7446]: internal error: qemu unexpectedly closed the monitor: 2022-12-07T16:06:59.603678Z qemu-kvm-one: Device has active dirty bitmaps.
- Also disk snapshots may render inconsistent bitmaps, example:
'virsh --connect qemu:///system checkpoint-delete 45d76e5f-25fa-416a-97fd-1624380d5e02 one-191-0': error: Failed to delete checkpoint one-191-0 error: internal error: unable to execute QEMU command 'query-named-block-nodes': Bitmap '' doesn't satisfy the constraints Error preparing disk files
This commits prevents snapshot operations when increment backups are configured, and vice versa.
co-authored-by: Pavel Czerný <pczerny@opennebula.systems>
co-authored-by: Frederick Borges <fborges@opennebula.io>
co-authored-by: Christian González <cgonzalez@opennebula.io>
* VNET updates trigger a driver action on running VMs with NICs in the
network.
* VNET includes a sets with VM status: updated, outdated, error and
updating. With VMs in each state.
* VNET flags error situations with a new state UPDATE_FAILURE.
* The same procedure is applied when an AR is updated (only VMs in that
AR are updated).
* A new options in the one.vn.recover API call enable to recover or
retry this VM update operations.
* The following attributes can be live-updated per VNET driver:
- PHYDEV (novlan, vlan, ovs driver)
- MTU (vlan, ovs driver)
- VLAN_ID (vlan, ovs driver)
- QINQ_TYPE (ovs driver)
- CVLANS (ovs driver)
- VLAN_TAGGED_ID (ovs driver)
- OUTER_VLAN_ID (ovs driver)
- INBOUND_AVG_BW (SG, ovs driver + KVM)
- INBOUND_PEAK_BW (SG, ovs driver + KVM)
- INBOUND_PEAK_KB (SG, ovs driver + KVM)
- OUTBOUND_AVG_BW (SG, ovs driver + KVM)
- OUTBOUND_PEAK_BW (SG, ovs driver + KVM)
- OUTBOUND_PEAK_KB (SG, ovs driver + KVM)
* New API call one.vm.updatenic, allows to update individual NICs
without the need of detach/attach (only QoS supported).
* Update operations for: 802.1Q, bridge, fw, ovswitch, ovswitch_vxlan
and vxlan network drivers.
* VNET attributes (old values) stored in VNET_UPDATE to allow
implementation of update operations. The attribute is removed after a
successful update.
* Updates to CLI onevnet (--retry option) / onevm (nicupdate command)
* XSD files updated to reflect the new data model
* Ruby and JAVA bindings updated: new VNET state and recover option, new
VM API call.
* Suntone and Fireedge implementation (lease status, recover option, new
states)
TODO: Virtual Functions does not support this functionality
iii
- Adds new configuration attribute MODE: FULL or INCREMENTAL for
BACKUP_CONFIG. FULL backups uses a differen backup image.
- INCREMENTAL backup information is together with the backup image.
Example:
<BACKUP_INCREMENTS>
<INCREMENT>
<DATE><![CDATA[1667770552]]></DATE>
<ID><![CDATA[0]]></ID>
<PARENT_ID><![CDATA[-1]]></PARENT_ID>
<SIZE><![CDATA[172]]></SIZE>
<SOURCE><![CDATA[bb828060]]></SOURCE>
<TYPE><![CDATA[FULL]]></TYPE>
</INCREMENT>
<INCREMENT>
<DATE><![CDATA[1667770604]]></DATE>
<ID><![CDATA[1]]></ID>
<PARENT_ID><![CDATA[0]]></PARENT_ID>
<SIZE><![CDATA[1]]></SIZE>
<SOURCE><![CDATA[ca0de5f6]]></SOURCE>
<TYPE><![CDATA[INCREMENT]]></TYPE>
</INCREMENT>
<INCREMENT>
<DATE><![CDATA[1667770700]]></DATE>
<ID><![CDATA[2]]></ID>
<PARENT_ID><![CDATA[1]]></PARENT_ID>
<SIZE><![CDATA[39]]></SIZE>
<SOURCE><![CDATA[e9897d6a]]></SOURCE>
<TYPE><![CDATA[INCREMENT]]></TYPE>
</INCREMENT>
</BACKUP_INCREMENTS>
This information only appears on incremental backups
- Internal BACKUP_CONFIG data includes information about the current
active backup and the last increment id.
- Backup operation includes a new parameter: reset. This "closes" the
current active incremental chain and creates a new FULL backup.
- Backup drivers store backups with increment index (0 = FULL) e.g.
disk.0.0.
- Incremental backups are only allowed for VMs using all disks in qcow2
format.
- Backup configuration cannot be changed while doing a VM backup.
- Downloader strings includes backup chains <inc_id>:<backup_ref>,...
- Restic downloader has been updated to support backup chains. Disk
images are rebased across increments.
Ruben S. Montero