Add DISK_IDS as part of the VM backup configuration so backups can target a
subset of eligible VM disks. The configuration parser now normalizes and
validates the list against the VM and resets the incremental chain when the
selected disk set changes.
co-authored-by: Pavel Czerny <pczerny@opennebula.io>
(cherry picked from commit 59c2cac474de545eddd642e03e3caac8fb47c1cc)
NAME = test_ds
DS_MAD = virtiofs
TM_MAD = local
TYPE = IMAGE_DS
DISK_TYPE = FILESYSTEM
oneadmin@ubuntu2204-kvm-ssh-7-0-f4a9-0:~$ cat img.txt
NAME="test_fs"
TYPE="FILESYSTEM"
PATH="/var/tmp/sharedfs_test"
PERSISTENT="YES"
PERSISTENT_TYPE="SHAREABLE"
DISK=[
IMAGE_ID="21" ]
Notes:
* Same FS image cannot be used twice in the same VirtualMachine, nor the
same tag twice
* OpenNebula will generate a MOUNT_POINT = /mnt/one-fs-<image_id> and a
MOUNT_TAG = one-tag-<image_id>. Both can be overwritten at DISK level.
* When using Hugepages in the VM that will be used for virtiofs and
memory access will default to shared. Do not use
MEMORY_ACCESS='private'
* If not using hugepages the VM will use memfd as memoryBacking source
if it includes a filesystem.
* OpenNebula will map FS access (UID/GID) to a given user that can be
configured per host, cluster or globally in vmm_exec_kvm.conf. By
default all access is mapped to oneadmin account.
* Datastore monitorization will report 0 free/used/total size. Images,
representing the FS folder, reports size as the result of a du -sb
command.
* Most of the disk operations do not apply to filesystem disks: resize, snapshots,
save-as, backup.
co-authored-by: MarioRobres <mrobres@opennebula.io>
co-authored-by: Marco Mancini <mmancini@opennebula.io>
Signed-off-by: MarioRobres <mrobres@opennebula.io>
Signed-off-by: Marco Mancini <mmancini@opennebula.io>
The following TM's now support storage live migrations:
- File system based (local and shared variants)
- LVM based (pure LVM, and LVM thin)
The LVM TM's does not support migrating a VM with disk snapshots.
co-authored-by: Guillermo Ramos <gramos@opennebula.io>
co-authored-by: josepselga <jselga@opennebula.io>
Author: Guillermo Ramos <gramos@opennebula.io>
This commit reverts the backup_incremental test. Now we check for not supported
configurations:
- qcow2 disks with snapshots
- raw disks in local/shared TM's
- lvm with lvm thin disabled
(cherry picked from commit 5c693d6741)
When a disk with snapshots is resized and a subsequent revert operation
is performed, the following actions are now done:
* DISK/SIZE is updated to reflect the active snapshot size
* Quotas are updated to account the space freed after the resize
operation
(cherry picked from commit 178142105dea30c7deffd1842d0f0ad23d7dc93b)
(cherry picked from commit d2aaf2c8823d4172b1be673f959c4c922266534f)
- Add suport for delete snapshots in the middle of the chain in qcow2
(ALLOW_ORPHANS=NO). Example, snapshot 2 below:
0 <- 1 <- 2 <- 3
- It supports removing the last snapshot and active (blockpull) and
first one (snapshot 0). The first snapshot cannot be deleted in
persistent images because of the underlying link setting.
- Snapshots are deleted by blockcommit. For example to delete 2, 3 is
committed into 2:
0 <- 1 <- 2: (2+3)
A new file 3.current is created in the filesystem to look for the
actual file name supporting snapshot 3.
- qcow2/shared operations snap_delete, snap_revert and cpds
(disk-saveas) has been updated to react to the .current files
- The kvm/deploy action has been updated to resolve links in source file
attributes. This is needed so the <backingStore> of the libvirt domain
does not contain the sysmlinks that changes on the revert/create
operations.
- The kvm/migrate action also includes a minor modification to resolv
the VM disk
- Snapshots with more than 1 child (or their relatives) cannot be deleted to not
break backing chains.
- It has been found that the snap-delete operation in suspend state is
insecure as the guest caches may not be updated accordingly. The
operation is now not allowed in this state.
- The associated state has been removed from oned. It needs to be
removed in CLI/Sunstone and APIs
- The changes are for VM disk-snapshots. A similar update can be made
for images in the datastore/fs snap_delete action. This operation is
not implemented in this commit.
Note: Output of xmllint may or may not include new lines when multiple matches
of an xpath expression occurs. Parsing of xmllint output adds additional new lines
to deal with different versions.
- Adds new configuration attribute MODE: FULL or INCREMENTAL for
BACKUP_CONFIG. FULL backups uses a differen backup image.
- INCREMENTAL backup information is together with the backup image.
Example:
<BACKUP_INCREMENTS>
<INCREMENT>
<DATE><![CDATA[1667770552]]></DATE>
<ID><![CDATA[0]]></ID>
<PARENT_ID><![CDATA[-1]]></PARENT_ID>
<SIZE><![CDATA[172]]></SIZE>
<SOURCE><![CDATA[bb828060]]></SOURCE>
<TYPE><![CDATA[FULL]]></TYPE>
</INCREMENT>
<INCREMENT>
<DATE><![CDATA[1667770604]]></DATE>
<ID><![CDATA[1]]></ID>
<PARENT_ID><![CDATA[0]]></PARENT_ID>
<SIZE><![CDATA[1]]></SIZE>
<SOURCE><![CDATA[ca0de5f6]]></SOURCE>
<TYPE><![CDATA[INCREMENT]]></TYPE>
</INCREMENT>
<INCREMENT>
<DATE><![CDATA[1667770700]]></DATE>
<ID><![CDATA[2]]></ID>
<PARENT_ID><![CDATA[1]]></PARENT_ID>
<SIZE><![CDATA[39]]></SIZE>
<SOURCE><![CDATA[e9897d6a]]></SOURCE>
<TYPE><![CDATA[INCREMENT]]></TYPE>
</INCREMENT>
</BACKUP_INCREMENTS>
This information only appears on incremental backups
- Internal BACKUP_CONFIG data includes information about the current
active backup and the last increment id.
- Backup operation includes a new parameter: reset. This "closes" the
current active incremental chain and creates a new FULL backup.
- Backup drivers store backups with increment index (0 = FULL) e.g.
disk.0.0.
- Incremental backups are only allowed for VMs using all disks in qcow2
format.
- Backup configuration cannot be changed while doing a VM backup.
- Downloader strings includes backup chains <inc_id>:<backup_ref>,...
- Restic downloader has been updated to support backup chains. Disk
images are rebased across increments.
co-authored-by: Frederick Borges <fborges@opennebula.io>
co-authored-by: Neal Hansen <nhansen@opennebula.io>
co-authored-by: Daniel Clavijo Coca <dclavijo@opennebula.io>
co-authored-by: Pavel Czerný <pczerny@opennebula.systems>
BACKUP INTERFACE
=================
* Backups are exposed through a a special Datastore (BACKUP_DS) and
Image (BACKUP) types. These new types can only be used for backup'ing
up VMs. This approach allows to:
- Implement tier based backup policies (backups made on different
locations).
- Leverage access control and quota systems
- Support differnt storage and backup technologies
* Backup interface for the VMs:
- VM configures backups with BACKUP_CONFIG. This attribute can be set
in the VM template or updated with updateconf API call. It can include:
+ BACKUP_VOLATILE: To backup or not volatile disks
+ FS_FREEZE: How the FS is freeze for running VMs (qemu-agent,
suspend or none). When possible backups are crash consistent.
+ KEEP_LAST: keep only a given number of backups.
- Backups are initiated by the one.vm.backup API call that requires
the target Datastore to perform the backup (one-shot). This is
exposed by the onevm backup command.
- Backups can be periodic through scheduled actions.
- Backup configuration is updated with one.vm.updateconf API call.
* Restore interface:
- Restores are initiated by the one.image.restore API call. This is
exposed by oneimage restore command.
- Restore include configurable options for the VM template
+ NO_IP: to not preserve IP addresses (but keep the NICs and network
mapping)
+ NO_NIC: to not preserve network mappings
- Other template attributes:
+ Clean PCI devices, including network configuration in case of TYPE=NIC
attributes. By default it removes SHORT_ADDRESS and leave the "auto"
selection attributes.
+ Clean NUMA_NODE, removes node id and cpu sets. It keeps the NUMA node
- It is possible to restore single files stored in the repository by
using the backup specific URL.
* Sunstone (Ruby version) has been updated to expose this feautres.
BACKUP DRIVERS & IMPLEMENTATION
===============================
* Backup operation is implemented by a combination of 3 driver operations:
- VMM. New (internal oned <-> one_vmm_exec.rb) to orchestrate
backups for RUNNING VMs.
- TM. This commit introduces 2 new operations (and their
corresponding _live variants):
+ pre_backup(_live): Prepares the disks to be back'ed up in the
repository. It is specific to the driver: (i) ceph uses the export
operation; (ii) qcow2/raw uses snapshot-create-as and fs_freeze as
needed.
+ post_backup(_live): Performs cleanning operations, i.e. KVM
snapshots or tmp dirs.
- DATASTORE. Each backup technology is represented by its
corresponfing driver, that needs to implement:
+ backup: it takes the VM disks in file (qcow2) format and stores it
the backup repository.
+ restore: it takes a backup image and restores the associated disks
and VM template.
+ monitor: to gather available space in the repository
+ rm: to remove existing backups
+ stat: to return the "restored" size of a disk stored in a backup
+ downloader pseudo-URL handler: in the form
<backup_proto>://<driver_snapshot_id>/<disk filename>
BACKUP MANAGEMENT
=================
Backup actions may potentially take some time, leaving some vmm_exec threads in
use for a long time, stucking other vmm operations. Backups are planned
by the scheduler through the sched action interface.
Two attributes has been added to sched.conf:
* MAX_BACKUPS max active backup operations in the cloud. No more
backups will be started beyond this limit.
* MAX_BACKUPS_HOST max number of backups per host
* Fix onevm CLI to properly show and manage schedule actions. --schedule
supports now, as well as relative times +<seconds_from_stime>
onvm backup --schedule now -d 100 63
* Backup is added as VM_ADMIN_ACTIONS in oned.conf. Regular users needs
to use the batch interface or request specific permissions
Internal restructure of Scheduler:
- All sched_actions interface is now in SchedActionsXML class and files.
This class uses references to VM XML, and MUST be used in the same
lifetime scope.
- XMLRPC API calls for sched actions has been moved to ScheduledActionXML.cc as
static functions.
- VirtualMachineActionPool includes counters for active backups (total
and per host).
SUPPORTED PLATFORMS
====================
* hypervisor: KVM
* TM: qcow2/shared/ssh, ceph
* backup: restic, rsync
Notes on Ceph
* Ceph backups are performed in the following steps:
1. A snapshot of each disk is taken (group snapshots cannot be used as
it seems we cannot export the disks afterwards)
2. Disks are export to a file
3. File is converted to qcow2 format
4. Disk files are upload to the backup repo
TODO:
* Confirm crash consistent snapshots cannot be used in Ceph
TODO:
* Check if using VM dir instead of full path is better to accomodate
DS migrations i.e.:
- Current path: /var/lib/one/datastores/100/53/backup/disk.0
- Proposal: 53/backup/disk.0
RESTIC DRIVER
=============
Developed together with this feature is part of the EE edtion.
* It supports the SFTP protocol, the following attributes are
supported:
- RESTIC_SFTP_SERVER
- RESTIC_SFTP_USER: only if different from oneadmin
- RESTIC_PASSWORD
- RESTIC_IONICE: Run restic under a given ionice priority (class 2)
- RESTIC_NICE: Run restic under a given nice
- RESTIC_BWLIMIT: Limit restic upload/download BW
- RESTIC_COMPRESSION: Restic 0.14 implements compression (three modes:
off, auto, max). This requires repositories version 2. By default,
auto is used (average compression without to much CPU usage)
- RESTIC_CONNECTIONS: Sets the number of concurrent connections to a
backend (5 by default). For high-latency backends this number can be
increased.
* downloader URL: restic://<datastore_id>/<snapshot_id>/<file_name>
snapshot_id is the restic snapshot hash. To recover single disk images
from a backup. This URLs support:
- RESTIC_CONNECTIONS
- RESTIC_BWLIMIT
- RESTIC_IONICE
- RESTIC_NICE
These options needs to be defined in the associated datastore.
RSYNC DRIVER
=============
A rsync driver is included as part of the CE distribution. It uses the
rsync tool to store backups in a remote server through SSH:
* The following attributes are supported to configure the backup
datastore:
- RSYNC_HOST
- RSYNC_USER
- RSYNC_ARGS: Arguments to perform the rsync operatin (-aS by default)
* downloader URL: rsync://<ds_id>/<vmid>/<hash>/<file> can be used to recover
single files from an existing backup. (RSYNC_HOST and RSYN_USER needs
to be set in ds_id
EMULATOR_CPUS
=============
This commit includes a non related backup feature:
* Add EMULATOR_CPUS (KVM). This host (or cluster attribute) defines the
CPU IDs where the emulator threads will be pinned. If this value is
not defined the allocated CPU wll be used when using a PIN policy.
(cherry picked from commit a9e6a8e000e9a5a2f56f80ce622ad9ffc9fa032b)
F OpenNebula/one#5516: adding rsync backup driver
(cherry picked from commit fb52edf5d009dc02b071063afb97c6519b9e8305)
F OpenNebula/one#5516: update install.sh, add vmid to source, some polish
Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 6fc6f8a67e435f7f92d5c40fdc3d1c825ab5581d)
F OpenNebula/one#5516: cleanup
Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 12f4333b833f23098142cd4762eb9e6c505e1340)
F OpenNebula/one#5516: update downloader, default args, size check
Signed-off-by: Neal Hansen <nhansen@opennebula.io>
(cherry picked from commit 510124ef2780a4e2e8c3d128c9a42945be38a305)
LL
(cherry picked from commit d4fcd134dc293f2b862086936db4d552792539fa)
support of SSH mode for a Ceph datastore.
Author: Vlastimil Holer <vholer@opennebula.org>
Author: juanmont <jjmontiel@opennebula.org>
(cherry picked from commit 8ab9c02102e2e7f72c4835aa819bab49c229ef45)
Author: MarioRobres