FreeBSD Manual Pages
VMSTATED(8) System Manager's Manual VMSTATED(8) NAME vmstated -- Manage and run virtual machine in supervised manner SYNOPSIS vmstated [-h] [-c configdir] [-f] [-p pidfile] [-v] DESCRIPTION vmstated is a user-space daemon that manages bhyve virtual machines. It provides hook scripts for easy network and storage setup and maintains records on a virtual machine's state while it is running. It provides a socket file on which utilities like vmstatedctl(1) can send commands and retrieve information about its state and about the virtual ma- chines vmstated is managing. Virtual Machine Configuration vmstated maintains a list of virtual machines, which are defined and con- figured via directories and files under /usr/local/etc/vmstated. It expects one sub-directory per virtual machine. The name of the sub di- rectory may be chosen arbitrarily and does not have any impact on virtual machine configuration, for example /usr/local/etc/vmstated/vm0. The contents of this virtual machine specific sub-directory define the con- figuration of the virtual machine and can also provide user defined hook scripts, which are called by vmstated when the associated virtual machine changes its state. vmstated expects a UCL-formatted configuration file called config within the sub-directory. This file contains the specification of the virtual ma- chine. The following variables can be set for a virtual machine: name Specify the name of the virtual machine configuration. If omitted, vmstated assumes the parent tag name as the virtual machine name. configfile path to the bhyve configuration file that specifies the layout and settings for the virtual machine. scriptpath define an alternate path holding hook scripts instead of the default /usr/local/etc/vmstated/<name>. os Specify the operating system of the virtual machine. osversion Specify the operating system version of the virtual machine. (owner) designate the name of the virtual machine's owner. This user may start and stop the virtual machine using vmstatedctl(1). (group) designate the name of the virtual machine's owning group. Any user in that group may start and stop the virtual machine us- ing vmstatedctl(1). description A brief description outlining the purpose and use of the vir- tual machine. maxrestart The number of times a virtual machine may restart within a set timeframe (see maxrestarttime) before it is put into failure mode. If no value is set, this value is set to 3 by default. maxrestarttime The timeframe in seconds in which the value "maxrestart" is being applied. For example, if maxrestart is set to 3 and maxrestarttime is set to 10, a virtual machine that attempts to restart 4 times within 10 seconds will put into failure mode. If no value is set, this value is set to 30 by default. (bootrom) The path to the bootrom file to use, i.e. /usr/local/share/uefi-firmware/BHYVE_UEFI.fd generate_acpi_tables Equivalent to the bhyve_config option "acpi_tables" vmexit_on_halt Equivalent to the bhyve_config option "x86.vmexit_on_hlt" memory Amount of memory in megabytes to use for the virtual machine cpus The number of VCPUs to assign to the virtual machine sockets The number of VCPU sockets to apply cores The number of VCPU cores to apply Variables in brackets are parsed by vmstated but are not yet applied to virtual machine configurations. They will be integrated in a future ver- sion. Settings in a config file that modify virtual machine configuration are combined with those in the file referenced by "configfile"; vmstated does this by appending the generated configuration to the file specified by "configfile". Generated settings thus override any predefined settings found in the "configfile" file. By default, vmstated takes the filename specified in variable "configfile" and appends ".generated" as suffix to construct the filename for the com- bined configuration file of original bhyve_config file settings and those listed in the config file. Hook Scripts vmstated allows placement of hook scripts within a virtual machine's con- figuration directory. It supports running scripts whenever the virtual ma- chine enters one of the following states: start_network The virtual machine's network is being prepared, i.e. tap interface(s) are being created and added to bridge devices. start_storage The virtual machine's storage is being prepared, i.e. connections to remote storage is established, a volume is snapshotted before the virtual machine starts, etc. running The virtual machine has reached its running state. restarting The virtual machine has begun the process of restarting; this state may be assumed at any state before and after the virtual machine process termi- nates, depending on whether the restart was triggered from within the virtual machine or from outside via a management command sent to vmstated. Hence there is no guarantee that the virtual machine is still running or has shut down when this state is assumed. restarted The virtual machine has completed its restart cycle and is about to reenter into the running state. stopping Similar to the restarting state, the stopping state may be assumed any time during the actual shutdown of a virtual machine process; depending on the origin of the shutdown trigger, this state is either assumed before a TERM signal is sent to the bhyve(8) process or after it has already terminated. stop_storage The virtual machine's storage is being shut down and disconnected; any clean up steps are taken to make sure a later restart can re-connect to the virtual machine's storage again. stop_network The virtual machine's network is being shut down and cleaned up; the steps taken during the start_network phase are reversed to restore the system to its' original state. stopped The virtual machine is now stopped. can be restarted by the user. Hook scripts are executed by vmstated synchronously - it waits for each process to complete before it continues. vmstated expects each hook script to return exit code 0 on successful com- pletion. Any other exit code will lead vmstated to assume the state change failed and move the virtual machine into a failure state. To make vmstated execute a script for a particular state, place an exe- cutable (binary or shell script) with the state's name in the virtual ma- chine's configuration directory. Output Logging vmstated creates one separate log file under /var/log/vmstated for each virtual machine recording the output of bhyve(8) as well as of any hook scripts it calls during state transitions. The log filenames are set to the virtual machine's name and the suffix ".log". The output from hook scripts is also included in each virtual machine's log file. Log files record stdout as well as stderr output. Failure Modes If a virtual machine fails to start correctly, i.e. either due to miscon- figuration, disk issues or other problems making bhyve(8) fail to complete the boot process, vmstated will keep track of the number of times it at- tempted to restart it. If the number of restart attempts exceeds the value of "maxrestart" within the last number of seconds set by value "maxrestart- time", vmstated will enter the virtual machine into failure mode and no longer attempt a restart. Virtual machines are also entered into failure mode when one of its hook scripts indicates a failure by returning a non-zero exit code. To restart a virtual machine that entered into failure mode, you first need to reset its state. You can do so by using vmstatedctl(1) with argument "failreset". OPTIONS -c configdir specify configdir to be used as configuration directory instead of /usr/local/etc/vmstated. -f run vmstated in foreground mode -h print usage summary and exit immediately -v be more verbose when logging FILES * /usr/local/etc/vmstated - default configuration directory path * /var/log/vmstated - default log directory path * /var/run/vmstated.pid - holds PID once vmstated has successfully started * /var/run/vmstated.sock - UNIX socket through which vmstatedctl(1) com- municates with vmstated EXAMPLES To configure a virtual machine called "bsdvm", create a directory /usr/local/etc/vmstated/bsdvm and then create a new configuration /usr/local/etc/vmstated/bsdvm/config with the following contents: bsdvm { configfile = /usr/local/etc/vmstated/bsdvm/bhyve_config; } Place a valid bhyve configuration file under /usr/local/etc/vmstated/bsdvm/bhyve_config. The file can be generated by calling bhyve(8) with the desired options as well as -o config.dump=1 and pipe the output into the bhyve_config file. bhyve -c 2 -m 4G -w -H \ -s 0,hostbridge \ -s 4,ahci-hd,disk.img \ -s 5,virtio-net,tap0 \ -s 29,fbuf,tcp=[::]:5900,w=800,h=600 \ -s 30,xhci,tablet \ -s 31,lpc -l com1,stdio \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ -o config.dump=1 \ bsdvm > /usr/local/etc/vmstated/bsdvm/bhyve_config Once the configuration file has been generated, make sure to remove the line "config.dump=1" from it: sed -i '' s@config.dump=1@@g \ /usr/local/etc/vmstated/bsdvm/bhyve_config To add tap device "tap0" to bridge "bridge0" before the virtual machine starts and to destroy this tap interface when it shuts down, we can create a new script /usr/local/etc/vmstated/bsdvm/start_network with the following contents: #!/bin/sh ifconfig tap0 create ifconfig bridge0 addm tap0 as well as /usr/local/etc/vmstated/bsdvm/stop_network: ifconfig tap0 destroy Make sure to make both scripts executable: chmod 755 /usr/local/etc/vmstated/bsdvm/start_network chmod 755 /usr/local/etc/vmstated/bsdvm/stop_network Finally, you can now use vmstatedctl(1) to start the virtual machine: vmstatedctl start bsdvm You can then check the virtual machine's log file for output and progress information: tail -f /var/log/vmstated/bsdvm.log EXIT STATUS The vmstated utility exits 0 on success, and >0 if an error occurs. It may fail for one of the following reasons: * It failed to read the configuration directory. * It failed to acquire necessary resources. * It wasn't launched with necessary credentials. SEE ALSO bhyve(8), vmstatedctl(1) HISTORY The vmstated command first appeared in FreeBSD 13.2. AUTHORS Christian Moerz <freebsd@ny-central.org> CAVEATS vmstated is work in progress. The programming interface, hook script archi- tecture and the inner workings may change in future releases. While some degree of testing has gone into its development, this is meant as a refer- ence implementation and brain teaser for those involved with bhyve and jails state management. Hence, it will likely change inside and outside due to new insights that were learned from the development process. Expect having to redo hook scripts and configuration files when upgrading to a new version in the fu- ture. At the moment, vmstated must be started as root. As of version 0.07, it is not making use of capsicum(4) to limit its system interaction capabilities. Feature variables like "owner" and "group" can be defined in a config file but are not yet respected by vmstated when authorizing users to communicate with vmstated via its socket. By default this socket is not world-writeable and thus requires root level privileges to communicate through the use of vmstatedctl(1). With its initial release, vmstated does not yet persist its state when shutting down. If you shut down vmstated without first shutting down the virtual machines it started, those bhyve(8) processes will become "or- phaned" and will have to be shut down and cleaned up manually - as no hook scripts will be called. Configuration names are not yet checked for their uniqueness by vmstated. Hence, if you label two virtual machines with the same name, expect unde- fined behavior. Finally, vmstated does not allow to be started multiple times. If another instance of it is already running, it will print an error message and ter- minate. BUGS There are some known issues: * There is no way yet to make vmstated reload its configuration files. If you modify configuration files, you need to restart vmstated for it to re-read settings from the file system. However, unless you first shut down any running virtual machines, you will end up with orphaned bhyve(8) processes. * if a hook script fails to complete within a reasonable amount of time, it may cause vmstated to hang because it waits for the completion of every hook script before it continues processing events and commands. * if vmstated fails to correctly initialize after reading in its configu- ration files and forking; this may lead to situations where vmstated actually fails to start correctly but still returns exit code zero upon daemonizing. There may be additional bugs lurking in the dark. Particularly with * parsing configuration files with syntax errors, * handling out of disk errors * handling out of memory errors * and more... FreeBSD ports 15.quarterly February 16, 2023 VMSTATED(8)
NAME | SYNOPSIS | DESCRIPTION | OPTIONS | FILES | EXAMPLES | EXIT STATUS | SEE ALSO | HISTORY | AUTHORS | CAVEATS | BUGS
Want to link to this manual page? Use this URL:
<https://man.freebsd.org/cgi/man.cgi?query=vmstated&sektion=8&manpath=FreeBSD+Ports+15.1.quarterly>