Using Metadata to Control Virtual Machine Placement

If your system administrator has enabled custom configurations of ESXi hosts in a resource pool that backs a VDC in your organization, you can apply object metadata in the VCENTER domain to virtual machines configured to exploit the special properties of those hosts. The system uses that metadata to place the virtual machines on the appropriate hosts.

You can create virtual machines with workload-specific configurations that require placement on hosts that have certain characteristics. The technical white paper Best Practices for Performance Tuning of Telco and NFV Workloads in vSphere (http://www.vmware.com/files/pdf/techpaper/vmware-tuning-telco-nfv-workloads-vsphere.pdf) provides several examples of virtual machine configurations that require specific host properties.

Because members of vCloud Director organizations do not typically have control over ESXi host configurations, administrators of the vCenter clusters that provide resource pools for organization VDCs must configure the hosts and apply vSphere custom attributes to indicate how they are configured. Organization members apply vCloud Director object metadata based on those custom attributes to virtual machines that require a host with a specific configuration.

Specifying Advanced Virtual Machine Settings with ExtraConfig Elements

OVF ExtraConfig elements provide a flexible way of including key=value pairs in the configuration of a virtual machine. The keys and values are interpreted by the system when the virtual machine is deployed, and can be used specify a variety of virtual machine properties. The XML fragment in Example: Applying ExtraConfig Values and VCENTER Metadata to a Virtual Machine shows how you can include an ExtraConfig element in the VirtualHardwareSection of a Vm.

The vCloud API supports several ExtraConfig key=value pairs that you can use to configure virtual machines for specific types of workloads. You can use OVFtool to create an OVF package that includes virtual machines with these ExtraConfig values.

Permission to upload or download a Vm that includes any of these keys requires one or more of the following rights.

ExtraConfig Keys, Values, and Required Rights
Key	Values	Required Right	Description
any key	any value	vApp: Preserve All ExtraConfig Elements During OVF Import and Export	A user with this right can upload or download an OVF package that contains an unlimited set of ExtraConfig `key`=`value` pairs. The set includes, but is not limited to, the pairs listed in this table.
any key that is a value of the vapp.allowed.extra.config system configuration property	any value	vApp: Allow Matching Extra Config	A user with this right can upload or download an OVF package that contains any of the ExtraConfig `key`=`value` pairs in which the `key` is a value of the vapp.allowed.extra.config system configuration property. See VMware Knowledge Base article https://kb.vmware.com/kb/2148573.
sched.cpu.latencySensitivity	high, normal (default)	vApp: Preserve Latency ExtraConfig Elements During OVF Import and Export	Set to `high` for virtual machines running latency-sensitive workloads.
ethernet`n`.coalescingScheme (where `n` is the number of the virtual NIC in the range 0-9)	enabled, disabled (default)	vApp: Preserve Ethernet-Coalescing ExtraConfig Elements During OVF Import and Export	Set to `enabled` and specify the virtual NICs that must disable interrupt coalescing.
numa.nodeAffinity	`n`, ..., where `n` is a NUMA node number on the host.	vApp: Preserve NUMA Node Affinity ExtraConfig Elements During OVF Import and Export	Constrains the set of NUMA nodes on which a virtual machine's virtual CPU and memory can be scheduled.

For detailed information about the effects of these settings, see Best Practices for Performance Tuning of Telco and NFV Workloads in vSphere (http://www.vmware.com/files/pdf/techpaper/vmware-tuning-telco-nfv-workloads-vsphere.pdf). For information about using the vCloud API to add rights to a role, see Create a Role in Your Organization.

Using vSphere Custom Attributes to Indicate Host Configurations

Administrators of vCenter clusters that provide resource pools for organization VDCs must apply vSphere custom attributes to indicate how ESXi hosts in those clusters have been configured. To apply vSphere custom attributes to a host, you must use the vSphere Client, vSphere API, or vSphere PowerCLI. The vSphere Web Client does not support viewing or modifying vSphere custom attributes. For more information about vSphere custom attributes, see https://pubs.vmware.com/vsphere-60/topic/com.vmware.vsphere.hostclient.doc/GUID-25244732-D473-4857-A471-579257B6D95F.html.

When used for controlling virtual machine placement in vCloud Director, vSphere custom attribute names must have the form system.service.tag, where tag is an arbitrary string that the cluster administrator has chosen. Within this constraint, cluster administrators are free to decide how custom attribute names and values indicate host capabilities. Cluster administrators must provide vCloud Director administrators with the following information, which users need to use when configuring virtual machines that must be placed on specific hosts.

■	Which VDCs are configured with resource pools that includes these hosts.
■	What vSphere custom attributes have been applied to indicate specific host configurations.
■	How custom attributes and their values indicate support for virtual machine ExtraConfig settings. See Example: Applying ExtraConfig Values and VCENTER Metadata to a Virtual Machine.

How vCloud Director Object Metadata Affects Virtual Machine Placement

In addition to using ExtraConfig elements to configure a virtual machine for a workload that requires a specific host configuration, you must apply vCloud Director object metadata in the VCENTER domain to that virtual machine. When this object metadata matches a vSphere custom attribute and value applied to a host by a vSphere administrator, the system deploys the virtual machine to that host.

Changing the value of vCloud Director object metadata in the VCENTER domain does not change the value of any vSphere custom attributes applied to hosts in a resource pool backing your VDC. A virtual machine to which you have applied vCloud Director object metadata in the VCENTER domain cannot be deployed unless the metadata key and value match a vSphere custom attribute name and value on at least one host in a resource pool that backs your VDC. If you change this metadata so that the key and value applied to the virtual machine do not match a vSphere custom attribute name and value on at least one host in a resource pool that backs your VDC, the virtual machine can never be powered on. In this case, the best practice is to delete the virtual machine and recreate it with the correct metadata.

For more information about vCloud Director object metadata and the VCENTER metadata domain, see Working With Object Metadata.

Important

When you apply object metadata in the VCENTER domain to a virtual machine that is also a subject of a vCloud Director affinity rule, make sure that the rule is compatible with the placement requirements specified by the metadata. For example, if your organization VDC has only a single host with a specific vSphere custom attribute name and value and you create two virtual machines with object metadata derived from that name and value, placing those virtual machines in an anti-affinity rule causes a conflict with the placement advice implied by the metadata.

Example: Applying ExtraConfig Values and VCENTER Metadata to a Virtual Machine

This example assumes that a vSphere administrator has configured one or more hosts to support latency-sensitive workloads, as described in Best Practices for Performance Tuning of Telco and NFV Workloads in vSphere (http://www.vmware.com/files/pdf/techpaper/vmware-tuning-telco-nfv-workloads-vsphere.pdf). At least one of the hosts must be in cluster that provides a resource pool that backs a VDC in your organization.

As a vSphere administrator, use the vSphere Client, vSphere API, or vSphere PowerCLI to create vSphere custom attributes and apply them to the ESXi hosts configured to support latency-sensitive workloads,.

■	Define a vSphere custom attribute of type `Host` that has the name `system.service.VCD-HOST-CONFIG`.
■	Define a vSphere custom attribute of type `Vm` that has the name `system.service.VCD-HOST-CONFIG`.
■	Set the value of this field to `VCD-VM-LATENCY-SENSITIVITY-HIGH` on at least one host in a cluster that provides a resource pool that backs a VDC to be used for metadata-based placement.

Use OVFtool to create an OVF package that contains a VirtualHardwareSection with an ExtraConfig setting like this one:.

<?xml version="1.0" encoding="UTF-8"?>
<Vm ...>
   ...
   <ovf:VirtualHardwareSection
      xmlns:vcloud="http://www.vmware.com/vcloud/v1.5"
      ovf:transport="">
      <ovf:Info>Virtual hardware requirements</ovf:Info>
      <vmw:ExtraConfig
         ovf:required="false"
         vmw:key="sched.cpu.latencySensitivity"
         vmw:value="high" />
      ...
   </ovf:VirtualHardwareSection>
   ...
</Vm>

Log in to vCloud Director as a user whose role includes the right vApp: Preserve Latency ExtraConfig Elements During OVF Import and Export and upload the OVF package you created in Step 2.

Log in to the vCloud API as a user whose role includes the right vApp: Allow metadata mapping domain to vCenter and make a request like this one to apply metadata to the Vm you defined in Step 2. Use the name of the custom attribute as the metadata Key, and the value of the custom attribute as the metadata Value.

POST https://vcloud.example.com/api/admin/vApp/vm-200/metadata
Content-Type: application/vnd.vmware.vcloud.metadata+xml
...
<?xml version="1.0" encoding="UTF-8"?>
<Metadata ... >
  <MetadataEntry
      ...
      <Domain
         visibility="READWRITE">VCENTER</Domain>
      <Key>system.service.VCD-HOST-CONFIG</Key>
      <TypedValue
         xsi:type="MetadataStringValue">
         <Value>VCD-VM-LATENCY-SENSITIVITY-HIGH</Value>
      </TypedValue>
   </MetadataEntry>
</Metadata>

Note

Object metadata in the VCENTER domain is not displayed in the vCloud Director Web Console.

Deploy the virtual machine to a VDC that is backed by a resource pool containing one or more of the hosts that were configured in Step 1. The system will place the virtual machine on one of those hosts.

To verify that the VDC you are using is backed by a resource pool containing the necessary host configurations, check with an organization administrator or your system administrator. Your vSphere administrator can provide the vCenter managed object reference of the cluster that contains the hosts, as well as the key and value names of the vSphere custom attributes that were applied to these hosts. Any organization VDC is backed by a Provider VDC that includes one of these resource pools (which are identified in the vCloud API by the value of a MoRef element) can support metadata-based placement of a virtual machine.