Linux – Manage oVirt environment

glusterfslinuxmaintenanceovirtvirtualization

I'm starting to deploy oVirt in one of my works, and I have a lot of questions about how to manage everything when you start using it.

First I'll describe my setup.
I have three machines, two ProLiant DL360e Gen8 servers and one common PC. All of them have 1TB of storage available for the OS, and have the same partition scheme. Following this tutorial and it's second part, I've managed to setup the three machines as follows:

Clusterized storage with gluster and CTDB exporting four volumes: engine, isos, data and meta.
Meta is used to maintain coherence among the cluster, engine is used to store oVirt's hosted engine data, data is used to store VM's images and isos to store iso images for OS vm provisioning.

Hosted engine has been set up in both ProLiant servers.

What I was looking for with this setup is to be able to keep on running VM's even whith one of the ProLiants being down. So I had to set up a third machine for storage, that way, I would not lose quorum in the exported gluster volumes if one of the ProLiants goes down.

Everything seems to be working fine. So let me ask a few questions:

Wich is the recommended way to make backups of the hosted engine and VM's, so I can recover from disaster??
I have four NIC's in each server and two in the PC. Right now I'm using just one interface in each machine. Wich would be the recommended network setup for this scenario? I'm buying a new managed gigabit switch next week, so I will be able to have VLAN's working with this machines!!
Whis is the recommended way to bring the hosts down for maintenance??? One at a time and all togheter??
All the equipment is connected to a single APC SmartUPS, and I've setted up apcupsd on another host that is network connected with these three machines. Which is the recommended way to bring them all down really quickly in an event of power failure??? (I have around 35 minutes of autonomy with this UPS for the whole setup!)
Reading around, I do not remember where, I remember that someone stated that in oVirt it was recommended to leave the default datacenter and cluster as they were, and start configuring everything in a new datacenter. I remeber trying to do that, but not being able to accomplish it, because the hosts (the ProLiants) where already memebers of the default cluster, so I've started to set up my VM's in the default cluster. Is this wrong??? I do not know if I missed something important with this.

Finally, please excuse my bad english! It's not my native language!

Thanks everyone in advance!!!

Best Answer

engine backup is done using a script aptly called engine-backup. Backing up VMs is trickier, there's a built-in API[1] but it will only be effective if you're using a host OS that is capable of libvirt blockcommit, i.e. latest Fedora or EL7.1. Otherwise, you can backup the VMs the old fashined way, using in-agent backups or stopping them to take a backup, if downtime isn't critical.
I would bond all 4 NICs using mode-4, if the switch can support that, and segregating the network into a management VLAN, VM network, display network and storage network. If you expect storage loads to interfere, maybe break that down into two bonds, one for storage and one for VMs, display and mgmt
One at a time. When you click the "maintenance" button, all VMs from the host will migrate away, so the host can be taken down. Storage connections, during maintenance are also getting shut down.
Whatever the UPS vendor recommends, this has nothing to do with oVirt. Basically, the pre-shutdown procedure would be to take all VMs down first (so the UPS might want to trigger an API script that would issue shutdown on all VMs) and then place all hosts in maintenance, so they can be shutdown cleanly. When all hosts are in maintenance, it is safe to pull the plug
Not wrong and not right either. In the older versions, there were some hard dependencies that could break if you deleted the default stuff. Using the defaults wasn't a problem. Since it's still better to have everything properly named, it is recommended to create your own DC and Clusters. To move a host between clusters, put it in maintenance, edit and change the cluster affinity. Activate the host - and it's in the new cluster.

Welcome to the wonderful world of oVirt :)

[1] http://www.ovirt.org/Features/Backup-Restore_API_Integration

Related Solutions

Amazon AutoScaling and GlusterFS

Is GlusterFS a good solution to share data across all the instances in the Autoscaling group?

Possibly.. The only way you'll get a definitive answer is with your own tests, however. In the past, I've set up a 4 node webserver cluster on Linode instances, using GlusterFS to distribute/share the assets directory of images and so on.
We found 2 main problems with this approach:

GlusterFS is pretty IO intensive, and works really well on hardware with uncontended IO
Occasionally, a Linode server would experience less-than-optimal access to the backend SAN, and IO-Wait time would go up dramatically. When this happened, Gluster would copy more data between the remaining nodes, which caused IO performance to suffer on those nodes in turn. The result of this was that a minor IO blip, caused by suboptimal SAN configuration, or timesharing would mean that the entire webserver cluster would go poot, and the entire shared filesystem might become unavailable.

Purely anecdotal evidence, but I'd not run GlusterFS on a virtual machine with SAN/shared storage ever again.

Does Gluster ensure there is no loss of data ?

It can... In Gluster 3.0, there's a better recognition of "replication pools" where you can define how many copies of the data exists throughout the cluster. Setting a replication level of 2, means that there's 2 copies on the entire cluster.. This effectively halves your storage capacity, but means that you've got greater resilience to node failure.
Importantly, it also means that you have to add more nodes as multiples of the replication level, in this case, pairs of nodes.

What will happen if all the instances in Autoscaling are terminated, will I lose user data ?

If the instances are only using ephemeral instance storage, yes. If they're EBS based, or using mounted EBS instances, then no.

What happens if a user uploads a image and the server processing the request goes down ?

That greatly depends on how your application is designed. I strongly suspect that the user would lose their data (almost certain in a naively architected solution.)

Is there an impact on IO if clients go down ?

See above.. If the client goes down because of backend storage problems, it can easily destroy the performance of the cluster entirely.

Centos – Gluster + ZFS, deadlock during benchmarking: zfs_iput_taskq 100% cpu

ZFS on Linux needs a bit of basic tuning in order to operate well under load. There's a bit of a struggle between the ZFS ARC and the Linux virtual memory subsystem.

For your CentOS systems, try the following:

Create an /etc/modprobe.d/zfs.conf configuration file. This is read during the module load/boot.

Add something like:

options zfs zfs_arc_max=40000000000
options zfs zfs_vdev_max_pending=24

Where zfs_arc_max is roughly 40% of your RAM in bytes (Edit: try zfs_arc_max=1200000000). The compiled-in default for zfs_vdev_max_pending is 8 or 10, depending on version. The value should be high (48) for SSD or low-latency drives. Maybe 12-24 for SAS. Otherwise, leave at default.

You'll want to also have some floor values in /etc/sysctl.conf

vm.swappiness = 10
vm.min_free_kbytes = 512000

Finally, with CentOS, you may want to install tuned and tuned-utils and set your profile to virtual-guest with tuned-adm profile virtual-guest.

Try these and see if the problem persists.

Edit:

Run zfs set xattr=sa storage. Here's why. You may have to wipe the volumes and start again (I'd definitely recommend doing so).

Best Answer

Related Solutions

Amazon AutoScaling and GlusterFS

Centos – Gluster + ZFS, deadlock during benchmarking: zfs_iput_taskq 100% cpu

Related Topic