Solution by TJ from launchpad bugs:
Create a new group “vm” and add users that require USB access for VMs to it:
sudo addgroup vm
sudo addgroup $USER vm
Log-out/log-in to effect the new group membership.
Add a udev rule to assign USB devices to the vm group:
# Virtual Machine hypervisor access to USB devices
SUBSYSTEM==”usb”, ENV{DEVTYPE}==”usb_device”, GROUP=”vm”
SUBSYSTEM==”usb_device”, GROUP=”vm”
Save the file as /etc/udev/rules.d/41-vm-usb.rules and reload udevd:
sudo /etc/init.d/udev restart