Checked some logs, couldn't really see anything there but there was a lot of pulseaudio on and of so I tried to switch of sound as I don't use it anyway and so far it has worked, only tried maybe 10 minutes, will try more when I have time.
Edit: I have now checked for some longer time, no issues as long as sound is of in the virtual machine.
For me it's ok like that don't need or want the sound anyway