I was trying to mount a shared folder in an Arch Linux guest OS under VirtualBox and kept receiving the same enigmatic error.
The problem turned out to be, believe it or not, that the shared folder and the mount location had the same name.
[trevor@arch ~]$ sudo mount -t vboxsf Music /home/trevor/Music /sbin/mount.vboxsf: mounting failed with the error: Protocol error
…change the share name in VirtualBox…
[trevor@arch ~]$ sudo mount -t vboxsf Music-Trevor /home/trevor/Music
And done. If there’s a good reason for this, I don’t know it. But that’s what it takes to get it working.