You could try to install this package, from the VMsvga2 project, inside the OS X Server guest:
VMsvga2_v1.2.5_OS_10.6-10.8.pkg
... and see if it adds some form of 3D acceleration (I'm not sure, but I remember having read something about that).
Probably, also this additional package is needed:
... which should solve window resizing problems after installing the first package.