I think it completely depends....
I think it completely depends on whether you even WANT to make fancy 3D games in the first place -- an issue we appear to have glossed over.
If you want to make menu-driven or tile-based games or have simple sprites (i.e. CIV, Heroes of Might and Magic, SimCity), VB is just fine.
Personally, I think computer gaming has been in a massive decline since the advent of 3D technology anyhow. Games today concentrate on developing these ridiculous 3D graphics engines that may or may not make the game better anyhow.
I can understand the use of 3D in first-person shooters and simulations and such, but they are popping up in all kinds of inappropriate situations where they really do nothing to make the game better, add useless functionality, and -- if you really think about it -- don't even make for as beautiful or as pleasing an environment as wonderful, drawn art.
I myself, when looking to purchase a computer game, just glance at the back of the box. Unless it's a shooter or simulation, 3D graphics has generally indicated a poor game, in my experience. I can think of a few exceptions (Myth/Railroad Tycoon II), but usually 3D graphics don't even necessarily make for a "prettier" ambience.