Convert HTML5/CSS3 site to WordPress Theme (need a lot of plugins) or just use the API?

I don’t really know what you mean by “using the WordPress API in a site as opposed to making it a theme” but I am assuming you mean something like “use the backend but not the front facing stuff”. So, given that, yes you can, but a lot of things will not work. There are too many hooks in the WordPress core– ‘pre_get_posts’ for example– and that are expected in themes– ‘the_content’, for example– that won’t fire. Some of these hooks are so ubiquitous that I’d expect a lot of plugins not to work. WordPress hooks its own functions to some of those so even some of Core won’t likely work. By the time you figure out how to compensate for all of that you may as well have just learned to write a theme. So, honestly, one of these:

  1. Learn to build a WordPress Theme and convert the site, integrate the
    plugins etc
  2. Hand this off to a WordPress Theme Developer 😉