Kohana Frontend, WordPress Backend

You could add the Kohana part as a WordPress plugin that filters template_include and returns a local file path to your Kohana controller. Then you all the regular WordPress variables and functions are available in your additional code.

If you want to use your own database layer you’d have to do the rewrite part probably too to find out which URL belongs to which content. And WordPress’ rewrite system is anything but simple. Also, your db layer has to be functional equivalent to WordPress’, because front-end requests are mixed with the back-end. Post previews or the theme customizer are to obvious examples.
Waste of time, error prone and probably not forward compatible.

But, honestly, this will be a mess. WordPress tries to stay compatible with older themes – but there is no guarantee. You have to test and to update two systems now instead of just one. They might become incompatible, for example when WordPress invents new constants or global variables.

WordPress themes are just a bunch of PHP code, no extra template language, and really easy to learn.

My recommendation: Use just one system. Do this right, within the paradigms of that system. The Kohana paradigm is MVC – in your case you would have to drop the M and the C almost completely. 🙂
The WordPress paradigm is the callback (filters and actions). Both can coexists, they overlap in some cases, but I think it is easier to stick to just one.