It is not impossible, but you need to be very careful to prevent conflicts. You can use my Rewrite analyzer plugin to look at your current rules.
The structure you propose, where the custom post type is stored in the URL hash (the part after the #), will not work because this part is not sent to the server. So the server will just see a request for my-category/!, not for my-category/!#/my-post. (Is it possible you switched the # and the !? The custom is to use #!/ as the hash start.)
How to solve this depends on whether you currently have verbose page rules, and this depends on your current permalink structure. Can you edit your question to add this?