How to solve minify js and css first time load delay?

In the end, I used speed booster plugin to defer javascript and inline CSS. For minifying, I manually minify the .js script and .css files before hand. This helped to improve my pagespeed score without going through first load issue.

Like Tom J Nowell suggested, do not use brute force hammer approach by using plugins like autoptimize which actually made the task more complex.