Registering different Post Formats for Blog Post and CPT

I found an alternative approach.

There is a unique class in body tag for each post type. e.g for portfolio post type i can use the CSS code something like mentioned below to hide the extra option.

.post-type-portfolio #post-format-audio, /* for radio button itself */
.post-type-portfolio .post-format-audio /* for option label */
{
    display: none;
}

Hope it helps anybody else.