Restrict category access to specific users/groups. Author always has access

You can assign meta values to users and can add a conditional statement in template to check if the user is having the permission to access the file, if not then redirect them to some other page.