This is what you want:
RewriteEngine On
RewriteCond %{HTTP_HOST} website\.com [NC]
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://website.com/$1 [R,L]
@elgoots, the reason yours was close but still had trouble is your HTTP_HOST has the www in it.