There is no difference in terms of file size between single and double quotes.
Generally speaking, you should use whatever quotes are called for in the coding standard defined by the project you’re using. If no coding standard is being used, at least be consistent with your usage.
WordPress specifies the following guidelines for using quotes in the CSS coding standards:
Selectors
Attribute selectors should use double quotes around values
Values
Use double quotes rather than single quotes, and only when needed,
such as when a font name has a space or for the values of the content
property.
I’d suggest following the WP coding standards if your project doesn’t already have a standard defined.