In short, there is no well defined convention for naming a theme’s directory, and all of the following are valid (among others):
my_wordpress_thememy-wordpress-theme(empirically the most popular option within the ecosystem)MyWordPressThememywordpresstheme(what the default themes use)
Details
The WordPress PHP Coding Standards Handbook states that filenames should be all lowercase and hyphen-separated. As Squish points out, various routines in the WordPress Template Hierarchy rely on this convention in order to auto-load templates for certain situations.
All of that said, you will not break Linux or Apache by using underscores instead of hyphens (though you will break conventionally loaded files in the template hierarchy). Directories don’t seem to have any well-defined conventions, but as a rule of thumb avoid spaces in all of your names. Dots may or may not have undesired effects in directory names, particularly as *NIX conventions traditionally handle directories beginning with a dot as “hidden.”
As the WP_Theme class’s scandir() function makes direct use of PHP’s scandir() followed by is_dir() in order to enumerate the individual themes in a directory rather than testing the naming conventions of entries, it shouldn’t actually matter what your directories are named as WordPress itself largely ignores the directory name for this purpose. You can read the comments on the is_dir() page for some specific caveats. More than anything, ensure that your directory names work in URL-format and are navigable by a web-browser.
The themes packaged with WordPress use a directory naming convention of all lowercase letters and no punctuation, i.e. twentytwelve, twentythirteen, etc.