can I add a custom format to the format option in the text panel?

The “classic” TinyMCE editor has two dropdowns: formatselect for paragraph styles and styleselect for character styles – which can also contain paragraph styles, to make it more confusing. The configuration in WordPress by default only shows the format dropdown. If you apply a custom stylesheet to the editor, TinyMCE can use it to pick up the classnames and add them to the style dropdown – but this did not work every time for me.

Since 3.0 you can call add_editor_style() in your functions.php to add a stylesheet to the editor. By default it’s editor-style.css in your theme directory. Before 3.0 you have to hook into the mce_css filter to add the URL to your editor stylesheet. This will end up in the content_css TinyMCE configuration value.

To add the style dropdown, the styleselect option must appear in one of the button bar configuration arrays (theme_advanced_buttons[1-4] in TinyMCE, filtered by mce_buttons_[1-4] in WordPress). The list of block formats is controlled by the theme_advanced_blockformats option of TinyMCE, which you can add to the control array in the tiny_mce_before_init filter. If you want to customize the names of the style dropdown (not just your CSS class names), look at the theme_advanced_styles option. You can also use the more advanced style_formats option which gives you more flexibility to define the styles.

The relevant PHP code with all the hooks and default configuration is in wp-admin/includes/post.php, in function wp_tiny_mce(). All together, your setup could look like this:

add_action( 'after_setup_theme', 'wpse3882_after_setup_theme' );
function wpse3882_after_setup_theme()
{
    add_editor_style();
}

add_filter('mce_buttons_2', 'wpse3882_mce_buttons_2');
function wpse3882_mce_buttons_2($buttons)
{
    array_unshift($buttons, 'styleselect');
    return $buttons;
}

add_filter('tiny_mce_before_init', 'wpse3882_tiny_mce_before_init');
function wpse3882_tiny_mce_before_init($settings)
{
    $settings['theme_advanced_blockformats'] = 'p,h1,h2,h3,h4';

    // From http://tinymce.moxiecode.com/examples/example_24.php
    $style_formats = array(
        array('title' => 'Bold text', 'inline' => 'b'),
        array('title' => 'Red text', 'inline' => 'span', 'styles' => array('color' => '#ff0000')),
        array('title' => 'Red header', 'block' => 'h1', 'styles' => array('color' => '#ff0000')),
        array('title' => 'Example 1', 'inline' => 'span', 'classes' => 'example1'),
        array('title' => 'Example 2', 'inline' => 'span', 'classes' => 'example2'),
        array('title' => 'Table styles'),
        array('title' => 'Table row 1', 'selector' => 'tr', 'classes' => 'tablerow1'),
    );
    // Before 3.1 you needed a special trick to send this array to the configuration.
    // See this post history for previous versions.
    $settings['style_formats'] = json_encode( $style_formats );

    return $settings;
}