How can I apply filters in my class that extends Walker_Nav_Menu?

apply_filters() applies a set of registered (with add_filter()) callbacks.

When you apply a filter make sure its name is not one of the reserved ones, the_title looks dangerously “collissible” with other filters.

Your filter application is almost correct, but the ‘modifyTitle’ bit is not required…

// apply the filter
apply_filters( 'wpse31787_the_title', $item->ID );

…instead you have to register a filter for wpse31787_the_title during your initialization

function __construct() {
    add_filter( 'wpse31787_the_title', array($this, 'modifyTitle') );
}

And your function definition would be

function modifyTitle( $item_id ) {
    // ...your code goes here
}

If you’re going to pass more than 1 parameter you will have to extend your filter addition like so:

// 2 arguments are expected
add_filter(  'wpse31787_the_title', array($this, 'modifyTitle'), null, 2 );

So, if I put the all the bits together I would have something like this:

class New_Walker_Nav_Menu extends Walker_Nav_Menu {
    function __construct() {
        add_filter(  'wpse31787_the_title', array($this, 'modifyTitle'), null, 2 );
    }

    function start_el( &$output, $item, $depth, $args ) {
        $output .= apply_filters( 'wpse31787_the_title', $item->title );
    }

    function modifyTitle( $title ){
        return $title . ' << ';
    }
}