Actions or filters fired when data is saved in a custom table

The plugin probably (or: hopefully) used the $wpdb object, an instance of the wpdb-class, to add the custom table. If so (and only if so), then its author probably also uses $wpdb->insert() to add rows/ entries to the table.

Taking a look at wpdb::insert(), and looking at its source, you will find out that this method just is a convenience wrapper around wpdb::_insert_replace_helper(), which sets the $type argument to INSERT. While this method does NOT have any filter, the source reveals that this function actually uses

return $this->query( $this->prepare( $sql, $values ) );

While wpdb::prepare() does not allow any modification via filters, let’s take a look at wpdb::query(). There you will find the following filter:

$query = apply_filters( 'query', $query );

Now that we have an entry point, where we can add modifications, let’s talk about how to identify the custom table that is filled with data. Looking at the SQL statement that gets shoveled into the statement for execution, you will notice the following line:

$sql = "$type INTO `$table` ($fields) VALUES ($formats)";

The $table variable comes from the function definition, which is the first argument of the $wpdb->insert() call. Normally any sane developer would use "{$wpdb->prefix}custom_table_name" there, to avoid a broken plugin if the table prefixed was changed away from wp_ inside wp-config.php.

Finally:

<?
/* Plugin Name: Do something when a custom table gets data added or altered */
add_filter( 'query', function( $query ) {
    // Return SQL unmodified. Not the table we are targeting.
    if ( false === strstr( $query, "{$wpdb->prefix}custom_table_name" ) ) {
        return $query;
    }

    // Return SQL unmodified. Not the actions we are targeting.
    # @TODO If you only want to act on either of them, modify the if-clause
    if ( 
        false === strstr( $query, "INSERT" ) 
        AND false === strstr( $query, "REPLACE" )
    ) {
        return $query;
    }

    # @TODO Do your custom task here and…
    # …modify according the query to your likings.

    return $query;
} );

Keep in mind that this plugin runs on each single query now. You might want to add checks like is_admin() and wrap the whole block of code in a function that is attached to a more exclusive hook to reduce the number of executions for it. Else you might slow down your site significantly.