The important thing to note is that action and filter hooks are mostly just standard PHP callback behaviour.
The second argument passed to add_filter() should be a PHP Callable. When the filter is applied with apply_filters(), that callback is just run through the core PHP function call_user_func_array(). Any additional arguments to apply_filters() are then passed as $args to your callback.
The WordPress specific behaviour is around defining the number of arguments your callback accepts. When you add a filter with add_filter(), the 4th argument defines the number of arguments that the callback accepts. WordPress then only passes that number of arguments when it calls call_user_func_array().
Normally passing more arguments than a callback accepts is perfectly fine. The exception is built-in PHP functions, which will throw an error. For example, this will throw an error:
call_user_func_array( 'addslashes', array( 'argument one', 'argument two' ) );
To prevent this WordPress makes you declare how many arguments your callback accepts, and trims the number passed to call_user_func_array() accordingly.
There is an old ticket about deprecating this behaviour here, and you can follow the discussion to see why that never happened. The short version is performance and backwards compatibility.