Custom category for posts via XMLRPC

You can do it by sending the 5th parameter to wp.newPost, like this:

$request = xmlrpc_encode_request( "wp.newPost", array( 1, $xmlrpc_username, $xmlrpc_password, $content, $terms) );

Please note the $terms should be in this format: ‘Taxonomy names as keys, array of term IDs as values’, according to the Codex.

FYI, here is another thread related to your question.