How can I list custom post by custom category?

Using categories isn’t the same as using a custom taxonomy. You should adapt your code to use get_terms and to query for your custom tax:

$categories = get_terms('article');
  foreach($categories as $category) { 
    $args=array(
     'article'=>$category->slug,
     'post_type' => 'journal',
     'post_status' => 'publish',
     'posts_per_page' => -1,
     'caller_get_posts'=> 1);
  }