First, get the current month and year:
$month = int(current_time('m'));
$year = int(current_time('Y'));
Next, get the posts in this month:
$query = new WP_Query( 'year=" . $year . "&monthnum=' . $month );
If the query returns empty query the month before and repeat this until you have found a non empty month:
while (empty($query)) {
$month = $month-1;
if ($month == 0) { $month = 12; $year=$year-1;}
$query = new WP_Query( 'year=" . $year . "&monthnum=' . $month );
}