I’ve recently came across this problem as well and managed to fix it.
In my case I created the order and added the line items and coupons programatically. When this is done, you calculate the totals and taxes. If there was no coupon to apply I would use: $order->calculate_totals( true ) and when a coupon was applied I did not use the calculate_totals method since $order->apply_coupons recalculates the totals and taxes automatically.
Doing this for percentage based coupons, the calculation went wrong and the total amount did not match the amount I was expecting. And like you, the calculation would be correct when following the normal checkout flow.
So what I did was calculate the totals after the line items were added and before the coupon was applied. This ensures that the taxes and totals were correct before the coupon was applied.
Simplified example:
// Create a new order.
$order = new WC_Order();
$order->set_prices_include_tax( true );
$order_id = $order->save();
// Add line item for order.
$line_item = new WC_Order_Item_Product();
$line_item->set_order_id( $order_id );
$line_item->set_name( 'Test product' );
$line_item->set_quantity( 1 );
$line_item->set_tax_class( 'tax-21' );
$line_item->set_subtotal( 1000 / 1.21 ); // Calc price pre tax: 21% VAT in this case.
$line_item->set_total( 1000 / 1.21 );
$line_item_id = $line_item->save();
// The next line is what fixed the problem:
// Calculate totals and taxes before a coupon is applied to ensure taxes and totals for order are correct.
$order->calculate_totals( true );
// Apply coupon.
$order->apply_coupon( 'MY_COUPON_CODE' );