What is the reason and how to avoid the [FIN, ACK] , [RST] and [RST, ACK]

Here is a rough explanation of the concepts.

[ACK] is the acknowledgement that the previously sent data packet was received.

[FIN] is sent by a host when it wants to terminate the connection; the TCP protocol requires both endpoints to send the termination request (i.e. FIN).

So, suppose

• host A sends a data packet to host B
• and then host B wants to close the connection.
• Host B (depending on timing) can respond with [FIN,ACK] indicating that it received the sent packet and wants to close the session.
• Host A should then respond with a [FIN,ACK] indicating that it received the termination request (the ACK part) and that it too will close the connection (the FIN part).

However, if host A wants to close the session after sending the packet, it would only send a [FIN] packet (nothing to acknowledge) but host B would respond with [FIN,ACK] (acknowledges the request and responds with FIN).

Finally, some TCP stacks perform half-duplex termination, meaning that they can send [RST] instead of the usual [FIN,ACK]. This happens when the host actively closes the session without processing all the data that was sent to it. Linux is one operating system which does just this.

You can find a more detailed and comprehensive explanation here.