How to extend a plugin like we do a theme?

There’s no standard method apart from hooks, and it’s up to the developer of the original plugin to add and support these. If the plugin is extendable they will have developer documentation (LearnDash’s is available here). The specifics will depend entirely on the plugin and how they’ve chosen to allow it.