Great question. There is no generic and comprehensive documentation workflow in WordPress circles. However there are some nice tools and options to make use of.
readme.txt
Markdown-based machine-readable file. It’s main purpose is to be parsed and presented by official plugin repository, but it’s not too convenient outside of that.
It is great and fitting place for initial overview, but not so much for advanced instructions.
Contextual help
Contextual help in administration panels had just recently (relatively) started to become useful. Great place for UI instructions, links to local and remote resources and maybe short FAQ.
Major issue that being new and underused thing it probably won’t occur to users to look there for information. Easy workaround is to trigger it open conditionally – for example on first time or until user performs required operations (setting up configuration, etc).
PHPDoc
Inline documentation is essential for developers. Makes it easy to learn or maintain code. Very useful in IDEs.
A little less obvious aspect is that with some effort inline documentation can be used as source to generate more human-friendly docs. Typical example are cross-references, but there are a lot of less common possibilities to explore like generating wiki pages (shameless plug).
Non-WP-specific tools
Other than what WP and PHP offer there is obviously ton of other things you can do, depending on your and your clients’ needs.
Typical out-of-WP solutions used are wikis (or just sites with docs), issue trackers, hosted community support services, etc.
Also there had been some exploration into mixing extensive documentation in WP itself. In simplest case this can be just including rich readme.html with plugin. But there are also more complex possibilities, like using of custom post types for docs.