Several days ago I rambled on regarding refactoring in relation to source code comments and documentation. To start, here are a couple of links from the C2 Wiki. These pages in turn link to others about related topics:
I'd post some quotes from CodeComplete, but unfortunately I don't have a copy handy. I recall there been very sane advice in that book, which generally is quite similiar to those found in the eXtreme Programming Wiki at the links above.
I think there are a couple of things to think of as logically seperate things.
First: "Source Code is for humans. Machine code is for computers."
Source code comments are stored interspersed throughout the code, wheras documentation covers installation, distributation, where the bits and pieces are, caveats, dependencies, recommended reading, and so forth. Using tools such as Javadoc or Perldoc that generate API documentation don't help with identifying related systems, scripts, design goals, and so on.
The various documentation formatting systems that ESR mentions are find and dandy, but if the information contained therein does not find its way to the person who needs it (the future developer who is maintaining the codebase) that it's pointless. Many months ago I wrote various build scripts that automatically unpacked, patched, configured, compiled, tested, stripped, and packaged various pieces of open software. These scripts were stored with obvious filenames (build.sh, etc) and documented in README files. The location and existence and usage was mentioned and demonstrated to maintainer taking it over... but several weeks later I was sad/mad to hear, that after the time I had put into this the future maintainer had rewritten them from scratch since he didn't know they existed. Is this a problem with poor documentation? Nope, it was there. A problem with insufficient training? Maybe; although the trainee probably should have taken notes.
... See also CodeReviews