'doctest' in Factor

The doctest module for Python scans your source file for code examples in docstrings, and evaluates them to check against the correct result. The documentation for this module seems to suggest it is a rather non-trivial affair (and looking at the code confirms this). In comparison, implementing this feature in Factor takes a few minutes:

: all-word-help ( -- seq )
    all-words [ word-help ] map [ ] subset ;

: all-examples ( words -- elements )
    [ \ $example swap elements ] map [ empty? not ] subset ;

: example-valid? ( elements -- ? )
    1 tail
    [ 1 head* "\n" join eval>string "\n" ?tail drop ] keep
    peek = ;

: incorrect-examples ( words -- elements )
    all-examples
    [ [ example-valid? ] all? not ] subset ;

all-word-help incorrect-examples [ . ] each

Factor's help is written in a structured markup language where examples are explicitly denoted as such (and in the UI, you can click them to evaluate them). So all you have to do is iterate over all words in the dictionary, extract their documentation, collect $example markup elements, and ensure they do the right thing. Note that $example elements look like so:

{ $example "2 2 + ." "4" }

All but the last element of the array are strings which are joined together with newlines in between to form the expression to be tested, and the last string is the expected output.

Using this tool I actually found some typos in the Factor documentation. I think the idea is great, and while Python makes it easier than something like Java (where you'd have to write a Doclet and use BeanShell, or something), Factor makes it even easier because it has an help system with a markup language, not just docstrings.

I have some other bits of code sitting around for checking consistency of parameter names in documentation, and checking for broken links and markup. One day I'm going to put them together into a single 'lint'-style tool for documentation.