Saturday, December 22, 2007

Going to Dominican Republic

I will be in the Dominican Republic from December 23rd to January 7th.

I wanted to blog about something I've been working on for the last week and a half, but didn't have time: basically, redefining a word will no longer recompile all words which depend on this word, so that loading a library such as math.ranges, which defines a new sequence type, no longer recompiles ~7000 words which directly or indirectly call length, nth, and so on. This will be a big productivity boost. It is a big change, but I'm trying to do it right, by re-designing the compiler to get rid of some cruft that's been building up for the last few years. When I return, I will write a detailed entry explaining exactly what's going on and what's changing.

I'm bringing my laptop with me, so if I get bored of sitting on the beach (and if the girl lets me) I'll try to do some hacking. However, I don't think I'll have Internet access. As usual, I'm going to leave Daniel Ehrenberg in charge of harvesting patches. If you have anything in your repository, ping him on IRC (he's littledan) so that he can pull. And if you want the latest changes, pull from his repository (

Have a good Christmas and New Year's everybody!

Thursday, December 13, 2007

Factor 0.91 now available

After several months of intense development, Factor 0.91 is available for download. Check out the change log.

Due to lack of interest, OpenBSD and Solaris are not supported anymore. However, to compensate, the Windows CE port has improved, and we have preliminary support for 64-bit Mac OS X Intel (no 64-bit UI yet; that's coming in 0.92.)

Don't forget to subscribe to the Planet Factor Atom feed!

Wednesday, December 12, 2007

Improved help.lint tool

The help.lint tool ensures that code examples in documentation produce the output that they claim to produce, and also checks that stack effects in documentation match the source, as well as finding broken links. I have made it easier to use, and no longer dependent on editor integration. Now, you just run "my-vocab" check-vocab-help, and any problematic words are listed. It is best to run this from the UI, so that the errors are clickable presentations. You can then right-click on any word and choose "Edit" from the menu.

I encourage all contributors to run this tool against their documentation. Just as you should always run unit tests before releasing changes, you should always run this tool too.

Saturday, December 08, 2007

Factor-powered pastebin with syntax highlighting for 100+ languages

I'm proud to announce a major new version of extra/webapps/pastebin. I've deployed the new pastebin on an experimental HTTP server. I hope it doesn't crash by the time you read this. The major new feature is syntax highlighting support.

For the last two weeks or so I've been working on a Factor port of jEdit's syntax highlighting engine. For those who have never used jEdit, syntax highlighting rules are specified in XML files. Rules either involve literal or regular expression matching. Factor already had an XML parser which works pretty well, but regexps were missing.

I've been working with Doug to implement a new regexp library; he will blog about it in the next few days. It suffices to say that it is coming along very well, and the code is very concise and readable, as is the norm with Factor. We're using Chris Double's parser-combinators library to parse the regexp itself, then construct parser combinators from the regexp.

There is quite a bit of history behind the jEdit syntax highlighting engine. jEdit 1.2, released in late 1998, was the first release to support syntax highlighting. It featured a small number of hand-coded "token markers" -- simple incremental parers -- all based on the original JavaTokenMarker contributed by Tal Davidson.

Around the time of jEdit 1.5 in 1999, Mike Dillon began developing a jEdit plugin named "XMode". This plugin implemented a generic, rule-driven token marker which read mode descriptions from XML files. XMode eventually matured to the point where it could replace the formerly hand-coded token markers.

With the release of jEdit 2.4, I merged XMode into the core and eliminated the old hand-coded token markers.

As you can guess from the age of the code, many design decisions date back to Java 1.1, a long-forgotten time when Java VMs with JIT compilers were relatively uncommon, object allocation was expensive, and heap space tight. As a result the parser is basically procedural spaghetti code, with lots of mutable state, the sort that gives Haskell programmers nightmares. So while the code is ugly and the parser design is archaic, the huge advantage is that there is a large suite of contributed modes ready to go.

I expected it would be a pain to port to Factor, but this hasn't been the case. Even though I kept the original design, warts and all, the resulting Factor code is pretty decent. I intend to refactor it to use a more modern design and take advantage of Factor's abstraction features instead of just representing parser state with an bunch of integer and boolean-valued variables. I also need to optimize it more.

Because XMode is an incremental parser, it would be easy to cook up an editor gadget with support for syntax highlighting in the UI. I'll probably do that at some point. Then it will be easy for a contributor to build a text editor in Factor, and we will have come full circle. :-)

Being able to implement a syntax highlighting pastebin in Factor -- one that manages to reuse jEdit mode files no less -- is a milestone. It means we can do XML, regular expressions, web applications, and not to mention the various minor bits and pieces which support these components, such as date/time support, various parsers, and so on. We have different people working on different libraries, and everyone is able to understand and reuse other people's work. It only took a handful of contributors to build all this infrastructure.

Wednesday, December 05, 2007 now has an Atom feed

Thanks to Daniel Ehrenberg, who extended our extra/rss RSS/Atom parser with Atom output support, now publishes an Atom feed. Subscribe away to receive your regular fix of Factor propaganda!