Tuesday, January 31, 2006

Now using DARCS for the Factor source repository

I got sick of SourceForge CVS. CVS sucks, the SF servers are always down, and their anonymous server lags. This is simply unacceptable, so I decided to take matters into my own hands.

I'm now using DARCS for the Factor source code repository now. I still haven't decided how patch submissions will be handled; I'll probably just ask people to post patches to the mailing list.

The repository is located at http://factorcode.org/repos. You can use a command like:
darcs get http://factorcode.org/repos
to download a copy. Then use darcs pull to stay up to date.

Monday, January 30, 2006

Factor 0.80 is out

Factor 0.80 is now available.
  • New help system, browsable in the UI and via the HTTP server (/responder/help). In the UI listener, invoke handbook to read the documentation root, and invoke \ foo help to look at documentation for the word foo.
  • Sequences:
    • Association list words assoc*, set-assoc, acons and remove-assoc are gone.
    • The repeated virtual sequence type is gone. Instead, the <array> word takes an initial element in addition to an initial size.
    • The fill word to create a new string with an initial character repeated a certain number of times has been renamed to <string>.
    • Add a new interleave ( seq quot between -- ) combinator that applies a quotation to each element of a sequence, calling another quotation in between each pair.
    • Add a new <=> ( obj1 obj2 -- n ) word for comparing two objects using an intrinsic order. For numbers this is the standard order, for strings this is lexicographic order, and for words, this compares word names.
    • The natural-sort ( seq -- seq ) word replaces number-sort, string-sort and word-sort.
  • Hashtables:
    • hash* ( key hash -- [[ key value ]] ) is now hash* ( key hash -- value ? )
    • hash-clear is now clear-hash
    • hash-each, hash-each-with, hash-all?, hash-all-with?, hash-subset, hash-subset-with now pass the key and value separately on the stack to the given quotation, instead of passing a cons cell
    • Literal syntax change: H{ [[ key value ]] ... } is now H{ { key value } }
  • Math:
    • The sum and product words have been moved to contrib/math/.
    • The mod word is now supported for ratios and floating point numbers.
    • The truncate, floor and ceiling words are now supported for floating point numbers.
    • The NaN, positive infinity and negative infinity floating point numbers now parse and unparse as 0.0/0.0, 1.0/0.0, and -1.0/0.0 respectively.
    • The NaN value is now equal to itself under =.
    • Negative and postive zero are no longer equal under =. However, the new zero? word tests if the top of the stack is a zero, and it tests for both positive and negative zero.
  • Streams:
    • stream-format ( string style stream -- ) now takes a hashtable rather than an association list for specifying style information.
    • stream-write and stream-terpri are now generic words, and there is a new with-nested-stream generic word. You can wrap your output streams in a <plain-writer> to avoid implementing these.
  • C library interface:
    • Some alien word changes:
      <foo> ==> "foo" <c-object> <foo-array> ==> "foo" <c-array>
    • Support for binding to Objective C libraries is now included.
      • Normal usage of Objective C classes and methods is done using the OBJC-CLASS: and OBJC-MESSAGE: parsing words. See the example in examples/cocoa-speech.factor.
      • Objective C runtime introspection functions and structures are defined in the objective-c vocabulary.
    • Added a pair of words for between Factor strings and C strings, alien>string and string>alien.
  • Compiler changes:
    • AMD64 compiler backend.
    • Fixed some problems with compilation of inlined recursive words.
  • UI changes:
    • Fixed invalid OpenGL calls which caused problems on Windows machines with ATI drivers, and Linux machines with the MesaGL implementation.
    • The listener looks different now. An expandable top area is used for browsing objects, words and help, and the stack display has been shrunk to a single status line at the bottom of the window.
    • A left click on a presentation now invokes the default command. A right click shows a menu of possibilities.
  • Bootstrap changes:
    • Source files are no longer loaded in the stage-2 bootstrap. Since stage-2 bootstrap runs in the interpreter, this reduces bootstrap time by a few minutes. Instead, all source files, including the compiler backend, are loaded in stage-1 bootstrap, and thus boot images are now CPU-specific. Boot images can be created as follows:

      USE: image
      "x86" make-image
      "ppc" make-image
      "amd64" make-image
  • Contributed libraries:
    • All libraries in contrib/ have been tested and updated for recent language changes, and you can run contrib/load.factor to load all of them at once (Trent Buck)
    • Updated contrib/x11/ with many more examples (Eduardo Cavazos)
    • Added splay tree library in contrib/splay-trees.factor (Mackenzie Straight)
    • Improved AJAX support in contrib/httpd/. The "prototype" JavaScript library is now included (Chris Double)

Factor UI now runs on AMD64

Factor 0.80 is complete. Check out the screenshot.

I dusted off the Factorbot IRC bot and spawned an instance to run in the #concatenative channel on irc.freenode.org. It is running in the same Factor instance that serves factorcode.org. At the moment the bot does not do much; you can ask it for a link to a browser view of a given word.

Saturday, January 28, 2006

Workaround for GCC 4.0 issue on x86

I have worked around the problem where GCC 4.0 would miscompile certain runtime primitives on x86 with the -fschedule-insns optimization. Factor 0.79 still has the problem, however Factor 0.80 can be compiled with full optimization.

I'm hoping I can finish off Factor 0.80 today. The only thing I need to do is get the UI running on AMD64. Enough of the FFI works to support native I/O, however the UI still has problems. Perhaps it is an FFI bug, or my SDL, OpenGL or FreeType bindings are not 64-bit clean. It should not be hard problem to solve.

Factor mailing list created

I have created a mailing list for Factor-related discussion. You can subscribe at https://lists.sourceforge.net/lists/listinfo/factor-talk.

Thursday, January 26, 2006

Factor 0.80 web server now live; AMD64 FFI progress

http://factorcode.org is now running Factor 0.80. I've also been working on the AMD64 FFI. Calling functions with integer and floating point arguments now works. Functions taking more than 7 parameters do not work, however; I haven't implemented stack parameter spilling yet. However, it is not too hard; I just have to adapt the existing PowerPC code that performs this task.

A release of 0.80 is just around the corner. As soon as the AMD64 FFI works well enough to run the Factor UI, I will release it.

Tuesday, January 24, 2006

Test instance of Factor HTTPD 0.80

I'm running a testing instance of Factor 0.80 on factorcode.org. You can access it via port 8889: http://factorcode.org:8889/.

There is a new responder of interest:
http://factorcode.org:8889/responder/help

The help responder for browsing articles. The help responder uses some JavaScript to implement expandable outliners. Thanks to Chris Double for implementing the "AJAX" code that does the trick.

The browser responder has also been overhauled and now shows word documentation:
http://factorcode.org:8889/responder/browser

The help and browser responders still have a few wrinkles. Once they're sorted out, and if I'm happy with the stability of the test server, I will make it the primary server.

The web site has seen 10,000 hits over the last 10 days. To date I haven't had a single issue with the Factor HTTP server. It is not a huge load, but it looks like the server is working okay.

I should mention my workflow when writing web code in Factor. I develop web apps locally on my workstation. I have the jEdit plugin run a Factor instance. I start the HTTP server in this instance, and while working on a responder, reload the source files with a jEdit command and immediately test the changes.

When I'm ready, I copy a bootstrapped image with the HTTP server loaded to my Linode box. On the Linode server, the Factor HTTP server runs inside a detached 'screen' instance. I can log in to the server, run screen -r, and I'm looking at the Factor listener. I can reload code and inspect server objects, all without restarting or 'redeploying' anything.

Being able to reload code in a running server without downtime is important. Look at the Java situation:

Here's the situation. I have a project that runs in JBoss. In normal/non-debug JBoss takes about a minute to startup and get ready to receive requests. In debug mode we're talking 2.5-3 minutes to startup even when no debugger has been attached. Given the severe difference in startup time I tend not to run the server in debug mode until I really really need to debug something. But then if I need to make a change and restart the server (hot-deploy and redeploy do not cover nearly enough) it's frustratingly slow!

Sunday, January 22, 2006

Problem with GCC 4.0 and -fschedule-insns

The random compiler tester found another problem, and this time the problem is with GCC! On GCC 4.0, enabling -fschedule-insns (which is implicitly enabled by -O2 and -O3) miscompiles the bits>double, bits>float, float>bits and double>bits primitives.

If you're using GCC 4.0, please do not use anything higher than -O1 to compile the Factor runtime until further notice.

Random compiler tester finds first bug

Doug Coleman's random compiler tester managed to generate a very large number of failing test cases, which are all found to be symptoms of the same bug: the overflow check for the fixnum-shift primitive was wrong, thus certain large negative fixnums, when shifted to the left, would not be upgraded to a bignum and instead overflow, Java-style. This bug manifested itself on x86 in a pretty bad way: because the x86 assembler uses bit shifting to assemble instructions, large negative literal fixnums in colon definitions would not compile correctly. This bug would have been hard to discover by hand since it is rare to find literal integers in code, other than the usual 0, -1, 1 and 2. Now the overflow bug is fixed.

Factor 0.80 already has a large number of fixes for compiler bugs which I found with manual testing, and now that we have an automated test case generator, I hope it finds many more. :)

Friday, January 20, 2006

Factor 0.80 almost ready

I'm considering the new help system done for now. The new handbook doesn't cover everything the old handbook did, however I'll take care of that over time. I'm currently working on HTML rendering for the help markup language; this will be ready shortly, and soon I'll update the Factor HTTP server instance running at factorcode.org to allow online browsing of documentation, which is a first since formely the handbook was only available in PDF format.

In other news, Trent Buck is the newest member of the Factor community. He's working on a Debian package, and he's also done a lot of cleanup of the contributed library code to make sure it loads and works with the language changes made in Factor 0.80.

Doug Coleman who has already contributed quite an amount of code is working on a random compiler tester. It generates random quotations, and compares compiled and interpreted output. It has already found some rather obscure bugs.

Thursday, January 12, 2006

3D graphics in Factor

Eduardo Cavazos has implemented a Factor program for rendering Lindenmayer Systems.

Here is a screenshot:


Eduardo is the latest Factor contributor. So far he's implemented a set of X11 bindings, an X11 window manager in Factor, and a series of graphical demos like this one. Factor 0.81 will use the X11 bindings for an UI backend in place of SDL.

Wednesday, January 11, 2006

Objective-C library interface with Apple text-to-speech example

In only a couple of hours, I managed to develop an interface to Objective-C libraries. This is now included in the Factor core. Objective-C is a static language with a runtime supporting limited introspection - you can list all defined classes, list all methods defined on a class, define new methods and classes at runtime, and invoke methods on objects dynamically. While it is not a great language, what makes it interesting is that, of course, the Cocoa API for Mac OS X is written in it.

The Factor interface is slightly awkward because method selectors have to be defined with parsing words first.

What follows is an example that uses Apple's text-to-speech library to speak a string.

We start by defining classes and methods we need.
OBJC-MESSAGE: id alloc ;
OBJC-CLASS: NSString
: NSASCIIStringEncoding 1 ; inline
OBJC-MESSAGE: id initWithCString: char* encoding: uint ;
OBJC-CLASS: NSSpeechSynthesizer
OBJC-MESSAGE: id initWithVoice: id ;
OBJC-MESSAGE: bool startSpeakingString: id ;

Then, a utility word to create new NSStrings:
: <NSString> ( string -- alien )
NSString [alloc]
swap NSASCIIStringEncoding [initWithCString:encoding:] ;

Finally, we can use the above words to speak a string:
NSSpeechSynthesizer [alloc]
f [initWithVoice:]
dup "Hello from Factor" <NSString> [startSpeakingString:]

I'm still trying to come up with a more natural way to fit this into Factor syntax. Merging the selector into a single word name is pretty tacky.

In a future release, probably 0.81, I will write a Cocoa bindings and a Mac OS backend for Factor's UI.

In other news, I'm slowly converting the handbook from LaTeX into the Factor online help markup. I guess I'm about half-way through.