Wednesday, July 25, 2007

Smoother stand-alone app deployment

I polished the deployment tool a bit. Now all it takes is a vocab name:

Each vocabulary has a deployment configuration. The configuration can be inspected and set with deploy-config and set-deploy-config:
( scratchpad ) "lsys.ui" deploy-config .
{ strip-word-props? t }
{ strip-word-names? t }
{ strip-dictionary? t }
{ strip-debugger? t }
{ strip-c-types? t }
{ deploy-math? t }
{ deploy-compiled? t }
{ deploy-io? f }
{ deploy-ui? t }
{ "bundle-name" "Lindenmayer" }

You can also change the configuration:
f strip-word-names? "lsys.ui" set-deploy-flag

The configuration is stored in a deploy.factor file of the vocabulary directory.

Deployment has been tested to work with the following vocabs; they all include sensible deploy configs now:
  • automata.ui
  • boids.ui
  • bunny
  • color-picker
  • gesture-logger
  • golden-section
  • hello-ui
  • lsys.ui
  • maze
  • nehe
  • tetris

To deploy your own app, you should only have to set two flags:
  • deploy-ui? - set this to true if your app uses the UI
  • deploy-io? - set this to true if you want non-blocking I/O or sockets

Specialized float arrays

I added float arrays to Factor. A float array behaves like an array of floats, except the representation is more efficient; individual elements are not boxed.

Literal float arrays look like F{ 0.64 0.85 0.43 0.16 0.37 }.

Float arrays can provide a performance benefit if the compiler is able to infer enough information to unbox float values. For example, consider the following word:
: v+ [ + ] 2map ;

It takes two arrays and adds elements pairwise. Let's try timing the performance of this word with normal arrays:
( scratchpad ) { 0.64 0.85 0.43 0.16 0.37 0.64 0.85 0.43 0.16 0.37 } dup [ 1000000 [ 2dup v+ drop ] times ] time
3200 ms run / 25 ms GC time

Now float arrays:
( scratchpad ) F{ 0.64 0.85 0.43 0.16 0.37 0.64 0.85 0.43 0.16 0.37 } dup [ 1000000 [ 2dup v+ drop ] times ] time
3653 ms run / 70 ms GC time

It is actually slower! This is because each element access has to allocate a new float on the heap. But now, lets use the new hints vocabulary to give a hint to the compiler that v+ should be optimized for float arrays:
HINTS: v+ float-array float-array ;

This has the effect of compiling a version of this word specialized to float arrays. Here, the compiler can work some magic and eliminate boxing altogether:
( scratchpad ) F{ 0.64 0.85 0.43 0.16 0.37 0.64 0.85 0.43 0.16 0.37 } dup [ 1000000 [ 2dup v+ drop ] times ] time
974 ms run / 10 ms GC time

I used float arrays to make the spectral norm benchmark faster: the run time went from 120 seconds to 30 seconds. The raytracer was not improved by float arrays, though; I need to investigate why.

Also float array operations are only compiled efficiently on PowerPC right now. I need to code some new assembly intrinsics for the other platforms.

Float arrays can be passed to C functions directly. Long-term, somebody should look into using SSE2 and AltiVec to optimize vector operations on float arrays. That would really rock.

Sunday, July 22, 2007

execve() returning ENOTSUP on Mac OS X

Apple's man page for execve() does not list ENOTSUP as a possible error code. This problem had me stumped for hours. After I was convinced I wasn't doing anything wrong, and there was no bug with the Factor FFI, I did some Googling and finally stumbled across this.

Turns out execve() returns this error if your program is multi-threaded. Makes perfect sense, but I wish Apple had kept the man page up to date!

In any case, I'm posting it here so that any other poor sucker who runs into this can find the solution with Google in less time than I did.

Saturday, July 21, 2007

Support for IPv6 and Unix domain sockets

Some very new code, not completely tested.

The <client> and <server> words used to take a host/port pair, and a port number, respectively. This was too limiting. Now both words take address specifiers, of which are there four types:
  • "/path/to/socket" <local> - an IPC socket with the given path (Unix domain sockets on Unix)
  • "dotted.quad" 123 <inet4> - an IPv4 address
  • "ipv6.addr" 1234 <inet6> - an IPv6 address
  • "hostname" "http" <inet> - Internet host named by DNS entry; will be either IPv4 or IPv6

Note that in most cases, <client> will be called with an instance of inet; this invokes the domain name resolver, which may produce a list of multiple IPv4 and IPv6 addresses. Factor tries each one in turn until a connection succeeds. This is the expected behavior for client sockets, since users generally input host names and not IP addresses, and don't care if the connection is made over IPv4 or IPv6.

The <server> word requires an instance of the more specific inet4 and inet6 classes. Since in most cases, a server interested in connections on, say, port 8080 wants to receive connections over both IPv4 and IPv6, the with-server combinator should be used instead of calling <server> directly. Here's a usage example which waits for connections on port 8080 and sends "Hello, client" to each one:
8080 internet-server [ "Hello, client." print ] with-server

If your system is configured for IPv6, this example will spawn two threads, for IPv4 and IPv6 connections.

Server sockets can now be restricted to the loopback network interface by passing 8080 local-server instead of 8080 internet-server. Before this was provided by the Unix-only loopback-server module; this module is now obsolete.

The domain name resolver can now be invoked directly with the resolve-host word. Given a host name and port name or number, it returns a list of address specifiers.

UDP/IP works as before; words which used to take host/port pairs now take address specifiers. Note that the inet address specifier is not supported for UDP/IP; clients must first resolve the destination host name, and pick the appropriate address specifier, whether it be an IPv4 or IPv6 address, from the result.

Unix domain sockets are accessed with the local address specifier.

Stream Unix domain sockets are very commonly used, but a little-known fact is that datagram Unix domain sockets are supported too. Here is some proof.

Enter this in one listener:
"/tmp/dgram-dst" <local> <datagram>
dup receive

Then this in another:
"/tmp/dgram-src" <local> <datagram>
B{ 1 2 3 4 5 } "/tmp/dgram-dst" <local> pick send

The first machine should now have the following two objects on the stack:
B{ 1 2 3 4 }
T{ local f "/tmp/dgram-src" }

I still have to fix assorted bugs in this code. Also Windows network support is lagging; not only does it have to be updated to support address specifiers, but it is also missing UDP hooks at this stage.

I won't be adding more features to networking for a little while, but the next thing I will do is probably raw sockets.

Thursday, July 19, 2007

Generating stand-alone Mac OS X applications from Factor

This is all subject to change, but right now, the following works:
USE: tools.deploy
"" "tetris" H{ { deploy-compiled? t } { deploy-ui? t } }

After a rather slow bootstrap and deploy process, you're left with, 2.2 Mb in size. Of this, 1.3 Mb is the FreeType font rendering library; soon, Factor will use native font rendering on Mac OS X, thus we won't have to ship FreeType anymore.

A lot remains to be done: we need a graphical way to configure the deployment tool (there are a lot of switches which can be toggled in the hashtable passed to, and there needs to be an automated way of packaging required resources and native libraries with the app. And of course, deployment of native Linux and Windows binaries is a whole other kettle of fish. But over time, all of this will be implemented.

Sunday, July 15, 2007

Back from New Zealand

Update: Sorry if some of the below reads like too much of a rant. I wrote this entry literally an hour after getting back into town, after spending 24+ hours in airports and airplanes. So I was somewhat grouchy. I had an amazing time in NZ, I really did.

My girlfriend and I just got back from New Zealand. We visited Auckland, Wellington, Napier, New Plymouth and the Coromandel Peninsula, all in a crappy rental car which surprisingly enough did not break down or explode once during the trip (we did mysteriously lose a hubcap, though). The weather was great for the most part, despite it being winter; we managed to go kayaking, visit the Auckland zoo, a farm in Wellington (lovely lambs), do some shopping and various other touristy things, and catch up with many of my old friends (I lived in Wellington for 8 years). I also got a chance to hang out with ├╝ber-hacker Chris Double on several occasions. Mostly just dinner, but we also sneaked in a bit of pair-programming while my girlfriend wasn't paying attention. Chris is a cool guy with a great sense of humor; if you ever get a chance to meet him, he'll tell you everything you ever wanted to know about online poker, the Firefox codebase, and penguins.

My friend Neil Bertram asked me to give a presentation about Factor at Catalyst NZ, an IT consulting shop. Every Thursday, they order a ton of pizza and have speakers, both in-house and from outside give short talks on various technical topics. The day I was presenting, there was a talk about a web-based content man management system, and Sun's ZFS. I don't really dig web-apps much but ZFS looks worth looking into more. I've known Neil since I was 8; we went to the same primary school and high school, and we've mostly stayed in touch since then. I had a lot of fun hanging out with Neil, too. He's a cool guy. If you're reading this Neil, I think you like Asian girls a bit too much, you'll have better luck if you broaden your horizons a bit when trying to pick up in clubs :-)

Mostly I slept my way through the flights and didn't feel jet-lagged upon arrival in Auckland; and I seem to be fine right now as well.

We also had two stopovers on Los Angeles, about 10 hours each, on the way there and back. Since LAX is pretty far from downtown LA, we didn't get a chance to visit the city of Los Angeles, however we did spend several hours in Santa Monica, which is pretty close to the airport (30 minute bus ride on the "Big Blue Bus"). Santa Monica is pretty; there's a (rather crowded) beach, eateries, shops, plenty of very expensive cars, and lots of homeless people.

I'll have some photos from NZ and LA to post later.

The flight from LA to Ottawa was rather unpleasant, for three reasons. First of all, we were within earshot of a number of crying babies. There's only one thing more annoying that an enterprisey Java programmer, and that's a child crying for 12 hours non-stop. I wish that children under 4 were not allowed on airplanes.

Also the entire entertainment system crashed in the middle of the flight, and took 20 minutes (!) to reboot. Chris Double told me that the same thing happened to him recently, so I wonder if this happens every time. Even when it was running, the user interface for movie selection was not very intutitive, and unresponsive. Apparently several passengers did not have functioning entertainment at all. While the whole thing runs on Windows CE, I don't think CE is the fault here, rather bottom of the barrel contractors hired by Air New Zealand to design and implement this thing.

While my luggage came through fine, my girlfriend's suitcase did not make it from LA to Ottawa. Seems that while the 10$/hour federal government employees do a great job of x-raying people's shoes and asking me why I'm bringing a small bottle of contact lens solution in carry-on luggage (hey, Mr Important TSA Guy: I wear contact lenses), they are incapable of placing bags in the correct plane. To make matters worse, the phone number they gave us for baggage inquiries is serviced by a typical cheap third-world call center. It starts off with an automated voice recognition system which asks you about 30 questions and you end up repeating everything 3 times because of poor voice recognition; then you get an operator whose grasp of English is even worse. He asked if we really needed this luggage or not, and he was convinced we missed our flight (when in fact we sat in the terminal for 4 hours). The sod just asks you a bunch of useless questions, then puts you on hold for 30 minutes, says he can't really help you, and asks you to call back tomorrow. Plenty of incompetence all round.

Daniel Ehrenberg did a great job of harvesting patches during my absence, and I will return to Factor hacking as soon as I get some rest. I will be releasing 0.90 ASAP. I'm looking forward to hacking Factor after 3 weeks away from the code!