Thursday, November 16, 2006

Most people who criticize Lisp don't understand it

Disclaimer: I don't program in Common Lisp these days and I don't claim to speak for the Lisp community at all, however I think I do know the Lisp language reasonably well.

Just looking the last few weeks of postings on reddit related to Lisp, people raise the following objections:
  • Lisp does not interface with the C-based world easily, unlike scripting languages -- this is clearly false, since most Lisp implementations have easy to use FFIs. To bind a C library to Ruby, you have to write C code which glues the library to the Ruby runtime system. With CFFI (which is portable across CL implementations), you just list the function prototypes in your Lisp file and you're ready to go. You can also pass structs, etc. Factor's C library interface works the same way.

  • Common Lisp was designed to perform well on Lisp machines only, and makes no concessions to ordinary computers. - this is partially true, but the CL standard did omit some features found on Lisp machines which would be difficult to implement efficiently on conventional hardware. However none of this can change the fact that Common Lisp implementations are an order of magnitude faster than scripting languages.

  • Look at all these parentheses! - of course if you don't need metaprogramming, macros, and code-as-data, may as well not use Lisp.


There are many other common misconceptions. The "Lisp only supports linked lists" and "Lisp is unsuitable for writing real-world programs" memes seem to have died down in the last few years, but more myths have taken their place. This leads me to believe that the main problem the Lisp community faces to ensure wider adoption (assuming wider adoption is what they want, which is not necessarily the case) is not a technical one, or an internal social problem as some suggest, but rather an education problem.

Most people simply don't know anything about Lisp, but feel qualified to discuss it and criticize it. Like I said, I'm not a member of the Lisp community (whatever that may be), but when advocating Lisp, Lispers should remember that a lot of things that they take for granted (sane language semantics, C library interface that works, metaprogramming, good performance from a high-level language) are completely foreign to most programmers! There's a communication gap.

And to all the programmers learning new languages, not just specifically Lisp, please hold off criticism until you know the language and have completed a significant project in it. If you did not get that far, don't criticize the language, and put it down to lack of time, lack of motivation or personal failing.

I can rag on Java all I want because I know it better than most Java advocates. But if you were first exposed to Lisp 2 weeks ago, then don't make yet another blog posting about parentheses, even if you work for Google. You're just wasting bandwidth.

11 comments:

Anonymous said...

For real though, java sucks.

Also, try executing a program from the shell portably in common lisp. Just can't be done (iirc!)

Ime, the biggest issues I have with modern lisp are the same ones most people have with C++:
- Lack of good standardized libraries

Uh... Of course, some people don't like the angle brackets in C++ code either. Templates are more like Lisp macros than you want to believe, though only by luck.

If the Lisp world had a boost (www.boost.org) equivalent, (which I think it kind of does, just not enough backing), I think Lisp would be more popular. That and the fact that whenever I mention it to other "professional" programmers, their eyes would stop glazing over.

taw said...

Lisp does not interface with the C-based world easily, unlike scripting languages

This is clearly still true. No-C solutions exists for pretty much all languages (like Ruby/DL for Ruby), but everybody who seriously tried to use them knows they suck.

You cannot use a C library without using a C compiler. To be more precise - the platform's C compiler. C APIs are defined only in C. There is no such thing as standard linker-level API, so that you could open .so and expect it to work. Low-level details can and will be different from platfrom to platform and from version to version even when C API stays stable.

A few examples from libgmp (it could have been any library). C API mpz_and is libgmp.so __gmpz_and, mpz_sgn is a macro and isn't in libgmp.so at all.
GMP_ERROR_DIVISION_BY_ZERO is 2 (you're not supposed to know it, but it's probably quite stable),
__GNU_MP_VERSION_MINOR is 2 too, what you are supposed to know, on the other hand it changes every minor version. C structs can be padded in any platform-specific way etc.

You can write everything by hand, and then update for every platform and minor release, but it's just way easier to generate some C code semi-automatically, and just get the real values from C compilers.

Anyway, low-level bindings are just part of it. Writing reasonable high-level API is much more work. It usually requires messing with both C and high-level language, and high-level languages aren't particularly good at it.

Just to make sure I didn't miss something revolutionary, I've taken a look at CFFI. curl bindings from the tutorial look way more complex than something you could write in a few lines of C with ruby.h, and that's the slowest and most low-level way. And if we actually used Inline::C instead or some code generator.

And as far as I can tell, there aren't any mature and portable C bindings in CFFI for any nontrivial library. Or are there

Slava Pestov said...

You're right that the C ABI is not standard between CPUs and OSes, but I'm not a pussy, so I implement it for every platform that Factor runs on. Only takes a few dozen lines of code.

We have bindings to a variety of C libraries in Factor and we haven't had to write any C glue code yet.

Slava Pestov said...

... and the same goes for Lisp implementations such as SBCL. Since they are compiled, they need CPU and OS specific code anyway, so writing a direct C library interface is no big deal.

Justin said...

"And as far as I can tell, there aren't any mature and portable C bindings in CFFI for any nontrivial library. Or are there?"

Yes. There are bindings for SDL, OpenGL, OpenRM

http://lispbuilder.sourceforge.net/

There are bindings for windows.h so you can do win32 programming.

Justin

Dan Nugent said...

I have to agree with u.int.32.t about the lack of Good standardized Libraries.

CL comes with a vast, feature-stuffed library of classes and data types. But it's not a good one. I mean, I don't claim to be a great Lisper, (I like Lisp in that I've read SICP and seen the beauty of it) but just trying to figure out how to use an Array from the Hyperspec is maddening.

I think Lisp would be the greatest language on Earth if it didn't have 40 years of convention to deal with. Whenever you're creating your own datatypes and functions, Lisp is a pleasure. Otherwise it gets to be a nightmare.

taw said...

And as far as I can tell, there aren't any mature and portable C bindings in CFFI for any nontrivial library. Or are there?"

Yes. There are bindings for SDL, OpenGL, OpenRM

From documentation of CL bindings for SDL:

The Common Lisp Application Builder project relies on SWIG to create the CFFI bindings for SDL. SWIG requires an interface (.i) file (lispbuilder-sdl/sdlswig.i) to create the CFFI bindings from the SDL C headers.

The same document claims it doesn't work on most implementations and OSes.

It's not 1.0 yet, everything is "In Progress", etc. As far as I can tell it wasn't used for a single popular game or other program.

Some time ago I wrote a game in Python with pygame, so I checked CL SDL bindings API whether it's more or less complete. It seems to miss pretty basic functionality like Font.render (rendering text to a new surface, you cannot render directly to preexisting surface, as in all nontrivial cases you want to do some layouting). I don't see anything like that in CL API.

So - it's not pure-CL, it's not portable, it's not mature, it's not complete, and it has pretty much no users. On the other hand pygame is perfectly portable (and creating a standalone zip or a one-click installer for Windows takes just a few minutes), is mature, complete, and very widely used.

If SDL and related libraries indeed represent the best of CFFI bindings, I think you should seriously retract your criticism of the way "scripting languages" do it.

Slava Pestov said...

taw, Can you show me an example of a GUI toolkit implemented entirely in Ruby, which compares to the Factor UI, which itself binds to OpenGL, FreeType, Cocoa, X11 and Win32 without any C code.

taw said...

Last time I checked we were talking about CL, not Factor, right ? As far as I can tell, Factor is written in C, not in CL.

And it isn't true that Factor UI contains no C code. What is this (gl.factor) if not gl.h massaged by a few regular expressions ?

: GL_ALL_CLIENT_ATTRIB_BITS HEX: FFFFFFFF ; inline
: GL_CLIENT_ALL_ATTRIB_BITS HEX: FFFFFFFF ; inline
FUNCTION: void glClearIndex ( GLfloat c ) ;
FUNCTION: void glClearColor ( GLclampf red, GLclampf green, GLclampf blue, GLclampf alpha ) ;

And what are Factor function from "alien" module other than thinly disguised C code ?

Languages like Ruby/Python typically use SWIG or real C compiler for parsing headers, and C instead of special low-level C-like language for handling low-level issues.

So there aren't any non-toy GUIs for Ruby/Python (or any other language) that don't contain any C code. Ruby X11 library doesn't use or even link to C code (it talks with X11 server using X11 wire protocol), but nobody uses X11 directly, so it counts as a toy.
There was also a Win32 library for Python that was purely ctypes, without any C. Also a toy.

FXRuby and Ruby bindings for Qt/KDE are Ruby + C automatically generated from API descriptions (by SWIG or special parser) + small amout of C related to garbage collection, library-specific low level type conversions etc.

Anyway: Factor UI isn't even CL. What's the big difference between C code automatically generated by SWIG from headers + small amount of low-level C code (the way most bindings are generated), and low-level not-C code automatically generated by something from headers + small amount of low-level pretends-not-to-be-C code (the way Factor uses) ? The only difference I see is reduced portability and having to maintain a new "I'm not C, really" language.

Justin said...

in response to TAW...

"The same document claims it doesn't work on most implementations and OSes."

There is a table showing what we have been confirmed to work on. I know we work on CLisp, SBCL, lispworks and allegro, and on mac, linux and windows.

"It's not 1.0 yet, everything is "In Progress", etc. As far as I can tell it wasn't used for a single popular game or other program."

No we don't have any users apart from the bug splat demo. I think that's more down to lack of demand for writing games in lisp, but Luke and I started this project to do something about that.

"Some time ago I wrote a game in Python with pygame, so I checked CL SDL bindings API whether it's more or less complete. It seems to miss pretty basic functionality like Font.render (rendering text to a new surface, you cannot render directly to preexisting surface, as in all nontrivial cases you want to do some layouting)."

That's not part of the basic SDL library, it's part of SDL_ttf, which we have generated bindings for and does work.

"If SDL and related libraries indeed represent the best of CFFI bindings, I think you should seriously retract your criticism of the way "scripting languages" do it."

I didn't criticize the way scripting languages do it. As it stands pygame is a great tool for making games, and I readily admit that lispbuilder needs users to mature, but nobody should be afraid to try it based on your comments, since mostly they are invalid.

Anonymous said...

"Most people who criticize Ruby don't understand it"

For the record, Ruby has a FFI included in the distribution, called dl (similar to CFFI, maybe not as powerful). Also, it has a slightly improved FFI on Win32, called Win32API.

Most people prefer writing extensions in C though, because the Ruby C API is similar to normal Ruby and quite comfortable.

You can also use normal C headers instead of converting them to some form your FFI requires.