Thursday, January 29, 2009

Rendering Unicode text to an OpenGL texture using Mac OS X CoreText

This post is really a follow-up to my previous post where I bragged about Factor supporting Unicode font rendering on Mac OS X. After writing that post I realized that there is not much information online about the particular use-case I implemented: rendering text to an offscreen bitmap context, and then making it into an OpenGL texture. So I decided to write up a blog post collecting the gory details in one place. This information should be useful to programmers who use any language on Mac OS X, but all the code examples are in Factor.

The code examples below are not self-contained; they use the FFI bindings I defined in the core-graphics and core-text vocabularies in the new_ui branch in the Factor GIT repository. Factor's FFI is very easy to use, for example here are some Core Graphics functions:
FUNCTION: CGContextRef CGBitmapContextCreate (
void* data,
size_t width,
size_t height,
size_t bitsPerComponent,
size_t bytesPerRow,
CGColorSpaceRef colorspace,
CGBitmapInfo bitmapInfo
) ;

FUNCTION: void CGContextSetRGBFillColor (
CGContextRef c,
CGFloat red,
CGFloat green,
CGFloat blue,
CGFloat alpha
) ;

Once the FFI bindings are in place, a lot of higher-level wrapper code has to be written, because these are low-level, C APIs. The first step is creating a bitmap context; after that, I'll discuss creating the various Core Text data types and rendering strings to a bitmap context. Finally, I'll discuss creating an OpenGL texture from a bitmap context.

Creating a bitmap context

The first step is to create a CoreGraphics bitmap context using CGBitmapContextCreate. When creating a bitmap context, you supply a byte array to render to. There are two main caveats here.

First, the memory backing the context should not move while the context is active. This means in GC languages with copying collectors, such as Factor, you have to pin your byte array (Factor doesn't support this) or allocate it in unmanaged memory (using malloc or similar). This is of course obvious (if the GC moves the object, it has no way of notifying Core Graphics that it moved) but its something to watch out for when using an FFI from a high-level language.

The second caveat, that took me a long time to figure out, is that you have to use a specific bitmap format if you want sub-pixel font smoothing to be enabled on LCD screens. The bitmapInfo parameter to CGBitmapContextCreate specifies the format of the bitmap in memory. The code path for sub-pixel font smoothing is only used if the following two flags are set in the bitmap info:
  • kCGImageAlphaPremultipliedFirst - put the alpha component first in each quadruple
  • kCGBitmapByteOrder32Host - use native endian to pack the ARGB quads, instead of the default which seems to be big-endian.

If these two values are not passed, then sub-pixel smoothing will not be enabled even if the user is using an LCD monitor and has requested that it be performed.

Below is the Factor code for creating a bitmap context, with some comments.

First, we make a word which constructs the flags mentioned above:

: bitmap-flags ( -- flags )
{ kCGImageAlphaPremultipliedFirst kCGBitmapByteOrder32Host } flags ;

Now, a word to compute the size of the bitmap in memory. This is used twice, when malloc'ing the bitmap, and after rendering, when copying it back to managed memory:
: bitmap-size ( dim -- n )
product "uint" heap-size * ;

Now, a word to malloc the bitmap. We add it the innermost destructor scope using &free; this ensures its deallocated when we're done with it, either after successfully rendering our text, or if an error is thrown:
: malloc-bitmap-data ( dim -- alien )
bitmap-size malloc &free ;

We also have to pass a color space; the default RGB space is sufficient for my purposes, and once again we define a destructor:
: bitmap-color-space ( -- color-space )
CGColorSpaceCreateDeviceRGB &CGColorSpaceRelease ;

Now, a word which uses the above to create a context backed by unmanaged memory:
: <CGBitmapContext> ( dim -- context )
[ malloc-bitmap-data ] [ first2 8 ] [ first 4 * ] tri
bitmap-color-space bitmap-flags CGBitmapContextCreate
[ "CGBitmapContextCreate failed" throw ] unless* ;

Once we're done rendering, we have to get the bitmap data from the context and copy it to managed memory, so that we can return a byte array object to the user. This is done with the below word:
: bitmap-data ( bitmap dim -- data )
[ CGBitmapContextGetData ] [ bitmap-size ] bi*
memory>byte-array ;


This ends the implementation detail. The last word here is a high-level word that is called by other code. This word completely hides the resource allocation and deallocation by establishing a destructor scope, creating a bitmap context, calling a quotation with the bitmap context, then copying the bitmap data back to managed memory and returning a byte array. All the external resources allocated during this operation -- the memory buffer, the color space, the context -- are deallocated for us as soon as with-destructors returns:
: with-bitmap-context ( dim quot -- data )
[ [ <CGBitmapContext> &CGContextRelease ] keep ] dip
[ nip call ] [ drop bitmap-data ] 3bi
] with-destructors ; inline

Now, higher-level code can simply wrap some Core Graphics rendering calls in a with-bitmap-context, and receive a Factor byte array as the result.

Creating a CTFont

Core Text defines the CTFont opaque type to represent a specific instance of a font with a style and size applied. I create CTFont instances with CTFontCreateWithName, and then apply bold and italic styles by calling CTFontCreateCopyWithSymbolicTraits. There is another API to do this in one shot, CTFontCreateWithFontDescriptor, but for some reason it ignores any set symbolic traits. Perhaps its a bug, or operator error, but it took me a lot of futzing around to get Core Text to respect my choice in font style. Another caveat to watch out for is that CTFontCreateCopyWithSymbolicTraits returns null if the traits could not be applied; for example, Apple's "Monaco" font does not have a bold variant, so if the user requests bold Monaco, you just have to return the original Monaco without the traits. There might be a way to synthetically increase the font weight, but I haven't discovered it yet.

Creating a CTLine from a string and a font

One of the central abstractions in Core Text is a CTLine. You create a CTLine from a CFAttributedString with CTLineCreateFromAttributedString, and render it to a CGContext with CTLineDraw.

Creating the CFAttributedString is an adventure in itself. For now, I'm only interested in rendering a string with a single font style and color, so I can use the CFAttributedStringCreate function to create the attributed string. It takes a CFString and CFDictionary. The dictionary can contain a number of keys, the two I'm using are kCTFontAttributeName and kCTForegroundColorAttributeName. These are global variables holding CFString references. Factor's FFI doesn't have a way to access global variables right now, so I made a quick hack specific to the core-text vocabulary. It lets you access global vars that hold void* values. Soon I'll generalize it and put it in alien.syntax, but for now it can live as a private utility in core-text:
dup name>> '[ _ f dlsym *void* ]
(( -- value )) define-declared ; parsing

Using this utility is easy:
C-GLOBAL: kCTFontAttributeName
C-GLOBAL: kCTForegroundColorAttributeName

Because I found myself constructing lots and lots of CF types: strings, numbers, dictionaries, colors, and so on, I made a utility word, >cf which converts a Factor object into a Core Foundation object. Using this word, I can define a <:CFAttributedString> that constructs a CFAttributedString from a Factor string and assoc:
: <CFAttributedString> ( string assoc -- alien )
[ >cf &CFRelease ] bi@
[ kCFAllocatorDefault ] 2dip CFAttributedStringCreate
] with-destructors ;

Again, notice the destructor usage: we want to release the string and dictionary after we return, since the attributed string retains them for us.

Now, we can create the CTLine, finally. Notice how we take a Factor string, a CTFont, and a color, and make the attributes as a Factor assoc; the <CFAttributedString> word takes care of converting it to Core Foundation types. And again, we use destructors for deterministic resource cleanup:
: <CTLine> ( string font color -- line )
kCTForegroundColorAttributeName set
kCTFontAttributeName set
] H{ } make-assoc <CFAttributedString> &CFRelease
] with-destructors ;

Rendering a CTLine to a bitmap context

Now we can put some things together and write a word which renders a Unicode string to a Factor byte array.

First, we make a data type to hold the CTLine, the rendered bitmap, the typographic bounds, and some other things:
TUPLE: line font line bounds dim bitmap age refs disposed ;

The refs and age slots are used for caching lines so that the UI doesn't have to render them over and over again; I'll discuss this in another blog post, maybe.

Now, we have some code which computes the typographic bounds of a CTLine, and packages the values up into a Factor tuple; there's also a word to compute the size of the bitmap to create. This doesn't take leading into account, I haven't decided how to handle that yet:
TUPLE: typographic-bounds width ascent descent leading ;

: line-typographic-bounds ( line -- typographic-bounds )
0 <CGFloat> 0 <CGFloat> 0 <CGFloat>
[ CTLineGetTypographicBounds ] 3keep [ *CGFloat ] tri@
typographic-bounds boa ;

: bounds>dim ( bounds -- dim )
[ width>> ] [ [ ascent>> ] [ descent>> ] bi + ] bi
[ ceiling >fixnum ]
bi@ 2array ;

Now that we have the above utilities, as well as all the code defined earlier, we can write a <line> constructor word which constructs a line tuple. Part of the construction involves rendering the text to a bitmap, and storing the resulting byte array in the line's bitmap slot. We're going to use the with-bitmap-context word written above here. There are a few steps to rendering a CTLine to a bitmap context:
  • First, you must fill the context with a background color, using CGContextSetRGBFillColor and CGContextFillRect. Leaving it transparent will disable sub-pixel rendering, so the line constructor takes a background color as a parameter.
  • You must set the text position with CGContextSetTextPosition.
  • Finally, you can render the text with CTLineDraw.

This word manipulates a lot of intermediate values, makes a fair number of Core Graphics calls, and has complex data flow, so I chose to implement it using locals. Notice the [let* "serial-binding" form; successive definitions can reference previously-defined values:
:: <line> ( string font foreground background -- line )
[let* | font [ font CFRetain |CFRelease ]
line [ string font foreground <CTLine> |CFRelease ]
bounds [ line line-typographic-bounds ]
dim [ bounds bounds>dim ] |
dim [
[ background >rgba-components CGContextSetRGBFillColor ]
[ 0 0 dim first2 <CGRect> CGContextFillRect ]
[ 0 bounds descent>> CGContextSetTextPosition ]
[ line swap CTLineDraw ]
} cleave
] with-bitmap-context
[ font line bounds dim ] dip 0 0 f
line boa
] with-destructors ;

Notice how it uses "on-error" destructors, named |foo. This means that if the construction fails for whatever reason, the CTLine and CTFont objects are released, but if construction succeeds, they are retained. This is done so that we can hold on to these objects and store them in the new tuple's slots. The tuple itself has a dispose* method which releases the font and line objects:
M: line dispose* [ font>> CFRelease ] [ line>> CFRelease ] bi ;

The Factor UI uses all the slots in the line object to perform operations such as measuring text, caching textures, and so on. For the purpose of this blog post, the only interesting slots are dim and bitmap. The bitmap slots holds a byte array with the rendered text, in the format we specified when creating the bitmap context; ARGB with native byte order.

Creating an OpenGL texture from a Core Graphics bitmap

Standard OpenGL does not support ARGB bitmaps, however Apple's implementation supports the GL_BGRA_ext extension. To use this extension, I had to add a couple of constants to our vocabulary:
! GL_BGRA_ext:

Now, we can pass CL_BGRA_EXT to glTexImage2D. Here is a utility word to ease creation of textures. Since glTexImage2D takes 9 parameters (!) I use locals so that I can name them:
:: make-texture ( dim pixmap format type -- id )
gen-texture [
GL_TEXTURE_2D swap glBindTexture
dim first2
] do-attribs
] keep ;

Once the texture has been created, we can display the text on the screen by binding the texture and rendering a quad. It is also important to call glEnable with GL_TEXTURE_2D, otherwise the quad won't be textured. I forgot this the first time and spent some 20 minutes looking at my code -- OpenGL is very picky like that.

Keen readers will observe that I make no attempt to ensure the texture dimensions are powers of 2. This is because all Macs capable of running OS X 10.5 support the GL_ARB_texture_non_power_of_two, so at least in this case, you don't have to worry about padding the bitmap out anymore.

I wrap all of this in a display list so that a previously-rendered piece of text can be displayed anywhere by performing a translation and calling the display list. Here is the code that does this:
: make-line-display-list ( rendered-line texture -- dlist )
white gl-color
GL_TEXTURE_2D swap glBindTexture
init-texture rect-texture-coords
dim>> fill-rect-vertices (gl-fill-rect)
GL_TEXTURE_2D 0 glBindTexture
] do-enabled-client-state
] do-attribs
] do-enabled
] make-dlist ;

All the rest

I won't discuss the details of the remaining parts of the font rendering implementation here, but briefly, they are:
  • A texture cache where frequently-used textures are held to avoid rendering the same text over and over again; text expires from the cache if its not used for five consecutive rendering runs
  • Code for measuring text, converting logical positions to screen co-ordinates, and vice versa
  • The high-level platform-independent text API in Factor's UI, and the mapping between this API and the abstractions I described here
  • The FreeType implementation of the high-level layer, for use on other platforms

If you want to dig deeper, feel free to peruse the core-text, core-graphics, ui.text and ui.text.core-text vocabularies in the new_ui branch in the GIT repository.

Appendix: alien destructors

For background about Factor's destructors, see the following:

With destructors, you can wrap some code in a with-destructors scope, and call &dispose on external resources; when the with-destructors form returns, dispose is called on all the resources marked for disposal with &dispose. This is very handy for working with such things as Factor streams, database connections, and so on, but its a problem for bare C references; they're all instances of alien, and there's no meaningful dispose method that works on all of them. Our idiom until recently has been to wrap the alien in a one-tuple slot, which defines a custom dispose method, but this is just boilerplate. Factor supports meta-programming, and this could be taken care of with parsing words and macros; but the functors abstraction I implemented a while ago, which basically gives you a very easy way to implement a certain type of parsing word, works really well here, and it becomes completely trivial to abstract out the creation of the one-slot class with a custom dispose method. The alien.destructors vocabulary defines a single parsing word, DESTRUCTOR:. Here is an example of its use from the core-foundation vocabulary; first we define an FFI binding to the Core Foundation CFRelease function, then we define it to be a destructor:
FUNCTION: void CFRelease ( CFTypeRef cf ) ;


The DESTRUCTOR: word defines two new words for us; &CFRelease and |CFRelease. You can see examples of their usage previously in this post. Here is the implementation of DESTRUCTOR: -- I won't go into the details, but you can see it looks like a template of what you'd write out every time if you didn't have the abstraction:
FUNCTOR: define-destructor ( F -- )

F-destructor DEFINES ${F}-destructor
<F-destructor> DEFINES <${F}-destructor>


TUPLE: F-destructor alien disposed ;

: <F-destructor> ( alien -- destructor ) f F-destructor boa ; inline

M: F-destructor dispose* alien>> F ;

: &F ( alien -- alien ) dup <F-destructor> &dispose drop ; inline

: |F ( alien -- alien ) dup <F-destructor> |dispose drop ; inline


: DESTRUCTOR: scan-word define-destructor ; parsing

Functors are just a hack and everything they do can be achieved using parsing words, but the code with functors, in the cases that they can handle, is simpler and more declarative.

Wednesday, January 21, 2009

Unicode font rendering in the Factor UI on Mac OS X

For the last month and a half, I've been working hard on a big top-to-bottom overhaul of the UI. The developer tools are getting a big facelift, and so is the gadget substrate and backend rendering code. When it is closer to completion, I will blog more about the new features and post a screencast showing off the new capabilities.

For now, here is a screenshot demonstrating just one of the many new features and improvements: Unicode font rendering on Mac OS X, with a cross-platform Factor API on top, and Apple's Core Text API under the covers:

Notice the Georgian text in the title bar. Non-ASCII window titles have already worked for a long time.

Throwing together bindings for Core Text and Core Graphics was pretty easy using Factor's FFI. Our destructors feature really helps bridge the gap between Factor's GC and C's manual memory management.

On Windows and X11, I'm still using a portable FreeType backend for ASCII-only text, but I plan on replacing that with Microsoft's UniScribe (on Windows) and Pango (on X11). This will free Factor's UI from external dependencies other than those shipped with the operating system; having to lug the FreeType library around was annoying for deployment.

Support for input methods is next, and together with Daniel Ehrenberg's excellent pure-Factor Unicode library, this will give Factor excellent support for international applications that works identically across platforms. I don't think any of the other "new" languages, which have had a similar amount of developer effort and time put into it as Factor, come close in this respect.

Monday, January 12, 2009

Bug in Intel video driver version 2.4 for Linux fixed in version 2.5

If you are running Linux with an Intel graphics card, you might get texture corruption issues, depending on what version of the driver you are using. The problem appears to be fixed in version 2.5. If you're using Ubuntu, here are some instructions for updating. The problem looks like this:

Sunday, January 11, 2009

Screencast: editing Factor code with Emacs and FUEL

For the last few months, Jose A. Ortega Ruiz has been working on the Factor mode for Emacs, which was originally written by Eduardo Cavazos. Jose has extended the mode considerably, and renamed it to FUEL, "Factor's Ultimate Emacs Library". FUEL extends Eduardo's Factor mode with features inspired by SLIME. Among them, we have:
  • Syntax highlighting, auto indent, s-expression navigation
  • Word completion
  • Jumping to definitions
  • Listing word usages
  • Some refactoring commands: extract word and extract vocab
  • Lots more; see the documentation
Some of FUEL's features duplicate functionality found in the Factor UI, but really, they're complementary; the backend code is the same (it's all written in Factor, and FUEL interogates a running Factor instance using reflection APIs).

I decided to record a screencast to demonstrate some of the more interesting capabilities offered by FUEL. You can watch it at Make sure to full-screen it so you can read the text on my screen.

Formerly I would use jEdit to edit Factor code. Once FUEL started to become more mature, I decided to bite the bullet and learn Emacs. So indeed, after writing my own text editor, and using it for almost 11 years, I've decided to switch, because of FUEL. That's how compelling FUEL is! FUEL's M-. feature, which jumps to the word at the caret, and its automatic display of stack effects in the status bar, are huge productivity boosters in themselves, but FUEL offers much more than that.

jEdit used to have something like FUEL, in the form of the now-obsolete jEdit Factor plugin. I could have continued maintaining this plugin, but really, I don't want to run any Java applications on my computer, if possible. I'm disappointed by Sun's latest antics (abandoning Swing for the last 7 years or so; bundling Yahoo and MSN adware with the Windows JRE), and Apple's half-hearted support for Java (really it's Sun's fault; why don't they develop the Mac OS X JRE?). Now that I've switched to Emacs and FUEL for editing Factor code, I can boycott Java entirely and never run a Java application again.

Of course, eventually I want to write a syntax-aware editor for Factor, in Factor. Until then, FUEL provides an adequate approximation.

Saturday, January 10, 2009

Java 7: too little, too late

So I'm reading an article about Java 7, and the new features just don't seem very compelling to me.

For example, look at this quote from the article:
JSR 203: NIO 2 extends and completes the work started in JDK 1.4 in the original NIO enhancements. The most obvious addition in NIO 2 is a complete overhaul of the file-access APIs. Most developers have used and are aware of the long list of shortcomings:
  • No support for symbolic links
  • No support for straightforward or atomic move and copy operations
  • Cumbersome APIs for walking and filtering directories
  • Very limited support for access to permissions and file attributes
  • No API for watching directories or files

Funny how Factor's IO library already has all of those features, with a cross-platform API on top, and more. Unlike language implementors who decided to host their language on the JVM, I don't have to sit around twiddling my thumbs while Sun implements basic functionality.

And then there's this quote from the article:
Given shrinking resources at Sun, the number of significant JSRs, and pressure to release, it seems that it's unlikely that closures will be included in the Java SE 7 release.

"Shrinking resources" indeed. During the dot-dom era, Sun pumped massive resources into Java, hiring an army of mediocre programmers to crank out feature after feature, half-assed API after half-assed API. Now all those bad design decisions, complete disregard for maintainability, and backwards compatibility is really weighing them down, and they're having trouble moving forward. This is why newer, better-designed languages and language implementations are able to evolve so much quicker than Java.

It's 2009 and only now they're adding APIs to work with symbolic links, and move and copy files? And no closures? What a joke.

Friday, January 02, 2009

Advantages of concatenative and stack-based languages

Factor has many features that make it stand out -- I've blogged extensively about some of these, such as interactive development, the compiler, various libraries, application deployment, and so on. In those respects, I think it is pretty clear that Factor is competitive, or even ahead of, existing languages.

However, I don't think I've ever written about the one aspect of Factor that many programmers find the most striking: the fact that it is concatenative language. For example, a recent blog post titled 10 programming languages worth checking out, the author decided to add Factor to the list not because of any of the features I mentioned in the first paragraph, but because it uses a stack to pass parameters; that alone make it "worth checking out" in his view.

Because many people have asked me the question, "why the stack", I've finally decided to put together an overview of what these languages, and especially Factor, are all about. I've mentioned most of the material below in presentations, comments on forums and blogs, mailing list posts, and on IRC.

Indeed, I believe that the underlying paradigm in Factor is the "secret weapon" which has allowed such a small community to build up a robust language implementation with a large collection of very capable libraries in only 5 years. There are inherent productivity advantages here, and we probably would have given up long ago if Factor was harder to use than existing languages.

Originally I was going to write a series of blog posts, but I decided to put it in the wiki instead, under the Concatenative language article there.

On another note, I haven't done much hacking for the last two weeks because of a vacation; our git repository has been pretty quiet of late. On January 5th I go back to work, and there is good stuff coming up for the UI developer tools, compiler, and web framework.