We Have The Construction!

Github Archive Now Available

2012-01-27T23:47:00.001-08:00

Based on popular request, I've published my full build tree for generating the WinCairoRequirements bundle needed to build the WinCairo port.

You can access it at Github.

It contains build targets for:

VS2005, which is needed for compatibility with the main WebKit development effort.
VS2008, which I used on my last shipping application, and
VS2010 for new development.

While I was at it, I also published CallJS, an example application exercising theWebKit framework, and provides an example of doing things like communicating between a WinApi application and the WebKit and JavaScriptCore instances.

Hopefully, these resources will make it easier for people to get started using, and perhaps contributing, to WebKit.

If you do try them out, please drop me a line and let me know if you had any difficulties.

Tests Active

2011-10-08T19:30:00.001-07:00

Looking back over this blog, I see that it was around a year ago that I got the initial WinCairo buildbot running. I'm very pleased to announce that I have gotten ahold of a much more powerful machine, and am now able to run a full build and tests in slightly under an hour -- a huge improvement over the old hardware which took over two hours just to build the software! In fact, the image on the left shows the new build box, but unfortunately the flash on my iPhone wiped out the machine's label ("Webkit") on the rack faceplate.

This is a big step, because we can now track regressions and gauge correctness compared to the other platforms. Up to now, testing has largely consisted of periodic manual runs of the test suite, and a separate set of high-level tests run as part of a larger application. This was not ideal, because it was easy for low-level functions in WebKit that I rarely use to be broken and missed.

All is not perfect, of course. Although over 12,000 tests now run (successfully) with each build, that is effectively two thirds of the full test suite. Most of the tests I have disabled are due to small differences in the output layout. I'm trying to understand why these differences exist, but I suspect many of them simply reflect small differences in Cairo compared to the CoreGraphics rendering layer.

If any of you lurkers are interested in helping out, trying out some of the tests I have disabled and figuring out why they fail would be a huge help!

Tests Active

2011-10-08T19:30:00.000-07:00

Looking back over this blog, I see that it was around a year ago that I got the initial WinCairo buildbot running. I'm very pleased to announce that I have gotten ahold of a much more powerful machine, and am now able to run a full build and tests in slightly under an hour -- a huge improvement over the old hardware which took over two hours just to build the software!

This is a big step, because we can now track regressions and gauge correctness compared to the other platforms. Up to now, testing has largely consisted of periodic manual runs of the test suite, and a separate set of high-level tests run as part of a larger application. This was not ideal, because it was easy for low-level functions in WebKit that I rarely use to be broken and missed.

All is not perfect, of course. Although over 12,000 tests now run (successfully) with each build, that is effectively two thirds of the full test suite. Most of the tests I have disabled are due to small differences in the output layout. I'm trying to understand why these differences exist, but I suspect many of them simply reflect small differences in Cairo compared to the CoreGraphics rendering layer.

If any of you lurkers are interested in helping out, trying out some of the tests I have disabled and figuring out why they fail would be a huge help!

WinCairoRequirements Sources Archive

2011-07-05T12:35:00.000-07:00

I've posted the 80 MB source archive of the requirements needed to build the WinCairo port of WebKit.

Note that you do NOT need these sources unless you plan on building them yourself or wish to archive the source code for these modules. The binaries are always present in the WinCairoRequirements.zip file, which is downloaded and unzipped to the proper place when you execute the update-webkit --wincairo command.

Updated WebKit SDK (@r89864)

2011-06-27T21:27:00.000-07:00

I have updated the WebKitSDK to correspond to SVN revision r8984.

Major changes in this revision:
* JavaScript engine improvements.
* Rendering improvements.
* New 'Transparent Web View' support.
* General performance and memory use improvements.

This ZIP file also contains updated versions of Zlib, OpenSSL, cURL, and OpenCFLite.

Note that I have stopped statically linking Cairo; I'm starting to integrate some more recent Cairo updates (working towards some new rendering features), and wanted to be able to update it incrementally as changes are made.

This package contains the same Cairo library (in DLL form) as used in previous versions.

As usual, please let me know if you encounter any problems with this build.

[Update] I forgot to include zlib1.dll! Fixed in the revised zip file.

An Unseasonable Snowfall

2011-04-29T23:54:00.000-07:00

A year or two ago I ported the Cocoa "CallJS" application to MFC for use with WebKit. The only feedback I ever got on the topic was a complaint that it would not build under the Visual Studio Express software many people used.

After seeing another few requests on the webkit-help mailing list for information on calling JavaScript from C++ (and vice-versa), I decided to dust off the old program and convert it to pure WINAPI calls so that VS Express would work with it.

Since my beloved Layered Window patches finally landed in WebKit, I also incorporated a transparent WebKit view floating over the main application window. Because I suck at art, I ~~stole~~ appropriated the Let It Snow animation example to give the transparent layer something to do.

Want to see what it looks like?

Transparent Windows

2011-04-18T17:13:00.001-07:00

Recently I've been hard at work reviving some old code originally written by the Appcelerator team. I tried to improve on the initial design by building the feature into the WebKit library itself. This reduces the need for clients of this functionality to have to figure out exactly how to construct their windows, and perform the layered window drawing logic.

With this patch in place, you can create 'chromeless' WebKit windows that you can fill with whatever rendering content you like. For example, you can tell WebKit to fill your entire desktop with a WebView that points at a copy of the awesome Snowstack implementation. Of course, to get a transparent view you have to be able to set the background color to something semi- (or fully) transparent. Consequently, I had to mirror Snowstack on a server I had access to so I could adjust the background. The screenshot shows the result of this effort.

I'm hoping to get the changes approved and landed soon, once I've created a decent example program that can be landed in the tree.

Oh, by the way, this change works on the Apple version of WebKit as well!

Towards a Simpler WinCairo Build

2011-04-12T19:47:00.000-07:00

For the past couple of years, anyone interested in trying to build the WinCairo port of WebKit had to track down a number of support libraries, place them in their development environment's include (and link search) paths, and then cross their fingers and hope everything built.

To make things a little easier, I wrapped up the libraries and headers I use for building and posted them as a zip file on my .Mac account. This made things a little easier, but you still had to figure out where to drop the files and figure out if I had secretly updated my 'requirements.zip' file without telling anyone. Not ideal.

A couple of days ago, while trolling through the open review queue, I ran across a Bug filed by Carl Lobo, which automated the task of downloading the requirements file when running build-webkit --wincairo. This was a huge improvement!

Today, I hijacked Carl's changes and railroaded the patch through the review process (making a few modifications along the way):

I renamed my requirements file WinCairoRequirements.zip.

I added a timestamp file, so that build-webkit --wincairo can check to see if the file changed, and download it if necessary.

I propagated Carl's changes to update-webkit, so that now by adding the --wincairo argument it will update the WinCairoRequirements file.

I'm really excited about this update. If you've been wanting to try out the WinCairo port of WebKit, this would be a great time to try it out. I'd love to hear your experiences!

Core Builder!

2011-03-30T19:50:00.000-07:00

Several months ago I posted to report that I had built and activated a WebKit BuildSlave. I'm pleased to report that our Build Slave is all grown up, and has been promoted to Core Builder status.

Fundamentally, nothing has really changed. It's just as easy for an errant check-in to cause the build to break. But on a social level this promotion is quite meaningful. It indicates that we (alas, the "Royal We") claim that the WinCairo port is an active, viable version of WebKit. Core builder status also helps apply social pressure for external developers to keep the WinCairo build green, as it is very poor form to break the build.

Now that we've reached this new lofty tier of project visibility, I'm hopeful that we will see fewer build failures. We will almost certainly get better feedback when the build is failing, and probably get help from parties interested in keeping the tree green.

I'm very excited about this change, and I hope you'll keep an eye on things to see how they progress!

At Last: A BuildBot!

2010-10-06T21:32:00.000-07:00

It's taken me nearly two years to get all of the stars in alignment, but I am very excited to announce that we now have a WinCairo build bot up and running. If you look at the WebKit Buildbot, you can find the new entry for it (and it's even currently green!) Better look quick -- who knows how long it will last!

I have not activated any test phases yet, since none of the pixel level tests have been generated, but this will come in time (hint, hint Mr. Martin Robinson...)

The whole process of getting things up and running was a rather arduous process:

First you have to get your port building from the build-webkit command line.

Next, you have to set up a Buildbot. You only really need the slave portion, but to test your configuration you will need to set up a local buildbot master.

Note: Here you will (perhaps) learn that the environment used by the pdevenv utility does not match the build environment you've been using to debug build problems.

Once you confirm your build works when executed by the buildbot, you need to get a patch landed in the WebKit archive that defines your build slave.

On the happy day your buildbot configuration lands, you have to start harassing _wms to schedule a restart of the master buildbot.

Being a great guy, _wms will get a restart scheduled, at which point a veritable avalanche of source changes will be suddenly kicked loose by the commit-queue. The restart will take upwards of 48 hours before things settle down enough for the service to restart.

But eventually, if you persevere, your bot will be live and GREEN. And with this work, comes some wonderful benefits as it will be much easier for the other WebKit folks to avoid breaks. And I can even configure the system to send me an e-mail when my build fails. So cool!

Let's hope that this will usher in a grand new era of clean WinCairo builds and better (and faster) code production!

Updated WebKit SDK (@r68150)

2010-10-02T22:34:00.000-07:00

I have updated the WebKitSDK to correspond to SVN revision r59726.
Major changes in this revision:
* More HTML5/CSS3/DOM3 compliance
* Web inspector improvements
* More Accessibility updates
* JavaScript engine improvements
* General rendering improvements
* SVG enhancements
* XMLHttpRequest Changes:
* Plugin support
* Better Windows 7 support.
* General performance and memory use improvements

As usual, this is a significant amount of new logic so there may be some regressions. Please let me know if you encounter any problems!

Side note: I still work primarily on Vista. If you have interest in better testing and support for Windows 7, and happen to have a spare Windows 7 license available that you could donate, please contact me.

I've also updated my build requirements zipfile.

Spelling!

2010-07-09T22:42:00.000-07:00

At long last we can perform spell checking in WinCairo! While the plumbing has long been in place to allow a spelling engine to be called from within the WebKit code, the WinCairo port has not had a working implementation of the 'squiggle' logic needed to indicate misspelled words.

This ~~functionality~~ has been landed in r62388, with a small correction a few days later.

With proper delegate use, you can hook up Hunspell and begin checking spelling and getting real-time suggestions. What could be more fun?

WWDC

2010-06-13T21:04:00.001-07:00

I spent the past week at Apple's amazing WWDC conference. I was primarily attending to learn about iPhone and iPad development, but it was also an opportunity to meet with the core Apple WebKit team. It was great putting faces to the various IRC nicks and e-mail addresses that I've been interacting with over the past couple of years. I learned some important facts, such as that Darin Adler is renovating his house, that mitzpettel is a baked goods aficionado, and that bdash is a really nice guy in person (despite his gruff online persona). Only Maciej managed to elude my stalking skills.

The guys were very accommodating, humoring me by sitting politely through a demonstration of my incomplete application so I could show off its multiple web view. Unfortunately, some nuance of the WWDC network preventing me from accessing my company's VPN, so I couldn't retrieve the current beta build to show off the current artwork and CSS. When we get closer to release I hope to post some videos of it.

The only disappointment (for me) was having to miss the annual WebKit party, attending instead a mandatory Oakland A's game with my Father-in-law. From all reports, it was a great evening (the party, not the A's game). Oh well -- next year!

If you haven't been to WWDC, I highly recommend it. It's a great way to get excited by the possibilities provided by the various Apple platforms, as well as an opportunity to be inspired by the great work being done by others in the industry. Arguably, the most valuable aspect of the conference is getting direct one-on-one contact with the various engineering teams at Apple. You can literally drop in with you buggy program, sit down with your laptop and a few engineers, and walk out of there with a solution.

Jake

2010-06-03T13:16:00.001-07:00

Jake in a box, originally uploaded by bfulgham.

Years ago, when the internet was largely unknown, USENET was king, and people still used America Online I was living in Virginia, near the headquarters of AOL. My wife and I were newlyweds, and decided to do some training for parenthood by getting a dog.

Answering an ad in the newspaper, we visited the cluttered basement of a home in nearby Vienna, VA. The room was filled with Cockatiel cages, and carpeted in old newspaper. The air was thick with dust, down, and the scent of air freshener that nearly masked the smell of wet dogs. This is where we first met Jake, who was to be our loyal companion for the next 15 and a half years.

Even then, Jake was his own person. He could have cared less about the two goofy visitors clamoring for his attention. He was focused on the lid from an air freshener spray, rolling it around the room and pouncing on it. I didn't think he was going to be people-focused enough, but my wife liked that he was tiny and I liked the fact that he was only $125 (a $25 savings!) because he was a runt. At six weeks of age he weighed only a pound and a half.

It turned out that he also came with a free case of Coccidia. Our new vet (Dr. Stork) prescribed a banana flavored medicine, which I would give to Jake each morning. I kept the fact that Jake loved the medicine a secret from my wife, hoping to get Jake to like me better (a plan which seemed to have largely worked.)

When he was very small, I could put him in the pocket of my coat, cinch the pocket loosely around his neck, and go to the grocery store. He greatly enjoyed visiting the deli department, as I recall.

Jake was wildly energetic, full of great humor, and always had a smile on his face. I remember when we got a second dog to be his playmate, Jake would climb on the couch and kick the throw pillows onto Dylan, who was too uncoordinated to climb up. I caught him doing this once, and I swear he winked at me while doing it.

Even in the most adverse conditions he maintained a positive attitude, and only ever presented signs of depression once when we lived with my parents for a number of months. Apparently he was fed up, and wanted his own house!

He was very friendly, and loved kids. When we finally brought a few home of our own, a new world was opened to him. He became adept at stealing food from the kids' high chairs, and soon grew fat (topping in at nearly 22 pounds at the high point.) He loved the chaos that two toddlers brings to a home.

As the kids and Jake grew older, he began to slow down. This was a great time for my wife and me, since he had never been capable of sitting still. At last we could hold him while watching TV or reading, and I soon learned that he had an almost magical power to put me to sleep if he sat in my lap.

Eventually his hearing faded, and then his eyes clouded over with cataracts. He stayed remarkably upbeat, memorizing the layout of the house, and never complaining about the fairly frequent knocks he took to the head from walking into furniture or walls. I took to carrying him outside to a safe spot in the yard, and switched him to a canned dog food when his normal dog food seemed to hurt his teeth. Even in his declining days he stayed a regal presence, barking to announce he was ready to be set on the couch, or ready to be put onto the bed, or ready to go outside.

Eventually he began to lose weight and became lethargic. He had a harder time finding his water, and seemed to frequently get lost in the house. The Vet was not sure what the problem was, aside from advanced age, and we did not wish to subject him to invasive testing that would likely be unhelpful, but would certainly cause pain.

This memorial weekend, we had to say farewell to Jake. He passed into the next life surrounded by his family (and Grandparents). He will be sorely missed.

Updated WebKit SDK (@r59726)

2010-05-20T09:11:00.000-07:00

I have updated the WebKitSDK to correspond to SVN revision r59726.
Major changes in this revision:
* More HTML5/CSS3/DOM3 compliance
- Updates to match current specification properties and naming.
- DOM3 XMLHttpRequest improvements
- HTML5 Database support improvements
//- Fill modes for CSS Animations
* Better multicolumn layout support
* Web inspector improvements
* More Accessibility updates
* JavaScript engine improvements (math, string optimizations)
* SVG enhancements
- feDiffuseLighting
- SVGFont vkern/hkern support
- More efficient drawing
* Better Windows 7 support.
* General performance and memory use improvements

As usual, this is a significant amount of new logic so there may be some regressions. Please let me know if you encounter any problems!

Side note: I still work primarily on Vista. If you have interest in better testing and support for Windows 7, and happen to have a spare Windows 7 license available that you could donate, please contact me.

Successful Print Preview

2010-05-18T17:15:00.000-07:00

I have been wrestling with WebKit for some time trying to get a proper 'Print Preview' thumbnail. Unfortunately, my various attempts generally resulted in good looking text, but offset and strangely scaled graphical elements, or vice versa.

During this process I learned that Windows (or specifically, MFC) implements a print preview by generating a device context for the selected printer, and a 'sister' display context that is configured to mimic the printed surface such that drawing commands intended for the full-sized printer surface are rendered into a small preview display.

This is accomplished by setting the mapping mode of the display context to MM_ANISOTROPIC, and setting a Window Extent (for the full size of the printed page) and a View Extent for the size of the display.

Unfortunately, if you pass a context created with these kinds of settings to Cairo, it improperly scales textual elements. In the attached image you can see that text written with cairo_show_text and cairo_show_glyphs get drawn at the same size no matter how the context is scaled. Text written with cairo_text_path and other graphical elements (e.g., the arc in the example) are scaled properly.

Thankfully, there is a workaround. After writing a test program to illustrate the issue, I discovered through some trial and error that everything works fine if I reset the preview window mapping mode to MM_TEXT, reset the HDC's WorldTransform to unity, and set the Cairo context's CTM to be scaled to match the ratio of the Viewport Extents to the Window Extents.

Voila! Properly scaled previews.

This change is implemented in ~~Bug 39329~~, which will hopefully be committed to the archive soon.

WinLauncher Sources

2010-04-07T20:59:00.000-07:00

A few people have contacted me asking about the WinLauncher program I use for testing. I didn't create this useful application; it's actually part of the standard WebKit source tree.

In fact, if you use the Visual Studio project to build WebKit (either the official Apple port, or my WinCairo variant), you can just set the WinLauncher project as the "Set as StartUp Project". Now, when you press the 'Run' button, WinLauncher will build and launch.

This can be extremely useful for testing various aspects of the framework, as well as playing with the official Apple build to see what it does under different conditions. It's also a lot easier to start and debug than the full-blown Safari process.

Over the last year or so I've worked to add a few new features, mainly related to stuff I was trying to get working under WinCairo. Most recently, that means that printing is partially supported.

Updated WebKit SDK (@r57152)

2010-04-06T13:48:00.000-07:00

I have updated the WebKitSDK to correspond to SVN revision r57152.
Major changes in this revision:
* More HTML5/CSS3/DOM3 compliance
- DOM3 custom events
- DOM3 XMLHttpRequest improvements
- element
- lang attribute added to

Printing!

2010-01-11T20:19:00.000-08:00

I've spent the better part of a week trying to understand why plugins were not printing properly from within WebKit. While this is probably not a critical topic for most WebKit users, I am using the plugin infrastructure to allow WebKit to host a UI widget from my main application. The really cool thing about this design is that I can call up a WebKit view in my main program, display some custom UI elements hosted by my main program inside the WebView, and even have the plugin (launched by WebKit) access internal objects in my running application!

Unfortunately, the whole grand vision teetered on the brink of failure, because I could not get the hosted widgets to appear in a printed copy of my WebKit instance. This was a real annoyance, after I'd *already* spent the better part of a week getting the main printing infrastructure working under WinCairo in the first place.

The plugins, which displayed so nicely on-screen, didn't show up at all in hard copy -- even when there was nothing else being displayed on the screen. Fearing some kind of strange interaction between the third-part binary I was embedding in my plugin, I wrote an extremely simple example NPAPI plugin (http://files.me.com/bfulgham/o7vlik), which renders a box in the region supplied by the calling web page. But again, the plugin rendered perfectly on-screen, but did not display in the printout.

It wasn't until I monkeyed with the margin, header, and footer that I saw tiny little rectangles being drawn in the upper-left-hand corner of the page. The plugins were getting drawn, but they were extremely tiny, and they were in the wrong location.

Searching through the code, I found the following in PluginViewWin:


    // The plugin expects the DC to be in client coordinates, so we translate
    // the DC to make that so.
    XFORM transform;
    transform.eDx = locationInWindow.x();
    transform.eDy = locationInWindow.y();

Of course! This was ignoring the scaling effects of printing to a page (~1600 dpi) versus on-screen (~96 dpi). I also discovered that header and footer extents were being ignored, which was why my tiny rectangles were getting drawn in the upper-left-hand corner of the page -- and only if I made the header big enough that the web page did not cover the region.

So, I happily filed ~~Bug 32909~~ and figured that would be the end of it.

Only, it wasn't. Once I corrected the scaling and translation issues, I found that although my on-screen display was correct, my print output continued to omit the plugin. Even worse, I could sometimes see the edges of rectangles peeking from around rendered portions of the page (when margin or header/footer sizes were manipulated). In at least one case, I was able to click on the edge of the rectangle in PDF Viewer, copy the object (hidden behind a blank area), and paste the *full* rectangle into another document. WTF?

After more trial and error, I discovered that changing the rendering type used for the page from cairo_win32_printing_surface_t to cairo_win32_surface_t my plugins *did* display properly! The only problem, was that my printing operations went from using very little memory, to using over 100 MB of RAM to print.

A bit of correspondence with the Cairo team helped me see that the problem was due to the dual nature of the drawing operations. The plugin is handed a Windows device context handle to the printing surface. But the majority of the drawing events in WinCairo are done against a cairo_t context. When this context is backed by a cairo_win32_printing_surface_t, drawing operations become slightly order-dependent. Drawing directly to the printer context (as my plugin was doing) would get drawn to the printer device context immediately. Later, the entire Cairo surface was shipped to the printer, overwriting (actually *overlaying*) the previous drawing operations.

And that explained it: when this multiple layered printing context was played back as a PDF file, the plugin contents were at the lowest Z-level, but the main page rendering was sitting on top of them making it appear as though the plugin was not rendering at all.

At last, a workable solution was found. Prior to the plugin drawing, I had to add the following logic:


#if PLATFORM(CAIRO)
    // Must flush drawings up to this point to the backing metafile, otherwise the
    // plugin region will be overwritten with any clear regions specified in the
    // cairo-controlled portions of the rendering.
    PlatformGraphicsContext* ctx = context->platformContext();
    cairo_show_page(ctx);
#endif

Now, each time I go to print a plugin, the Cairo context is flushed to the printing surface. Then the plugin is drawn (on *top* of this existing rendering), then control returns to WebKit for more Cairo rendering, followed by more flushes (if more plugins are being drawn), and so forth.

In this way, I was finally able to get a correct print version of my plugin-containing page. See ~~Bug 33022~~.

I also pushed some general printing adjustments to the WInLauncher program to help other see the correct steps to take to execute printing. ~~Bug 33537~~.

ClearType Italics

2009-12-28T19:50:00.000-08:00

A commenter recommended that we double-check that italic characters render properly, as these are frequently clipped. You can see from this attached image that italic characters are rendered properly.

Proper ClearType Support!

2009-12-22T21:14:00.000-08:00

Several months ago, the Appcelerator team noticed that the WinCairo build of WebKit failed to properly render ClearType glyphs at the end of lines. At the time, frankly, I was just pleased to have rendering of *anything* on the screen, and basically brushed their report aside as perfectionism better addressed at a later date.

Well, today that 'later date' finally arrived. After chatting with Martin Robinson about the issue, and finally recognizing the very poor font quality when rendering with CAIRO_ANTIALIAS_GRAY (rather than the desired CAIRO_ANTIALIAS_SUBPIXEL) I did a bit of Googling. It turns out some similar issues were discussed in early 2006 (and apparently resolved) on the cairo development list. After mailing the Cairo guys to ask about the status of this particular bug, I received a message from The_Decryptor (my new best friend), pointing out that Firefox has encountered the same problem back in February, and had resolved it.

The underlying problem is perhaps best explained by the comment on the patch:

The bounding box reported by Windows supposedly contains the glyph's "black" area; however, antialiasing (especially with ClearType) means that the actual image that needs to be rendered may "bleed" into the adjacent pixels, mainly on the right side. To avoid clipping the glyphs when drawn by _cairo_surface_fallback_show_glyphs, for example, or other code that uses glyph extents to determine the area to update, we add a pixel of "slop" to left side of the nominal "black" area returned by GDI, and two pixels to the right (as tests show some glyphs bleed into this column).

How well does this work? The attached image shows the difference. The left-hand side shows the strange end-of-line behavior of our stock Cairo build. It's especially pronounced in the numeral '1', but can be seen to a lesser extent in the lower-case 'j'. The right-hand side is the patched build, which is nearly identical to Safari on my installation.

Many thanks to Marshall Culpepper and Martin Robinson for coming up with the initial workaround, and to The_Decryptor and the Firefox team for their careful and thorough review of the issue, and the solution!

Updated Requirements

2009-11-30T10:27:00.000-08:00

If someone wanted to build and run WebKit using Visual Studio 2008, they might find this requirements_2008.zip file to be useful.

Pixar!

2009-11-25T19:24:00.001-08:00

Several months ago, I received an excited phone call from my father-in-law.

"We won the auction!" he shouted.

What auction?

"An auction for a tour of Pixar! You and David are going to go!"

Apparently, my wife's parents (who obviously have a much different relationship with money than I do) had attended a charity auction for a school affiliated with their Church, and had decided they were going to land the Pixar tour. Even more amazing, they were planning on sending me and my brother-in-law for the visit. To be honest, I was pretty sure that schedule, fate, or a myriad of other events would conspire to prevent me from attending, but today the stars were apparently in alignment, for I did indeed take my tour of Pixar.

The nice folks at Pixar asked us to refrain from commenting on anything particularly interesting we might notice, but they allowed us to take some photos in specific areas (above is a photo of David and me hamming it up with the Incredibles.)

The place is incredible, with a huge central concourse designed to encourage staff to mingle (example: all the bathrooms and snack areas are here.) Much of the 'public' spaces are decorated with cool conceptual art from the current release ('Up" in this case), although I did find some neat story boards from the Incredibles (my personal favorite from their film catalog.)

Being a computational analysis guy, I was really interested in seeing their famous render farm. I remember back in the dark, distant, early days of Linux (back when Bruce Perens was still deeply involved in Debian GNU/Linux) how exciting it was to learn that Linux was being used to run those rendering nodes. So it was with a bit of a lump in my throat that I stood besides those now-vast computing resources to get a quick picture under the neon lights.

Later, they treated us to a great lunch in their cafeteria. It reminded me a bit of Amgen, in that they had created a great immersive environment aimed at keeping people in-house and interacting with each other.

There's something really inspiring about being surrounded by people who are so passionate about what they are doing. Clearly, the people of Pixar love their work -- a fact that shines through the finished products in a way you don't see in many other production houses.

Another element that deeply impressed me was the wide array of talents arrayed in their organization. From sculptors, to sketch artists, to the animators whose final products we see on the big screen; all of them make huge contributions to the whole.

All in all, it was a really exciting day. While I didn't get to peek behind the veil into the secret Pixar underworld where the magic really happens, just getting to absorb a bit of the energy and atmosphere of the place was totally worth the trip.

I actually found myself feeling a little depressed as I left; sort of like the let-down after the holidays when the fun is all over, and all that's left are the bills to pay. I envy the team at Pixar and their creative endeavors, and their ability to reach out and touch people's lives.

One of the things that I love about my company is that our products are involved in a vast range of important work, ranging from alternative fuels, to nanotech, to cancer and HIV research. So it's a little funny (and probably more than a little misguided) that I was more impressed by meeting a few of the people who brought Up to life, than when I met my first Nobel laureate.

But perhaps that is just the nature of art, in all its forms. Its ability to reach out and touch lives, however briefly, seems far more profound in the moment than the plodding march of research, conducted in the dark recesses of academia (or worse, industry), inching ever onwards into the future.

Apple Store Customer Support Fail

2009-11-24T15:39:00.000-08:00

Over the years I've had a number of Apple computers, iPods, and more recently, iPhones. Based on the support sites I've looked at, I've been quite lucky. I've never experienced a significant hardware failure, the closest being a two-CPU Mac Pro that stopped booting from the first drive bay for some reason, but was perfectly content to boot from the second. No big deal. I even own one of the original G5 iMacs with the exploding capacitors, and have had perfectly good experience with this unit.

Therefore, it was with some embarrassment that I mulled over the non-functioning bluetooth feature on my wife's new (well, two month old) iPhone. Every time you switch the bluetooth setting to "on", it obstinantly flips back to the "off" setting. She had already been to the AT&T store where we purchassed the phone; they ran through the usual steps: (a) reset the phone, (b) restore the phone. After playing with it myself, I noticed that the Wifi features were not working either (not seeing or connecting to the home network, or anywhere else I tried). AT&T basically washed their hands of it, instructing us to visit the Apple store who had the technicians to resolve the problem.

And so today I found myself waiting for an Apple Store employee to take notice of me. I stood near the Genius Bar, watching as an elderly lady was shown how to construct a colorful, yet tasteful newsletter in Pages. Another guy was busy being indoctrinated into the dark arcana of connecting a camera to his Macintosh with a USB cable. Still another person was learning how to post pictures to their iWeb page, and seemed to be deeply suspicious that the 'Genius' was trying to play some sort of trick on them.

I finally flagged down someone walking by. "Can you help me?", I asked. She peered over my shoulder before answering, making sure she wasn't missing out on a potential buyer hovering near the 27" iMacs. "Um, sure."

I walked her through the problem with the phone, and asked if someone could take a look at it? Or maybe could this malfunctioning unit just be swapped out for a working version?

No, and No.

"Did you try resetting the phone?", she asked. Yes.
"Did you try doing a full restore?" Yes.

I explained how the AT&T store had already done that level of troubleshooting, and had sent me to the Apple store to resolve the issue. According to the clerk, however, no one with any troubleshooting skills would even glance at the phone until I made an appointment with the "Genius Bar" staff, first available slot being sometime next week. It didn't matter that the phone was malfunctioning NOW, and that we did not live in the area and were going to be stuck with the malfunctioning phone until we got home.

And so I fumed as I watched one customer having her new iPhone activated (no appointment necessary!), and another person sidle up to the bar to be dazzled with the complexities of red-eye removal, and a flock of Apple staffers swarmed the mulling over a MacBook Pro. Meanwhile I and my in-warranty, malfunctioning iPhone was sent packing.

And so I still fume.

Windows 7 Request

2009-11-21T18:59:00.000-08:00

I've recently started enjoying Windows 7 at the office, and I must say I am very pleased. There is a huge difference in performance (at least running under VMWare) when compared to Vista. In fact, I'm now fairly spoiled and find working in Vista to be absolutely horrible! :-)

I was lucky enough to have someone donate a brand new copy of Windows Vista to me in July, which I've been using for WebKit hacking on the weekends. It's been a huge help, allowing me to finish up the recent -webkit-box-shadow coding and tests. Alas, now that I've been exposed to Windows 7, I'm a bit ruined! I'd love to upgrade to Windows 7, but the current upgrade options are price prohibitive ($119 for an upgrade).

I had hoped that a new Vista box purchased in July would make me eligible for some kind of low price upgrade, but apparently not.

If anyone has an inside scoop on whether there might be cheaper upgrade options, I'd love to hear them. Maybe there's a great deal I'm missing somewhere? Also, since I have a 64-bit machine I'd really like to start using Windows 7 64-bit, as that is where most new development will now be focused.

I run Vista inside a virtual machine (since I also do Mac development). I was curious if anyone has any experience with the VMWare Fusion 3 software, specifically if the performance is noticeably improved on Snow Leopard?

Updated WebKit SDK (@r51062)

2009-11-17T10:23:00.000-08:00

I have updated the WebKitSDK to correspond to SVN revision r51062.
Edit: I stupidly wrote "-webkit-block-shadow", when I meant "-webkit-box-shadow". Thanks to Brad K. for catching this!
Major changes in this revision:
* Support for -webkit-box-shadow.
* General support for Canvas and SVG shadows in Cairo.
* New SVG filter effects: feComposite, feDisplacementMap, feGaussianBlur, feMorphology
* Many SVG rendering improvements, including support for SourceAlpha.
* Support for CSS in Text Kerning and ligature.
* Ability to specify the number of visible lines in -webkit-line-clamp.
* Implement DOM Level 3 composition events.
* Implement CSSOM Range.getClientRects/getBoundingClientRect.
* @import now works in user stylesheets.
* Protect Canvas methods from NaN and Infinity.
* Web Inspector improvements (see this blog entry for details), but many cool new features!
* Some support for -webkit-color-correction (needs further testing on this platform).
* JavaScript engine speed-ups.
* Initial <ruby> support (see this blog entry for details)
* Accessibility improvements.

Clearly this is a significant amount of new logic, so there may be some regressions. Please let me know if you encounter any problems with this new version.

Unfortunately I have not been able to complete work on 3D CSS or WebGL, but stay tuned!

WebKit-Box-Shadows At Last!

2009-11-16T13:06:00.000-08:00

With the landing of ~~Bug 26102~~, all Cairo ports of WebKit can now enjoy full webkit-box-shadow support. Even the multi-line wrapped bordered text show shadows, and the geometric primitives in the "border-radius-big.html" test work.

It took a long time, but the results are definitely worth it!

Functioning Box-Shadow

2009-11-15T21:31:00.000-08:00

It's taken a couple of weeks to get everything put together and landed in the archive, but tonight I finally got webkit-box-shadow functioning in the Cairo build.

This original work was done by Dirk Schulze, who put together most of the shadow logic. There were a few dark corners that had to be examined and resolved before the feature worked properly.

The main problem boiled down to a very simple coding error:


===================================================================
--- GraphicsContextCairo.cpp    (revision 50998)
+++ GraphicsContextCairo.cpp    (working copy)
@@ -1113,7 +1155,7 @@
     cairo_t* cr = m_data->cr;
     double x1, y1, x2, y2;
     cairo_clip_extents(cr, &x1, &y1, &x2, &y2);
-    cairo_rectangle(cr, x1, x2, x2 - x1, y2 - y1);
+    cairo_rectangle(cr, x1, y1, x2 - x1, y2 - y1);
     cairo_rectangle(cr, r.x(), r.y(), r.width(), r.height());
     cairo_fill_rule_t savedFillRule = cairo_get_fill_rule(cr);
     cairo_set_fill_rule(cr, CAIRO_FILL_RULE_EVEN_ODD);

The clipping rectangle was being computed with the right-hand x-position of the current clipping bounds, rather than the y-position per the signature. This was causing the wrong regions of the screen to be clipped out, yielding bizarrely inverted shadow effects (see my last post for an example.)

The final results look pretty good. They generally match the reference implementation (Safari), at least as closely as WinCairo does in other respects (e.g., fonts), as shown in the following overlay:

Once this patch lands, both the Linux and Windows builds of WebKit (Cairo) should begin enjoying most of the cool features provided by webkit-box-shadow. There are still several missing bits of functionality:

Shadows on various geometries don't seem to work (see the "border-radius-big.html" test).

Multi-line shadow effects (see the first series in "basic-shadows.html".)

Strangely, this effect works properly when the shadow is inset in the image.

More Cairo Shadow Work

2009-10-25T23:12:00.000-07:00

WebKit Uber-Hacker Dirk Schulze has been hard at work on his SVG (and general graphic) filter modifications in WebKit. It now seems that drop-shadow support will be very doable using the existing Cairo backend. I am still spending a bit of time porting the work over to the Skia backend, but took a break from that to try out a few of Dirks updates, as well as attempting to correct some WebKit bugs manifested by enabling filters under Windows.

As you can see from the first image, Dirk has gotten very nice looking stroke shadows. Likewise, he has gotten proper image masked shadows working, shown in the second image. I'm quite impressed with the facility's ability to generate a correct shadow from the generated gradient. The page shows a few other examples, but these highlight the main idea.

On the Skia front, I've gotten WebCore building, and a partially functioning WebKit build. However there's still a few things I need to reconnect. It looks like there are a few things that live in "platform/graphics/chromium" that might make more sense in "platform/graphics/skia" (namely ImageBufferData.h, and perhaps some of the font and icon handling.) It also seems like the Skia backend is not as fast as the Cairo backend. This is currently a purely subjective observation, and of course neither backend is currently taking advantage of any hardware acceleration (well, actually Cairo probably is). However, the presence of the compositing and 3D transform support in Skia still make this look like a good choice, especially when attempting to address 29813.

Experiments with Cairo Shadows

2009-10-23T18:01:00.000-07:00

Fellow WebKit hacker Dirk Schulze pointed out to me that the various filter work he was doing would largely provide the functions needed to support box-shadow, and that they might even work on the WinCairo port.

I went ahead and enabled the filter support in WebKit, corrected some build problems caused by the Windows ports' non-use of the filter features until now. I then fired up the resulting binary and tried out the box-shadow support. The screenshot shows the result of this effort -- not so great. The shadows seem to be inverted so that they are cast inside the box. I'm not quite sure what's wrong there!

I also tried out the HTML5 test page he pointed out. As you can see in this second screenshot, the filled areas shadow nicely (pretty much matching what Safari does), but that stroked areas are not shadowed.

Dirk points out that stroke support will be coming, though I'm not sure how quickly that work will be completed.

I'm also enabling the Skia support in WebKit for Windows, and will try these same examples on that build, hopefully this weekend.

Exploring Skia

2009-10-18T20:06:00.000-07:00

Someone recently asked me about the state of shadow support in the Windows Cairo ports of WebKit. As it turns out, the answer is a firm 'it doesn't work.' The reason for this lies in the underlying Cairo graphics library. It doesn't support shadow effects as part of the path rendering. Consequently, programs like Firefox have to generate shadow effects in a separate drawing pass, rather than specifying the 'form' of the shadow and allowing it to be drawn with the main drawing element.

Unfortunately, WebKit's drawing model assumes an underlying graphing system that supports toggling shadows on and off during the rendering of a page. Attempting to shoehorn shadow effects into WebKit (using Cairo) would involve adding extra logic into the drawing system to perform shadow strokes, manually tracking the shadow state somehow and making sure to only perform the extra shadow drawing when needed.

However, another option is to look at using Skia, Google's custom drawing library used on their Chrome browser's port of WebKit. In case you are unfamiliar with the project, Skia is a 2D graphic library for drawing Text, Geometries, and Images. It has built in support for some 3D perspective effects, antialiasing, transparency, and various effects. It has good support for hardware acceleration, and runs on a wide range of hardware platforms.

The bad news is that information about building and running Skia under Windows is pretty non-existant. Clearly it is possible, since all ports of Chrome (including Windows) use Skia, but they project provides no project files (or scons or gyp files), and the gyp files used in the Chromium project don't work outside of the Chromium source tree.

Happily, it's not too hard to figure out which files need to be included in the build, and I found a post by a guy who had gotten things building back in May. Consequently, after a couple of hours of work I was able to get a test application up and running to play with the core library and confirm that it functions properly. I was then able to explore the API a little bit, and confirm that shadows work as expected.

My initial impressions of Skia are very positive. It's nice working with a C++ framework, rather than Cairo's very strict C interface. One puzzling thing about Skia is that the Chromium project has built an extremely useful set of abstractions on top of Skia that seem like they would be well-suited for inclusion. For example, constructing a context for a typical Windows device involves a few steps. But the skia::PlatformCanvasPaint template class takes care of many of these details (as well as cleaning up after itself). Equivalent templates are provided for GTK+ and Mac use. Why leave this in Chromium's private repository? This is useful for everyone!

Over the next week or so I intend to revise the Windows Cairo port to use Skia for its graphical back-end. As a first step, I have updated my requirements.zip file with Skia and it's include files. I have also posted my source tree for Skia, including the Visual Studio solution and my test application (based on WinLauncher).

Updated WebKit SDK (@r48761)

2009-09-25T16:24:00.000-07:00

I have updated the WebKitSDK and requirements.zip to correspond to SVN revision r48761.

Major changes in this revision:
* Support for user stylesheet injection.
* Fixes for various layout/rendering issues.
* Various stability and memory improvements.

This bundle also includes Snow Leopard-era CoreFoundation (CFLite), so please let me know if you encounter any strange problems.

The recent WebKit improvements have corrected a few rendering issues I have experienced in the past, so please give this build a try and let me know if you encounter any problems.

Towards a Reopened CFNetwork

2009-09-09T19:54:00.000-07:00

I think one of the coolest things about Apple, Inc. in their modern incarnation is how great a member of the Open Source community they have become. The fact that the true core of Mac OS, the Darwin BSD implementation, is open and available for all interested parties to review, modify, and learn from is unprecedented in any commercial company I am aware of (at least, none that started off life as a closed-source entity.)

In addition to the core OS components from the Darwin layer, they have contributed improvements to the GNU compiler collection, their amazing WebKit rendering engine, spearheaded the OpenCL initiative, shepherded LLVM into an absolutely amazing compiler system, provided a fantastic local host resolution and routing system (Bonjour), and have contributed the equivalent of millions of dollars in development value to the world at large.

Amazing!

So of course I am writing this post to ask for EVEN MORE.

While working on a native Windows port of WebKit (one unencumbered by proprietary Apple libraries that prohibit redistribution), I started working with CoreFoundation. Along with the many other products I listed above, they opened a large subset of the full CoreFoundation product as CF-Lite, large enough in fact to support running WebKit. They even provided useful examples and arguments for basing a porting strategy around CF-Lite. While they make dumps of the CF-Lite source tree available with each Mac OS release, they did not generally release Linux or Windows sources (which had to be recreated externally). They were also uninterested in opening it up as a WebKit-style open source project, so I started hosting them as the opencflite project on SourceForge. We now fully support Mac OS X (a no-brainer), Windows, and Linux.

CoreFoundation is really neat because of its various data types and operating system abstractions, but it also has a nice modular system for modelling various operating system elements (such as sockets, files, timers) as part of a common runloop, allowing easy asynchronous operations. What is even neater is when you can make use of CoreFoundation's sister framework, CFNetwork, which extends this idea to various network features, such as Host lookup, DNS, and Bonjour.

Up through Mac OS X 10.4 (Tiger), CFNetwork was Open Source software. Alas, those salad days were not to last. With the release of Mac OS X 10.5 (Leopard), CFNetwork closed up and has not been seen in the wild since.

Initially I thought this was a mistake, but after corresponding with Apple's Open Source Product Manager I was informed that it was indeed closed on purpose. After a couple of years of e-mailing various parties associated with the CoreFoundation/CFNetwork product line, I eventually filed rdar://7195752 to request the source to be reopened.

I'm not sure why CFNetwork was closed. But even if it contains Secret Alien Technology, I believe that portions of it could be made available in an Open Source form. In my bug report, I gave an impassioned plea that basically boiled down to the following bullet points:

Increased uptake of CoreFoundation as a general purpose, cross-platform library would facilitate porting of applications to the Mac OS.
Security engineers would be able to audit and validate the source code, increasing overall platform security.
Presence of the source and examples on more development platforms would increase developer mind share, and help extend the visibility and utility of Apple development tools.
More eyes looking at the code would benefit the source base as a whole.

As I conclude in my bug report, I sincerely hope that Apple could be convinced to open source at least portions of CFNetwork so that its use could be extended outside of Apple. This would greatly aid in my efforts to keep the Windows Cairo port of WebKit in sync with the full Apple version, as well as allowing work on the PureDarwin project to continue.

Updated WebKit SDK (@r48212)

2009-09-09T15:13:00.001-07:00

A new WebKit SDK, now new and improved with no multiple-MSVCRT linkage!

This build corresponds to the new requirements.zip I uploaded last night. This should be suitable for running on an end-user system (assuming it has been updated to the current patchlevel for MSVCRT). If the user is running 'Windows Update' this should be a safe assumption.

Please try it out and let me know if you have any problems.

Revised Requirements

2009-09-09T00:18:00.000-07:00

A couple of people who have been working with the WebKitSDK reported problems when running WinLauncher (or other applications based on WebKit.) This was tracked down to the WebKit.dll manifest, which contained dual entries for the Visual Studio runtime:


assemblyIdentity type="win32" name="Microsoft.VC80.CRT" version="8.0.50727.4053" processorArchitecture="x86" publicKeyToken="1fc8b3b9a1e18e3b"

versus


assemblyIdentity type="win32" name="Microsoft.VC80.CRT" version="8.0.50727.762" processorArchitecture="x86" publicKeyToken="1fc8b3b9a1e18e3b"

I initially planned on rebuilding everything using static linkage to the CRT, but quickly found that this would require changing far too many project files (including every existing WebKit project). This quickly got out of hand, so I simply rebuilt against the current set of libraries and called it a day.

You can grab the new requirements here. If you update to current SVN you will have proper link instructions to build a correct set of executables and DLLs.

Updated Build

2009-08-25T23:21:00.001-07:00

I haven't posted much recently, but rest assured I am continuing to work on WebKit in a variety of areas. I found (and corrected) a few issues in the libcurl backend, but you were only likely to run into them if you were working a lot with local files (e.g., stuff in your filesystem) which happens to be something I'm doing a lot of right now. The build now contains corrections for these items.

At the moment, I am working on making it easier for people to use WebKit in projects without having to invest massive amounts of time to figure out how to build the framework themselves.

To that end, I am ditching the separate Debug and Release bundles I had been producing, and am paring things down to a single WebKitSDK zip file, which contains the Debug and Release binaries in a single package. I first had to modify WebKit to emit dll's with "_debug" names so that everything could coexist peacefully in a single bin folder. Now that this is all bundled together, I have removed the original Debug and Release-only zip files from my iDisk.

Now that I've got that finished up, I'm going to focus on shrinking the runtime dependencies by trying to link more of the components statically. I am also trying to get the libcurl support into a background worker to improve throughput.

Of course, if any of you have more pressing concerns, let me know or file a bug!

WebKit (r47083) Build Pushed to iDisk

2009-08-11T23:33:00.000-07:00

I've pushed an updated build (with HTTPS support) to my iDisk. A Debug version is also available at the same place.

If you intend to build WebKit yourself, you should use the current requirements.zip file for your building pleasure. In addition to the SSL-enabled cURL, it contains a slightly updated CFLite that brings it up to Mac OS 10.5.8 level.

These builds include the cURL cacert.pem file in WebKit.resources/certificates/cacert.pem. If you have another certificate bundle you would like to use, just copy it to this location (and name).

WinCairo WebKit Supports SSL

2009-08-11T16:16:00.000-07:00

The guys at Appcelerator figured out how to enable SSL support in their local build of the WinCairo port, and passed along the three-line change required to do so.

Of course, there was a little bit more to it than that, mainly involving figuring out how to (a) build OpenSSL on Windows, (2) how to build libcurl with SSL enabled, and (iii) how to link WebKit with all of this stuff.

I managed to swell their three-line modification into around twenty lines, but in my defense I wanted to make it possible to have WebKit look for the SSL certificate in the WebKit bundle directory (falling back to the CURL_CA_BUNDLE_PATH environment variable only if necessary).

So now you can locate a certificate bundle in "WebKit.resources/certificates/cacert.pem", and WebKit will find it and use it. If not, you have to set the CURL_CA_BUNDLE_PATH environment variable to the desired location.

The upshot of all this is that I am posting this blog note via WinLauncher (after logging into Blogger.com using HTTPS), and I closed out ~~Bug 28171~~ via WinLauncher (over HTTPS) as well!

A Thank You!

2009-07-16T23:22:00.000-07:00

A few weeks ago I posted a request for a valid, legal license for Microsoft Windows so that I could do more WebKit work on the weekends or when I only had my MacBook Pro at hand.

I'm shocked and delighted to report that a kind donor (who wishes to remain anonymous) has provided me with a brand new Windows Vista boxed set. This is an expensive and very helpful donation, and I greatly apreciate it.

So many thanks to my anonymous benefactor, and I'm off to buy a larger hard drive so I can set up a respectable partition for it.

WebKit Cairo Windows Port Landed!

2009-07-13T17:12:00.000-07:00

It's been a long, hard road, but today marks the commit of the last missing element of the Windows Cairo port.

Literally every bit of code I use on my system to build WebKit on Windows using the non-Apple support libraries is now part of the official WebKit archive. This means that you can now embed a WebView in your Windows application that harnesses the full power and features of the best rendering engine out there.

To celebrate, I am posting here an image of Satine.org's CSS animation demo"SnowStack". It runs fairly well on my local Debug build.

Out of sheer meanness, I am also attaching the same website viewed with IE 8...

Updated Builds (r45683)

2009-07-10T10:42:00.000-07:00

I have updated the binaries for the Windows Cairo (need a name!) port:

Release Build
Debug Build
CallJS Example
Build Requirements

Please let me know if you encounter any problems with these binaries. All font assertions have been fixed, scrolling and panning issues are resolved, and the Google Maps issue mentioned previously seems to work fine for me.

I Need Port Name Ideas!

2009-07-09T23:50:00.000-07:00

One of the few remaining tasks holding up complete integration of the Redistributable Windows WebKit port is an official name. I tried using the term "Redistributable WebKit", but this met with somewhat negative feedback.

So, what do all of YOU think would be a good name?

Maybe I could commandeer the abandoned SquirrelFish name? ;-)

Helping Out -- Windows License

2009-06-28T18:24:00.001-07:00

A few people who have found the Windows Cairo WebKit port useful have asked if there was any way to 'donate' to the project or otherwise help out. While the most useful thing anyone can do is to contribute debugging or coding help, there one thing that would really help me out would be if someone could contribute a valid (i.e., Legally transferrable) Windows XP, Windows Vista, or (someday) Windows 7 license so I could do more coding after hours or one weekends.

To help motivate your desire to donate such a license, I present here a photo I took while on vacation this past week. This is a nice example of what awaits you if you rely on Internet Explorer for your web kiosk needs! ;-)

DumpRenderTree: Windows Cairo 'Mostly' Good

2009-06-11T17:22:00.000-07:00

I recently found and corrected the cause of the hated scrolling bug. A question by Jonghoon Kim about the inconsistent use of SetGraphicsMode(bitmapDC, GM_ADVANCED); prompted me to review the handling of graphics context between Windows and Cairo. This led to the discovery that the Windows HDC and the Cairo cairo_t contexts were not being kept in sync in all cases, and so various transformations were being inconsistently applied, resulting in the strange behavior that prompted me to file the bug. Now that the only major rendering discrepancy was resolved, it's time to start solidifying the Cairo port.

I finally got around to updating DumpRenderTree so that it can run the enormous suite of layout tests included with the WebKit sources. I was pretty sure that things would be fairly broken, but I am pretty happy that DumpRenderTree decided that the Windows Cairo build desired a grade of "D".

Here are the test results:

7907.74s total testing time

5598 test cases (54%) succeeded

4597 test cases (44%) had incorrect layout

19 test cases (<1%)>

4 test cases (<1%)>

409 test cases (4%) had stderr output

Clearly there is a lot of room for improvement. However, as you can see in the screen grab above, many of the differences are a matter of one or two pixels.

I'll probably target the crashes first.

CFLite Corrections, Web Inspector

2009-06-03T14:42:00.001-07:00

Apple recently released an update of the CoreFoundation sources to their Open Source website, which prompted me to take a pass over the OpenCFLite project to merge the (minor) bits of new code. While doing so, I decided to track down the cause of the bundle loading failure which was preventing WebKit from loading resources from the bundle.

The problem turned out to be due to an incorrect file access method (it turns out that you have to pass the FILE_FLAG_BACKUP_SEMANTICS flag to CreateFile if you want to get a handle to a directory. If you just ask for general permissions you get an "Access Denied" error.)

Lo and behold, once this bug was corrected, CFLite.dll began happily loading WebKit resources which caused the myriad of context menu options to suddenly start working.

The benefit? The AWESOME WEB INSPECTOR is now live and functional! The attached screenshot shows what this looks like under the Windows Cairo build. Not too shabby! :-)

I've updated the requirements.zip file with the revised CFLite.dll.

CallJS Updated

2009-06-01T14:09:00.000-07:00

I've updated the CallJS sample project with current WebKit binaries.

* Now uses the WebKit COM API call "WebKitCreateInstance", which greatly simplifies using WebKit. No more having to figure out the right incantation to get the manifest files built properly.
* Cleaned up some of the MFC code.
* Updated to current ICU and other support files.

Enjoy!

UPDATE: Corrected CallJS example link.

Recent WebKit Updates

2009-05-29T16:30:00.000-07:00

I've beaten back the recent bit-rot that is endemic in tracking the WebKit project. This project moves so fast that if you don't stay on top of things it rapidly leaves you behind!

Recent updates now provide proper support for synthetic bold/oblique fonts, as well as using more of the Chromium team's image decoders. I also corrected various build settings that were causing the Cairo build to attempt to link against old ICU libraries.

The never-ending patch is now down to just slightly over 2 KB. Once I get the printing stuff sorted out this patch should no longer be needed.

Next week I should get the sample JavaScript example application working. This little app will probably start linking against the new JavaScriptCore.dll, since WebKit is now broken into two DLL's to simplify access to the core JavaScript routines.

Updated WebKit Build

2009-03-10T12:02:00.000-07:00

I have updated the Windows builds of the redistributable WebKit dll (Debug, Release).

I have also updated the set of build dependencies, in case you need them.

Updates with this release are:

Cairo updated from 1.8.4 to 1.8.6.

CFLite (CoreFoundation) updated to match Leopard 10.5.6

ICU updated from 3.6 to 4.0

For some reason, the Scrolling Bug is gone for Debug builds, but alas seems to occur in Release builds.

OpenCFLite 476.17.1 Release

2009-02-14T23:05:00.001-08:00

A few weeks ago, another Linux hacker named Grant Erickson contacted me to suggest joining forces on the CF-lite front. He had done some work porting the 299 version of CF-lite to Linux, and had stumbled across my more recent port of the 476.17 sources to the Windows platform (needed in support of my redistributable Windows WebKit project.) Grant has now successfully merged his sources into the archive, and we are now feature-compatible on Mac OS X, Windows, and Linux.

We are attempting to track Apple's upstream CF-lite source tree, and so our number scheme is to use their version with a third digit added to represent our revision. We initially released as 476.17.0, but now have fixed a few bugs, added some more examples, and cleaned things up.

CF-lite isn't as sexy as WebKit, or even Bonjour, but is a quite useful, cross-platform abstraction that provides a generally useful toolkit that is the basis for a variety of Apple products. Although it is pure C, it has a nice Object Oriented design and is quite easy to learn.

If you haven't looked into it, please give it a glance. Mac OS X developers can especially benefit, since its objects are toll-free-bridged to Objective C, making it a breeze to compose high and low-level code in your software.

Callback Examples

2009-01-29T15:29:00.000-08:00

Now that there is just a small patch left until all the WebKit Cairo changes are in the main tree, I've been spending a bit of time attempting to incorporate WebKit into actual applications.

Apple provides some documentation on using WebKit inside other programs, as well as a few samples to show how it's done.

Unfortunately, the examples don't help much if you are needing to use WebKit on the Windows platform, so I have ported the CallJS sample to MFC. While this may be a violation of one of the Geneva Conventions, I hope that this example will be educational, and will help in the uptake of the WebKit project.

I've posted a set of Debug binaries that include the CallJS example. I haven't tried them outside my own build environment, but they should give a bit of a head start to using the component.

Google and Chromium

2009-01-11T14:24:00.000-08:00

Update:Proving once again the Google is a company that tries to do no evil, the Chromium team have updated their announcement to more clearly identify the origins of the full page zoom and autoscroll features. Thanks guys!

I think Google is a really cool company, and I use their products and services every day. From their excellent search engine, to the great technical lectures they show on You Tube, to discussing the burning questions of the day (such as Pirates-vs-Ninjas), they provide a huge amount of benefit to the world at large. I love Google! (In fact, I couldn't even write this screed were it not for the services Google provides ;-)

However, just like when a family member does something crappy to you, I do feel that the publicity for their Chrome browser project has been somewhat disingenuous in their discussion of the benefits derived from the WebKit project.

Take for example their recent announcement, or their ongoing blog. Among their new features and bug fixes attributed to the Chromium team, they highlight two significant new features that were both provided by the WebKit project:

Full page zoom

Autoscroll

There is also a bit of vague hand waving about the great tools in Chrome, including the Web Inspector and ACID 3 compliance, both of which are actually provided by WebKit.

If you delve down into the detailed release notes, you can find reference to these facts, but to the typical reader this work all seems to have been done by Google.

Now, things certainly go both ways -- for example, we hope to get new bitmap rendering support in Windows from the Chromium team, and their threading and other low-level updates have been very useful. But I would hope that both of our projects would want to provide correct attribution, and at least for now this does not seem to be true for the Chromium publicists.

But is it so much to ask for there to be proper attribution?

Closing in on full merge

2009-01-11T14:04:00.000-08:00

It's been a few weeks since my last post, but there has been a lot of activity going on. Thanks to some recent commits by the chromium team, much of the pthreads dependencies that crept in have been removed. Unfortunately, there is still some bits of it left over in FastMalloc.h and Allocator.h, but hopefully these can be removed in the near future.

Even more exciting for me is that thanks to the patient reviews by Darin Adler and Adam Roben, I have finally gotten the bulk of the CFNetwork code in WebKit.dll separated into separate implementation files. Right now the cURL implementation is mostly stub, but this will be changing over the next few weeks.

The redistributable port of WebKit is now a mere 7.5 kB of changes from the ToT.

We're almost there!

A Redistributable WebKit

2008-11-25T16:22:00.000-08:00

Well, it's taken a long time to get to this point, but at long last I have managed to get a working version of WebKit that does not use any proprietary Apple libraries. I have posted a Release and Debug build, with the WinLauncher program that produced the accompanying image.

I have updated 17484 with the necessary diff to build the software. You will also need various build requirements, including builds of CFLite.dll that take the place of CoreFoundation.dll.

This WebKit differs from the true Safari webkit as follows:

No support for QuickTime movies.
CFNetwork is replaced by cURL, which implies some reduced functionality and poorer support for concurrent/asynchronous connections.
CoreFoundation is replaced by CFLite, which for the purposes of WebKit should not imply any performance or functionality regression.
CoreGraphics is replaced by Cairo. This generally trails the CoreGraphics implementation, but provides reasonable performance and functionality.

Please give these binaries a try and let me know if I forgot any dependencies.

Core Foundation Moved

2008-11-25T12:21:00.000-08:00

As many of you may know, Google has taken a strong stance against the proliferation of software licenses. Clearly this is a good thing, as a great many of these licenses are virtually identical, modulo some trademark or name changes.

Unfortunately, one of the licenses not officially supported by Google is the Apple Public Source License, under which the CF Lite sources are released. Consequently, I was asked to move the source hosting to a location where this license can be clearly documented.

The most obvious location for an APSL project would be MacOS Forge. However, Apple preferred to not provide hosting on Mac OS Forge until CF Lite was proven to have a meaningful set of users, and that it seemed to be actively maintained.

Happily, Sourceforge does support the APSL as a valid licensing option on their projects, so the code has been moved there.

CoreFoundation Gone?

2008-11-22T23:34:00.000-08:00

One of the major impediments to using WebKit on Windows (assuming you are not writing software for Apple) is that the implementation relies on a suite of proprietary support libraries, such as CoreGraphics, CoreFoundation, and CFNetwork.

Readers of this blog will be familiar with some of the steps I have taken to attempt to mitigate this by using Cairo (rather than CoreGraphics) and Curl (rather than CFNetwork). However, there still remained large bits of logic written to use various useful data types and operating system support logic found in CoreFoundation. While performing the tedious task of replacing these routines with equivalent code held limited appeal, the vision of maintaining these mirror routines in perpetuity while the Apple WebKit team marched ever forward with new features and functionality left me curled in the fetal position calling for my mother.

So, rather than attempt to replicate the FontDatabase routines and other logic in WebKit proper, I decided to start a new project to bring the Open Source Core Foundation (CF Lite) into closer compliance with the 'real' CoreFoundation.

It took about a week of on-and-off hacking to take the raw tarball provided by Apple on their Open Source site and get it building and running on Windows. Since I mostly hack on my Mac, I also created an Xcode project so I could confirm function on both platforms. I intend to also port it to Linux once I have some spare time.

The sources are mostly complete, and easy to build on OS X using darwinbuild. However, the instructions on Apple's website are out of date, and the posted example program doesn't even work in the version of CF-Lite (299) referenced on the page. There are no working instructions or Makefiles for building the current Leopard-based sources on Windows (bug filed.) Furthermore, a lot of the Windows-specific code that appeared in earlier releases seems to have been stripped out in later releases.

Thankfully, by reviewing every one of the released CF-Lite tarballs, I was able to piece together much of what was missing. It is unfortunate that the Open Source source omits (or in some cases excludes via "#if 0" stanzas) the Windows versions of these routines. I took the opportunity to file some radar bugs about this as well. To add insult to injury, after getting the project building I discovered that someone else had gotten parts of it working (though with some bugs), which would have saved some time if I had been more proficient in my Googling.

The upshot of this long post is that I have successfully launched WinLauncher and browsed to the cf-lite project using WebKit built with the redistributable CFLite.dll (see image above!).

Now, I'm starting to glance meaningfully towards the original CFNetwork source releases ...

Update: I have had to move the project to SourceForge, as Google Code does not support the use of the Apple Public Source License for its projects.

Factor

2008-11-09T21:00:00.000-08:00

I finally managed to finish watching the Google Techtalk Slava Pestov recently presented at Google. If you are not familiar with the Factor programming language, let me make the tasteless move of quoting myself from two years ago:

The main author, Slava, has poured a tremendous amount of energy into this project, and the results show. Where many similar projects chug along as console-only systems, building on a single platform, and limited to simple console interaction, Factor provides a built-in GUI system (based on an OpenGL widget set), embedded help, searchable function reference, tutorials, and examples. If a language can be judged on its merits (rather than peer pressure), Factor sure seems to have a lot of things going for it.

If anything, this is more true now than then. The language has grown in features, efficiency, and clarity and seems to have a small, yet prolific, kernel of extremely talented developers moving it forward.

While I am no Factor expert, I am continually amazed at the quality of work produced on this project, and am enjoying learning more about it. If you haven't had a chance yet, please check it out!

Fonts Fixed (sort-of)

2008-10-22T21:41:00.001-07:00

With the help of the good Dave Hyatt, I managed to track down the cause of the strange font behavior in the Cairo port. As it turns out, the problem was actually also present in the CG port, but had a slightly more benign failure mode.

You can read the gory details here.

You will also need the following two patches:
17484 and ~~21822~~. These should be the only two patches needed, as 17484 contains a diff against my current sources, and should apply cleanly against svn trunk (at least for the next few minutes! :-))

You may also need the following changes in your WebCore/config.h to flip the proper compiler switches. I hope to someday get this addressed properly in WebKit itself:


Index: WebCore/config.h
===================================================================
--- WebCore/config.h (revision 37798)
+++ WebCore/config.h (working copy)
@@ -96,9 +96,16 @@
 #endif
 
 #if PLATFORM(WIN)
+#if 0
 #define WTF_PLATFORM_CG 1
 #undef WTF_PLATFORM_CAIRO
 #define WTF_USE_CFNETWORK 1
+#else
+#undef WTF_PLATFORM_CG
+#define WTF_PLATFORM_CAIRO 1
+#undef WTF_USE_CFNETWORK
+#define WTF_USE_CURL 1
+#endif
 #undef WTF_USE_WININET
 #define WTF_PLATFORM_CF 1
 #define WTF_USE_PTHREADS 0

Performance

2008-10-18T19:44:00.000-07:00

I was recently asked about the performance of the Cairo-based Webkit. As a first cut I tried running SunSpider to see what would happen. Unfortunately, it identified a serious bug in the WebKit Cairo port, in that the font handling appears to be screwed up. Consider the following screenshot...

It appears that the build is not dealing with code elements properly.

By the way, anyone interested in playing with the Release or Debug binaries of this webkit is welcome to grab the Debug or Release ZIP files. They have been tested on one external system, so I believe all dependencies are in the archives, but please let me know if you have any problems.

BitRot Beaten Back

2008-10-17T22:46:00.000-07:00

I spent a few hours this evening getting the current source base of WebKit to build as a Cairo-based project. I updated my Cairo library to 1.8, which gave the added benefit of getting the photo.svg test to work properly.

Whee! Spinning, twirling, alpha-blending fun.

See Bug ~~21724~~ for the first set of patches.

Additional patches are still needed to get WebKit working, as there is a ton of CFNetwork, WebKitSystemInterface, and other Apple-restricted dependencies that have to be excised.

Succumbing to the Madness

2008-07-14T21:26:00.000-07:00

This isn't really WebKit related, unless you include things that *use* WebKit. I got up early on 7/11, stood in line, and plunked down my cash for a new iPhone 3G. I knew it was going to be cool from playing with my brother-in-law's original iPhone, but until I had my own unit I really had no idea how absolutely amazing this device is.

In some ways, I'm glad I ended up having to grovel for a year to convince my wife that I deserved to blow hundreds of dollars on a cell phone. Although the guts of the phone are mostly unchanged (i.e., it's not as though it has a faster ARM chip or a built-in death ray), it seems to have matured a bit in its first year. It does have very nice network capability, the best sound quality of any mobile phone I have every used, and what seems to be a nice GPS subsystem; at least, it finds me on the Google map in seconds and provides nice a nice map overlay with a cool blinking dot showing my position as I drive around.

But I think the best way to sum up the experience is one word: Joy. The iPhone is just a joy to use. The interface fairly oozes fun and cleverness. The interactions are quick, simple, and obvious. The screen is large enough that meaningful information is presented clearly; no more cryptic error codes blinking sourly in pixellated monochrome.

So it was with no small pleasure that I stumbled across a great article revisiting some naysaying technical writers predicting the iPhone's sure failure. Most come across as buffoons (typical follow-up question, "Are you surprised at how well the iPhone has done?", answer from writer "No, I'm not surprised -- it's a great phone." Huh? Then why did you think it would fail in the first place !?!). I think the big problem is that most of these 'visionaries' are the kind of vaguely creepy men of a certain age who sport suspiciously bushy mustaches, mid-70's tinted eyeglasses, and an inexplicable penchant for secreting multiple black plastic devices on their person. I remember the IT guy at my last company who carried no fewer than three pagers, a cell phone, and a blackberry (on an improvised batman-style utility belt) to meet his exacting telecommunications needs. To guys like these, I'm sure the iPhone does seem undesirable as no single piece of consumer electronics can possibly provide the same level of bulky electronic gravitas

To me, it just feels so pleasant to have a single device that satisfies these many tasks, and doesn't leave you looking like you still live with your mother.

Bit Rot

2008-05-28T12:50:00.000-07:00

After a nearly two-month self-imposed death march in "the real world", I've gotten back to my WebKit hacking. As expected, several aspects of the build process were broken, but it was exciting to see the massive progress on almost all fronts.

First, daddy-o Hyatt managed to get CSS gradients working nicely, and I'm pleased to report that this came through in the Cairo build flawlessly:

Second, Dave implemented shadows for fonts, which also works properly:

Finally, image reflection is working:

It's really amazing how much functionality is present in a modern web engine, and that so much of it works without having to do much under the Cairo-backed Windows build.

To recreate this build on your own (and I hope you do so we can verify everything is getting properly committed to the archive) you will have to add a few pending patches to the WebKit SVN source tree.:

As before, you will need to add 17730 to get cURL support, as well as some odds-and-ends in 17484

You may still want to extend build-webkit with the --cairo flag, as coded in 17952.

cURL, Grokking those URL's

2008-03-21T14:00:00.000-07:00

With help from Daniel Zucker at Wake3.com, the wxWindows and GTK+ teams, and the folks on #webkit we've finally got a building version of WebKit that does not rely on CFNetwork. Instead, it uses the same cURL backend as wxWindows and GTK+ to handle network interaction, URL parsing, and so forth.

Currently this work is pending review at WebKit.org, under 17730 and ~~17985~~. I've also extended the build-webkit interface to support a new --cairo flag that generates a Cairo/cURL build. You can find this (also pending) under 17952.

Note that you will probably need to apply some final WebKit patches to get a clean build. See 17484 for this handful of changes.

Acid Trip

2008-03-07T20:22:00.000-08:00

There's been a lot of press recently, announcing the excellent support for the Acid 2 test. Since I've recently become a bit of a Web Implementation Enthusiast, I thought I'd see how the nascent Cairo WebKit implementation would do:

How about that? Even with all the code we have commented out of WebKit's Cairo version, it still achieves a perfect Acid2 score.

But what of Acid3? Perhaps unsurprisingly, it scores quite well -- 90/100. This is the same score as the main WebKit implementation. It's better than I would have expected, given that there are certainly several things that are not completely implemented in the Cairo back-end. However, these are apparently not important to the score of this particular test:

This is also a bit disingenuous; there is a font destruction problem that (depending on timing) causes the implementation to crash. But after loading the page a couple of times, I managed to get the test to run to completion. There's another bit of tomfoolery; this build of Cairo Webkit is still linking against Apple's proprietary CFNetwork. Once we cut over to the Curl back-end, this is certain to cause new woes.

But for today, we can chuckle smugly at IE 8 and its standards compliance.

JavaScriptCore Benchmarking

2008-02-26T22:35:00.000-08:00

Although the core WebKit team has already created their own comprehensive javascript testing suite, I thought it would be fun to throw it into the pile of language implementations I like to play with in my other favorite waste of time.

Of course, it wasn't quite possible to run all tests initially. JavaScriptCore did not have support for handling command line arguments, and was not built to use readline to handle standard input. But a few hours after the initial discussion on #webkit, a revised command line utility was available satisfying all the necessary requirements.

The results are a bit mixed, though the overall score for JavaScriptCore beats a recent CVS version of Spidermonkey. Let the arm's race wage on!

WebKit Scrollbars!

2008-02-21T14:20:00.001-08:00

It was fairly easy to revive the existing Windows rendering under WebKit. Scrollbars, however, turned out to be fairly tricky. The basic problem is that unlike all other rendered objects in the frame, Scrollbars are not rendered through the usual pipeline. Instead, a separate infrastructure is in place to hook into the scrollbar controls.

Eventually, the theme logic in WebKit will be refactored such that rendering is done mostly the same way across the board, with individual 'theme' objects that do the actual work of drawing the widgets.

For now, I ended up having to duplicate logic from RenderThemeWin in PlatformScrollbarWin to load the them library and draw.

I also attempted to retain the 'legacy' windows drawing commands, though this has received no testing.

CURLing -- Not just for the Winter Olympics

2008-02-14T20:38:00.000-08:00

Despite historical precedent, I was told today that Apple would "not be able to make the CFNetwork sources available going forward." It appears that Mac OS 10.4.10 appears to be the last version in which the sources are available.

What does that mean for the WebKit port? In practice, not a whole lot. I had been intending to use the Windows INET features to implement this code layer. However, the amount of effort required is probably greater than I want to take on, while having an equivalently low level of "fun" such that I don't intend to do much to hook it together.

Instead, I plan on using the existing CURL logic under Windows, which solves the CFNetwork problem and allows me to reuse the existing code built by other users.

Of course, there is some hope that one of the Windows Mobile ports might have something available that could be used, but so far there has been lots of talk but not much code actually submitted by any of the WebKit mobile porters. Hopefully that will change soon.

Finally, I have somewhat working scrollbars. You can't tell from the above screenshot, but the while blank area on the right-hand side is a functioning scrollbar that doesn't now how to paint itself. Although it's invisible, it does move the page around as desired.

WebKit Success!

2008-02-12T10:26:00.001-08:00

Well, it only took uber-hacker Alp about 5 minutes to figure out the problem I had that was causing the blank-box fonts in WebKit (see below). Once he pointed out my problem (a stubbed-out "GlyphPage::fill" method), I was able to dredge up old Apple code, convert it to the current code base, and away we go.

Patch attached to ~~this~~.

Another small step

2008-02-11T21:59:00.000-08:00

After pulling together more of the original Cairo code from the GTK, Apollo, and original Apple ports I've got WebKit/Cairo properly sizing fonts based on the metrics provided by the web layout. Unfortunately, somewhere along the line I broke the actual rendering of the Glyphs, so now everything displays as boxes. But at least things lay out properly, and you can actually click on the links (identifiable due to the hover effects) and visit additional pages.

So here is a screenshot of Spinneret looking at the Safari page on Apple's website. All of that nice text appears courtesy of the fact that Apple uses a lot of text images on their pages. :-)

Partial Text

2008-02-11T14:55:00.000-08:00

I'm moving closer to having a functional browser. I'm definitely getting text output now. Unfortunately, something is wrong with the layout logic such that the letters are written one on top of the other, such that the result is reminiscent of some sort of Asian glyph.

SVG Works Pretty Well!

2008-02-09T22:46:00.000-08:00

I'm pretty happy with how well Alp Toker's Cairo SVG backend implementation works on Windows!

Now if we could just figure out a good way to get CG and Cairo WebKit to build out of a common project file!

But who cares -- I can already generate/view some great SVG output that would be directly applicable to my work goals:

1. Bezier curves:

2. Chemical Structure:

3. And various geometrical drawings:

WebKit on Windows

2008-02-09T21:29:00.000-08:00

I've been working off and on for the past few weeks on getting WebKit to build and run under Windows without requiring any non-redistributable Apple libraries. It's been pretty easy, which is almost completely due to the fact that Apple had this working originally under Cairo. Later, they ported their own amazing CoreGraphics (see their Quartz and Image Kit for some ideas of this great framework) to Windows, and based Safari on it. This has the great advantage of providing nearly per-pixel accuracy on both platforms, resulting in a great user experience.

I have been hoping to strip out Internet Explorer from a product at work, and make use of WebKit instead. Today, I took a big first step by getting their old Spinneret test program to work:

This is no great accomplishment on my part; Apple's engineers had already created all of this infrastructure years ago. They tossed it when their own superior libraries became available. However, since they have not yet seen fit to allow these tools to be redistributed, we mere mortals must base our WebKit hopes on the Cairo back-end. Of course, over the last few years, the GTK+ port has brought the once-buggy Cairo back-end to be nearly per-pixel accurate with the CoreGraphics logic. If all goes well, it may soon be impossible to tell the two apart.

Currently, my own great accomplishment is having commented out the entire text rendering system yielding the entirely blank example of http://webkit.org. But stay tuned -- bigger and better things are to come soon!

Leopard is Scary-Cool

2007-11-03T23:06:00.001-07:00

I was trying out John Wiseman's "Lisp Metadata Importer", which I rebuilt to run under Intel OS X. I expected to have various source files display based on my search, but I didn't expect them to be rendered as shown here. Look how cool this is; they are tiny renderings (blown up here) of the source files, where you can even make out some of the text on the page.

Kudos to Apple

2007-09-18T21:19:00.000-07:00

Looks like I'm 0 for 2 this week. I thought I had found a bug in the way CIImage data was processed via NSBitmapImageRep, but it turns out that the problem was entirely my own. The good folks at Apple reviewed my original bug report, and provided an explanation of what I was doing wrong, which completely resolved my issue.

My problems were as follows:

1. The GL_UNPACK_ROW_LENGTH parameter needs to be set to the pixel count equivalent of the number of bytes per row, given the number of bytes per pixel. I was doing something like setting this value to the number of bytes per row, which is not necessarily the same thing (depending on image depth, etc.)

2. My use of the bitmap size (height and width) to the glTexImage2D function was wrong. This was using the image physical size in points.

With these changes, all texture mapping works properly.

Text working (mostly) again...

2007-09-17T21:29:00.000-07:00

Switching to the Cocoa-based backend to the Open Agent Engine was more work than I expected. Typically, most of the problems ended up being caused by my own lack of understanding. Case in point: Text. The image in the preceeding post shows what I was getting when attempting to render Text: a mirror image of the text!

I initially thought this reversal was due to some endian foolishness in porting my PPC implementation to Intel. And to be sure, there have been a few areas where this has been an issue (mainly with interpretation of bytes in the image textures).

But it turns out, that it was due to two errors. (1) I decided to ditch the manual byte-swapping logic from the original implementation in favor of a sexier Affine Transform (violating the "If it ain't broke, don't fix it" rule), and (2) me (probably late at night) not recognizing that rotating an image 180-degrees does NOT give the same result as inverting the Y axis and drawing the image.

Thankfully, after staring at the logic a second (well, fifth) time the problem became obvious, and now I can display massive amounts of meaningless text. Rather than rotating 180-degrees, I need to apply a scaling affine transform with a Y-axis scale of -1.0, which flips the Y axis:

Of course, it's not perfect. Note that the height of the text row is wrong, causing parts of the text to be obscured. Still, it's a huge step forward.

Flipping/Endianness?

2007-09-12T22:40:00.000-07:00

The OpenMCL texture examples work fine under PowerPC, but when I load them as textures on a rectangle on my new MacBook Pro, the images are inverted. I did not expect the bytes to result in a mirror image -- I just assumed I would see mishandled Alpha/color values.

Once I figure this out, there is only one outstanding 'bug' in the implementation related to panning.

Open Agent Engine: Now in 64-bit

2007-09-03T19:07:00.000-07:00

I had a bit of a setback with my Open Agent Engine work when I switched to my new Macbook Pro. OpenMCL is only available on Intel as a 64-bit application, which means that none of the Cocoa (and other frameworks) are available in Mac OS X (Tiger). As announced in the WWDC 2007 keynote, the full set of frameworks will be available as 64-bit frameworks in Mac OS X (Leopard) which means that everyone will finally get to play with all of this stuff in a few weeks when Leopard is in general release.

As I was working through the changes needed to use more Cocoa and CGL routines (rather than Carbon and AGL), I started to get interested in Quartz Composer, and how it might be used to visualize data in 3D. To get an idea of baseline OpenGL performance, I mocked up a few simple clouds of data to see how rapidly the data could be displayed and examined (via fly-overs and zooming).

My first cut was using OpenGL vertices as points. I could only display about 2 Million points before performance got too slow to display with any meaningful feedback (first image).

The data could be zoomed in to see details of the individual points, but nothing was in color.

I then changed the implementation to use small spheres to represent the points in the file. Now the data showed color, but the image was much slower -- I had to drop the point size down to no more than 20 thousand points.

Even 20 thousand points it probably too much for real interactive use, but it is at least sufficient for actual examination of the data. With a MacPro or faster system (perhaps next year!) there should be little if any problem viewing such data.

Now, it's on to a Quartz Composer implementation.

Face Morpher

2007-03-29T21:37:00.000-07:00

My port of the Common Lisp "Open Agent Engine" library to OpenMCL has proceeded very smoothly. Fonts, windowing, and a lot of other neat stuff now work properly, though there are lots of buggy edge conditions and murky corners where I manage to crash my Lisp Image due to OpenGL foo.

This has been my first non-trivial Lisp project, and I'm definitely impressed with the language. I really like how I can leave things unimplemented for later, and the compiler politely notifies me, but does not throw any fits as long as I don't attempt to call those methods. Very useful for initial testing; an excellent iterative development language.

For a recent example: The "Face Morpher". I'm not quite sure if my code runs the original "Face Morpher" program properly. I can move the cursor around, and drop nice little inflection points on James Dean's image, but it doesn't seem to do anything aside place the marks (and note some coordinates in the listener). I understand that the h/u keys should create smiles and frowns, but what I actually see is a jittery James Dean that seems to be suffering from some kind of horrible neurological ailment rather than a happy or dour individual. I think something is going wrong in the update routine causing the face to act as though I'm hammering away at my h/u keys.

Fonts!

2007-03-22T20:31:00.000-07:00

Fonts under Mac OS X are kind of confusing to me. Tasks that seem like they should be easy (such as finding a font, or rendering text using a particular font) usually require you to navigate through several layers of code. I'm sure this is much easier if you are working with X-Code, but as I am tinkering with OpenMCL using VIM and a console for testing, it's often hard to find the specific classes and methods necessary to do what I want.

For example, attempting to determine the width of a particular string (in pixels) for a given font, etc., requires a fairly hefty chunk of Objc-Bridge code:


;;
;;  Some AD3D stuff.  A Font-Spec is a list:  (list "font-name" font-size :font-style)
;;       e.g., ("helvetica" 16 :plain) 
;;
(defun STRING-WIDTH (String &optional Font-Spec)
  "Returns the width in pixels of String, as if it were displayed in the font,
   size, and style of font-spec."
  (if (null font-spec)
      (format t "No spec!")
      ;; NSDictionary* attributes = [NSDictionary dictionaryWithObjectsAndKeys: font, nil];
      ;; NSString foo = String;
      ;; Size = foo :sizeWithAttributes attributes
      (let ((size (float (second font-spec))))
        (with-cstrs ((c-font-name (first font-spec))
                     (c-string    String))
          (with-autorelease-pool
             (let* ((tempname   (ccl::send (@class ns-string) :string-with-c-string c-font-name))
                    (fontname   (ccl::send tempname 'capitalized-string))
                    (nsstring   (ccl::send (@class ns-string) :string-with-c-string c-string))
                    (font       (ccl::send (ccl::send (find-class 'ns:ns-font)  :font-with-name fontname :size size) 'retain)))
                (rlet ((glyph-size :ize))
                   (let ((dict (ccl::send (ccl::send (find-class 'ns:ns-dictionary) :dictionary-with-objects-and-keys font (:id #&NSFontAttributeName :address (%null-ptr))) 'retain)))
                      (ccl::send/stret glyph-size nsstring :size-with-attributes dict)
                      (let ((width (ccl::pref glyph-size :ize.width)))
                         (ccl::send dict 'release)
                         (ccl::send font 'release)
                         width)))))))))

But the results are very cool!

More Open Agent Engine Fun

2007-03-03T23:06:00.000-08:00

The inimitable Alexander Repenning is constantly tinkering with the real Open Agent Engine (under MCL). Sometimes he shows one of the cooler ones to me, which I then try to run under my OpenMCL version of the engine. Here's just such an example: Animations and textures are fun!


? (load "voice rings")
#P"/Users/brent/Projects/Lisp/Open_Agent_Engine/Examples/voice rings.lisp"
? (defparameter *RW* (make-instance 'ccl::ring-window))
*RW*
? (ccl::start-animation *RW*)
(#<ccl::ring-window x8e5f5a6>)
?

Squeak: Cool, but Confusing!

2007-01-12T22:04:00.000-08:00

I went ahead and started playing with Squeak, to get a feel for the Croquet project and how it might relate to the Open Agent Engine. The first thing I wanted to try was to play the included Chess game. Unfortunately, I couldn't find anything to explain how to do this.

There were HOWTO's to describe:

How to run an IRC client.
How to use the web browser.
How to install updates.
How to write your own SmallTalk code.

But how could I start an existing program that did not create any menu entries to launch them?

Answer:

1. World-click.
2. Select 'new morph'
3. Select 'from alphabetical list'
4. Find 'ChessMorph'
5. Voila...

Textures!

2006-12-28T21:55:00.000-08:00

I'm very pleased to finally have working textures under the OpenMCL port of the Open Agent Engine. I initially had a lot of trouble getting them to load properly, but some help from Gary Byers quickly resolved the issue.

Following is a brief bit of eye candy showing the results:

Event Handling

2006-12-22T21:31:00.000-08:00

I basically scrapped my initial Carbon implementation, even though it was much easier to follow the original MCL code following that model. However, it rapidly became obvious that implementing the event-handling code would be a lot of work. So, I switched to the nascent Cocoa infrastructure and was able to make much better progress.

Now I can zoom, tilt, and pan across images and models.

OpenGL for OpenMCL

2006-11-22T21:21:00.000-08:00

I've been working on getting the Open Agent Engine infrastructure working under OpenMCL. You can visit their website for examples of the working product, but the graphical infrastructure is pretty neat, and is a nice example of what Lisp can do.

Since I don't own MCL, but *do* have OpenMCL, I naturally figured that it would be relatively easy to port over. But this was a mistaken assumption -- much of the 'hard' work is already done, but OpenMCL does not provide a lot of window and event handling infrastructure, resulting in a bit of work.

I was certainly excited when I finally got a simple OpenGL triangle to display!

Debian and CMUCL: Not Getting Along

2006-07-28T21:07:00.000-07:00

CMUCL is a great Lisp implementation, created by the fine people at Carnegie Mellon University, who are also responsible for the excellent Dylan implementation. It has excellent performance, often meeting or exceeding native C code in the shootout benchmarks.

Unfortunately, changes in the ABI for the Linux Kernel seem to have rendered it unusable under Debian GNU/Linux.

For Example:


brent@hopper:~$ cmucl
Couldn't mmap at 0xbe000000, len 1048576; got mapping at 0xa7cbb000 insteadensure_space: Failed to validate 1048576 bytes at 0xbe000000
brent@hopper:~$

Even worse, rather than seek to resolve the problem, both sides seem intent on pointer fingers at one another. I hope this issue is resolved soon, or else CMUCL's rankings in the shootout will continue to be questioned.

Three Strange Days

2006-07-21T12:38:00.000-07:00

We noticed a few days ago that traffic to the Shootout had spiked recently:

**Shootout Traffic for Three Strange Days**
Date	Count
16 Jul 2006	909
17 Jul 2006	1,955
18 Jul 2006	11,409

At first I was worried that Spam postings were being injected into the website, but we later discovered that a posting I had made to the haskell-cafe mailing list had been posted to Reddit, which had driven the traffic spike.

How cool that around ten thousand people read an e-mail I wrote!

Factor

2006-07-17T21:42:00.000-07:00

A really interesting language is Factor. You might think of Factor as a variant of Forth, and in some ways it is. However, it's also an extremely well-built, featureful, and downright neat system.

The main author, Slava, has poured a tremendous amount of energy into this project, and the results show. Where many similar projects chug along as console-only systems, building on a single platform, and limited to simple console interaction, Factor provides a built-in GUI system (based on an OpenGL widget set), embedded help, searchable function reference, tutorials, and examples. If a language can be judged on its merits (rather than peer pressure), Factor sure seems to have a lot of things going for it.

Things I'm interested to find out:

How well does it handle concurrency? (The key to the thousand's).
Does it scale?
Does it have a cool mascot?

I have created my own Icon for my Mac so I don't have to look at the stupid default OS X Icon.

Want to see? Okay:

Isn't it great? It bears no relation to the program, doesn't tell you what it does, and I can't even remember the name of the piece. But I think it looks cool!

Dry Spell

2006-07-17T20:47:00.000-07:00

I hadn't realized how long it had been since I wrote something here. It's not as though I dropped off the face of the earth -- I actually have been doing something during that time.

In fact, I also changed jobs about a year and a half ago. I started working for a cool little company in Santa Barbara working on scientific stuff. The best part is eating dinner once a year with members of the American Academy of Arts & Sciences, or realizing a year or two later that someone you met won the Nobel Prize.

A lot has happened since I last posted, especially in the language world. Several cool new languages are now available, and I finally got a mac.

Stupid Is as Stupid Does

2004-03-18T12:36:00.000-08:00

Like most of you, I laughed when I learned of the Aliso Viejo City Council's desire to ban styrofoam cups due to the use of dihydrogen monoxide in their manufacture. The story was carried far and wide, from Alaska to Auckland.

However, I'm actually a bit troubled by City Manager David J. Norman's quote in the AP article:

"It's embarrassing," said City Manager David J. Norman. "We had a paralegal who did bad research."

Now, I realize that people have to delegate to get things done in a reasonable time period. But the paralegal certainly was not responsible for the City Council's ability to reason and study information. The fact that the whole council could be swayed by the contents of a single web site does not show good judgement, and should call into question the ability of this government body to weigh the merits of virtually anything placed before them. Where was the independent verification of the facts? If they did not know what dihydrogen monoxide was, why did they feel competent to ban its use?

The fact is, most people think they know a lot more than they do. And we all too frequently have uneducated/ignorant people making decisions on topics far outside their capabilities.

Bless the RIP

2004-01-22T11:04:00.000-08:00

Since I've always had closet Macintosh-envy (but can't afford to buy a system), I've been trying to get the x86 build of Darwin working. Unfortunately, that means I keep hammering my boot sector. Last night, due to fatigue, I forgot to re-write the boot sector and now I can only boot into Darwin. This would not be so bad if Darwin networking wasn't broken right now, so I can get onto the network.

Thank goodness for the RIP recue disk!

Alas, Poor Canon, I Barely Knew Ye

2004-01-05T15:22:00.000-08:00

Here's a sad little note I wrote to Canon USA owing to their terrible Linux/BSD support policy. I can't fault a company for not providing drivers to Linux (since it's an admittedly small portion of the population). But for Canon to rebuff repeated developer requests for documentation on their scanner API is very dissappointing.

Here's a note I wrote to Canon describing my decision to return the scanner:

I recently purchased a Canon 5000F scanner, based on the high quality of my Canon camera products, and the specifications found on the product literature.

It was not until I had taken the unit home and attempted to connect it to my Linux system that I realized you did not provide Linux support. These days, most vendors do provide this support, or at least documentation for the scanner interface so that support can be added to an existing driver framework. My web inquiries found that Canon was not making documentation of the driver interface available, and so that the only way this scanner would be supported would be through a reverse-engineering effort.

Regretfully, I had to return the unit and exchange it for an EPSON, which does provide full support.

I realize that Windows and Mac OS are far and away the most common operating systems, and that it makes good business sense to prioritize your product support accordingly. However, the lack of driver support (or documentation) makes your products effectively worthless for my use.

Based on this experience, I will probably not be purchasing a Canon digital camera (which had been my next planned purchase). This is unfortunate, because I have found your 35mm camera optics to be superior, but in the digital arena I must stay compatible with the rest of my system configuration.

Hat's Off to Apple

2004-01-05T11:18:00.000-08:00

So, my mom got a new Apple for Christmas this year (an eMac) and I got to spend a few hours tinkering with it to get her various applications working (in the OS 9 emulation environment), connect to the wireless network, and so forth.

I must say I am quite impressed with Mac OS X. It's a solid, fast, and absolutely beautiful interface. Things work well, and are relatively easy to figure out. When you consider that OS X is sitting on top of BSD, it's absolutely amazing how simple it is to set things up.

Egad! When Will They Learn?

2003-12-30T20:44:00.000-08:00

So I received a really nice scanner for Christmas this year (Canon CanoScan 5000F). Unfortunately, they don't have active support for Linux yet. Somehow I was actually surprised by this (it's been a while since I ran into hardware I couldn't use.)
Bummer.

Clueless Websites at Christmas

2003-12-18T23:41:00.000-08:00

So, I'm trying to purchase a few niceties from Williams-Sonoma, a wonderful cooking site. But my Linux machine can't connect, though I can get there from my work box. After lots of poking around reading TCP packets between my machine and WS, I discover that they are rejecting all of my connection attempts because my system (properly) includes ECN information in the packets it sends.

ECN (Early Congestion Notification) is a current RFC blessed by the IEC for helping to reduce congestion and better balance load over the internet. Williams-Sonoma must really be running some old firewall software to be complaining about this (since it was ratified back in 2001). I sent them a nasty-gram about the business they are probably losing from people staying up-to-date with Internet Protocols.

Bah, Humbug.

What Did They Do Before The Internet?

2003-12-03T09:40:00.000-08:00

Found while reading A Little Smalltalk by Timothy Budd:

The Little Smalltalk system can be obtained directly from the author. The system is distributed on 9-track tapes in tar format (the standard unix distribution format).

Ahh... memories. Now that the IRS refuses to accept 9-track media, my company's finally been able to discontinue support for these beasts.

Microsoft Monad

2003-11-11T09:25:00.000-08:00

Lots of traffic this week regarding the PDC slide deck about Microsoft's new command shell. My reaction after reading through this whole stack was "so what?" It looks like they have created something of an interpreter for their C#/.NET environment, which allows you to whack out some quick scripts from the command line.

The one useful thing about this is that it nicely integrates various parts of Windows in an easy-to-automate fashion. But this is evolutionary, rather than revolutionary. You won't be able to do anything new the day after you install this stuff than you could the day before -- it's just (arguably) easier.

It's not even novel. The Clean team has already thought of all of this -- and in a more novel way.

Clean Updates

2003-11-10T09:55:00.000-08:00

I've updated my Clean ObjectIO Gtk+ Status page. Lots more stuff is working.

Everyone's Doing It

2003-11-07T16:56:00.000-08:00

So, which founding father are you?

Which Founding Father Are You?

Excellent Windows Compiler Set

2003-10-09T10:02:00.000-07:00

A great project (that I continue to forget to mention) is the Dev-C++ toolset for Windows users. It's a slick package, with an easy update facility and extension packages (for libraries and so forth.)

Request-Response: Mail Filters Sigh

2003-09-09T15:23:00.000-07:00

Request:

> -----Original Message-----
> From: Brent Fulgham
> Sent: Tuesday, September 09, 2003 2:40 PM
> To: ((Hapless Build Technician))
> Subject: Build Log
>
> ((Hapless Build Technician)),
>
> Can you e-mail me the latest MSP Build Log for our SP5 stuff?
>
> Thanks,
>
> -Brent

Response:

> -----Original Message-----
> From: ((Hapless Build Technician))
> Sent: Tuesday, September 09, 2003 2:49 PM
> To: Brent Fulgham
> Subject: RE: Build Log
>
> (( File: Quarantined Attachment Report ))
>

Request:

> -----Original Message-----
> From: Brent Fulgham
> Sent: Tuesday, September 09, 2003 3:02 PM
> To: ((System Administrator))
> Subject: FW: Build Log
>
> ((System Administrator)),
>
> I need to be able to get e-mail from people with the name "LOG" in it:
>
> BUILDMSP_1.0.1.32.LOG
>
> This was apparently quarantined by the server.
>
> Thanks,
>
> -Brent
>

Response:

> -----Original Message-----
> From: ((System Administrator))
> Sent: Tuesday, September 09, 2003 3:09 PM
> To: Brent Fulgham
> Subject: RE: Build Log
>
> ((Hapless Build Technician)) will have to resend the document to you. She should
> ensure attachments have only one period (.) within the name
> (e.g. .doc) before sending it through the email system.
>

Back to the Important Stuff

2003-09-05T17:20:00.000-07:00

Enough of this crap. Here's what I'm currently reading:
Lisp (3rd Edition)

Lisp by Patrick Henry Winston and Berthold Horn is a paragon of clarity. Although I enjoyed books like The Little schemer and The Seasoned Schemer, this book has explained far more about Common Lisp in the first 200 pages then I learned about Scheme from both of those other slim volumes.