Chad Austin's Blog

Thriving Sphere community

Wed, 16 Jul 2008 07:05:28 GMT

I just found out that Sphere, the RPG creation system that I started developing eleven years ago, has a thriving community! A good 2/3 of all of the hits to aegisknight.org are to http://aegisknight.org/sphere, so I made sure to point people on that page to spheredev.org and the spheredev.org forums. Yeah open source!

Finding the shortest path between two objects in Python

Sun, 13 Jul 2008 21:26:09 GMT

I've been doing a bit of memory profiling and debugging in the IMVU client lately, and we're starting to collect a nice set of tools. Today, I'm adding a function that can find the shortest path from one Python object to another. No guarantees about performance or correctness, but it seems to work so far.

The findShortestPath function.

Its tests.

Update: Fixed some ordering bugs in longer paths and optimized it a tad.

More Python trivia

Thu, 10 Jul 2008 02:44:51 GMT

Given a set a, what's the difference between:

a |= set([1])

and:

a = a | set([1])

New Sphere Game

Mon, 30 Jun 2008 03:32:02 GMT

Metallix just let me know his new game Aquatis has been released! Check it out!

IMVU Client's Automated Crash Reporting System: Catching Python Exceptions

Sat, 24 May 2008 21:20:51 GMT

For years now, I have been meaning to write a series of articles on the automated crash reporting system in the IMVU client. This first article will give a bit of background on the structure of the client and show how we handle Python exceptions.

At IMVU, we generally subscribe to the Fail Fast philosophy of handling errors: when the client encounters an unexpected error, we immediately crash the program and ask the user to submit a crash report. As part of the crash report, we send log files, stack traces, system information, and anything else that might help us debug the failure.

You might wonder why we crash the program whenever anything goes wrong rather than trying to catch the error and continue running. Counterintuitively, crashing the program forces us to act on crashes and immediately exposes bugs that might trigger unwanted behavior or lost data down the road.

Now let's talk a little bit about how the client is structured. The IMVU client is written primarily in Python, with time-critical components such as the 3D renderer written in C++. Since the client is a cross between a normal interactive Windows program and a real-time game, the main loop looks something like this:

def main():
	while running:
		pumpWindowsMessages() # for 1/30th of a second
		updateAnimations()
		redrawWindows()

This structure assumes that no exceptions bubble into or out of the main loop. Let's imagine that updateAnimations() has a bug and occasionally raises an uncaught exception. If running the client with a standard command-line python invocation, the program would print the exception and stack trace to the console window and exit. That's all great, but our users don't launch our client by invoking python from the command line: we use py2exe to build a standalone executable that users ultimately run. With an unmodified py2exe application, uncaught exceptions are printed to sys.stderr (as above), except there is no console window to display the error. Thus, the py2exe bootstrap code registers a handler so that errors are logged to a file, and when the program shuts down, a dialog box shows something like "An error has been logged. Please see IMVUClient.exe.log."

From a crash reporting standpoint, this is not good enough. We can't be asking our users to manually hunt down some log files on their hard drives and mail them to us. It's just too much work - they will simply stop using our product. (Unfortunately, most of the software out there asks users to do exactly this!) We need a way for the client to automatically handle errors and prompt the users to submit the reports back to us. So let's rejigger main() a bit:

def mainLoop():
	while running:
		pumpWindowsMessages()
		updateAnimations()
		redrawWindows()

def main():
	try:
		mainLoop()
	except:
		error_information = sys.exc_info()
		if OK == askUserForPermission():
			submitError(error_information)

This time, if a bug in updateAnimations() raises an exception, the top-level try: except: clause catches the error and handles it intelligently. In our current implementation, we post the error report to a Bugzilla instance, where we have built custom tools to analyze and prioritize the failures in the field.

This is the main gist of how the IMVU client automatically reports failures. The next post in this series will cover automatic detection of errors in our C++ libraries.

Open sourced our pstats viewer!

Thu, 22 May 2008 05:46:41 GMT

IMVU has benefited greatly from open source software. Large components of our client and website are built on top of various open source projects. In fact, you can see the list of software on our technology page.

Well, I'm happy to announce that we're beginning to contribute source code back! We have created a project on SourceForge and released our first standalone tool: pstats_viewer.py. pstats_viewer is a tool for browsing the data stored in a .pstats file generated by the Python profiler.

Blogs and blogs and blogs

Sat, 17 May 2008 19:16:43 GMT

I've been asked several times which blogs I actively read, so I wrote a little program to automatically synchronize my RSS subscriptions with my web site. You can see the list here.

Tasks and Futures and Concurrency in the IMVU Client

Thu, 15 May 2008 08:48:27 GMT

The IMVU client has a platform for developing asynchronous or long-running operations that we call the task system. The following describes why we built it. (Chris finally convinced me to post this information to the web.)

Most programs that we write can be split into two categories. There are short-lived programs like gzip that take command line arguments and files and inputs and output to other files or stdout.

There are also longer-lived programs that run indefinitely, handling events that happen to it. Think about MySQL or memcached. It takes inputs from the network, time, and files; doing work and spitting output back onto the network or the file system.

The operating system kernel is one of those event-driven, long-lived programs. It schedules the execution of subsidiary short- and long-lived programs.

We can certainly map from one type another. Take PHP for example. It is running within the context of a long-running server process, but the programming model for a single page is a lot closer to gzip. The program takes inputs (session data, request variables) and outputs HTML.

You can also think of a web service as a set of computers all running a bunch of short-lived processes that combine into one large program. PHP is the CPU, memcached is memory, the database is the file system, etc.

Desktop applications, including the IMVU client and other "rich internet clients" are structured a lot like servers: they are long-lived programs that respond to events from the keyboard, mouse, time, the windowing system, network requests... and interact with its users via graphics, sound, files, and the network.

The key notion here is that these are abstractions on top of an enormous state machine known as your processors and memory.

Now let's talk a little bit about concepts and modeling. When you come into a code base, say, our client, you expect to see structures that look and behave like concepts you know: avatars, chat windows, buddy list, inventory, products from the catalog, etc. This is one reason OOP is so powerful - it's an effective method for describing concepts and the way they interact.

Now let's talk about a simpler concept. One so ingrained you probably don't even think about it. Functions (in the imperative sense). Functions are an abstraction on top of the state machine that is your processor. We're all very familiar with functions: "This operation takes X and Y as inputs, does steps A, B, and C, and returns Z." Even if you throw in a few branches and loops and call some other functions, we're good at following the logic. For example:

def _readNumber(self):
    isfloat = False
    result = self._next()
    peek = self._peek()
    while peek is not None and (peek.isdigit() or peek == "."):
        isfloat = isfloat or peek == "."
        result = result + self._next()
        peek = self._peek()
    try:
        if isfloat:
            return float(result)
        else:
            return int(result)
    except ValueError:
        raise ReadException, "Not a valid JSON number: '%s'" % result

This a fairly elaborate function, but we have no problem following it. Still, it's an abstraction on top of the processor and memory: it's compiled by Python into bytecode, executed by the interpreter loop, which is written in C, which is compiled to x86 assembly, which is being converted by the processor into micro ops... The point here is that we can build our own abstractions if they make us more effective at doing our jobs.

There is a concept in the IMVU client that may be unfamiliar to some of you: long-running processes. These are like functions, except that many can be running at the same time, and they can be blocked, waiting for results from the network or disk. Or perhaps just waiting for some time to elapse. Pseudocode for one of these processes might look like this:

def loadAvatar(chatId, userId):
    avatarInfo = getAvatarInfo(userId)
    if failure, show dialog box and stop loading
    products = getCurrentOutfit(chatId, userId)
    for p in products:
        loadProduct(p)
    loadAvatarIntoScene(products) # this might take a while

In fact, these processes can be fairly easily implemented on top of threads, and many people do it that way, to varying degrees of success.

However, threads have several disadvantages:

Each thread (on CPython/Windows) requires 1 MB of memory for the stack (plus any kernel objects), limiting you to at most 2000.
Because threads are scheduled by the operating system, they're prone to nondeterminism and race conditions. In our case, threads are a large source of intermittent test failures and real bugs.
Threads in CPython don't actually buy you any concurrency because of the global interpreter lock, so you might as well just run everything on the main thread.
Threads are GC roots and must be explicitly cleaned up.
You have to be extremely careful when interacting with the windowing system or any third-party system from threads. Many APIs are not thread safe.

Another way to manage these types of long-running operations is by building an explicit state machine and periodically pumping it, when it will check and update its state if possible. This is generally how interactive games work, because they can get away with using up all of the CPU and having a fixed frame rate. Much of the code in the IMVU client uses this model, but we've found it 1) hard to understand the interactions between states and 2) inefficient. Most of the time you're _not_ updating the state of a particular object, so you don't need to check every single frame. For example, some of our most avid users have thousands of items in their inventory, and our old inventory UI walks that entire list 30 times per second, asking “does this one have any events to process? does this one have any events to process? does this one have any events to process?” This takes up to 10% of the CPU on the user's computer without a 3D window open! Finally, this approach generally makes the application feel sluggish, because the latency between an event (say, a mouse click) being sent to the client and it being processed depends on the rate at which the application is pumped, rather than having the client immediately handle messages as they come in.

A third technique you might call continuation-passing style or "onComplete callbacks". Basically, any function that can be asynchronous takes an onComplete callback function that it runs (with the result or error) when the operation is completed. The problem with this style is that hearkens back to the days of goto with spaghetti callback loops and non-linear control flow. Moreover, if you accidentally forget to call an onComplete, or you call it twice, you've just introduced a very hard-to-find bug. We have had both of these types of bugs multiple times in our client.

We did not find any of the above approaches particularly compelling, so we built the task system on top of Python's bidirectional generators. First, I'll enumerate the advantages:

Tasks are a lightweight resource (sorta) – you can start as many as you'd like.
Tasks look like normal functions with 'yield' statements sprinkled in.
There's no chance you'll forget to call onComplete or call it twice.
Exceptions propagate automatically -- with stack traces!
Tasks can be implicitly cancelled by releasing the reference to their result, so you won't leak them as is so easy with threads.
All tasks run interleaved on the main thread, so you don't have to worry about most kinds of race conditions or calling into external libraries.
Tasks run as part of the Windows event loop, which means that popping up modal dialog boxes or modal menus does not block them from executing, keeping the application responsive.

Here's what the hypothetical example from above would look like as a task:

@task
def loadAvatar(chatId, userId):
    try:
        avatarInfo = yield getAvatarInfo(userId)
    except Exception:
        showErrorDialog()
        return

    products = yield getCurrentOutfit(chatId, userId)
    for p in products:
        try:
            yield loadProduct(p)
        except Exception:
            pass # ignore products that fail to load?
    yield loadAvatarIntoScene(products)

At every 'yield', other tasks get a chance to run, and if that task does not return immediately, then the task is paused until the asynchronous call is done. getAvatarInfo, for example, will contact the server in order to fetch the avatar name. Tasks can yield on other task calls, suspending their execution until the subsidiary call returns a value or raises an exception. (Sound familiar?)

Tasks can start other tasks, with the admittedly strange “yield Start(task)” syntax which gives back a Future object. Think of futures as values that may not have materialized yet. You can later wait for the result of a future by yielding on it.

@task
def printFuture(future):
    # this will throw an exception if the future
    # contains an error
    result = yield future
    print result

@task
def printAvatarInfo():
    avatarInfo = yield Start(getAvatarInfo(userId))
    print 'started fetching avatarInfo'
    yield printFuture(avatarInfo)

You can use futures to start several tasks in parallel and then wait on all of them, making efficient use of parallel resources. Some examples with real code might help. Unfortunately you can't run them, as we have not (yet) open sourced the task system.

Today, much of the client's code uses threads, polling, and onCompletes, but we're slowly converting things to tasks, usually to fix bugs. I hope this means that the quality of the client from a user's perspective will continue to improve and engineers will be able to work in the code more easily.

Twitter

Sun, 27 Apr 2008 20:31:18 GMT

After several abortive attempts, I've finally started using Twitter. Tweet tweet.

A Wiki For Reviewing Third-Party Libraries

Sat, 26 Apr 2008 03:54:45 GMT

In Don't Invest in the Wrong Code, Tom Kleinpeter has created a wiki for pitfalls in third-party libraries. Sort of a consumer reports for software engineers. I think this is a great idea, and would have helped us a lot at IMVU. Please contribute! Let's see this thing get critical mass!

More Updates (and testing facebook rss integration)

Sun, 20 Apr 2008 01:25:50 GMT

I replaced my lame and outdated 'about me' section with links to Facebook, LinkedIn, and Last.fm.

This morning, at 6:00 a.m., I was woken by a text message from Andres, asking if I was in the Minneapolis airport. He heard my name in an announcement. I guess there are more Chad Austins in the world than I'd like to think.

Site Updates

Sat, 19 Apr 2008 12:54:06 GMT

aegisknight.org's been pretty stale in recent years, so I changed the 'links' link to point to my del.icio.us and 'pics' to Laura's Picasa account.

Tasks, object lifetimes, and implicit cancellation

Thu, 03 Apr 2008 12:34:07 GMT

Background: The IMVU client has a system that we call the 'task system' which works a lot like lightweight threads or coroutines. You can schedule up some work which gives you back a Future object on which you can either wait or request the actual result (if it has materialized, that is).

I just read Joe Duffy's IDisposable, finalization, and concurrency. The parallels between TPL and IMVU's task system surprised me.

When we built the task system, there were two ways to cancel execution of a task: explicit and implicit. You could either call future.cancel() or just release your reference to the future and wait for it to be collected by the GC. In either case, the scheduler would notice and stop running your task. I've often wondered if support for implicit task cancellation was a good design.

After reading Joe's article, I'm convinced. If you believe that resource utilization is also a resource, and that implicit resource disposal (garbage collection) is the right default, then implicit cancellation is the only sane default. In fact, we recently removed support for explicit cancellation, because we haven't even 'really' used it yet. (Some of Dusty's code used it, but he said it was fine to take it out.)

We may have to implement explicit cancellation again someday, but now I'm happy to wait until there is a compelling use case.

Why are old, but popular, games so hard to find?

Mon, 09 Apr 2007 06:42:51 GMT

Back in 2002, when I was in college, I was walking down the street from one of my classes, and a friend randomly walked up to me and handed me a video game. It had a shiny box, so I accepted it, but for whatever reason did not even try playing until recently. The game was Age of Mythology, and as I found out in the last few weeks, it is awesome. It's so much fun, that I want to recommend it to all of my friends. And there's where the problem lies. As far as I can tell, you simply can't buy it anymore. I have looked at all of the major online retailers, but all of them either say temporarily out of stock or unavailable. It's not like age of mythology is some rare game that you have to look especially hard for -- it won several awards in 2002 including "game of the year". And now it's just gone.

Earlier this year I had the same problem with We Love Katamari. After trying to buy it from six or seven places that claimed they had it -- each time it turned out they had miscounted their inventory -- Amazon got a new shipment and I managed to snag a copy before it went out of stock again. Once again, this isn't some obscure PlayStation RPG, this is a recent and well-received game.

So what's the deal? Will this trend continue? Am I going to have to purchase every game that I think I might want to play in the future? You would think that digital distribution would solve this problem, and maybe it will eventually, but right now I just want to give somebody some dollars in exchange for a couple plastic discs. Is that too much to ask?

(This entry voice-recognized, courtesy of Dragon NaturallySpeaking. Apologies for any weird wordos.)

Hawaiian Honeymoon

Mon, 19 Feb 2007 07:56:46 GMT

Over the past six months, Laura and I have written an account of our honeymoon in Hawaii. Pictures included. You can read (or just look at the pictures) over here.

Consumer Electronics

Tue, 13 Feb 2007 07:24:50 GMT

Laura and I got a Wii on Sunday morning. (Barely. Luckily, Target got a shipment of 88.) And digital cable today. We already had: DVD player, VCR, TV-out (from my computer), PlayStation 2, N64, and a GameCube (which the Wii will replace soon). So... I'm looking for an 8-way A/V switch. Definitely needs composite video, s-video a nice-to-have. Manual switches are better. We don't need a full-fledged receiver yet, at least until we get an HDTV.

I can't find anything. Any recommendations?

Open Source Development Tip #1

Sun, 11 Feb 2007 01:10:00 GMT

A couple years ago, when I was heavily involved in several open source projects, I accumulated a set of thoughts that might be interesting to someone starting their own open source project. I had grandiose dreams of writing a series of articles along these lines, but as you might expect, real life and a lack of ambition nixed that. Now, I've forgotten even what I was interested in saying, so let's try a different approach. Whenever I have a thought that I think somebody might possibly care about, I will write it as a tip. If the set of tips grows sufficiently large, and enough people care, I may combine them into something more organization. So here we go. Open Source Development Tip #1:

Subscribe to a Google Alert with your project's name and watch users "in the wild". Mailing lists are fine and all, but many people won't bother sending their problems or even first impressions to you or the project mailing lists, even if they'll write in other forums or blogs. Google Alerts will give you a broader perspective.

Prerequisites:

Release your software. (This, of course, requires a sufficient feature set to convince yourself that somebody might possibly want to use it.)
Give your software a sufficiently google-able name. (Audiere, iPodExtract, yes. Corona, Sphere, no.)

Windows is looking pretty good today

Sun, 28 Jan 2007 01:51:38 GMT

Microsoft released a new theme for Windows XP, in the style of the Zune.

It looks pretty good.

Quality With A Name

Sun, 28 Jan 2007 00:32:09 GMT

As IMVU and its engineering team have grown, we've encountered an increasing amount of overhead while implementing features that affect the original code for the client and web site (you may even call some of it prototype code). You know, this effort even goes back to when I started. Most of my early days at IMVU involved fixing bugs, adding test coverage, and making small changes to the product. There were a lot of things about the code that I didn't particularly like, but I chalked that up to the first time I've ever had to deal intimately with a large code base that I did not write. That doesn't mean I didn't try to improve the things that I found; in fact, it's been an excellent exercise in articulating engineering practices I've internalized so deeply that I take them for granted.

At IMVU, we practice test driven development, which means that we always* add a unit test exposing a bug in the system before fixing the bug. In practice, this means we should never see the same bug twice. Some areas of the code have seen more love than others, of course.

Recently, Andy has been trying to implement a tiny new feature in the buddy window. The kind of change that should only take 15 minutes to write the tests and maybe another 15 to implement. (And we wouldn't have to worry about other features breaking, or other changes breaking this feature, and all the other TDD goodness.) That is, if we have good tests in that area of the code. But we don't, so it's not that easy. So he gets to be the lucky engineer, adding tests to this module, which is involving a lot of refactoring. The code's style has a lot of internal consistency, and there definitely is a design, but adding tests around it has proven to be a huge time loss. So we've been trying to articulate why the code is such a pain. But James Shore explains it exactly:

"...the goodness of a design is inversely proportional to the cost of change."

And there we go. That sums it up.

* Ideally. We're always improving.

My father, continued

Sun, 21 Jan 2007 07:00:00 GMT

Laura and I just returned from San Diego, where my father is in the cardiac care unit of UCSD Medical Center. On Wednesday, he was at a leadership conference, and out of nowhere, he collapsed to the floor. Somebody noticed his heart had stopped, so they started CPR, but CPR doesn't do a whole lot for the condition my dad experienced. Luckily, the hotel he was in happened to have an automated external defibrillator (AED) on hand, which they placed on his chest, and after two jolts, started his heart again and brought him back alive. Within hours, he was basically fine again.

Normal heartbeat

Ventricular fibrillation

The condition my father experienced is ventricular fibrillation (VF), where the electrical activity of the heart causes it to lose its coordination and stop pumping blood. This is not the same as a heart attack, which is usually a problem with the muscle or blockage. The survival rate on VF is 4% outside of the hospital. Four percent. Scary. The AED saved his life. We're so lucky that one was nearby and that we still have a father.

So, since not many victims of cardiac arrest and VF in particular survive, my dad's a bit of a celebrity now, and various doctors and interns have come in to say hello. His picture and story are even going to be used in publications for San Diego's Project Heart Beat, a program to make AEDs ubiquitous. My dad's company, Terex, is going to place AEDs in every plant and office building. They're so cheap (you can even order them on amazon.com) and so easy to use (basically automatic) that there's no reason public places, hotels, office buildings, etc. shouldn't have them. So yeah, we're big fans now.

Anyway, we're blessed that my father is still with us, and I guarantee we won't take any of our family for granted anymore. Events like this seem to have that effect.

Update: I guess my sister wrote an update too.

My dad's in the hospital

Thu, 18 Jan 2007 09:09:55 GMT

My dad is in the hospital in San Diego right now. My sister explains. Laura is flying down tomorrow morning, and I will be joining them the day after. Please keep our family in your thoughts.

Quotes

Thu, 18 Jan 2007 09:07:59 GMT

Creating a lot of technical debt quickly is not exactly what "rapid development" is supposed to mean. -- dnicolet99 on Scrum mailing list

Scrum is great for either fixed-date variable-scope, or "fixed-scope" (which always grows) variable-date. If you're doing fixed-date fixed-scope, I recommend waterfall or RUP, which will buy you a few months to look for a new job. -- Michael James

Help Me Pick a Game

Sat, 16 Dec 2006 22:02:27 GMT

In the recent past, I have finished Dragon Quest 8, Bahamut Lagoon, and Far Cry, basically flushing my game queue. So I'm looking for advice on what to play next. I'll give a list of games, but feel free to suggest others. Optimizing for fun divided by number of hours is best. For example, Katamari Damacy was supremely fun and short, giving it a very high enjoyment quotient. Here are the games, in no particular order:

The Main List

Half-Life 2
Zelda: Twilight Princess
Final Fantasy XII
Rogue Galaxy
We Love Katamari
Shadow of the Colossus
Chrono Cross

The Secondary List

Age of Mythology
Final Fantasy III
Rudra no Hihou (Treasure of the Rudras)
Devil May Cry 3
Ico
Psychonauts
Metroid Prime
Kingdom Hearts
Phantom Brave
God of War
Lego Star Wars
Xenogears
Valkyrie Profile (if playing it before VP2 is important)
Zelda: Majora's Mask

Help! (But there are always "eternal games", like Guild Wars and Gran Turismo that will consume all time in the meanwhile.)

Update: Added Shadow of the Colossus, Chrono Cross, Ico, Metroid Prime, Xenogears, and Majora's Mask.

Agile Databases

Tue, 28 Nov 2006 05:43:45 GMT

There is a subgroup of the agile community focused on agile database development and database refactoring. Just do a search. At IMVU, we definitely do agile development of our databases and the application layer code that talks to them. We expect most new engineers to be capable of writing a database schema and pushing it to production within a week of starting, with automated tests that let us change things without fear of breakage, of course. We have no traditional DBAs, although one of our engineers acts as one, reviewing schemas before they're applied in production.

The database refactoring resources that we have seen have been deficient in one area: they assume that you have small amounts of data, or the ability to take down your site or service so that you can apply schema changes. (Which lock tables until the schema is applied.) The problem that we have not seen addressed yet is how to change table structure when you have millions of customers and heavy usage. Each hour the site is down translates into lost revenue. Yeah, there are ways to get around this, such as applying schemas to slaves and then failing the master over to the slave, but these require a fair amount of infrastructure and pain to support.

But if I could choose one thing to get from "the agile database community", it would be a modification to MySQL that let you add or remove indices or columns in the background, even if that particular table was under heavy use, and even if the table had degraded performance during the alter. Maybe it could be implemented as a table with special metadata that shows how to get the data from the old table if it does not exist in the new one. And maybe Oracle supports this already, and I just want a new feature in MySQL.

Update: It turns out that if you have partitioned your customers across enough databases, and have some spare capacity, you can apply schema changes across your databases in parallel, and it's not so bad. So maybe that's how to be agile and have users. :)

backup-sourceforge.py

Sun, 26 Nov 2006 09:10:08 GMT

Today I updated my backup-sourceforge.py script to reflect the new SourceForge backup policies, including support for Subversion repositories. You can download it on my scripts page.

It used to be a bash script, but the second you're using commands like trap "error" ERR and set -e and cd "`dirname \"$0\"`", it's worth rewriting in Python. Interestingly, the Python came out surprisingly terse and clear, unlike past experiences writing shell scripts in Python. Also, it used to call into the adocman project (some Perl scripts provided by SourceForge to make backups easier), but it was such a pain to set up CPAN and install the required Perl packages that I decided I would reimplement their xml_export tool in Python and urllib2. I managed to replace xml_export with fewer than 20 lines of code. Python really does have "batteries included"!