The Fury² Game Development System: A post-post-post-mortem

This article is inspired by the format of and spirit behind the many wonderful ‘Postmortems’ published in Game Developer magazine. I will attempt to collect the most significant failures and successes that occurred during the life of my project, and try to give you a glimpse of what the project looked like and how it felt to use it.

To begin, the many failures and unexpected challenges I encountered:

Mitigated Disasters

Passing by Reference versus Passing by Value, and the evils of Variants

Originally all of the APIs exposed to script used the VB default of ‘ByRef‘ (pass-by-reference) with static types. This had the unfortunate side effect of disabling automatic type conversion when calling into those APIs from scripts. As a workaround, I changed the types of all those parameters to Variant and did the type coercion inside the APIs.

Bad idea. Performance took a hit, and passing the values by reference caused some extremely confusing bugs. In most cases, ‘ByVal’ (pass-by-value) would have been the correct choice, but I didn’t do enough research to discover how to use ByVal correctly with the scripting interface. As a result I wasted a lot of time dealing with coercion-related bugs, and then had to sink another few weeks of effort into moving things over to the correct model (ByVal with explicit types, ByRef where appropriate) much later on, which broke some code. Even by the time I stopped maintaining Fury², some portions of the API still had to accept variants as arguments for backwards compatibility reasons.

Some of the worst bugs in my engine were a result of this decision as well – out of my top 10 worst bugs, one of them was caused by an object reference being passed ByRef (that’s right – a reference to a reference) down a stack of function calls, about 12 levels deep. All the way down the stack, a function assigned null to an object reference before returning – a reference that had been passed in ByRef. The end result was that the null travelled all the way up the stack and obliterated the contents of a member variable elsewhere, causing multiple data structures to get out of sync with each other. This was worsened by the fact that the variable in question happened to be part of a linked list that was shared with C++ – more on that later…

Lacking understanding of COM (Component Object Model)

As a result of VB6 being built on top of COM, I was somewhat familiar with its feature set and how to use it at the beginning of the project. Unfortunately, I lacked deep understanding of a number of important details that turned out to be critical to getting my engine to perform well and run reliably. The main problems I encountered were:

• Circular references – while I had a vague notion of what a circular reference was and how it could cause resource leaks, it took a while before I finally realized that COM provided no built-in mechanisms for freeing object cycles. This meant that somewhat far into the life of my project, I started noticing that I was leaking huge amounts of memory on transitions between different maps, when I shouldn’t have been leaking anything at all. The ensuing cycle-hunt (using manual inspection of reference counts, hand-coded weak references, and all sorts of things that are extremely painful to do in VB6) took a lot of time, and while it yielded results, I still never got to the point where my application was cycle-free, and the introduction of weak references late in development created numerous elusive crash bugs.
• Creation/destruction overhead – Due to the lack of good support for structs in the scripting framework I was using, basically every type in my engine’s APIs was a class, which meant it incurred COM reference-counting overhead. This ended up being extremely painful for plain-old-data types like rectangles, which were created and destroyed extremely often and stored in many places. Even after some heavy optimization (reusing rectangles and similar objects whenever possible, and implementing object pooling where possible) this was still a significant bottleneck that could have been avoided with a better design.
• Type library registration and versioning – I originally believed that I could deploy all my components using what VB6 called ‘binary compatibility’, where I could deploy a new version of a given type library that could be dropped in to replace the old one without breaking any existing binaries. This worked for a few months, but eventually I realized that it was impossible to maintain binary compatibility given VB’s restrictions and the rapid pace at which I was making changes to my APIs. Once binary compatibility went out the window, distribution and installation became a serious issue – having to register over a dozen type libraries for a given version of the engine was extremely painful, creating significant performance issues and making it impossible to run games without being an administrator. Later, I even ran into versioning conflicts with a couple Microsoft type libraries where the latest version of a library was not backwards compatible with the previous version, but the previous version had a crippling bug – which meant I couldn’t run on a machine with the old version, but I couldn’t forcibly upgrade it without breaking other applications.
• Apartment models and RPC – VB6 does a fairly terrible job of abstracting away the many nuances of COM. Other than type library versioning, mentioned above, the worst examples in this area are probably apartment models and RPC – both options are exposed to users of VB6 via radio buttons in the project properties window; unfortunately, the documentation almost completely glosses over the significance of these options, making it easy to select the wrong one and not realize your mistake until much later. I spent weeks developing my tools pipeline using COM RPC to communicate with a running instance of the game engine, resulting in some extremely confusing bugs and awful performance characteristics. It was only once I realized what was going on behind the scenes, due to the awful performance (why is this so slow? wait, the engine is running in *another process* and COM is marshalling all my data between processes… uh oh) that I understood the importance of those settings and learned to leave them set to the defaults. I had some utterly mindrending bugs where I was allocating GDI bitmaps or large arrays of data in an Engine class for a map layer or an entity, and passing the class instance across the RPC boundary to the editor, and then scratching my head in confusion when BitBlt or CopyMemory failed to do what they should have done, due to the fact that I had just marshalled pointers and handles across process boundaries in an unsupported way, without even realizing it.
• Quality issues – unfortunately, many of the implementations of COM that I had the misfortune of dealing with were subpar. Some were downright broken. I discovered dozens of bugs in various COM libraries and APIs, some of them ones provided by Microsoft themselves. In most cases my only choices were to hack around the bugs or reimplement the entire component myself, and I ended up choosing the latter. While most of the implementations in Windows itself and the VB6 runtime were fairly stable, some had extremely bad performance characteristics that necessitated writing my own implementations – and reimplementing COM library functions is *not* a walk in the park.
  One positive note in this section, though, is that I happened to stumble across a bug in PyWin32′s support for COM exceptions, and the maintainer was very quick to merge my fix into the trunk. This was one of my first real introductions to the value of open source.

The wrong model for rendering

The object model I chose for my graphics API had some strengths, but it also had one primary weakness that caused most of the graphics-related stability and performance problems the engine had later on: I had no distinction between mutable and immutable images. During the early stages of the project, this was liberating, because it meant that I could easily whip up prototypes with complex visual effects and that it was straightforward to create reusable pieces of rendering code that could be applied to anything.

The downside, which didn’t become apparent until much, much later, was that having an object model where everything is mutable makes it extremely hard to accelerate graphics rendering using hardware, because neither OpenGL, DirectDraw or Direct3D offer any way to get high-performance mutable surfaces that can still be used as textures. I ended up sinking at least a month of coding time into getting acceptable framerates out of OpenGL acceleration, and even after that, some of my games and prototypes simply would not run well on hardware due to heavy use of mutable surfaces.

If I had taken a more conservative approach I would have had a much easier time adapting my code to exploit hardware acceleration. In many cases the mutable nature of images wasn’t useful at all, and I paid a significant price for providing it. Hardware textures had to have software backing stores at all times, and I had to jump through significant hoops to ensure that the two were kept in sync and flagged as ‘dirty’ after rendering operations. Copying texture data back and forth between CPU and GPU was an enormous performance killer, especially on older hardware – hardware that would have otherwise been able to accelerate my engine just fine.

I also failed to abstract out my rendering operations in a sane manner until it was too late to do anything about it; the result was that hardware acceleration depended on a convoluted set of per-function ‘overrides’ for the entire graphics API, all of which had to be hand-written for each rendering API and tuned to behave similarly to the built-in software implementations. All of the typical issues people are used to from working with virtual functions and inheritance applied here, except it was significantly worse due to the fact that I was implementing it from scratch without a concrete model or understanding of all the implications. Even after significant optimization and refactoring, the overhead this approach created was responsible for making my OpenGL accelerated renderer perform 2-5x worse than it would have otherwise, sometimes even worse than the native software implementation.

Active Scripting

Microsoft’s Active Scripting library made it really easy to get up and running, and provided a fairly solid feature set, so it was a good choice for a scripting platform initially. Unfortunately, as the engine matured, Active Scripting started becoming a bad choice. It lacked support for various portions of COM that I hadn’t realized I would need, and had some severe performance problems that were extremely hard to compensate for. It also had some sneaky bugs and reliability issues that took a long time to track down once discovered.

I also initially believed, based on some testing and documentation, that Active Scripting gave me a way to transition to a better language when the need arose – it was possible to use it to run Ruby, JavaScript or Python scripts instead of VBScript, which gave me a good path for transitioning to another scripting framework in the future if I so desired. In practice, this didn’t turn out to be true – there were subtle, crippling issues in the Active Scripting implementations of all three languages, leaving VBScript as the only feasible option. In the end, this miscalculation cost me weeks of time writing workarounds for Active Scripting bugs and trying to get other languages to work as well as VBScript did, and the cost of ending up stuck with VBScript is hard to estimate.

The lack of expressiveness that initially made VBScript ‘accessible’ to new users and quick to prototype with also ended up being a hindrance. VBScript provided no usable implementations of concepts like delegates, interfaces, inheritance, exception handling, or strongly typed arrays – all things that don’t seem particularly important early on in a project, but become increasingly important as your architecture grows more complex and your performance requirements are more demanding. In some cases, the language provided a functioning implementation of a feature, but it was implemented in such a way that it was impossible to use it with native APIs, making it useless in the context of a game engine.

The worst example by far was delegates; at first, it seemed that the performance of invoking a script function by name was acceptable, so I built my event-handling model around having a list of event handler names to invoke at runtime. Later I began to realize that every invocation was resulting in expensive name lookups and type conversions, which resulted in severe performance hits when you were having scripts respond to thousands of events per frame. In the end I had to build a fairly complex code generation system to construct ‘trampoline’ functions any time I needed to have scripts handle events, which resulted in a small but steady leakage of memory to generated scripts (which couldn’t be garbage collected or reclaimed).

Larger games also suffered from inefficiencies in VBScript’s runtime, with the cost of individual variable lookups increasing based on the number of variables defined in the entire game, due to a lack of support for usable namespacing or modules.

Wide feature set and endless ‘scope creep’

From the beginning, I had a habit of allowing the scope of the engine’s feature set to increase rather than stay fixed, and it took me a very long time to recognize the problems that this caused. By the end of its development, Fury² had nearly 200 unique classes spanning over 100000 lines of VB6 code, with thousands of unique methods and properties. Many of the classes interacted in ways that were unclear or unpredictable, and some of them had decayed into various forms of brokenness due to neglect. Near the end of the project I regularly discovered that I had just reimplemented a function or feature that already existed elsewhere in the codebase, which was a clear sign that things had gotten out of hand.

The most representative example was the ‘Engine’ class, which would have probably been more appropriately named ‘KitchenSink’. Due to Active Scripting limitations, the Engine class held any and all variables or functions that needed to be accessible at global scope, which resulted in a steady growth of new public variables, properties, and functions, many of which were interdependent or affected the behavior of classes elsewhere in the codebase. Keeping all of the responsibilities and features of the Engine class in my head soon became impossible, and it was far more difficult than I had anticipated to factor out individual responsibilities into unique classes without breaking existing games or degrading performance.

Even after aggressive refactoring and elimination of dead code, by the time I put the project to bed, the Fury2Engine class was over 6000 lines of code, containing over a hundred member variables and hundreds of member functions. Almost every class in the framework held a reference to an Engine instance for one reason or another, making it difficult to destroy dead Engine instances to reclaim memory, and leading to subtle bugs where multiple instances were alive at once. Basically any change to a part of Engine was likely to break other parts of the codebase, unless the change involved adding completely new functions and variables (a downward spiral if I’ve ever seen one.)

While the wide feature set meant that it was often possible to build a functioning game prototype with a high level of polish in a few hours, it also meant that it was impossible to build such a prototype without finding at least one bug in the engine (usually more like two or three), and the resulting interruption to my flow almost completely outweighed the benefits I got from rapid prototyping. In many cases I used my engine to build a game for a competition with a short time limit – as short as 48 or 72 hours to build a game from scratch – and had a large chunk of content and code complete, only to find a showstopper bug hours before the deadline that ended up taking days to fix. The reasons for this weren’t immediately clear to me, but over time it became obvious that it was due to the breadth of the codebase.

Large footprint

As a result of using COM and VB6, the size of my engine’s binaries alone quickly grew into the megabytes. This was pretty severe – it meant that merely downloading my engine on a typical connection at the time would take as long as an hour, without even including a game. I went to great lengths to reduce the size of my binaries, by trimming out dependencies, applying code compressors like UPX, and attempting to eliminate as much code as possible from the largest sections of my codebase. In the end, though, the real problem was that the combined overhead of COM and VB6 meant that even simple code resulted in a large binary, and when you applied that overhead to my already large codebase, you ended up with something that was almost impossible to trim down to a reasonable size. For this reason I ended up having to apply UPX to all of my binaries to be able to get them to a size that was remotely small enough for a typical end-user to download, even after halving the size of some of them and eliminating entirely unneeded libraries. Even by the end of the project, the engine and development tools combined totaled around 12 megabytes of executable code – somewhat large by modern standards, and unreasonably huge by the standards of someone using a 56k modem.

The footprint also had less obvious consequences – the size of the binaries meant that more and more often, I ran into strange bugs in the compiler and debugger, and overall build times grew to the point that going from the IDE to a running version of my engine could take up to a minute. This severely impacted my ability to iterate and track down problems.

The worst side effect of this by far, though, was the impact it had on actually using the engine to make games. While the size didn’t really impact my ability to build a game, it meant that I had to somehow get dozens of megabytes of data onto a user’s machine before they could play my game. Attempts at reducing the download size typically resulted in compatibility issues, as I discovered that ‘optional’ libraries weren’t so optional because particular OS versions didn’t include them, or included outdated versions. Troubleshooting these issues was next to impossible since getting a user to download a dozen megabyte installer twice is exponentially more difficult than getting them to download it in the first place – many of my users gave up and simply did not have time to help me track down their issues, so those issues went unsolved until I could reproduce them on a personal machine – if I ever solved them at all.

Instability

One of the most significant disadvantages to using VB6 was the near-complete lack of error handling mechanisms. The only viable options when building a large VB6 application were to hand-write complex error handlers at the bottom of every function, or ‘delegate’ errors up the stack using On Error Resume Next. Unfortunately, both approaches are actually worse in practice than they sound on paper – hand-written error handlers could not easily delegate to common error handler implementations due to language limitations, which meant that they had to be manually copy-pasted throughout the codebase, or machine-generated by automated tools. Neither option was sustainable, which meant that I ended up using On Error Resume Next to effectively ignore errors in any part of the codebase where they weren’t expected.

Even with the most diligent approach to error-checking, ignoring errors by default is not a sustainable way to build an application, and when you’re trying to build a reusable library or game engine, it’s outright disastrous. Up until the very last day I stopped working on it, Fury² suffered from crippling, unpredictable stability issues where a minor failure somewhere in the application could result in a catastrophic failure seconds later in an unrelated piece of code, with few clues as to the actual cause of the problem. In many ways, this was as bad as the issues that C/C++ programmers can run into through misuse of pointers and unchecked operations, but I managed to end up with this level of instability despite using a language that offered fairly safe string and array primitives along with null reference checking.

With more foresight, the sheer impossibility of handling errors properly in this environment would have resulted in the project coming to a halt and never resuming, or being restarted in another language. Unfortunately, I opted to continue down the path I was on, methodically hunting down bugs one by one, adding thousands of unique error handlers and ‘armoring’ my code against unexpected conditions by checking for nulls anywhere I had an object reference and validating the input to any given function. While the resulting approach of ‘defensive programming’ did have benefits, it also resulted in significantly slower code and didn’t help improve the quality of the rest of the codebase – 1000 input checks don’t do you any good if you forget one in an important function.

This approach to error-checking was worsened by the necessity of integrating with native code written in C/C++ – while a null reference could cause Visual Basic code to fail to work, the impact on a snippet of C/C++ was far worse: Due to the fact that my native code was being called from Visual Basic, I couldn’t rely on C++ exceptions and had to resort to return codes, and all of the problems that entailed. Furthermore, in cases where my error handling was faulty or nonexistent, a null object reference in Visual Basic would often transform into catastrophic failures in C++ – overwriting random parts of memory with garbage, smashing the stack, or simply throwing access violations. All of these failures inevitably brought down the application, some of them even resulting in data loss from unpredictable crashes of long-running editing tool sessions.

Reinventing the wheel

As mentioned multiple times previously, I ended up having to reimplement many components that I had taken for granted. This included a large portion of the VB6 UI stack, and significant portions of the COM API and the VB6 runtime. This effort cost me a significant amount of time – many months of engineering work in total – and resulted in an experience that felt inconsistent and often lacked key features that I could have gotten for free if I had been able to use the stock implementations. Trying to reimplement complex UI primitives like cascading dropdown menus was an exercise in torment, where every bug fixed revealed two more bugs waiting to be fixed, and the sheer fatigue that resulted was one of the main reasons I ended up abandoning the project.

Among some the many things I had to reimplement from scratch to solve issues, here are a few of the major ones:

• Listbox, Toolbar, Menu, Tab Strip, Scrollbar and Button controls – because the stock VB6 implementations had severe performance issues, crash bugs, and missing features. Most of these issues were the result of bugs in the VB6 wrapper code that talked to the underlying native APIs, so in some cases I was able to use the native APIs directly – the longest-lived example of this was the VB6 scrollbar control, which ended up being replaced by a very complex wrapper for the native Win32 scrollbar. In other cases the native APIs were simply too difficult to use, so I had to reimplement them from scratch – I invested a significant amount of effort into building a good toolbar component, and ultimately it was still not as good as its equivalent in modern toolsets like Windows Forms or SWT.
• COM type library registration, loading, and unregistration - Yes, you read that correctly. I reimplemented a significant portion of COM and the related portions of the VB6 runtime to work around significant limitations and bugs in the stock implementations. This meant that I had to learn far more about the inner workings of COM and the Windows Registry than I ever wanted to know, and it meant that I had to resort to hooking dozens of native Win32 APIs in order to fool native COM components into thinking that they were running in a normal COM environment. The end result met most of my needs – I was able to load and unload COM libraries on demand without running as an administrator, and the performance of my COM implementation was generally superior to Microsoft’s – but it caused hundreds of issues with COM components that depended on quirky undocumented traits of the Win32 API, or components that were simply written incorrectly. All said, I spent at least a month coding and troubleshooting something that would have been almost completely free if I had been using almost any other comparable toolset.
• Rendering - I ended up implementing my entire graphics stack from scratch, only using third party libraries in cases where I could absolutely rely on them or where it wasn’t feasible to implement things myself – I used GDI or DirectX or OpenGL to render my framebuffer to the screen, and an open source library named Corona to load images – but everything else, I basically wrote from scratch. I spent an extremely large amount of time discovering the many nuances involved in rendering, and struggling with the Visual C++ compiler trying to get it to turn my C into performant assembly so that my games would run at a solid framerate. Despite the benefits I got from having complete control over my graphics pipeline, in the end I would have been better off learning an API like OpenGL extremely well, and taking advantage of pixel shaders once they were available to get the precise control over rendering that I wanted. While I thought that software rendering would protect me from driver issues and hardware bugs, in practice, it actually didn’t, and I lost many weeks tracking down obscene bugs in GDI and DirectDraw drivers that made simple things like drawing a framebuffer to the screen produce incorrect results or fail to work completely, often without any error messages or debugging information. A single bug in the DirectDraw drivers for a common Intel chipset cost me over a week of testing and debugging due to the fact that I had no access to a machine with the problem, despite it preventing hundreds of people from being able to run my game.
• Text – Despite the fact that VB6 used UCS-2 as its internal string representation, I ended up having to write my own UTF-8 decoder and decoder, along with many other assorted utility functions and my own text rendering/measuring APIs to actually get functioning unicode support so that localized text could be put into a game. This was a huge, mostly unanticipated cost, because VB6 appeared to have top-notch Unicode support on the surface but was actually barely serviceable.

Incomplete and incorrect knowledge

I learned a lot of things during the course of the project, but unfortunately, much of the knowledge I gained was incomplete, or subtly incorrect. This is a common problem when self-teaching, and in the case of a project like this, it’s easy to form bad mental habits that are hard to break out of later. I don’t think it outweighs the benefits of what I learned, but it definitely poses a continuing problem, as I often have to relearn concepts and techniques that I already believed I understood. If you’re a particularly opinionated or strong-willed person, this can be especially dangerous as you may never realize how flawed your understanding is, despite the fact that it will cripple your professional career and even your personal life.

Now that you have an idea of the challenges I faced and the mistakes I made, it’s time to try and share some of my triumphs and hard-won lessons:

Silver Linings

Rapid UI development

Despite all its flaws, VB6 was absolutely the right choice for building the UI for my editing tools. It provided a near-perfect mix of ease of use, simplicity, and ease of integration that let me build highly usable UIs without having to sink time into authoring message handlers, digging through API documentation, or struggling with resource management. I could open the visual basic editor, create a new form, place some simple UI components on it (many of which I had written to abstract out common UI elements, like lists of generic game objects or property grids), and rig up functioning event handlers to them in a matter of minutes. In most cases, the majority of the UI code worked right as soon as I hit save, and the majority of the bugs that did make it into my UI were simple and possible to avoid with enough diligence.

VB6 also scaled up suprisingly well as my UI grew more complex. Despite a few crippling issues, for the most part, it performed admirably – I had hundreds of unique dialogs and tens of thousands of unique event handlers, and most of them did not require any maintenance once they were fully implemented. I was able to reuse an extremely large amount of code between individual dialogs, allowing me to build extremely usable and powerful UI primitives that made it even easier to build more UI in the future. Many pieces of UI that would have otherwise taken weeks of effort to implement from scratch were finished in a matter of hours, leaving me time to spend on user experience, reliability, and performance. To this day, the editing tools that resulted from this have extremely good performance, a smooth workflow, and a robust feature set – things that weren’t available in any of the comparable tools at the time.

Ad-hoc, iterative design

While my initial approach was to design large sections of the engine up-front and then implement them, I quickly shifted to an iterative approach where I designed simple solutions to problems as they arose, and built on them as I continued. I prototyped many of my features in minutes, sometimes even using VBScript to implement them without worrying about static typing or having to compile code. When those prototypes were meeting my needs, it was easy to move the scripts into my codebase and add the necessary static types to compile them to native code, which made for an excellent feedback loop and allowed me to kill many bad ideas without having to spend days implementing them first. Being willing to throw out bad ideas and bad code was much easier because the code took less time to write, and I hadn’t gotten attached to it. Despite the numerous stability and performance issues I encountered, by the end of the project I had an extremely versatile set of prototyping tools, with complex primitives like functors and mixins that wouldn’t have otherwise been available to me.

In almost every case where I applied this approach intelligently, the end results were great, even without the benefit of a rigid development process, unit testing, or the kind of prototyping tools I now have access to in environments like Python and .NET. Being able to prototype rapidly allowed me to maintain an extremely high level of morale and build a large amount of usable code in a short amount of time, which was essential for maintaining my interest in the project and being able to carry it along to the point that I did. If I had been prototyping in an environment with a longer, more painful edit/compile/test cycle, like C++, I probably would have abandoned the project after months, instead of the nearly 7 years I maintained it for.

Software rendering and the lowest common denominator

Early on, my decision to use software rendering meant that games and prototypes I built would run the same on any machine (provided they ran at all) without any need for drivers, libraries, or special hardware. I was able to test and demonstrate prototypes on absurdly low-end machines, like a 486 running Windows 95, and then go back and make changes to them using my Windows 2000 or Windows XP machines at home, without seeing any difference in rendering behavior or gameplay. The performance tended to scale badly on the low end, but the games always ran and were often playable on the worst machines if I was careful enough about how I built them, which was something I couldn’t get at the time using OpenGL or DirectX. As time went on, this benefit grew less and less important, but in the early stages of the project it was essential in allowing me to get important feedback on usability, gameplay, and the overall experience from people who didn’t have powerful computers or have the necessary hardware to run most PC games, and it minimized the time I spent having to track down bizarre issues with video drivers and video hardware.

Software rendering also gave me the ability to produce visuals that were out of the reach of most indie developers at the time, with the ability to combine various blending modes, filters, masks, and transformations to create scenes approaching the level of quality that I could get with layers and filters in photoshop. The effort involved in getting these visuals to run at a decent framerate is hard to overstate, but the end result was also extremely satisfying – to this date, a few of the rendering techniques I experimented with and used in prototypes have only been used in a scant few games, due the near-impossibility of implementing them on pre-DX9 hardware, and how difficult it is to implement them even on modern hardware.

The process of learning how to implement the many graphics primitives and techniques that I had taken for granted was an extremely valuable one as well, and helped shore up the many academic weaknesses that had resulted from my lack of formal math education. Becoming intimately familiar with image processing and the mathematical models behind lighting and other common graphics effects gave me a strong appreciation for the value of math that I would not have had otherwise, and made it easier for me to begin learning more complex techniques and algorithms to apply to hardware rendering and 3D.

The desire to innovate

I regularly spent weeks or even months in pursuit of an implementation for a wild idea or concept that I had, digging through relevant articles and books, experimenting with formulas, and simply fiddling with things to try and come up with something new or entertaining. As a result, I ended up wandering into areas of computer science, graphics, and other areas like physics and artificial intelligence that I would have never been exposed to otherwise. I literally spent an entire summer attempting to form an understanding of how light and color worked, building a complex lighting model that could run in realtime and produce realistic, smooth-looking colored lighting for 2D scenes. By the end of the summer I had rewritten individual algorithms hundreds of times, and created and abandoned dozens of different lighting models. The process was difficult, but it was also one of the most satisfying things I’ve ever done.

Without the framework I had in place for rapid prototyping, and the willingness to investigate dead-ends in the search for fun new things, I would have missed out on a lot of valuable knowledge and experience. Once I was done, the resulting lighting model ended up being the springboard I needed to build many of the prototypes and game projects I’m most proud of, and without having spent all of that time in pursuit of something interesting, I wouldn’t have been able to build those prototypes. Even today, working in a medium-sized engineering team on a somewhat ‘normal’ application, the thirst for knowledge and willingness to keep trying to solve a problem that I gained from these efforts are extremely valuable.

The downside, of course, is that I didn’t have time to watch any of those TV shows people were talking about. Is the cast of LOST still on that island?

Language interoperation

The need to combine tools and libraries written in multiple languages required that I form a deep understanding of the issues involved in getting multiple programming languages and environments to interact. As a result, I ended up digging into various parts of my toolchain and into the inner workings of various language environments. By the end of it, I had acquired a working knowledge of assembly, a fairly in-depth understanding of COM, and a grasp of how complex tools I had previously taken for granted truly were. Debugging issues with stack corruption and library load failures resulted in learning how the Win32 call stack is laid out, how PE executables are organized, and how the OS actually loads libraries and executables – things I otherwise might not have ever had to know that have ended up being extremely valuable in my professional career.

In no small part due to this project, in the span of a decade I went from not knowing anything about computer programming, to being able to write and debug software in dozens of different languages and environments. Being exposed to so many different mindsets and approaches helped counteract the many bad habits that I would have otherwise formed while working on my projects, and made it easier for me to adapt to the often strange and rigid processes I’ve encountered working on video games and commodity software. Being able to recognize similarities to existing tools and languages is extremely valuable when trying to learn a new skill or tool for a job, and has helped me develop my career without the benefit of a college degree or a significant academic education.

Collaboration and documentation

As my engine and toolset grew larger and more people became interested in trying them out, I quickly learned the value of collaboration with both programmers and non-programmers, and the importance of good documentation. Maintaining such a large project for such a large span of time, and making it possible for the uninitiated to learn how to use it and build things with it, was quite possibly the largest challenge I faced during the project. As a result, I developed some extremely valuable habits and skills, learning how to document APIs, reproduce bugs, provide workarounds, and track changes. I got better and better at all of these things as time went on, and as my skills improved, the results improved as well.

Near the end of the project, multiple people had successfully built playable game demos and prototypes using the engine, despite the subpar documentation, lack of stability, and other numerous problems. The experience of providing them with support when they encountered problems, and helping then understand how to use my tools, helped open my mind to the challenges involved in designing truly useful tools, and the value of open and honest communication with both customers and coworkers. Without them, I would not have been able to accomplish nearly as much as I did, and my personal projects and career would have suffered for it.

In retrospect, the people who spent their time trying to decipher my often-cryptic documentation and spent time finding bugs and workarounds were as important to the project as I was, and it was a blessing to have their help. The downside is that none of them were able to complete a game using my tools, but even that taught me a valuable lesson: My personal standards of quality aren’t enough to produce a quality product – I need to understand my customers’ needs and listen to *them*, instead of focusing purely on my vision of how things should be. The point of building tools is to improve the quality of others’ lives, and enable them to do things they couldn’t have done otherwise; if they succeed, you succeed with them, and if they fail, the failure is yours as well. Out of all the lessons I learned, this may be the one I treasure the most.