So, I tried those things. I rolled back a few revisions, rebuilt, and it still crashed. I tried rolling back four weeks to a known-good revision, and it still crashed. I went through all the commits over that time period to try and identify any obvious suspects, but didn’t find anything. I tried running the browser with a new, empty profile, on the assumption that perhaps one of my add-ons was causing the problem. Still crashed.

At this point, I could have thrown out my clone of the mozilla source code, along with my build folders, in the hopes that starting over from scratch would fix the problem. I’ve done this before, and sometimes it does make problems go away. But in practice, if everyone does this every time they hit a problem, insidious bugs can creep their way into your build system (or worse, your code base), and when you have a few hundred developers hacking on one project, the time wasted by using that solution adds up. So I decided to try and figure out why FF was crashing on shutdown.

The first thing I needed to figure out was whether the heap was actually corrupted. I started up Firefox inside the Visual Studio debugger and configured it to break on error, so that I could inspect the state of the application. At the point of the crash, here’s what the state looked like, along with some other useful information:

The most immediately obvious piece of information is that the ‘crash’ in this case is actually a breakpoint, triggered by ntdll.dll!_RtlValidateHeap. Furthermore, if we walk up the stack a bit, we can see that this is from msvcr100d.dll!_free_dbg. This tells us that we’re using the debug heap provided by the Visual C++ Debug Runtime, and the validations it performs when freeing a block of memory have failed. This can happen for various reasons, but if it happens, it almost always means something is wrong.

Next, we need to figure out why this is failing. There are a few obvious candidates, so let’s try to rule them out.

Memory corruption would be a likely cause. Corrupting memory that is infrequently used will probably not cause an application to fail, but when you try to free the memory, if the bookkeeping data used by the heap was also corrupted, the free call will fail. To rule that out, I walked up the stack to find free()’s caller and figure out what kind of data I was dealing with:

  free(lib->name);
  lib->name = NULL;

Based on this, we can guess that the data is a name (and if you read more code, it’s clear that it is in fact a single-byte-per-character null terminated string). Finding the pointer being freed within the crashed process and looking at it in the debugger’s Memory window shows a well-formed string (see the Memory 1 and Locals panes in the screenshot above).

So, casual inspection suggests that this is not memory corruption, but it’s still possible. What if the library in question is corrupting memory near the string, and that’s why free is failing when we unload the library?

To rule that out, I climbed further up the stack and discovered that there’s an environment variable you can set that prevents the library from being unloaded. Interestingly enough, setting this variable causes the crash to go away… but one of Firefox’s child processes, plugin-container, then crashes in the same way, unloading a different library. This suggests that it’s at least not specific to one single library.

At this point, I decided I wanted to know where lib->name was coming from. I tried a couple quick searches with DXR (for lib->name, then ->name), but the first didn’t provide any results, and the second provided a bunch of results.

So, instead, I started Firefox up under Heap Profiler, and then attached a debugger and shut it down normally. I followed the above steps to get the address of the pointer being freed, and then opened up the Heap view of the latest snapshot I had taken in Heap Profiler. It was at this point I realized that I had completely failed to anticipate this use case when I was working on Heap Profiler, so I had to manually scroll through the heap, looking for the address I wanted. Whoops. Luckily, the address was near the beginning of the heap, so I found it easily:

Now, given the stack, I knew exactly who was responsible for allocating the string. Closer inspection even revealed why I was getting a crash: The string was being allocated by msvcr100!malloc, but freed by msvcr100d!free. Wait, how is that even possible?

I set breakpoints on the respective calls and started Firefox under the debugger, to ensure that I wasn’t being tricked by DLL load/unload operations that were occurring between the malloc and free calls. After seeing that the data from Heap Profiler matched reality, I decided to do some more digging. I opened up the Modules window to confirm that yes, both the debug and release C++ runtimes were loaded, and used the watch window to find the addresses of free and strdup, respectively (as strdup was the function responsible for calling the release malloc, in this case).

As you can see in the highlighted sections, __imp__free, the import symbol for free, points into MSVCR100D, while __imp___strdup, the import symbol for _strdup, points into MSVCR100. This is pretty confusing, and definitely confirms that the crash we’re seeing is caused by mismatched heaps.

At this point, there are two major questions:

1. How are we loading both versions of the runtime in a debug build?
2. And, how is this causing strdup and free to get imported from different runtimes?

This is especially troubling, because my local build of Firefox seems to run perfectly, other than this shutdown bug. You would expect mismatched heaps to cause all sorts of trouble. I did some digging around, and arrived at part of the answer to both questions:

This screenshot shows that nspr4.dll is linked against both runtimes, but only imports _putenv and _strdup from the release runtime. Everything else comes from the debug runtime! This explains why the heap mismatch is limited to the one crash I’m seeing, instead of causing widespread problems with Firefox – the effects of this linking problem are (theoretically) limited to code within NSPR, and further limited to code that uses _strdup or _putenv. Very mysterious.

After confirming that NSPR4.dll didn’t link against both libraries when compiled on my home machine, I set out to try and get verbose output from the linker, to identify just why I was getting linked against both runtime libraries. And this is where I made a mistake.

Earlier in the debugging process, I had convinced myself that the problem was reproducible, because checking out various revisions of the Firefox source code and doing full builds had caused the problem to persist. In order to get the linker to actually re-link nspr4.dll from scratch, I decided that I had to delete the contents of my build folder – I couldn’t get it to perform the link operation by simply deleting the output files (probably because I was deleting the wrong ones). So, I deleted my build folder, and performed a full build from scratch. Pouring over the log files for the build, I found the link command in question:

c:\mozilla\nsprpub\config\rules.mk:330:0$ rm -f nspr4.dll
c:\mozilla\nsprpub\config\rules.mk:345:0$ link -nologo -DLL -SUBSYSTEM:WINDOWS -DYNAMICBASE -OUT:"nspr4.dll" -DEBUG -MAP   advapi32.lib wsock32.lib winmm.lib ./prvrsion.obj io/./prfdcach.obj io/./prmwait.obj io/./prmapopt.obj io/./priometh.obj io/./pripv6.obj io/./prlayer.obj io/./prlog.obj io/./prmmap.obj io/./prpolevt.obj io/./prprf.obj io/./prscanf.obj io/./prstdio.obj threads/./prcmon.obj threads/./prrwlock.obj threads/./prtpd.obj linking/./prlink.obj malloc/./prmalloc.obj malloc/./prmem.obj md/./prosdep.obj memory/./prshm.obj memory/./prshma.obj memory/./prseg.obj misc/./pralarm.obj misc/./pratom.obj misc/./prcountr.obj misc/./prdtoa.obj misc/./prenv.obj misc/./prerr.obj misc/./prerror.obj misc/./prerrortable.obj misc/./prinit.obj misc/./prinrval.obj misc/./pripc.obj misc/./prlog2.obj misc/./prlong.obj misc/./prnetdb.obj misc/./praton.obj misc/./prolock.obj misc/./prrng.obj misc/./prsystem.obj misc/./prthinfo.obj misc/./prtpool.obj misc/./prtrace.obj misc/./prtime.obj io/./prdir.obj io/./prfile.obj io/./prio.obj io/./prsocket.obj misc/./pripcsem.obj threads/./prcthr.obj threads/./prdump.obj threads/./prmon.obj threads/./prsem.obj threads/combined/./prucpu.obj threads/combined/./prucv.obj threads/combined/./prulock.obj threads/combined/./prustack.obj threads/combined/./pruthr.obj md/windows/./ntmisc.obj md/windows/./ntsec.obj md/windows/./ntsem.obj md/windows/./ntinrval.obj md/windows/./ntgc.obj md/windows/./w95thred.obj md/windows/./w95io.obj md/windows/./w95cv.obj md/windows/./w95sock.obj md/windows/./win32_errors.obj md/windows/./w32ipcsem.obj md/windows/./w32poll.obj md/windows/./w32rng.obj md/windows/./w32shm.obj md/windows/./w95dllmain.obj  ./nspr.res

I dug around further in the log files to locate the nspr4.dll in question, deleted it, and then ran the link command with the additional -VERBOSE flag to get verbose linker output. Pouring through the verbose output from the linker, I located the two functions in question:

      Found __imp__free
        Referenced in prlog.obj
        Referenced in prlink.obj
        Referenced in prmem.obj
        Referenced in prdtoa.obj
        Loaded MSVCRTD.lib(MSVCR100D.dll)

      Found __imp__strdup
        Referenced in prlog.obj
        Referenced in prlink.obj
        Loaded OLDNAMES.lib(strdup.obi)

      Found __imp___strdup
        Referenced in OLDNAMES.lib(strdup.obi)
        Loaded MSVCRTD.lib(MSVCR100D.dll)

It was at this point that I realized my mistake: NSPR4.dll was now linking correctly. Somehow, my previous attempts at building older revisions of the code had not fixed the build, but entirely wiping out my build folder had fixed it. Now I felt like a moron.

On the other hand, the mystery grew slightly more mysterious: Somehow, strdup was different from free – the verbose linker output, even from this valid build, was different for the two! strdup, for some reason, was defined in OLDNAMES.lib, while free was defined in MSVCRTD.lib. Why the difference?

A quick search of MSDN reveals the truth: The POSIX strdup function is deprecated beginning in Visual C++ 2005, and you should use the ISO C++ conformant _strdup function instead. strdup is an alias to _strdup! This means that the way the linker treats it is different from the way the linker treats free.

Armed with this knowledge, I stumbled across a discussion forum post where someone encountered the same problem, by accidentally linking against both runtimes, and that post linked to a bug report on Microsoft’s Connect website. The bug report not only confirms that the crash in question is caused by having both heaps loaded, but describes the behavior as by design, and points out that strdup and free are being loaded from different runtimes.

And now, at the end of this long, frustrating journey, I have acquired many valuable things:

• A further distrust for huge build systems, and linkers in particular
• The knowledge that loading both debug and release C++ runtimes is definitely not a good idea
• The knowledge that aliases and exported functions in libraries are treated differently by the linker
• A Heap Profiler to-do item: Add support for searching the heap by address
• And most importantly: Don’t delete your build folder while debugging, even if you think you can reproduce the problem!

(As an aside, while debugging this problem I also discovered that debug builds of Firefox crash on my personal machine but not my work machine – but the crash only occurs if I run firefox.exe from within a bash shell. I kind of hate computers.)

I hope this short tale of woe has given you some new ideas about how to debug the next mysterious crash you encounter. If it hasn’t, I hope it at least entertained you!

In many graphical applications, a common pattern is the need to remember user preferences or inputs. Modern languages provide useful tools that can make this task simpler – for example, both Java and C# provide ways to automatically serialize an entire data structure into XML, which eliminates the need to manually read/write your preference values. Unfortunately, even with the aid of these features, you still find yourself with a lot of duplication – even if the runtime can automatically serialize your data structure, you still have to fill the structure out with your preferences, or read them back in manually. I’m going to demonstrate how you can use a little-known aspect of the new LINQ feature in C# 3.5 to eliminate duplication with the aid of the compiler.

As an example, we’ll be using the simple application shown above. It’s a Windows Forms dialog box, with a few different controls, set up to remember user inputs between sessions. You can download the source code for this application below:

Preferences Example (v1)

If you run the application, input some values, and click OK, you’ll find that it stores your inputs in a JSON file in the %AppData%\PreferencesExample folder. If you run it again, you’ll see that it pulls your inputs back out of the JSON file to populate the UI.

Let’s take a look at the relevant parts of the source code:

public void LoadPreferences () {
  if (!File.Exists(GetPrefsPath()))
    return;

  var serializer = new JavaScriptSerializer();
  var preferences = serializer.Deserialize<Dictionary<string, object>>(File.ReadAllText(GetPrefsPath()));

  checkBox1.Checked = (bool)preferences["checkBox1"];
  radioButton1.Checked = (bool)preferences["radioButton1"];
  radioButton2.Checked = (bool)preferences["radioButton2"];

  listBox1.SelectedIndices.Clear();
  var indices = preferences["listBox1"] as ArrayList;
  foreach (var index in indices)
    listBox1.SelectedIndices.Add((int)index);

  textBox1.Text = (string)preferences["textBox1"];
  dateTimePicker1.Value = (DateTime)preferences["dateTimePicker1"];
}

public void SavePreferences () {
  var preferences = new Dictionary<string, object>();

  preferences["checkBox1"] = checkBox1.Checked;
  preferences["radioButton1"] = radioButton1.Checked;
  preferences["radioButton2"] = radioButton2.Checked;
  preferences["listBox1"] = listBox1.SelectedIndices.Cast<int>().ToArray();
  preferences["textBox1"] = textBox1.Text;
  preferences["dateTimePicker1"] = dateTimePicker1.Value;

  if (!Directory.Exists(Path.GetDirectoryName(GetPrefsPath())))
    Directory.CreateDirectory(Path.GetDirectoryName(GetPrefsPath()));

  var serializer = new JavaScriptSerializer();
  File.WriteAllText(GetPrefsPath(), serializer.Serialize(preferences));
}

This application is relatively small – in particular, the JavaScriptSerializer handles a lot of the work involved in storing preferences for us. But regardless, we find ourselves having to list out all the control names multiple times, and some of those references to the controls are strings. Since they’re strings, that means that automatic refactoring tools, like the Rename tool in Visual Studio, won’t find them. As a result, this sort of code becomes difficult to maintain, especially in a larger codebase – changes are extremely error-prone and require careful, manual work.

What we want is a way to eliminate this duplication – a way to pull the list of controls and values out and store them in one place, so that we don’t have to specify them multiple times. Unfortunately, there’s no obvious way to do this in either Java or C#. While you can add the controls to a list quite easily, and walk over the list to get control names – eliminating part of the duplication – there’s no clean way to specify the particular fields/properties you wish to store.

Those of you familiar with reflection techniques may realize that you can eliminate the duplication by using reflection. By specifying a control and the name of the field we care about, we can use reflection to find that field and fetch its value. This technique will work, and other than the performance overhead of reflection, it will work well enough to ship in an application. However, it still suffers from the downside of breaking automated refactoring tools, and it’s somewhat error-prone due to the complexity involved in finding fields/properties via reflection.

One of the new features in C# 3.5 is support for ‘Expression Trees‘. Expression trees allow you to convert a C# expression into a tree of objects that describe the expression and allow you to evaluate it. This feature is used to support LINQ – for example, when you write ‘where i < 5′ in a LINQ query, the ‘i < 5′ portion of the query is turned into an expression tree, that LINQ can then use to run the query – for example, by turning it into SQL. As it happens, we can use this feature to solve our problem!

In C# 3.5, virtually any expression we can write can be converted into a delegate using lambda syntax, like so:

Func<int, int> func = (x) => x * 5;

And, as part of the Expression Tree feature, we can convert that lambda into an expression tree, at compile time:

Expression<Func<int, int>> expr = (x) => x * 5;

The compiler provides us all the information we need to evaluate an expression within the expression tree – if the expression references a field, the expression tree will contain the object containing the field, along with the FieldInfo object representing the field. Using this knowledge, we can eliminate the duplication from our sample application: By using a single expression to represent each preference we want to store, we can list all of our preferences in a single place in our application, and use that list of expressions to both load and save preference values. Best of all, because the compiler gives us full information on the expressions, we know the exact data types involved, and we can find out the name of the field being stored, and even the name of the object containing the field – eliminating the need to name each preference manually.

Using this knowledge, we can write a simple helper class that represents a field or property reference, and construct instances of that class by passing it an expression. You can find the source code to the class here:
http://code.google.com/p/fracture/source/browse/trunk/Squared/Util/Bind.cs
And we’ll be using it to eliminate the duplication from our sample application. The code is relatively small – a couple hundred lines – so by reading it you can easily see how to apply this technique for your own purposes, even if you don’t end up using my code.

The first step is to use expressions to build a list of all the values we want to store. We’ll do this by creating an array of bound members in the dialog box’s constructor:

IBoundMember[] BoundMembers;

public MainDialog () {
  InitializeComponent();

  BoundMembers = new IBoundMember[] {
    BoundMember.New(() => checkBox1.Checked),
    BoundMember.New(() => radioButton1.Checked),
    BoundMember.New(() => radioButton2.Checked),
    BoundMember.New(() => listBox1.SelectedIndices),
    BoundMember.New(() => textBox1.Text),
    BoundMember.New(() => dateTimePicker1.Value)
  };
}

Note that all we have to do is create an expression that evaluates to the value we wish to store. The C# compiler will convert it into an expression tree that tells us which object contains the value, and the FieldInfo or PropertyInfo we can use to fetch the value.

To eliminate the need to specify the name of each preference, we can combine the name of the field/property along with the name of the control containing the field/property, like so:

string GetMemberName (IBoundMember member) {
  return String.Format("{0}.{1}", ((Control)member.Target).Name, member.Name);
}

If we specify the Checked property of a RadioButton named radioButton2, this function will evaluate to ‘radioButton2.Checked’.

With these two pieces of functionality, we can now eliminate all the duplication from the old LoadPreferences and SavePreferences functions:

public void LoadPreferences () {
  if (!File.Exists(GetPrefsPath()))
    return;

  var serializer = new JavaScriptSerializer();
  var preferences = serializer.Deserialize<Dictionary<string, object>>(File.ReadAllText(GetPrefsPath()));

  foreach (var bm in BoundMembers)
    bm.Value = preferences[GetMemberName(bm)];
}

public void SavePreferences () {
  var preferences = new Dictionary<string, object>();

  foreach (var bm in BoundMembers)
    preferences[GetMemberName(bm)] = bm.Value;

  if (!Directory.Exists(Path.GetDirectoryName(GetPrefsPath())))
    Directory.CreateDirectory(Path.GetDirectoryName(GetPrefsPath()));

  var serializer = new JavaScriptSerializer();
  File.WriteAllText(GetPrefsPath(), serializer.Serialize(preferences));
}

Another thing you’ll notice is that the type casting from the original version is gone – since we have type information, we can automatically cast preference values to the appropriate type – and if we were using a strongly typed datastore, instead of JSON, we could actually store preferences directly in their correct type, without having to do any boxing or type casting.

If you download the source code for the revised example:
Preferences Example (v2)
You can run it and see that while it works the first time, and correctly saves preferences to JSON, it fails when loading preferences back from JSON. This turns out to be because of the JSON serializer: While listBox1.SelectedIndices is of type SelectedIndexCollection, the list we get back from the JSON serializer is of type ArrayList. The two types aren’t convertible, so the application fails.

Luckily, we can easily solve this problem. Since this technique works for any field or property, we can simply define a property that handles type conversion so that we don’t get an error when trying to load the preference value from JSON, like this:

// This has to be ArrayList since that's what we get back from the JSON serializer
ArrayList ListBox1_SelectedIndices {
  get {
    var result = new ArrayList();
    foreach (var index in listBox1.SelectedIndices)
      result.Add(index);
    return result;
  }
  set {
    listBox1.SelectedIndices.Clear();
    foreach (var index in value)
      listBox1.SelectedIndices.Add((int)index);
  }
}

If we change the list in the dialog box constructor to reference this new property instead of referencing the listbox directly, we can run the application and it will now successfully load and save preferences.
Preferences Example (v3)

With this technique, you can easily handle automatically loading and saving values from almost any datastore – and do so efficiently, without any unnecessary duplication, thanks to the information we’re given by the compiler. As an example, here’s a simple implementation of an object that loads and saves preferences from a database using this technique:
http://code.google.com/p/fracture/source/browse/trunk/Squared/TaskLib/Data/PropertySerializer.cs

I hope this post has shown you something useful! In the future I’ll be showing some other interesting applications of this technique.

I’ve released the source under the BSD license (just like Berkelium) and included two simple examples (one for Windows Forms, one for XNA). It’s not a Chrome competitor but it’s surprisingly easy to get a lot of complicated things working – you can watch Hulu videos if you have Flash installed, and Flash games like Dino Run work in the Windows Forms example since it implements keyboard input.

You can download pre-built demos to test it out, or grab the source code over at Google Code.

Have fun!

P.S. If you try out the examples, be aware that content that opens pop-up windows doesn’t seem to work. I think this is a bug in Berkelium.
Edit: Figured out how to build Chromium and Berkelium myself and fixed the bug. Open source is awesome!

However, doing this made it clear that I had some architectural issues to tackle: All of these mechanics were heavily dependent on the tunable constants for the creature in question, which meant I couldn’t just pull methods and variables out of my entity classes into classes of their own.

To solve the problem of accessing an object’s constants, I came up with a solution based on reflection. I can define a helper object designed to handle an aspect of an entity’s mechanics – for example, a HealthPool object to manage the creature’s health, along with associated aspects like regeneration. The helper object can define instance variables for the constants it needs access to, like so:

public class HealthPool {
    public Constant<float> HealthMax = null;
    public Constant<float> HealthPassiveRegen = null;
    public Constant<float> HealthRegenDelayTime = null;
    public Constant<float> HealthRegenRampTime = null;
    public Constant<float> FlinchThreshold = null;
    public Constant<float> FlinchThresholdDecay = null;

    public readonly RuntimeEntity Entity;
    public readonly ITimeProvider TimeProvider;
    public float Health = 0.0f;

Note that these variables are the same name and type as the actual tunable constants – the difference is that instead of being static, they’re instance variables. Doing this allows me to pull a function out of an entity’s source code without needing to change the way it references particular constants, since the constants have the exact same names as before.

Of course, since these variables default to null, we need some way to fill them in with references to the actual tunable constants we want to use. To do that, we apply reflection:

public static void BindConstants (object destination, params Type[] sourceTypes) {
    var genericConstant = typeof(Constant<>);
    var destinationType = destination.GetType();
    var destinationFields = new Dictionary<string, FieldInfo>();

    foreach (var field in
        destinationType.GetFields(BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Instance | BindingFlags.FlattenHierarchy)
    ) {
        var fieldName = field.Name;
        var fieldType = field.FieldType;

        if (!fieldType.IsGenericType || fieldType.GetGenericTypeDefinition() != genericConstant)
            continue;

        destinationFields[fieldName] = field;
    }

    foreach (var sourceType in sourceTypes) {
        foreach (var field in
            sourceType.GetFields(BindingFlags.Public | BindingFlags.NonPublic | BindingFlags.Static | BindingFlags.FlattenHierarchy)
        ) {
            var fieldName = field.Name;
            var fieldType = field.FieldType;

            if (!fieldType.IsGenericType || fieldType.GetGenericTypeDefinition() != genericConstant)
                continue;

            FieldInfo destinationField = null;
            if (destinationFields.TryGetValue(fieldName, out destinationField)) {
                destinationField.SetValue(destination, field.GetValue(null));
                destinationFields.Remove(fieldName);
            }
        }
    }

    if (destinationFields.Count > 0) {
        var constants = String.Join(", ", (from key in destinationFields.Keys select key).ToArray());
        var types = String.Join(", ", (from type in sourceTypes select type.Name).ToArray());

        throw new InvalidDataException(String.Format("Type(s) {0} do not declare the following constants:\n{1}", types, constants));
    }
}

What we’re doing here is pretty simple: We accept a reference to an object that has constants requiring binding, and a list of source types to retrieve constants from. The function operates in two stages: First, we enumerate all the instance variables defined in the target object, and build a list of all the tunable constants it has that need to be bound. After that, we enumerate all the static fields of the provided source types, looking for constants that have names matching those of the instance variables on the target object, binding them where appropriate. After this, we can simply check to see if our list of constants is empty or not – if it’s empty, we successfully bound all our constants, and if it’s not, we know that one or more of the desired constants was missing.

We get a few useful things out of this: First, accepting a list of types allows us to do simple inheritance of constants. If we first check the most-derived type and then the base type of an entity, that allows us to define a ‘default value’ for a particular constant, like ‘Maximum Health’, in a base class, and then define a new constant with the same name in the derived type. This also allows us to create ‘global defaults’ for a given constant – for example, if we always put Game at the end of the type list, we can have global game-wide constants for things like physics parameters, and only override them in specific classes if necessary.

Finally, to wire things up, we just need to do a little work in the constructor for our helper object:

    public HealthPool (RuntimeEntity entity, ITimeProvider timeProvider) {
        Entity = entity;
        TimeProvider = timeProvider;

        ConstantManager.BindConstants(this, entity.GetType(), entity.Game.GetType());

        Health = HealthMax;
    }

In this case, we’re initializing the HealthPool using the constants defined in the entity, and falling back to any constants defined in the Game when the entity doesn’t specify them. If a necessary constant is missing, we’ll get an exception thrown when constructing our helper object. Once we’ve bound the constants, we can just use them like we would otherwise – in this case, the HealthPool automatically initializes itself based on the HealthMax constant.

This is definitely preferable to the approach I used to use for exposing an entity’s constants – previously, for important constants like an entity’s bounding box size, I’d define an abstract property in a base class, and override it in each derived type to return the value of the constant. Now, I don’t need to use any abstract members or interfaces; I can just bind to the constants once when I initialize my helper objects.

One thing of note is that since this technique uses reflection, you might run into performance issues if you’re binding constants repeatedly. This is pretty trivial to solve, however; you can just cache the results of a constant binding operation based on the destination and source types, since those aren’t going to change at runtime.

So, at one point we decided to run a weekend promotion to promote our upcoming title. We temporarily enabled access to the content from our next game, and allowed people who didn’t own the game to create characters and play for the duration of the weekend. Great idea – get people in there at no cost, try and convince them your game is fun and worth playing. Works good for existing customers, too, because now they get excited about what’s coming down the pipe because you’ve let them play it for a couple days.

I was pretty excited about it, since it was the first promotion of its kind that we’d run since I had started working there – which meant I could finally show some of my friends what I was working on, and try and convince them it was worth playing. Most of them hadn’t even looked at the game since they had to pay for it first – and who can blame them, really?

Friday rolls around and I head home late and get some rest. The rest of the team does the same.

On Saturday morning, I send a message to a couple of my friends explaining how to download our client and try out the game. They’re both pretty enthusiastic about it and get started. A few minutes later, one of them says:

‘Why can’t I use any skills?’

What? What do you mean you can’t use any skills? Are there buttons on the bar at the bottom of the screen?

‘Yeah, but they have little lock icons over them. Nothing happens when I press them.’

Well, uh, s–t. That’s not supposed to happen. I emailed one of my coworkers to ask – uh, did anyone try creating a brand new account for the promotion to see if it worked? – and the answer is depressing: Nope. Luckily, it wasn’t a total loss – our existing customers were still able to participate in the event, because they already had working accounts. But for new, potential customers – the game didn’t work. They downloaded and installed our game, and it was mostly useless to them.

The worst possible scenario is that your game breaks and you don’t know about it. A modern game company can be quite hard to communicate with sometimes – when a person’s only choices for getting support are outsourced email support in India or ‘support forums’ full of angry trolls, they might just give up instead of trying to get help. Even worse, if they bought the game with a credit card, they might just issue a chargeback instead of trying to get the game fixed – which costs you more than it would have to issue a refund – all because they either couldn’t, or simply didn’t, get in touch with you about their issues.

As a developer, you want to do everything within your power to make sure that a customer with a broken game remains a customer. Here are some important steps you can take:

Provide a visible, straightforward channel for user feedback and support. An email address printed in the front of the manual, placed at the bottom of error dialogs, or on the front page of your website are a good start.

You can get extravagant by using things like forums, twitter, blogs, etc. to communicate with your customers – some of them will probably appreciate it – but it is tremendously important that whatever tool you provide for customer feedback be as simple and straightforward as possible, and that it be easy to find. Many game companies have switched to hugely complicated ‘automated support systems’ that require logins, registration, password management, support tickets, etc. While these systems make sense from the perspective of a studio, most gamers simply won’t bother and will issue a chargeback instead of trying to figure out why your automated support system’s registration page is sending them an HTTP 500. If you want to manage support tickets using an automated system, go for it, but you absoutely need to accept customer feedback via email or some equally accessible method.

Also, when considering the use of twitter, forums, or blogs, keep in mind that visible complaints and negative feedback cost you more than invisible ones. You should strive to encourage your customers to contact you privately before taking their issue public for the world to see. Ideally, you can resolve it in private before anyone else ever sees it, which limits the negative impact the issue has on your image as a developer.

Prompt responses to feedback are also important. Nobody expects you to solve a bug in 2 hours, but you should strive to send your customer an initial response to their feedback as quickly as possible, even if the response is ‘We’re taking a look at your issue, please be patient – here’s a link to the ticket in our support system’.

Provide news to your customers about ongoing issues with your game in an obvious and accessible location (your website’s homepage or the homepage for your title are good choices).

Running a website is, contrary to popular opinion, quite difficult. As a result, some developers tend to … forget to update their website, or simply don’t offer support information for customers in a visible place, relying on blogs, forums, or news sites to serve as their method for communicating with customers.

This is inadequate. It sucks, but you need to be putting regular updates out there for your customers in the first place they’re probably going to look – your website. If possible, you should put that news in the launcher for your game, if it has one. When you’re showing your customers an error message because the game failed to start, you should make sure to point them at your website, too, just to be sure that they get any information you have for them.

It costs you time and money to communicate with your customers, but it’s time well spent and money that will pay for itself by keeping those customers from leaving.

Design your game so that when it breaks, it breaks gracefully. Present clear, informative error messages that customers can pop into Google and send you via email.

So, let’s be honest. Software in general does not break gracefully. People are used to cryptic error messages by now. Regardless, you should strive to do better.

Every failure should present an error message that:

• Tells your customers what to do to resolve the problem (even if it is ‘contact customer support at support@mycompany.com’)
• Clearly identifies the failure. Providing an error number or error code is good, so is providing the location where the error occurred. However, if possible you should avoid presenting gigantic stack traces or error dumps to the customer – they’re overwhelming, not helpful. Get them back to you, the developer, some other way.
• Lists any user-actionable problems that contributed to the failure – ‘your microphone isn’t plugged in’ is actionable, while ‘your Direct3D driver returned D3DERR_INVALIDCALL’ is not.

Do whatever you can to find out about failures.

When your game fails, if possible, it should ‘phone home’ to notify you. At the very least, it should report the version number of your game, the error that occurred, and a stacktrace of where it occurred.

Sometimes it’s not possible to phone home – no internet connection, for example – and in that case, it’s okay to give up. If you’re concerned that failures happen a lot in this environment, consider storing the most recent failure message in your game’s AppData directory and reporting it the next time your game is launched with an internet connection. Better than nothing, right?

In many cases you may find it useful to report detailed information on the user’s system configuration, and the circumstances under which the failure occurred – remaining memory and disk space available, etc. This is a good idea, but if you do it, you must be absolutely certain that you do not report a single shred of your customer’s personal/private information along with the crash report. If you do, they will be completely justified in kicking up an annoying s–tstorm about it on the internet.

Some customers will complain about your game phoning home no matter what you do. As long as you only phone home when your game fails, you can safely ignore them. You’re doing your best to make sure the game works for everybody, and you’re not compromising their privacy. If they’re worried, they can unplug the ethernet cable.

Note that if your game requires internet access to work anyway, you should probably also consider reporting a successful start and exit of your game as well. This will help you track down situations where your automated crash reporting also fails to work correctly (these are more common than you might think).

Don’t compromise your game’s reliability for business reasons.

Now, to begin with – I’m not a fan of DRM, but I’m not going to try and convince you to abandon it here. If you’re already using DRM, you probably have a reason for doing so.

However, there is no excuse for allowing DRM – or any other addition to a game title – to compromise the reliability of your game. Any addition -  DRM, an automatic updater, a ‘compatibility check’, etc – should never prevent customers who purchased your title from playing it.

Nobody’s perfect, of course – sometimes your game is going to break despite your best efforts, and it will be due to some secondary part of the game, like DRM. In these situations, the previous rules apply: Provide clear, informative error messages, and communicate with your customers.

When a game breaks because of a business decision, customers tend to be less willing to forgive. They just want to play your game; they don’t care about your revenue. Sorry. Therefore, while a good business decision is still a good business decision, you should always strive to make sure that you don’t make your game more likely to break.

If the addition of DRM reduces piracy by 2% but causes the game to break for 4% of your legitimate customers, you just kneecapped the effectiveness of that DRM package, which doesn’t benefit anyone.

]]> http://www.luminance.org/blog/code/2009/07/15/shipping-a-brick/feed 0 http://www.luminance.org/blog/code/2009/07/01/this-application-is-not-configured-to-use-perfhud-how-to-fix-it http://www.luminance.org/blog/code/2009/07/01/this-application-is-not-configured-to-use-perfhud-how-to-fix-it#comments Thu, 02 Jul 2009 01:35:21 +0000 Kael http://www.luminance.org/?p=536 For anyone trying to use NVPerfHud, if you get stuck trying to figure out this error message:

This application is not configured to use PerfHUD.

Consult the User’s Guide for more information.

You may have noticed that there is no reference to this error message, or ‘configuring your application’ in the User’s Guide. The solution is actually documented in the ‘NVPerfHud Quick Tutorial‘, and involves changing the parameters you pass to CreateDevice.

Hopefully posting this here will help other people find this information, since googling the error message turned up zero results for me. Good work, NVidia!

P.S. Preventing ‘unauthorized profiling by third parties’? What the hell is this, the cold war? This is computer graphics, not espionage. Can you imagine if other software worked this way?

To prevent unauthorized pixel modification by third parties, any images you wish to view with Adobe Photoshop must contain a hand-written sample of dialogue from the second act of A Midsummer Night’s Dream, by William Shakespeare. Please note that dialogue from modern adaptations like the 1999 version featuring Kevin Kline is not sufficient. Also please note that handwriting in sufficiently obscure cursive scripts is unlikely to be recognized.

One of the problems with using vertical sync in a video game is that it eats into the available CPU time for performing game updates. The way vsync is implemented in most graphics APIs, it causes your Present/EndDraw/SwapBuffers call to block until the card enters vertical blank and the frame is shown to the user. While this is ideal from a correctness perspective, it’s a tremendous waste since it means you can end up sitting there for up to 16 milliseconds, waiting for vertical blank. If your game spends lots of time doing both updating and drawing, all that time could be spent performing updates instead. Ouch.

Currently, my game spends about as much time drawing as it does updating. A significant portion of the time spent drawing (20-30%) is within the EndDraw function. Turning off vertical sync drops the amount of time spent in EndDraw considerably, but introduces tearing. So, as a potential solution, why not call EndDraw on a background thread? While the thread waits for vertical blank, I can begin performing the next frame’s Update, and in the event that I finish updating before the previous frame is visible, I simply wait for that previous EndDraw call before beginning to paint the next frame. In the optimal case, this means I can come much closer to the best possible framerate without introducing tearing, and in the worst case, the cost of rendering an individual frame is only *slightly* increased by the use of thread synchronization. The fact that I’m only doing EndDraw on another thread means that I don’t have to worry about protecting my game data with locks and other synchronization techniques, since the GraphicsDevice doesn’t use any of my game data when performing the EndDraw operation.

So, to test this out, I overrode my Game class’s BeginDraw and EndDraw methods. This turns out to be all we have to do to change the way drawing is performed, because the XNA Framework developers were kind enough to make both of these methods virtual.

        protected override bool BeginDraw () {
            _DrawCompleteEvent.WaitOne();
            _DrawCompleteEvent.Reset();
            return base.BeginDraw();
        }

        protected override void EndDraw () {
            _DrawRequiredEvent.Set();
        }

Of course, at this point, the two events used here are never set, so this code won’t work. Thus, we add a background thread to perform our painting:

        AutoResetEvent _DrawRequiredEvent = new AutoResetEvent(false);
        ManualResetEvent _DrawCompleteEvent = new ManualResetEvent(true);

        public Game () {
            ...

            _DrawThread = new Thread(DrawThreadFunc);
            _DrawThread.IsBackground = true;
            _DrawThread.Start();
        }

        protected void DrawThreadFunc () {
            while (true) {
                _DrawRequiredEvent.WaitOne();
                base.EndDraw();
                _DrawCompleteEvent.Set();
            }
        }

Fairly simple thread programming here: We create a thread, and set IsBackground to true so that it will stop as soon as the main thread exits. The thread spends all of its time waiting for a ‘required draw’ signal, and then performs an EndDraw call. Once the call is complete, it sets another signal to inform the Game class that the previous draw has finished and it’s safe to perform a BeginDraw call (this lets us make sure that we never use the GraphicsDevice on the main thread while the background thread is performing an EndDraw).

Once this is all done, I start up my game, and… the framerate isn’t any different. Huh? What’s more, my frame profiler indicates that Update is now taking ten times as long as it used to, while Draw isn’t any faster. Huh???

In situations like this, it’s always good to consult a profiler to see if you’re missing something important:

So, we can see that the DrawThread is definitely doing its job – it calls EndDraw, then waits for a signal asking it to draw again. Both are taking about as much time as we’d expect. But why is Update taking so long…?

… oh. Oops.

So it turns out that I was using GraphicsDevice.Viewport.Width and GraphicsDevice.Viewport.Height in my camera code. Accessing the Viewport property caused the XNA framework to call into Direct3D to retrieve the viewport, which acquired the exact same lock being used by EndDraw, causing my main thread to stall until the draw completed. WHOOPS.

This is especially embarassing since the viewport size never changes anyway, so I could have just stored the width/height into constants. After doing just that and starting the game again, the profile looks more like you’d expect:

What’s more, this is actually an improvement: With vertical sync enabled, this results in a significant reduction in the amount of time spent inside the BeginDraw/Draw/EndDraw functions on the main thread, which means there’s more time left to perform Updates. This means that I can maintain a solid, smooth 60fps easier on dual-core/hyperthreaded machines.

Even with vertical sync disabled, this is still an improvement, though not as significant – apparently other things are happening inside EndDraw (not a big surprise), so by shifting that work off onto a second thread, I’m still gaining some time to spend performing the next update. When I disable the built in framerate balancer, this brings my framerate from ~350fps up to ~380fps. Not bad for a couple dozen lines of code!

Of course, it’s worth pointing out that the XNA Framework documentation doesn’t make any promises here, so it’s possible that this technique is unsafe. When it comes to concurrency, it’s very easy to do the wrong thing and get away with it – as you might have noticed here, I was doing something utterly stupid and unsafe in my Update function, and I got away with it because the DirectX developers had the foresight to put a lock in the right place. If they hadn’t, my game might have corrupted state from accessing the GraphicsDevice on two threads, and crashed intermittently.

Regardless, this is a handy technique – once I’ve had the chance to do lots of testing on various PC configurations (and the XBox 360), I’ll probably be using it in my game when I ship.

It was good enough for me, and the performance wasn’t *terrible*, so I didn’t worry about it much. But I always had this nagging doubt in my mind – this solution is obviously a hack. Isn’t there a way to do this with pure math?

Well, I’m not very good at math, but it turns out there is a way to do it with pure math. So today I threw out all the motion and collision code I had written for the platformer and decided to try a pure math approach, and see if I could resolve collisions non-iteratively. Suprisingly enough, it wasn’t actually that hard.

Here’s a summary of how it works:

I already do my collision detection using the separating axis theorem (SAT). To check for collision between a pair of convex polygons, I compute a set of axes based on the polygons (a vector perpendicular to every edge of each polygon) and then for each axis in the set, I project both polygons onto that axis and see if they overlap. As soon as I find an axis in which the polygons are not overlapping, I know that they are not intersecting – no need to check all of them.

Pretty simple, and actually quite easy to code – Around 20 lines of C# for a relatively efficient implementation of the check itself, and another 20 lines to compute the list of axes without any duplicates, since there’s no point in checking the same axis twice.

With the algorithm described above, it’s actually quite simple to build motion on top of it. Once I’ve projected both polygons onto a given axis, and confirmed that they don’t intersect, I can do another projection – I can project the velocity of the moving polygon onto that same axis, and add that to the original projection of the moving polygon to get the projection of its new, desired location. At that point, I know whether the polygon will intersect on this axis as a result of moving, and not only that, I can compute the amount of intersection by determining how much the two intervals overlap. Once I have that value, I can use it to compute a vector that will separate the two polygons.

Once I iterate over all the axes and compute separating vectors, I can pick the smallest separating vector and add it to the original velocity of the polygon. At this point I can do some checks to see if the new velocity is acceptable – for example, did the separating vector cause the polygon to change directions entirely? In that case, it’s probably not acceptable, and we can just flat-out reject movement and say that the polygon cannot move in this direction at all. Likewise, if the separating vector is enormous, that means the polygon moved all the way through the obstruction to avoid collision, which we obviously don’t want either, so we can reject it.

The end result was essentially this:

• For every axis, project movingPolygon and stationaryPolygon onto that axis. This produces two intervals – movingInterval and stationaryInterval.
• Check to see if movingInterval and stationaryInterval overlap. This tells us whether or not the polygons were already colliding.
• Project the velocity of movingPolygon onto the axis, and add that projected value to movingInterval. This gives us the new interval of the moving polygon – newInterval.
• Check to see if newInterval and stationaryInterval overlap. If they overlap, determine how much they overlap by – overlapAmount. At this point we know whether or not the two objects would have collided.
• If the newInterval is overlapping, we can now compute a vector that will separate the two polygons, by multiplying the current axis by the amount of overlap (the axis is a unit vector, and the amount of overlap is a distance).
• Add the separating vector to the original velocity of the polygon, to compute the new candidate velocity – newVector.
• Check to see if Magnitude(newVector) <= Magnitude(movingPolygonVelocity). If not, reject the new vector.
• Check to see if Normalize(newVector) ·dot· Normalize(movingPolygonVelocity) >= 0. If not, reject the new vector. (This ensures that we don’t push the object in the opposite direction to resolve the potential collision – in this case, the magnitude would have still passed the previous check)
• If the new vector was not rejected, we know that the two objects will not be colliding.
• Once you’ve iterated over all the axes, select the largest non-colliding vector (if any). That’s your resultVector.

At the end of this algorithm, you have four outputs:

• areColliding (right now, before applying the velocity)
• wouldHaveCollided (if we applied the original velocity)
• willCollide (if you apply the result velocity)
• resultVelocity

In the case where the collision could not be resolved, the resultVelocity will be 0. In the case where there was no collision, the resultVelocity will be the input velocity. In every other case, the resultVelocity is a vector that safely moves the polygon in the desired direction, without producing a collision.

One detail to note is that the quality of the result velocity depends on the axes that you check. If you only check axes generated from the shapes of the colliding objects, you can get strange (though valid) results – for example, two boxes will not produce any axes except X and Y to check, so if the object is moving at an angle, the algorithm will miss the ideal axis to compute the resultVelocity from. There are a few ways to solve this – you can pre-compute a set of ‘basic axes’ to check in every case, or you can derive a set of axes from the velocity of the object. Either way, though, the algorithm seems to work quite well – it works as well as the old one did in the platformer, and passes all my unit tests.

It’s always nice to go from 100 lines of code to 10 lines of code. Unfortunately, I’m not done yet. My previous implementation supported walking up and down slopes in a natural manner, and it also supported moving platforms. The new implementation supports neither, so I have some more work to do. Hopefully I can solve both of those problems using pure math, too – I’m working on sloped surfaces right now, and it looks like I can make it work with pure math by improving on my existing algorithm.

You can see the source code for my implementation here. Note that there’s some commented out bits in there from my work on sloped surfaces.

Starting at the beginning of high school, through my few years of college, and into the first year of my professional career, I spent a significant portion of my free time building and maintaining a toolset and game engine called Fury². It was built as a way for me to quickly prototype games and ideas, to learn about game design and engine design, and most importantly, to let me build games that I didn’t think I could create using any of the tools that were already out there.
It was a long and painful development project, and by the end, I realized that I had made numerous mistakes; despite that, I also gained many valuable lessons and learned things that I might not have learned any other way, and with the clearer perspective that hindsight brings, I consider it one of the things that made it possible for me to start a career doing things that I love without having to struggle with debt or worry about how I’m going to afford my next meal. For me, this project embodies the famous Nietzsche quote, ‘What does not kill me, makes me stronger‘.

Learning how much sheer effort goes into the process of creating even a simple piece of software has changed me for the better in many ways, so I’m writing this article in hopes that I can share some of the lessons I’ve learned with others. Unfortunately, the benefit of hindsight also brings with it the fact that I don’t remember many of the things that happened during the 8 or so years I spent developing the project. I hope that despite this, you can benefit from what I’ve learned as much as I have.

Of course, I must preface the contents of this article with a warning: It’s written in fairly technical terminology, and does not have any particular narrative structure. I have tried to compile the most important issues and triumphs from the history of the project, and present them in a form that a programmer can understand. Regardless of whether you’re a programmer or not, I encourage you to respond via email or comments and let me know what parts of this you couldn’t understand, so that I can improve this article and make it worthy of your time.

- Kevin Gadd