In many graphical applications, a common pattern is the need to remember user preferences or inputs. Modern languages provide useful tools that can make this task simpler – for example, both Java and C# provide ways to automatically serialize an entire data structure into XML, which eliminates the need to manually read/write your preference values. Unfortunately, even with the aid of these features, you still find yourself with a lot of duplication – even if the runtime can automatically serialize your data structure, you still have to fill the structure out with your preferences, or read them back in manually. I’m going to demonstrate how you can use a little-known aspect of the new LINQ feature in C# 3.5 to eliminate duplication with the aid of the compiler.

As an example, we’ll be using the simple application shown above. It’s a Windows Forms dialog box, with a few different controls, set up to remember user inputs between sessions. You can download the source code for this application below:

Preferences Example (v1)

If you run the application, input some values, and click OK, you’ll find that it stores your inputs in a JSON file in the %AppData%\PreferencesExample folder. If you run it again, you’ll see that it pulls your inputs back out of the JSON file to populate the UI.

Let’s take a look at the relevant parts of the source code:

public void LoadPreferences () {
  if (!File.Exists(GetPrefsPath()))
    return;

  var serializer = new JavaScriptSerializer();
  var preferences = serializer.Deserialize<Dictionary<string, object>>(File.ReadAllText(GetPrefsPath()));

  checkBox1.Checked = (bool)preferences["checkBox1"];
  radioButton1.Checked = (bool)preferences["radioButton1"];
  radioButton2.Checked = (bool)preferences["radioButton2"];

  listBox1.SelectedIndices.Clear();
  var indices = preferences["listBox1"] as ArrayList;
  foreach (var index in indices)
    listBox1.SelectedIndices.Add((int)index);

  textBox1.Text = (string)preferences["textBox1"];
  dateTimePicker1.Value = (DateTime)preferences["dateTimePicker1"];
}

public void SavePreferences () {
  var preferences = new Dictionary<string, object>();

  preferences["checkBox1"] = checkBox1.Checked;
  preferences["radioButton1"] = radioButton1.Checked;
  preferences["radioButton2"] = radioButton2.Checked;
  preferences["listBox1"] = listBox1.SelectedIndices.Cast<int>().ToArray();
  preferences["textBox1"] = textBox1.Text;
  preferences["dateTimePicker1"] = dateTimePicker1.Value;

  if (!Directory.Exists(Path.GetDirectoryName(GetPrefsPath())))
    Directory.CreateDirectory(Path.GetDirectoryName(GetPrefsPath()));

  var serializer = new JavaScriptSerializer();
  File.WriteAllText(GetPrefsPath(), serializer.Serialize(preferences));
}

This application is relatively small – in particular, the JavaScriptSerializer handles a lot of the work involved in storing preferences for us. But regardless, we find ourselves having to list out all the control names multiple times, and some of those references to the controls are strings. Since they’re strings, that means that automatic refactoring tools, like the Rename tool in Visual Studio, won’t find them. As a result, this sort of code becomes difficult to maintain, especially in a larger codebase – changes are extremely error-prone and require careful, manual work.

What we want is a way to eliminate this duplication – a way to pull the list of controls and values out and store them in one place, so that we don’t have to specify them multiple times. Unfortunately, there’s no obvious way to do this in either Java or C#. While you can add the controls to a list quite easily, and walk over the list to get control names – eliminating part of the duplication – there’s no clean way to specify the particular fields/properties you wish to store.

Those of you familiar with reflection techniques may realize that you can eliminate the duplication by using reflection. By specifying a control and the name of the field we care about, we can use reflection to find that field and fetch its value. This technique will work, and other than the performance overhead of reflection, it will work well enough to ship in an application. However, it still suffers from the downside of breaking automated refactoring tools, and it’s somewhat error-prone due to the complexity involved in finding fields/properties via reflection.

One of the new features in C# 3.5 is support for ‘Expression Trees‘. Expression trees allow you to convert a C# expression into a tree of objects that describe the expression and allow you to evaluate it. This feature is used to support LINQ – for example, when you write ‘where i < 5′ in a LINQ query, the ‘i < 5′ portion of the query is turned into an expression tree, that LINQ can then use to run the query – for example, by turning it into SQL. As it happens, we can use this feature to solve our problem!

In C# 3.5, virtually any expression we can write can be converted into a delegate using lambda syntax, like so:

Func<int, int> func = (x) => x * 5;

And, as part of the Expression Tree feature, we can convert that lambda into an expression tree, at compile time:

Expression<Func<int, int>> expr = (x) => x * 5;

The compiler provides us all the information we need to evaluate an expression within the expression tree – if the expression references a field, the expression tree will contain the object containing the field, along with the FieldInfo object representing the field. Using this knowledge, we can eliminate the duplication from our sample application: By using a single expression to represent each preference we want to store, we can list all of our preferences in a single place in our application, and use that list of expressions to both load and save preference values. Best of all, because the compiler gives us full information on the expressions, we know the exact data types involved, and we can find out the name of the field being stored, and even the name of the object containing the field – eliminating the need to name each preference manually.

Using this knowledge, we can write a simple helper class that represents a field or property reference, and construct instances of that class by passing it an expression. You can find the source code to the class here:
http://code.google.com/p/fracture/source/browse/trunk/Squared/Util/Bind.cs
And we’ll be using it to eliminate the duplication from our sample application. The code is relatively small – a couple hundred lines – so by reading it you can easily see how to apply this technique for your own purposes, even if you don’t end up using my code.

The first step is to use expressions to build a list of all the values we want to store. We’ll do this by creating an array of bound members in the dialog box’s constructor:

IBoundMember[] BoundMembers;

public MainDialog () {
  InitializeComponent();

  BoundMembers = new IBoundMember[] {
    BoundMember.New(() => checkBox1.Checked),
    BoundMember.New(() => radioButton1.Checked),
    BoundMember.New(() => radioButton2.Checked),
    BoundMember.New(() => listBox1.SelectedIndices),
    BoundMember.New(() => textBox1.Text),
    BoundMember.New(() => dateTimePicker1.Value)
  };
}

Note that all we have to do is create an expression that evaluates to the value we wish to store. The C# compiler will convert it into an expression tree that tells us which object contains the value, and the FieldInfo or PropertyInfo we can use to fetch the value.

To eliminate the need to specify the name of each preference, we can combine the name of the field/property along with the name of the control containing the field/property, like so:

string GetMemberName (IBoundMember member) {
  return String.Format("{0}.{1}", ((Control)member.Target).Name, member.Name);
}

If we specify the Checked property of a RadioButton named radioButton2, this function will evaluate to ‘radioButton2.Checked’.

With these two pieces of functionality, we can now eliminate all the duplication from the old LoadPreferences and SavePreferences functions:

public void LoadPreferences () {
  if (!File.Exists(GetPrefsPath()))
    return;

  var serializer = new JavaScriptSerializer();
  var preferences = serializer.Deserialize<Dictionary<string, object>>(File.ReadAllText(GetPrefsPath()));

  foreach (var bm in BoundMembers)
    bm.Value = preferences[GetMemberName(bm)];
}

public void SavePreferences () {
  var preferences = new Dictionary<string, object>();

  foreach (var bm in BoundMembers)
    preferences[GetMemberName(bm)] = bm.Value;

  if (!Directory.Exists(Path.GetDirectoryName(GetPrefsPath())))
    Directory.CreateDirectory(Path.GetDirectoryName(GetPrefsPath()));

  var serializer = new JavaScriptSerializer();
  File.WriteAllText(GetPrefsPath(), serializer.Serialize(preferences));
}

Another thing you’ll notice is that the type casting from the original version is gone – since we have type information, we can automatically cast preference values to the appropriate type – and if we were using a strongly typed datastore, instead of JSON, we could actually store preferences directly in their correct type, without having to do any boxing or type casting.

If you download the source code for the revised example:
Preferences Example (v2)
You can run it and see that while it works the first time, and correctly saves preferences to JSON, it fails when loading preferences back from JSON. This turns out to be because of the JSON serializer: While listBox1.SelectedIndices is of type SelectedIndexCollection, the list we get back from the JSON serializer is of type ArrayList. The two types aren’t convertible, so the application fails.

Luckily, we can easily solve this problem. Since this technique works for any field or property, we can simply define a property that handles type conversion so that we don’t get an error when trying to load the preference value from JSON, like this:

// This has to be ArrayList since that's what we get back from the JSON serializer
ArrayList ListBox1_SelectedIndices {
  get {
    var result = new ArrayList();
    foreach (var index in listBox1.SelectedIndices)
      result.Add(index);
    return result;
  }
  set {
    listBox1.SelectedIndices.Clear();
    foreach (var index in value)
      listBox1.SelectedIndices.Add((int)index);
  }
}

If we change the list in the dialog box constructor to reference this new property instead of referencing the listbox directly, we can run the application and it will now successfully load and save preferences.
Preferences Example (v3)

With this technique, you can easily handle automatically loading and saving values from almost any datastore – and do so efficiently, without any unnecessary duplication, thanks to the information we’re given by the compiler. As an example, here’s a simple implementation of an object that loads and saves preferences from a database using this technique:
http://code.google.com/p/fracture/source/browse/trunk/Squared/TaskLib/Data/PropertySerializer.cs

I hope this post has shown you something useful! In the future I’ll be showing some other interesting applications of this technique.

For a system like this you have a few important pieces:

• Your game needs to collect data during play sessions – in my case, I track important events like player/creature death, and periodically sample the player’s position to get a general idea of where players are in a given level during their session.
• You need a way for your game to periodically report collected data to a remote server. You have to handle lots of edge cases here – for example, it’s likely some people will play without an active internet connection or suffer temporary loss of connection, so you need to batch up events to report later in this case – you definitely don’t want the game to fall over and choke without access to the internet, and if possible you want to avoid losing data too.
• You need to build a set of services to collect data from the game. This typically means you also need services to handle things like uniquely identifying individual play sessions and computers, so that you can track achievements for individual players and perform analysis on individual sessions instead of only on a player’s entire play history.
• You need to build an in-game frontend, to expose collected data to a player. This means turning your event data into user-friendly statistics – like a kill counter, or an achievement for killing a particular boss. One interesting challenge here is that you probably want to expose this information online, too, so that players can share their profiles and achievements.

In this post, I’m going to cover the first two.

Collecting Data

So, first, you need a way to collect important player data. For it to be useful, you need to collect it in as consistent a format as possible; you don’t want every piece of data to have a unique set of parameters attached to it, for example, because that would make it impossible to perform any generic analysis of your data. Likewise, you want to try and keep your data ‘denormalized’, by storing as much relevant data together as possible. If you achieve both of these goals, you end up with a stream of ‘events’ that represent the things you care about in a form that lets you inspect them individually or analyze them in large groups.

In my case, this step was actually quite straightforward. I already had a simple ‘event bus’ integrated into the game, that I was previously using to synchronize animation and sound effects with gameplay. Given this, it was simple to create a data collector that attaches to the event bus and listens for events that it wants to collect.

I did, however, have to make some changes – many of my events didn’t expose the information necessary to be useful; for example, the event representing a player death didn’t contain any information to tell you who (or what) killed the player, so I had to tweak the game code to attach that to the event. Likewise, many other events didn’t include position information, which made it hard to get meaningful information about them – knowing that the player grabbed onto a ledge isn’t particularly useful if you don’t know *where* or *which ledge*. In some cases, the data collector attached important parameters into the events so that game objects wouldn’t have to; for example any event coming from the Player automatically has PlayerX and PlayerY parameters attached to it (since the Player is the source of the event, it is simple to extract a position and attach it to the event).

One thing that required some special treatment as well was recording the player’s position over time – while attaching position information to events gives you a general idea of the player’s location, it’s not very useful for figuring out where players are getting stuck or confused. To address this, I built a simple class that periodically samples the player’s position and adds it to a list. Every time the list reaches a certain size, its contents are batched up into a single event – a ‘HistoryBatch’ – and sent to the data collector. This allows me to get fairly high-resolution position data without having to report hundreds of individual events every minute (which would have contained lots and lots of redundant information).

Reporting Data

Once you have your player data, you need to report it to your server (and probably store some of it locally, too). I honestly haven’t tackled the latter part yet, but here’s how I handled the former:

To handle periodically reporting events, I created a simple ‘Event Reporter’ class that collaborates with the data collector. Essentially, the event reporter maintains a queue of pending events, and runs a thread that is responsible for sending batches of events to the remote server once the queue gets large enough. It automatically waits a certain amount of time between requests, and attempts to batch events into groups instead of reporting them one at a time as they occur. This allows me to get relatively low latency (if I want it) but still remain relatively efficient in my use of the network. Whenever the data collector gets an event, it adds it to the event reporter’s queue.

Once I have a batch of events ready to send to the server, I pack them together into a class that’s laid out optimally for network transmission. I then use .NET 3.5’s built in JSON serializer to convert the batch into a blob of JSON data ready to send to my remote server. A typical blob looks like this:

{
  "PlayerId":xxxx,
  "SessionId":yyyy,
  "Events":[
    {"Type":"ActiveControllerChanged","Time":0,"Data":{
      "IsKeyboard":true,
      "ConnectedGamepads":0,
      "ActiveController":0
    }},
    {"Type":"LevelLoaded","Time":5436,"Data":"troupe"},
    {"Type":"HistoryBlock","Time":7788,"Data":{
      "LevelName":"troupe",
      "PlayerX":[-136,-74,35,192,209,209],
      "PlayerY":[588,875,920,920,920,920]
    }}
  ]
}

As you can see, events are small when possible – they simply store the event type and the timestamp at which the event occurred, along with their associated information. In cases where an event has lots of information attached, I send a dictionary, but in some cases I can just send a single value. In the case of a history block, I make sure to send the X and Y coordinates as individual lists, so that I don’t have to waste time encoding ‘PlayerX’ and ‘PlayerY’ along with every individual coordinate. A batch of events has a player ID and a session ID attached, which allows me to analyze individual play sessions and track achievements. (Note that both of these IDs are randomly generated on the server, so they’re anonymous.)

The reporter starts an HTTP POST to send the JSON to the server, and goes to sleep until it’s finished. If the send fails, the events are left on the queue and the thread sleeps for a while so it can retry sending them later. Once a send completes, the reporter checks to see if the queue is empty – if the queue is now empty, it goes to sleep until the queue contains items again. If the queue still contains items, it sleeps for a short while and prepares to send another batch – this allows the event reporter to spend the majority of its time asleep, but still handle large bursts of events without sending a huge blob of events at the server in one POST (since that would be very likely to fail or time out.)

One important detail is that the reporter needs to be able to correctly report events even if the player quits. In my case, the player can quit at any time, and I can end up with a very large number of events in the queue at the time. To deal with this, once the game has finished tearing down, the event reporter is given around 30 seconds to report any remaining events to the server. This usually allows events to get through, but prevents the game from hanging indefinitely on exit (in the event that there’s some sort of network failure, or the event queue is really full). Since this occurs after teardown, the game window is gone so the player doesn’t get the impression that the game has hung, and since the reporter spends most of its time sleeping, they won’t see any significant CPU usage either.

The reporter also runs in its own isolated thread and uses its own task scheduler, instead of sharing resources with the game. This increases the likelihood that the game will be able to successfully report events in the case of a bug or crash, and also means that if the reporter fails for some reason – probably a network outage – it won’t take the game down with it.

In my next post, I’ll cover the remaining two items and show you some of the things you can do once you’ve gathered player data. In the interim, you can take a look at my player profile!