So, this past weekend I participated in the 22nd Ludum Dare competition. You have just 48 hours over the weekend in which to create a game. The crux of the competition is that you create all code and content within that 48 hour period. No re-used assets at all. Only engine/middleware/framework code is permitted to be prepared beforehand.

I wrote my game with Unity. It’s the first project I’ve ever undertaken in Unity, and I had very little experience of it. I came out of the other side with a very positive impression, I really liked using it. As much as I love C/C++; given the time constraints, it seemed a good idea to go with Unity. The other big upside is that you can deploy the game to be played via a web browser. This was definitely what I wanted. I didn’t want people to have to go to the hassle of a download-and-install, just to play the little toy game I made over a weekend.

I wrote a turn-based strategy RPG game. My inspiration was the “Shining Force” series of games. The theme of Ludum Dare this time was “Alone”; every game submitted had to follow this theme. This obviously doesn’t fit the team-based gameplay of Shining Force, but mine has a twist. You just play as a single character. It just takes the combat part of those RPG games, there’s no walking around and talking to villagers. There is a shop to purchase and sell items between battles, but that’s it.

The Game

Here’s a link to play the finished game:

“Adventures of One”

I was disappointed with what I ended up with, mostly due to how much work I had to squeeze into the time. In my eyes the game had two major failings:

• A lack of polish. The art is my placeholder art, and the GUI is the default-Unity GUI.

• Little game balancing. The first world is decently balanced, but after that it gets way too easy. The game is completely procedurally-generated. I just didn’t have the time to playtest it.

The biggest achievement though was:

• I wrote a game in 48 hours, that’s actually playable; and a little bit fun!

That’s my introduction out of the way. Now, onto the postmortem…

What Went Right?

• Choosing an idea which I’d see through to the end. I actually lost most of the first two hours to thinking over what type of game I’d do. The theme ‘Alone’ was tricky for me, as most of the ideas I had before it was announced didn’t fit it. I ended up taking one of those ideas and simplified it. In retrospect doing a full-blown party of player characters would have been ridiculously hard to do in 48 hours. But yeah, I was excited by the idea and that helped me keep my motivation.
• Using a higher level language like C#. Considering the game didn’t make too much use of Unity-specific ‘scene’ features, and was mostly generated in code; I could have had similar results using something like XNA instead. Either way, this sort of rapid development lends itself to something higher-level like C#. Out of habit, I think I would have been too anal about my code if I wrote it in C++. C# just lets you throw caution to the wind that little bit easier!
• Enemy AI. Pretty close to “Shining Force”’s enemy AI. It slotted in well to the turn system I had set up; so it followed the same rules and states the player had. The turn system I wrote had its positives and negatives, but since I left the AI to pretty late on, I’m happy I spent the time earlier making the turn system generic.
• The UI. I left it to pretty late on. I hate coding UI. I’m glad though I just used the stock Unity functions and didn’t try any fancy rendering techniques. I went for the basics, and that was the right decision. I’d hoped I’d get the chance at the end during the polish phase to ‘skin’ it, but I had no polish phase.
• Save and load. Lost close to an hour implementing it, but it’s a pretty cool feature. It autosaves after each battle (Unity stores web-app info on your hard-drive, so it’s transparent). I also got in a crude Base64 load/save screen, to bring your save to other computers.
• Bug free? Well, it seems to be for me. I kept it in a pretty stable condition, considering the game’s complexity and reliance on procedural generation, and AI.
• Food, drink, and breaks. I think I was sensible with what I ate and drank. I took pretty regular short breaks. Even if it was to just chill on the sofa with a notepad for five minutes, and plan out my next coding session. I’m not a coffee or energy-drink guy, so just cups of tea kept me going. I’m unsure how I wasn’t totally fatigued by the end, I’m guessing it was adrenaline.
• The ‘all nighter’. It saved me from not finishing. I’m an optimist when it comes to scheduling time. Re-reading my journal log, I hardly meet any of my set goals (but to be honest I did surprise myself with getting ‘unscheduled’ stuff done really quickly). If I’d had lost say, another five hours to a second nap, there is no way I’d have had a functional game come submission time.
• Placeholder work. Crappy art, blocks instead of characters, a checkerboard playing area for 90% of development. These sorts of things meant I had systems playable sooner. This helps my motivation, there’s nothing worse than building up to a point, where you’ve seen nothing on screen and hoping ‘everything will work out at the end’…. And then it inevitably doesn’t. My longest blind-coding run without actually seeing anything on the screen working, was about half an hour; for the turn system. Everything was short and sweet to rapidly iterate on.
• The audio. I think it went well. I dropped in the sound effects just after the halfway point, which provided good battling feedback, with the lack of particles/visual effects. The music I also procedurally generated. I lucked out with some good classical piano pieces which fit the style of game well. The voiceovers by the wife I threw in at the last minute. They worked out well, but I’d have liked to try and fix the audio balance between her and the music a bit more.
• Not caring about the code quality. It’s awful. Terribly hacked-about stuff that reminds me of the sort of thing I’d have written in college. I however still coded pretty defensively though. I just didn’t worry about things like commenting, and whitespace/indentation being off. Fix something ‘properly’, or throw in a sure-fire hack? A no brainer. I also committed many cardinal sins with global static singletons, instead of jumping through Unity’s component hoops. My typing fingers thanked me!
• Stand-up desk. I have an IKEA Jerker desk. It’s not adjustable, but I have a high stool which I use when I need a rest. Being able to constantly switch between two was awesome. When I’d feel lethargic in the chair, I could stand. When my feet got tired of the standing, I’d be able to take a load off and sit. I probably was about 50/50 in each position, time-wise.
• No internet distractions. I know some people who do these completions regularly like to use Twitter and IRC extensively during the dev time. I’m a bit of a procrastinator; those sorts of things can be a big timesink. I told myself I would just check my emails on my cellphone, and do nothing more than that. It worked out well. I’m real surprised that I didn’t feed my Chaos Engine addiction, throughout the whole 48 hours!

What Went Wrong?

• I left some important features until way near the end. In particular the attack logic I rushed through, such that it doesn’t scale into later battles. Enemy placement and the map terrain generation I did with less than two hours to go. I should have had all the procedural-generation stuff done very early, and other risky things like the battle logic calculations too. Instead I did the bread-and-butter stuff that’s hard to get wrong first, and did the risky stuff later.
• Very little playtesting. It’s a slow game to play through to test leveling into later battles. I should have made leveling stats deterministic and had some way of starting myself off in the later, harder battles to test balancing.
• Graphics/art. I spent hardly any time at all on the graphics. I can’t draw for shit, but I can usually fudge things to look very aesthetically pleasing despite that lack of talent. I didn’t even get any time to try that.
• A lack of preparation. I’d planned each day during the week prior to “look at Unity tonight”. I hadn’t got around to it. So I had to learn as I went. Some aspects like hooking up audio took a little longer than they would have, if I knew how to do it beforehand. I also found myself dropping in and out of the ‘component’ model that Unity has for objects, because it didn’t ‘click’ for me until a few hours in. I think I would have likely pen-and-paper planned out my components to be better organized, than the ad-hoc mess they ended up as.
• AOE (area-of-effect) magic; stuff that affects more than one tile. Probably lost a couple of hours spread over the whole period, to implementing it, and dealing with fallout from it. I envisioned it as being useful in later battles when there are tons of enemies; which drove me to add it in the first place. It’s a shame later battles are such a walkover, which kind of renders it just a time-saving device.
• I started working at about 11am on the Saturday, after about five hours of sleep. I took it easy in the morning, and watched some football as I coded. That time was actually pretty productive. I then carried on for well over 24 hours and worked up to the 6pm deadline the next day. I really didn’t feel too tired, but my productivity certainly slowed. I didn’t have many bugs to fix, but one in particular was niggling in the last four hours. I only fixed it shortly before submission. I think with a fresher head, I’d probably have not caused the bug in the first place.
• UI and state machines. I hate coding UI. I hate coding state machines. This game had plenty of both. I’m surprised it didn’t demotivate me. State machines always seem such a chore. So much code for what has very simple intentions. Scripting languages can make this a lot easier with their micro-threading capabilities. I remember reading something about C# and the ‘yield’ statement lending itself well to state machines, but it’s a little less straightforward to grok. I should have probably checked that out again.
• The scope. Too big. Too ambitious. Given my lack of preparation too, it was silly to think I could do the idea justice in less than 48 hours. I had a good stab at it, though.

“Picasso” – Look at that work of art!

My TODO List At The End

I had to prioritize the last few tasks, so I inevitably had some on the cutting room floor. Here are the features I really wanted to add:

• Particle effects on attacks. I’d seen some demos of particles in Unity, and they looked really simple to set up. A good bang-for-buck. Didn’t even give it a slight look.
• More/better character graphics. I also rushed through naming and giving stats to the enemies that did make it in. The art in the game isn’t good, and I don’t think I’d have made it that much better; but I’d have liked to added more varied enemies for sure. I think I probably spent a grand total of about twenty minutes the whole project (if that), making art.
• I had functionality to ‘slow’ the player movement over harsh terrain, and collision for impassable squares. I always planned to procedurally generate the terrain, but I didn’t get the chance to hook this stuff up. Impassable squares would have been tricky, in order to avoid blocking the player out of areas. ‘Slow’ terrain though wouldn’t have been much work.
• Healing spells for enemies. ‘Buff’ and ‘debuff’ spells for the hero player, as well as the enemies. Similar ‘buff’ and ‘debuff’ items to use as well. Some of the turn-logic for this sort of thing was in place; it would have probably taken less than an hour to add these.
• A better looking GUI. I’d have liked to try changing the font and colour used. I did Google search at one point, but it looked pretty involved, and would require me to go through all my UI code and hookup references to a ‘GUISkin’. Probably not much time to do at all, but every minute counted at this stage.

Of course, all of these things still fall way below ‘fixing the balancing’, and some other things mentioned in the “What went wrong” section. The Perlin-noise, colored and textured terrain was actually a wishlist item for me originally. I wonder if I’d have left that on the wishlist, I would have been able to fix the balancing?

Final Thoughts

In all, I enjoyed the experience. I was disappointed that the game I was intending to be challenging and provide seemingly ‘endless hours’ of battling, only holds up for about ten minutes. But I had fun coding something that quickly under a time pressure. Sure, the code looks absolutely horrible, but it’s a thrilling experience just churning out stuff at that speed, and having it all come together.

I would consider doing another Ludum Dare in future, but I’d definitely want to do more preparation. I’d also certainly scope my idea much better. I think a shorter, tighter game is the way to go. Something that would be doable with two ‘sleeps’ in there as well. I’d like to get a good twelve hours of sleep next time, instead of five!

My Journal Log

You’ll notice the log ends at about 2pm, four hours before my 6pm deadline. That last four hours was just a whirlwind, the time flew by ridiculously fast!

log.txt

I found it pretty useful to keep this log. It helped with motivation, and helped me focus on what I should be doing next.

I also have half a dozen mini-notepads full of ramblings about movement, AI, and the turn system. Taking a break from the computer and sitting down with a pad and paper is pretty valuable. Just gives you time to refocus, so you don’t just sit there coding down a blind alley.

My Time-lapse Video

~38 hours in ~4 minutes:

The first session was about 10 hours; the second session was about 28.

If you’re a developer and you’ve picked up the Kindle Fire this past week, you may still be in the Honeymoon period with the device. If that is starting to wear off by now, you may have the inkling to try writing code for it. If you’ve previously developed software for Android, you’ll be a little worried when you first connect it up. Out of the box, you can’t develop with it right away.

Thankfully, it’s not too complex to get things working. Thanks to some existing blog posts, it was hardly difficult to figure it out. I thought I’d write up my install process, for anyone specifically developing on a Windows machine. I’m sure by now though, a week after launch there’s plenty of info out there. I’m also wanting to flex my blog-writing muscle again, just to get something written and maintain my recent uptick in pace.

For this blog post I’m assuming you’re using Windows 7. Earlier versions of Windows should work fine too, but the steps may be subtly different.

Unknown USB Device – Kindle

The first thing to address is that there’s one ‘Unknown Device’ when you connect the Kindle Fire to a Windows PC. I’m assuming you have the Android SDK installed, so if you haven’t got it you’ll want to install it from here.

Unknown Device: “Kindle”

Now, you’ll want driver support for this ‘Unknown Device’ so that you can use ‘adb’ with it. So, download this file:

kindlefire_winusb.inf – Right-click and choose to “Save Link as…”

You’ll want to place it at a specific location, relative to the directory the Android SDK is installed to. It should go here:

• \extras\google\usb_driver

There should already be an ‘android_winusb.inf’ file in the same directory. My ‘kindlefire_winusb.inf’ is actually largely a copy of the contents of this other inf file; it simply has the USB hardware device IDs for the Kindle Fire listed now.

You can point Windows to this ‘usb_driver’ directory, if it’s currently trying to install that ‘Unknown Device’. Otherwise follow these steps:

• Go to Control Panel -> Administrative Tools -> Computer Management.
• Select the ‘Device Manager’ in the left-hand pane.
• Find ‘Kindle’ under ‘Other Devices’ (as shown in the above screenshot).
• Right-click on it, choose ‘Update Driver Software’.
• Tell these dialogs to locate the driver inside the same directory you copied the .inf file to: \extras\google\usb_driver under the Android SDK directory.
• That’s it. The device should now no longer appear under ‘Other Devices’. It ends up being ‘Android Phone’ -> ‘Android Composite ADB Interface’.

adb: No device found when Kindle Fire is connected

Though the Windows device itself is installed, this isn’t quite enough. If you attempt to deploy to it through adb, it won’t work yet.

The problem here is that adb doesn’t know about the Kindle Fire USB Device ID. It has to be given that ID explicitly via a .ini file. The “adb_usb.ini” file in question is located in this directory:

• c:\Users\*USERNAME*\.android

Editing: adb_usb.ini

So, open up “adb_usb.ini” in Notepad. Add this one single line, as shown above:

0x1949

Almost there now! Fire up a command prompt and navigate to your Android SDK /tools directory. Type the following commands:

• adb kill-server
• adb start-server
• adb devices

Restarting the adb server: The Kindle Fire lives!

If all went well, it should look like the above screenshot. There should be one device listed, which in this case would be your Kindle Fire.

Aside: Interestingly ‘android update adb’ as suggested in the text of ‘adb_usb.ini’, generates an error with the latest SDK R15 tools. ‘adb’ isn’t an accepted parameter to ‘update’. My Google-fu has failed to find a more permanent solution to adding this line. All references I can find to various other Android devices, encountering the same issues as the Kindle Fire, is just the same edit to ‘adb_usb.ini’.

All Done

That’s it! Your Kindle Fire will now behave just like any other Android device you may have developed with before.

File Transfer / Disconnection screen

One thing to note, is that big black screen which the Kindle Fire displays: “You can now transfer files from your computer to Kindle”. This is just to serve the internal storage over the USB connection, as a mounted device. You can safely press ‘Disconnect’ and leave this screen, and still be able to debug without issue. As long as the cable remains connected, ‘adb’ will work fine.

“San Angeles” demo running on my Kindle Fire, built by vs-android

Thankfully, vs-android worked just fine. Given I’ve only ever tested my personal cellphone with it; even though there’s no reason it shouldn’t have worked, I was nonetheless relieved that the Kindle fired up “San Angeles” without issue.

References

• Act Local Media: Developing on the Kindle Fire
• xda-developers: [Q] Creative ZiiO doesn’t appear on adb devices list

In an effort to post more often in this blog, I’m writing up a minor vs-android update. I don’t usually write here about the minor version changes, I usually just fire out a tweet when I do those.

I actually started work on this one, the evening after I published XNACPC. I was going to fix some of the issues of vs-android, which were posted to the Google Code page. One of them was related to a new Android SDK (the Java/apk building counterpart to C++’s NDK), which caused issues when building the vs-android samples. So I went and downloaded the new SDK. I also figured I should grab the r6b NDK, which had been out a while now but I hadn’t updated to yet. To my surprise NDK r7 was out! Coincidentally it was posted up the very same day I decided to check it. Spooky.

Breaking Changes

Unfortunately NDK r7 introduced breaking changes for vs-android. Specifically the names of support libraries I was linking with, were renamed. This, along with the SDK R15 issues were a stark reminder that when Google change things like this, they are likely to break vs-android. Fortunately addressing these changes was straightforward.

Interestingly NDK r7 brings with it a ‘Cygwin-free’ build method. Since the compiler and linker have always worked without Cygwin, it only seems like ‘make’ was an issue for them. So I’d imagine the inclusion of a Win32 native make is what allows them to do this now. Glad they’ve seen sense that Cygwin isn’t at all popular with most Windows devs.

NDK r7 also brings support for Android 4.0: “Ice Cream Sandwich”. I’m not entirely up to speed with what’s changed, but the API’s available for the NDK jumped from “android-9″ to “android-14″. With nothing in between. ’9′ being support for Android 2.3.2, and ’14′ being Android 4.0. So oddly enough they skipped the 3.x releases entirely for the NDK. I guess nothing changed to anything exposed to the NDK, during that period?

v0.93 Changes

Here is the list of changes made in v0.93. Pretty much all just bug fixes, and support for some small additions to the new NDK:

• NDK r7 was a breaking change for vs-android. This version now requires r7 or newer to be installed.
• Fixed breaking changes to the location of STL libraries. Also fixed new linking issues introduced by STL changes.
• Removed support for defunct arm 4.4.0 toolset.
• Added support for android-14, Android API v4.0.
• Added support for the dynamic (shared) version of the GNU libstdc++ STL.
• Tested against newest JDK – jdk-7u1-windows-i586.
• Added support for building assembly files. ‘.s’ and ‘.asm’ extensions will be treated as assembly code.
• Correct passing of ANT_OPTS to the Ant Build step. Thanks to ‘mark.bozeman’.
• Corrected expected apk name for release builds.
• Added to Ant Build property page; the ability to add extra flags to the calls to adb.
• Fixed bug with arm arch preprocessor defines not making it onto the command line.
• Fixed bad quote removal on paths, in the C# code. Thanks to ‘hoesing@kleinbottling’.
• Removed stlport project from Google Code – This was an oversight by Google in the r6 NDK, prebuilt is back again.
• Updated sample projects to work with R15 SDK tools.

You can get the latest vs-android here:

• http://code.google.com/p/vs-android/

Kindle Fire

I have my second Android device arriving soon. I ordered a ‘Kindle Fire’; I’m yet to try using a tablet computer. At $200 the price was right for me to take the plunge.

I’m looking forward to coding for it. I’d imagine it will help speed the development of vs-android. I haven’t been working on my personal Android project for a while, so it has slowed how often I update it. Certainly helps to ‘eat your own dog food’.

On and off during the last month or so, I’ve worked toward finishing a project I wrote about last year. It’s an Amstrad CPC Emulator, written in C# using the Microsoft XNA library. I’ve written plenty about the background and motivation in the previous blog post. This post is just here to announce the finished product, and touch a little on the things I’ve worked on recently.

The picture above is from Hewson’s “Nebulus”. Now that the emulator has a pretty complete feature set; games like these are very playable now. It makes a world of difference having the audio fully working now, for example. It also is great to finally be able to play at the original, full frame rate on Xbox 360. Surprisingly the 360 had trouble emulating this seemingly primitive machine!

Anyway, after the jump is a video showing the emulator in action on the 360…

XNACPC In Action

Here you go! This video shows a number of games running on the emulator. It’s taken with a point-and-shoot camera, in a poorly lit room. So the quality isn’t too great, unfortunately. Some games have pretty washed out colors and bad contrast; some look fine though.

I guess I could have captured it from my PC, but the whole point of the emulator was to get it running on the Xbox 360. The fun part is showing it off on the real thing.

CRTC – Good Timing and Accurate Video

My first pass at XNACPC last year, used the same technique which I employed when I wrote my CPC emulator back in the 90′s. I used a pretty naive approach, that had surprisingly good results for the early crop of launch games for the CPC.

I fudged the interrupts simply so that they occurred at 300Hz, which they’d be at on a real CPC. I wasn’t emulating the HSYNC correctly, nor the counters which are based off the HSYNC and generate interrupts. I also had to fudge the VSYNC signal which is read from one of the PPI ports. Whilst it was a primitive approach, it worked pretty well. Even ‘Sorcery’; a split-screen game, looked and played like the original.

Fixing the CRTC to behave properly was at the top of my list. I was pleased that I managed it. The best reference I found for it was probably here:

• http://www.grimware.org/doku.php/documentations/devices/crtc

Grimware CRTC Timing Reference

So I threw away my old video implementation and did it the proper way. Many games which previously didn’t work, now played just fine. I also used to draw the entire screen at the end of the frame. When I switched to drawing it line-by-line after each HSYNC, it eliminated flickering artifacts in many games. “Monty on the Run” for instance, looked awful in many places when I was updating the whole screen at once.

“Monty On The Run” – Fullscreen vs. Line-by-line

Audio – Silence Isn’t Golden

I actually traded off the audio work with the CRTC work. I’d switch between them when I hit a wall, to keep things interesting. When I first worked on the emulator last year it actually wasn’t possible to implement emulated audio at all. That only became possible when XNA 4.0 was released, with the addition of DynamicSoundEffectInstance.

The audio on a CPC is pretty simple in it’s concept, but it did take me a little while to get the timing correct. It’s essentially based on square waves, where the audio signal is either on or off. The signal does have volume control, so it’s possible to produce pretty decent sounding music with it.

AY8912 Audio Square Waves

There’s three main parts to the AY audio chip used in the CPC:

• Three channels of tone generators. Producing square waves at given independent periods.
• A single noise generator. It generates random noise, which can be used by any combination of channels instead of the tone generator output.
• A single envelope generator. This allows any combination of channels to progressively change the volume over time, in a manner that doesn’t need any CPU assistance.

Of the three channels; two of them represent the left and right speakers, and the other one is mixed between each speaker.

That’s the gist of it. The audio was pretty satisfying to implement; since I’d never tried to get it working, back in the emulator I developed in the 90′s. Unfortunately the addition of the audio was taxing enough for the Xbox to drop under the 50fps that a CPC runs at. This meant I had to refactor the audio code somewhat, from a nice C#-style to a more brute-force C-style.

Z80, PPI and Other Hardware Bugs

There was a fun little period after I had the CRTC emulation all working correctly. With the last big hurdle overcome, I now had to fix bugs in my hardware implementation. This was the period where suddenly lots of other dead non-working games would spring to life.

Coming back to the source code after over a year was fine for the most part, but I did forget about a number of functions which I stubbed out. The PPI was a big one, I just did the basics to get the most primitive games running. So there I had to go back to the hardware documents and fill in the more obscure parts of the implementation. After doing that, a bunch more games would run.

One other key fix I remember was to the Z80 emulation. The HALT instruction was broken. I took the Z80 CPU emulator originally from an older C# Spectrum emulator. I’d made a number of modifications to get it to play nice with the CPC; mostly around timing and interrupts. When I noticed a couple of games dying on the HALT instruction, I went back to the Z80 documentation and implemented HALT correctly. As well as getting some games to run, it also amusingly fixed the broken ‘pelvic thrust’ animation in the intro to ‘Rocky Horror Picture Show’.

“The Rocky Horror Picture Show” – Intro

CRT Shader – A Bit of Polish

Once I finally had the majority of the games looking pretty much perfect, I had an inkling to go a little further. I’m not much of a graphics programmer, by today’s standards. I used to be really into graphics programming back in the mid to late 90′s. Michael Abrash’s “Zen of Graphics Programming” was one of my favorite books. I also remember fondly working through the ‘Asphixia VGA Trainers’. It’s about the last time I think I wrote more than just a handful of lines of assembly code at once.

It’d been a few years since I last experimented with shaders, so I decided to try writing a simple shader to give the emulator a ‘CRT effect’. I only spent a small amount of time on it; it’s nothing fancy. The gist of my effect works as follows:

• Each CPC pixel is exactly three ‘XNA pixels’ high. My shader uses modf() to modolo based on this, so I can do something different for each of the three rows of pixels.
• Row One: I leave the color alone. Sample goes straight through.
• Row Two: 90% of the color I take from original sampled value. 5% I actually sample from the next CPC pixel scanline. Then, 5% left out to make it darker.
• Row Three: 75% of the color I take from original sampled value. 15% I sample from the next scanline. Then, 10% left out to make it darker.

Sampling a little of the next scanline down, introduces a little color bleeding; which seems to work well. It’s pretty decent-looking considering how simple my shader was. Here’s a zoomed picture, below. It looks it’s best though at 1:1 resolution, as you can see from the “Nebulus” shot at the very top of this blog post.

“Ikari Warriors” – 2x Zoomed Shot of my CRT Shader Effect

Xbox 360 – Compact Framework Creaking Under Pressure

Getting the emulator running at the full frame rate of the original CPC, wasn’t much fun. On PC it was fine, no problems there. I think Task Manager shows ~2% CPU usage on my PC when it’s running. When I turned the frame-rate limiting off, I think it ran at >300fps. I’m talking specifically here about emulation of the CPC, not just drawing and updating the screen. So when I run it with the frame-limiter off, the machine runs so fast that any game is essentially unplayable.

Now, the Xbox 360. Some of you with experience with C# and the 360, will likely think I’ll mention garbage collection? Actually this wasn’t an issue for me. For any emulation runtime code, I made sure I wasn’t doing anything bad there. I generate absolutely no garbage when the game runs. I do allow for garbage in my “Pause Menu” code, however. But that’s really nice and easy to handle, just perform a full collection when unpausing. I like that approach for being able to write quick-and-dirty UI code, and keep the actual game clean.

So, not garbage? Nah, it’s just a raw CPU throughput problem. I hear someone shouting “JIT”…? Indeed; it does help a lot. The version of the emulator I’ve uploaded today runs at about half the speed of a CPC, when I have the debugger attached to the Xbox. So yeah: Release Build. No debugger attached. No garbage collection. This is as good as it gets.

I did a few things to drag the emulator kicking and screaming, to meet the 50fps that the CPC runs at:

• General optimizations which seem to benefit 360: Lookup tables seem to win over a handful of computational instructions. A general mistrust of the optimizer/JIT works well too. Cache variables yourself when doing multiple lookups from other classes, for example.
• Changed the routine which writes screen memory out to the XNA texture, to use unsafe code.
• Refactored my audio code to have the tone, noise and envelope generators as structs instead of classes (no vcalls). Ripped out my nice methods and just let my audio code access public members instead.
• The biggest win by far was hand-unrolling a simple for loop. It was an inner loop inside my graphics rendering code. It was walking over eight pixels for each byte of CPC memory. Instead of looping, I copy and pasted the line over eight times. My frame rate jumped from ~40fps to ~50fps.

Anyway. This was a bit of an eye-opener. I’m used to writing native code on 360, and I’m sure the thing wouldn’t break a sweat with a C++ version of my emulator. It’s odd to think that when I first worked on my emulator back in 1998, in C++; it was on a Pentium 120. That P120 ran the emulation fine, taxing the CPU at around 50% if I recall. Compare that to the ~95% I’m shipping XNACPC on the Xbox with.

So, the Compact Framework on 360 isn’t too hot. To be honest though, I’d have no qualms with developing something like this in C# if I was targeting PC. The full .NET Framework seems to be a good performer. Of course, straight-game code is going to be a whole different kettle of fish; compared to something as ridiculous as I’ve coded here. I don’t mean to slag off the Compact Framework; I’d imagine other XNA devs don’t see the huge disparity that I did, when they’re coding their games.

The Source

In an ideal world I’d be able to release it on the XBLIG channel. But given both copyright (and moral) issues with distributing games that other people have worked on, it’s not a possibility. This is a shame for people interested in playing with it though, since it means your only option requires a ‘Creators Club’ subscription. This costs $99 a year!

That said, I’d likely expect the few of you that actually read this blog (and the fewer that read this far down!) might be XNA coders. I’ve a zip file setup which contains the solution and project files, with the emulator ready to deploy to your Xbox. You can download it here:
xnacpc_1_0_source.zip

If you just want to browse the code online, I’ve put it up here at Google Code too:

• Google Code – XNACPC

If you’re interested in taking XNACPC and releasing something (legal and authorized) on XBL Indie Games, or Windows Phone 7. Please contact me about it, I’d love to work on the project!

PC Version

The PC version is playable without any sort of coding experience, nor requiring paying any money to Microsoft. You can download it here:
xnacpc_1_0_pc_binaries.zip

Just unzip it somewhere, and run ‘XNACPC.exe’. That’s it. There’s a number of games in the package, all setup ready to play. You may also need to download the XNA Runtime, if the app complains about missing something when you run it.

What’s Next?

Well, I really don’t plan to do any more work on XNACPC. There’s plenty of other CPC emulators which are more complete. They emulate more features, and run pretty much any game you can throw at them. The intention wasn’t to compete with those emulators, it was just to prove it’s possible to do the same in C# with XNA. I think I’ve done that, so there’s no real motivation to continue with the project.

It’s been fun; but I’ve been tinkering with CPC emulators in various forms since 1998. Right now I’m happy to close the door on this, and keep working on ‘new’ stuff instead of having one eye on the past.

Hope you enjoy poking around the source code!

So, it’s been quite a while since I last posted on this blog. I’ve been pretty busy the past few months with various things. At work we had an important milestone during that time, which thankfully passed with flying colors. Outside of work though, I actually got married a couple of months ago! My wife and I eloped to Hawaii, just the two of us, and tied the knot.

So it’s been quite a busy period recently for me. Starting on vs-android back in January probably wasn’t the best timing. Getting it done wasn’t too bad; it was more the follow-up support email period that was tougher to deal with. More so, given the marriage preparations and what have you, which I was going through at the time. So vs-android got left by the wayside a while, and rightfully so!

A couple of weeks ago though, I picked up the project again. I was determined primarily to fix the dependency issues the previous version had. After looking again at what Microsoft had done with their Win32 toolchain, I decided to massively change my approach. I went from a system that was built primarily with MSBuild, with a little C# to sanitize pathnames and switches. To a system which had almost all of it’s moving parts in C#, with MSBuild just ferrying the project data into it.

That was pretty much the basis for the latest iteration of vs-android. I bolted on a few other things along the way, but for the most part it was just about getting it into a stable and reliable state.

vs-android v0.9

So yeah, I’ve just wrapped up a new version of vs-android. I didn’t bother posting about the last couple of version revisions I did in the past, but this one is a little different. It got a pretty major overhaul. In my eyes has moved from an experimental toy, to something people can actually write production code with. So this is why the version number jumped up to point-nine.

To get an idea of the extent of said overhaul, here’s a list of the changes made:

• Verified working with Android NDK r5b, r5c, and r6.
• Much of vs-android functionality moved from MSBuild script to C# tasks. Similar approach now to Microsoft’s existing Win32 setup.
• Dependency checking rewritten to use tracking log files.
• Dependency issues fixed, dependency checking also now far quicker.
• Android Property sheets now completely replace the Microsoft ones, no more rafts of unused sheets.
• Property sheets populated with many options. Switches are no longer hard-coded within vs-android script.
• STL support added. Choice between ‘None’, ‘Minimal’, ‘libstdc++’, and ‘stlport’.
• Support for x86 compilation with r6 NDK.
• Full support for v7-a arm architecture, as well as the existing v5.
• Support for Android API directories other than just ‘android-9′.
• Separated support for ‘dynamic libraries’ and ‘applications’. Applications build to apk files.
• Response files used in build, no more command-line length limitations.
• Deploy and run within Visual Studio, adb is now invoked by vs-android.
• ‘Echo command lines’ feature fixed.
• All support SDK/libs (NDK, SDK, Ant, JDK) are okay living in directories with spaces in them now.
• All bugs logged within Google Code are addressed.

About six weeks ago I moved into that club which everyone else seemed to be part of, but not me. I joined the ranks of smartphone owners! No more 90′s flip-phone; I picked up a recent-ish Android handset. I’d never really been too into the whole concept of smartphone, but was starting to feel a little left behind. I must say that after those few weeks I’m sold, these things are pretty cool.

So, what’s a programmer to do? I think I may have actually downloaded the Android SDK before I even received the phone. Yeah, just a little bit eager! It’s actually the second handheld device I’ve tried to code on. The first being the Tapwave Zodiac, also with an arm processor and touchscreen. I never really got anything worthwhile going on it, my excuse being a job hunt and subsequent move to the United States. But nonetheless, it was a fun little handheld to code for. I was certainly looking forward to coding on a much more popular device.

Total Eclipse of the IDE

Now if you’ve ever tried to write anything for the Android, you’ll know the preferred development environment is Eclipse. If you’re writing your stuff completely in Java, you have everything you’d ever need in Eclipse. It seems to do all the nice syntax highlighting, and intellisense-like functionality that Visual Studio offers. Of course the shortcut keys are completely wrong, and must have been setup by a madman , but it does seem a pretty good IDE to develop in. Better yet, it’s free!

But is it for me? Nah. It’s not for me.

I just couldn’t get on with it. Primarily I want to write my code in C/C++ too. So I started to convince myself that I needn’t bother with Eclipse. I can get all the highlighting and quick-reference stuff in Visual Studio, and I could run batch files to build my code. After all, with C++ under Eclipse you still have to install Cygwin and effectively build from the command-line. (or setup batch/makefiles as you would with Visual Studio).

I’ve used Visual Studio for a very long time now, since it’s older guise as simply: ‘Visual C++’. Was version 4.0 I think that I started with, when I moved from DOS to Windows programming. Now it’s my primary coding IDE; I’ve used it at both games companies I’ve worked at in the past decade. It’s an odd one to admit, but it’s too hard to let the thing go.

Makefile Project

So I set off and started a ‘Makefile Project’ in Visual Studio. I could update the system include paths under Visual Studio 2010 (something I think was trickier with earlier versions, or at least not set on a per-project basis, I forget). This meant I’d get properly working intellisense of the Android NDK headers, the ‘External Dependencies’ list in my solution was all completely correct too. All in all, a pretty nice setup.

I put together a batch file, which invoked a cygwin session running ‘ndk-build’. I then setup some further ones which would build the entire package using ‘ant’, and then install and run the application on the Android device connected to my PC. Not much to argue with here. For someone who was longing for Visual Studio back, after a few days with Eclipse. It was lovely.

If you want to try the makefile method, take a look at this website:

• http://ian-ni-lewis.blogspot.com/2011/01/adventures-on-android-ndk-part-i.html

His method is a little cleaner than the one I used, my batch/script files were hard-coded messes of experimentation. Check out the debugging article on his page too, he’s got a way of using WinGDB to debug Android NDK code in Visual Studio. It’s awesome.

What about when it gets big?

Unfortunately Android’s NDK build scripts under Cygwin on Windows, isn’t exactly the performer. When your file count gets into triple figures the initial dependency checks take an age. I also wasn’t too enamored with not knowing what was passed to my compiler. You get a limited set of proprietary settings you can change in your makefile, but it did feel like a black box.

Coming from a console background, I like to know what my debug and release builds actually are. I also like to setup further build variants too… Usually one specific to profiling the game, essentially release with some limited modules enabled to do the profiling. The other major one would be a variant of my debug build, with asserts on but the optimization cranked up to max.

So it did feel a little confining. My realization about the build slowdowns with lots of files came when I attempted to integrate third-party code… Specifically what I’m wanting to use is the Irrlicht engine. It’s a well-structured open source rendering engine. Very versatile, and runs on a variety of platforms. Android included, of course.

Runing ndk-build, the thing would sit there for well over a minute before even starting to compile files. I’ve a recent i7-based laptop too, so it’s no slouch. I had an itching to try and delve into a more integrated method of building with Visual Studio, and that tipped the balance for me.

Enter MSBuild

With Visual Studio 2010 Microsoft made a radical change to the way their C++ environment worked. They moved over the entire thing to use MSBuild. Having worked with a certain non-Microsoft gaming console, you could see that they had to do a little hoop-jumping in previous versions of Visual Studio. Integrating custom compilers with the older versions really appeared to be quite a task. With VS2010, it had been hinted on some preview posts from Microsoft that this would be far easier.

Unfortunately though when I started to look into this, I found next to no information on the web about it. Well, really I found absolutely nothing in the end. I found a couple of questions on stackoverflow.com about the same subject, but with no answers. I ended up searching through the Visual Studio directories to find the existing scripts for Microsoft’s compiler. From there I duplicated the directory layout and began tinkering with setting up a new platform.

I’d never used MSBuild before this. It was completely new to me, never even seen a MSBuild file before. I hear some game companies use it, but ours certainly doesn’t. I can definitely recommend it though, it’s very powerful and flexible. It does have quite a learning curve, and Google searching for documentation can lead you to very bad explanations of how it works. If you’re at all interested in it, I recommend this book:

• “Inside the Microsoft Build Engine: Using MSBuild and Team Foundation Build” by Sayed Ibrahim Hashimi, William Bartholomew.

As well as being a very good guide to MSBuild, it actually had a small chapter about exactly what I was trying to do! Something I couldn’t find at all with Google. Unfortunately I picked the book up after I’d done the majority of my scripting, but it at least showed me I was barking up the right tree.

vs-android

So, vs-android. Once I’d gotten all the features I wanted, and all the bugs I could find ironed out, I decided to release it as open-source. The regular download package just consists of a collection of MSBuild scripts, which need to be copied within a certain sub-directory of your Visual Studio installation.

Alongside the scripts is a single DLL file. When you’re working with MSBuild, you can code up custom tasks in a .NET language, such as C#. The header dependency scanning code lives here, which invokes gcc to find all the headers that the c/cpp files would be dependent on. As well as some other command-line switch manipulation code. The full source to the C# DLL is up on my Google code page.

You can download vs-android here:

• Google Code: vs-android

There’s full documentation on those pages too. Along with a step-by-step example of getting Android’s ‘san-angeles’ sample app compiling and linking. The Google Code page also contains more technical info about the implementation, if you’re interested. There’s a link on the main page to ‘tech notes’, which expands on some of the points I’ve touched on here.

One thing I’d like to mention is that you don’t need Cygwin at all any more, if you use vs-android. That in itself is a godsend! It’s been a few years since I last used it. It was compulsory for our build process a few years ago, when I was working in the United Kingdom. For whatever reason the install took far longer than I remember it taking, and it has a horrible long pause when the thing starts up. I’m definitely glad I don’t need to deal with it again for Android dev.

Finally, I’ll leave you with a comparison of the build times, versus using ‘ndk-build’ (sans long Cygwin pause). They’re pretty good, but I think with some more work I can get the thing running even faster.

What’s next?

I’d hope to continue updating these scripts in future. I’ve got a nice list of things I’d like to tackle already. For now though, I’m going to take a little break from the build scripts. I’m very happy that I can change gears back to writing game code again. Getting a ‘release’ together, with all that it entails, documentation and everything is pretty draining.

I think the last time I put something up on the internet like this was in 1997, when I worked on an editor for Command and Conquer: Red Alert. If you do click onto the garish Geocities website, keep in mind that I was a teenager at the time.

So yeah, this is another post detailing a change made in the latest version of XNA.

This one was inspired by some comments on a previous blog post of mine: “StringBuilder to String with no garbage”. A developer named Matt, who was targeting Windows 7 Phones, pointed out that this particular trick with reflection was not working. He’d get a ‘FieldAccessException’ when attempting to access the internal private System.String object.

I’d just got round to installing the Windows Phone stuff to check this out, and indeed this is the case. I tried various different flags in the GetField() call, but had no luck in getting it to work. I tried setting up a trivial case with my own class containing a private ‘System.String’ and had the exact same problem.

So for whatever reason Windows Phone’s Compact Framework doesn’t allow access to private members via the GetValue() reflection call. In my aforementioned test with my own custom class, public members worked fine. In both cases the GetField() method was all good, it was just the GetValue() that throws ‘FieldAccessException’ in the case of private members.

Well… That’s a bit of a fly in the ointment right there.

Windows PC: A totally new version of StringBuilder?

Upon further inspection with the new XNA, I saw that the Windows setup had changed too. The StringBuilder object contents had changed completely.

For reference, here’s what I’ll call ‘version one’ of the StringBuilder. The one we’re used to from good ‘ol XNA 3.1:

StringBuilder version one

Here’s ‘version two’:

StringBuilder version two

Very, very different.

This new version maintains a series of chunks. As chunks are added it actually ends up linking together multiple new StringBuilder objects. The member ‘m_ChunkPrevious’ is this link.

From checking out the XNA 4.0 platforms, only the PC Windows build target uses this new StringBuilder. It could be argued that given that the GC performs well on PC that you need not use any tricks to grab internal strings anyway.

If you’re desperate though, I did come up with a crude method of grabbing the internal string without generating garbage. Be warned though, it’s far uglier than the previous ’3.1′ method I had.

Here goes:

const int MAX_STRING_SIZE = 64;
 
StringBuilder my_string_builder;
char[] my_char_array;
string my_dest_string = new string(' ', MAX_STRING_SIZE);
 
public static unsafe void CopyIntoString( string dest_string, char[] char_buffer, int length )
{
    System.Diagnostics.Debug.Assert( dest_string.Length >= length );
 
    unsafe
    {
        fixed ( char* dest_fixed = dest_string )
        {
                // Copy in the string data
                for ( int i = 0; i < length; i++ )
                {
                    dest_fixed[i] = char_buffer[i];
                }
 
                // NULL terminate the dest string
                if ( length < dest_string.Length )
                {
                    dest_fixed[length] = (char)0;
                }
        }
    }
}
 
public void PrepareSB()
{            
    my_string_builder = new StringBuilder(MAX_STRING_SIZE, MAX_STRING_SIZE);
 
    // Ensure this StringBuilder only has one chunk
    StringBuilder sanity_check = (StringBuilder)my_string_builder.GetType().GetField(
        "m_ChunkPrevious", BindingFlags.NonPublic | BindingFlags.Instance )
        .GetValue(my_string_builder);
    System.Diagnostics.Debug.Assert( sanity_check == null );
 
    // Grab the char array from the StringBuilder
    my_char_array = (char[])my_string_builder.GetType().GetField(
        "m_ChunkChars", BindingFlags.NonPublic | BindingFlags.Instance )
        .GetValue(my_string_builder);
}
 
//! This function can be called iteratively and generates no garbage
public void TestSB()
{
    // Put something in there
    my_string_builder.Length = 0;
    my_string_builder.Append("Hello World!");
 
    // Copy into our dest string
    CopyIntoString(my_dest_string, my_char_array, my_string_builder.Length);
 
    // Note this dest string is a little odd, the NULL terminator shows in the 
    // debugger with all the spaces after it.
    // On use of the string though in code, I'm seeing the correct amount of 
    // characters used. It stops at the NULL.
}

My eyes!

It works… But really; I wouldn’t touch this monstrosity.
 

Summary by Version and Platform

 

So what follows here is a summary of the version of StringBuilder used on each platform, under both XNA 3.1 and 4.0. I also note the version of ‘mscorlib’ as listed in the project references section; this is the assembly which contains the System.Text.StringBuilder class.

Visual Studio 2008, and XNA v3.1

Windows

mscorlib v2.0.0.0, Runtime Version v2.0.50727
Uses StringBuilder version one. The technique detailed here works fine.

Xbox 360

mscorlib v3.5.0.0, Runtime Version v2.0.50727
Uses StringBuilder version one. The technique detailed here works fine.

Visual Studio 2010, and XNA v4.0

Windows Phone 7

mscorlib v2.0.5.0, Runtime Version v2.0.50727
Uses StringBuilder version one. Unfortunately reflection access to the internal System.String is disallowed. You’re stuck with using ToString() and incurring garbage.

Windows

mscorlib v4.0.0.0, Runtime Version v4.0.30319
Uses StringBuilder version two. This new implementation of StringBuilder requires a convoluted way of converting the StringBuilder to a System.String, without generating garbage. I’d say incurring the garbage hit on PC is preferable to using the hacky stuff I’ve published in this post.

Xbox 360

mscorlib v2.0.5.0, Runtime Version v2.0.50727
Uses StringBuilder version one. The technique detailed here works fine.

Oddly enough the main version number dropped to v2.0.5.0, from v3.5.0.0. A little confusing. The versions though match the Windows Phone 7 assemblies exactly.

Conclusions

So, under XNA 4.0 you’re out of luck on Windows Phone 7 unfortunately. Please let me know though if this isn’t the case, if you find some other way of getting round it please drop a comment on this post.

For Xbox 360 we’re fortunately still good to use the same methods which worked under XNA 3.1.

It’s pretty much inevitable though that in future both Phone 7 and Xbox 360 are bumped up to the new .NET StringBuilder class, which Windows currently uses. Possibly even Xbox 360 may get the same reflection shackles that Phone 7 has now too.

Can only hope that when we get a new Compact Framework, it comes with a shiny new generational garbage collector implementation.

References

• String and StringBuilder in .NET 4
• How the StringBuilder class is implemented? Does it internally create new string objects each time we append?
• Does StringBuilder become immutable after a call to ToString?
• XNA/C# – StringBuilder to String with no garbage

I initially begun writing this particular article back in April. Back then with XNA 3.1 to play with, there was actually a method of ‘thread-local storage’ for Xbox. With the release of XNA 4.0, this last remaining native method was removed. The [ThreadLocal] attribute though was added to the Compact Framework, but it seems to be a stubbed non-functional attribute on Xbox 360.

So, where does that leave us? There is various ways you can manually code up a thread-local storage solution. The whole crux of it is data-retrieval and storage on a per-thread basis. The most immediately obvious way to go is just some sort of associative container, a Dictionary<> of some sort of thread ‘id’ mapped to the data you wish to store.

Just something like:

Dictionary< System.Threading.Thread, MyClass > my_tls_data;

There’s a lot of room for freedom in implementation though. Is Dictionary<> really the most suitable container? Should items be auto-created on the first access, or explicitly created by the user? Most of all I want to keep things performing well. Given some functionality performs extremely poorly on the Xbox 360, I want to be careful what I throw into my implementation.

The profiling approach

So, here’s how I set things up to profile changes I made throughout development:

• My test case sets up ten threads.
• Each thread has TLS data which consists of one simple integer.
• I kick off all ten threads at once, and each runs a loop 100,000 times incrementing their integer by one each time. (I verified the loop is not optimized out for the release-JIT’d builds).
• Once I see all ten threads have finished their loops I conclude the test and take the timing.
• I run the tests over and over and calculate a running average time. I display the results on screen, as opposed to debug output. So I can see the results on a release build which isn’t connected to the debugger. Pretty simple stuff, it just looks like this:
  
• Xbox times have a steady average very quickly after just a dozen or so tests. Windows times were more erratic and I generally left them for a couple of minutes to settle down.

Implementation choices

So, given the myriad of different ways this thing could be done, I broke down the choices I’d need to make. Here’s a breakdown piece-by-piece of the decisions I made along the way. Forgive the language, they’re just lifted verbatim from notes I made:

• Array over Dictionary<>
• For both Windows and Xbox with ten test threads, using an array instead of a Dictionary<> was a little over twice as fast. 74ms versus 151ms on Xbox.
• From additional testing on Xbox, array became a comparable speed to Dictionary<> at around 32 threads. I threw in a couple of different access patterns to test with too. Only if you’re going crazy with threads and data access, will Dictionary<> be a win.
• See the source code for the array implementation. It’s just essentially a simple associative container of ‘Thread’->’Data’. Potentially the whole array needs to be searched on each access. But keeping the array size small makes this a win over Dictionary<>.
• For vs foreach on array access
• A for loop incrementing an integer was faster.
• Doing the search with foreach() instead roughly doubled the times. 74ms versus 141ms.
• Thread object vs integer thread ID (ManagedThreadId)
• Each of these approaches were within a millisecond of one other. Something like 73.5ms versus 74ms. The integer version was the slight winner.
• I kind of like using the object reference, it makes debugging easier. So I went with that instead of using the integer ID.
• Call to Thread.CurrentThread on data access.
• Cost is prohibitive on Xbox, see a blog post I wrote a while back:
  http://www.gavpugh.com/2010/04/30/xnac-thread-currentthread-is-slow-on-xbox-360/
• This is still the case under XNA 4.0, figures haven’t changed. For the testing I was doing for these classes, calling Thread.CurrentThread on Xbox each access increases the time taken in one example from 74ms up to 7248ms. The equivalent test on Windows went from 26ms up to 32ms.
• My class has to provide a data accessor which takes a thread object param.
• Explicit cleanup, versus WeakReference to determine threads are done with.
• WeakReference is slow on Xbox unfortunately too. Even worse than Thread.CurrentThread in my tests. The example case I used shot up from 74ms to over 40,000ms. Windows jumped from 26ms to 76ms.
• I chose to use explicit cleanup.
• Side note: To make cleanup easier on thread destruction, consider a TLS manager class. The TLS objects can register with the manager upon creation, and the manager can walk them all destroy any data associated to a specific thread.
• Explicit construction, versus ‘auto’ construction.
• Auto construction isn’t too much of a performance drop. It just adds an extra branch to the code if the data access fails to find its data. Which you probably would have in there anyway to throw an Exception, or fire an assert.
• In my tests removing auto construction from my code, dropped the reference 74ms case down to 73ms. Good tradeoff in my opinion, so I kept auto-construction.
• Avoid thread locks for data access
• Locks are only required for adding new array entries, or removing them. The data access function should be able to execute in the general case without a lock.
• With auto-construction though, a new entry could be added within the accessing function. There’s a risk of a race condition if two concurrent threads try to auto-construct the data for one specific thread.
• As long as we can be sure that threads only access their own data, everything is fine. So the GetData(Thread) version should verify that the thread matches Thread.CurrentThread.
• The check only needs to be made after the search fails, just as it’s about to do the auto-construction. I ended up just putting this code within #if DEBUG, so I didn’t have to worry about any performance hit.

Explicit cleanup

I wasn’t too happy with requiring explicit cleanup. It means the cleanest solution would be some kind of global static function which ensured all various TLS variables are freed. Something like:

public static void CleanupTLS()
{
    Profiler.GetCurrentNodeStack().FreeData();
    Debug.Output.GetStringBuilder().FreeData();
    Renderer.DebugLines.FreeData();
 
    // etc...
}

Consider too that some of these variables may be initially at private scope, so they would need to be specifically exposed somehow so the freeing can occur. At best you’d need to add explicit public freeing functions to various affected modules.

As hinted in my notes, a global manager class which registers all TLS variables could make this a lot cleaner. As TLS data is instantiated, I will register with the manager. Where this occurs is not performance-critical. The freeing isn’t performance-critical either. The freeing though is pretty simple, the manager just walks all the TLS references it’s been given, and attempts to purge any with data for a specific thread.

So instead of calling the above mess, you can just call:

Core.ThreadLocalManager.FreeData();

Bit happier with it now.

Profiling figures

I explained my profiling setup earlier in this article. Here are the figures with my final implementation, compared against the existing methods of TLS. The ‘No param’ version of my code makes a call to ‘Thread.CurrentThread’ each time to determine what data to pass back. The ‘Thread param’ version uses a passed in thread reference to retrieve the data instead.

The figures are for ten concurrent threads making 100,000 accesses each:

 
* Using ‘Thread.CurrentThread’ on Xbox 360 is very slow.
** The GetData/SetData() (LocalDataStoreSlot) method doesn’t work under XNA 4.0 on Xbox 360. These figures are from when I ran the same tests months ago under XNA 3.1 on 360.

Given these figures I’m happy to use my new implementation for both Windows and Xbox 360. There’s no alternatives currently for Xbox 360, so given I’ve explored many optimization possibilities it’s pretty much as good as it gets. For Windows it actually is slightly faster than even [ThreadStatic], so using the same code across both totally makes sense.

Show me the code

Here’s the code for download:

ThreadLocal.cs

It also includes the class for ‘ThreadLocalManager’, which as detailed earlier assists with easier cleanup of data from destroyed threads.

Here’s a quick contrived bit of code to show how these classes should be used:

private static Core.ThreadLocal<StringBuilder> ms_tls_stringbuilder = 
    new Core.ThreadLocal<StringBuilder>();
 
//! Entry function for new worker threads
public static void ThreadRun()
{
    // Grab the TLS stringbuilder
    StringBuilder sb = ms_tls_stringbuilder.Data;
 
    // Do bunch of work
    ...
 
    // Free all TLS data associated with this thread
    Core.ThreadLocalManager.FreeData();
}

The TLS data access has a function which takes a thread parameter. This should be used in performance-critical code, given my findings detailed in this article. Just note the ‘Thread.CurrentThread’ once in your new thread, and pass it around to any performance-critical functions that require working with TLS.

Value Types

My implementation is currently tied to just using reference types, value types will not work with it. An alternative is to just simply wrap value types in a class, which is what I’ve been happy to do with my engine.

If you absolutely need a thread-local implementation which does work natively with value types, it’s not too hard to do. You’d want to set up an alternative class, with pretty much the same code in it. Change the ‘where’ constraint to simply: ‘struct’. There’s a couple of different ways I could see the implementation going:

• Implement separate Get and Set data methods. Maybe just a single property?
• Have a get method not return a value, but instead pass it back through a reference parameter.

There’s no reason why you couldn’t have both these types of interface for it.

References

• MSDN: Namespaces, Types, and Members in the .NET Compact Framework for Xbox 360
• MSDN: Thread Local Storage: Thread-Relative Static Fields and Data Slots
• XNA Forums: possible to have an array on stack, not in heap?
• XNA Forums: Why no ThreadStaticAttribute on XNA?
• MSDN: Constraints on Type Parameters

Continuing on from my last XNA post, I had a go at using [ThreadStatic] on the Xbox. Specifically this attribute was not available with XNA 3.1, but was introduced in the new 4.0 version.

At first glace it worked fine, I threw in the same code I’d used on Windows. Unlike under 3.1 it compiles without issue. However, the attribute appears to actually do nothing it all. It behaves just like the attribute wasn’t there, just like a regular plain static variable.

I found this issue from attempting to use [ThreadStatic] within some code I had written for a future article here. The code was to profile various methods of implementing Thread-local storage on the Xbox 360. Switching the code to add a test case for [ThreadStatic] worked fine at first, but I was seeing very odd profile numbers coming back from it. On closer inspection in the debugger I could see that the TLS values weren’t thread-specific, they were getting written to at runtime by other executing threads.

Show me the code

So, I spliced out some code and came up with a contrived test to check how [ThreadStatic] is working. Here’s the code:

public class ThreadTest
{
    [ThreadStatic] private static int m_int_counter = 0;
 
    private Thread m_thread;
    private int m_affinity;
    private int m_thread_num;
 
    public ThreadTest( int thread_num, int affinity )
    {
        m_affinity = affinity;
        m_thread_num = thread_num;
        m_thread = new Thread( Run );
    }
 
    public void Start()
    {
        m_thread.Start( (object)this );
    }
 
    public static void Run( object passed_obj )
    {
        ThreadTest this_obj = (ThreadTest)passed_obj;
 
        Debug.WriteLine( "Started thread #" +
            this_obj.m_thread_num + " Affinity=" + this_obj.m_affinity );
 
#if XBOX
        Thread.CurrentThread.SetProcessorAffinity( this_obj.m_affinity );
#endif //XBOX
 
        Debug.WriteLine( 
            "Thread #" + this_obj.m_thread_num + 
            ": Before m_int_counter " + m_int_counter );
 
        m_int_counter++;
 
        Debug.WriteLine( 
            "Thread #" + this_obj.m_thread_num + 
            ": After m_int_counter " + m_int_counter );
    }
}
 
// The four affinities that are okay to use. See:
// http://msdn.microsoft.com/en-us/library/microsoft.xna.net_cf.system.threading.thread.setprocessoraffinity.aspx
private static readonly int[] m_affinities = new int[4] { 1, 3, 4, 5 };
 
private static bool m_started = false;
 
// Call this from anywhere, I just threw it in Update() in a new XNA game project
public void Test()
{
    if ( m_started )
    {
        // Just do it once...
        return;
    }
    m_started = true;
 
    const int NUM_THREADS = 10;
 
    ThreadTest[] threads = new ThreadTest[NUM_THREADS];
 
    // Create ten threads and kick them off near-enough at the same time.
    for ( int i = 0; i < NUM_THREADS; i++ )
    {
        int affinity = m_affinities[i % 4];
        threads[i] = new ThreadTest( i, affinity );
        threads[i].Start();
    }
}

What’s up

Firstly, here’s the output under Windows. The ‘before’ number is always zero, and the ‘after’ is always one. The [ThreadStatic] variable is indeed unique to each thread instance.

Windows

Started thread #0 Affinity=1
Started thread #6 Affinity=4
Started thread #4 Affinity=1
Started thread #7 Affinity=5
Started thread #8 Affinity=1
Started thread #2 Affinity=4
Started thread #9 Affinity=3
Thread #7: Before m_int_counter 0
Thread #8: Before m_int_counter 0
Thread #0: Before m_int_counter 0
Thread #6: Before m_int_counter 0
Thread #4: Before m_int_counter 0
Thread #2: Before m_int_counter 0
Thread #9: Before m_int_counter 0
Thread #8: After m_int_counter 1
Thread #7: After m_int_counter 1
Thread #0: After m_int_counter 1
Thread #6: After m_int_counter 1
Thread #4: After m_int_counter 1
Thread #9: After m_int_counter 1
Thread #2: After m_int_counter 1
Started thread #5 Affinity=3
Started thread #3 Affinity=5
Started thread #1 Affinity=3
Thread #1: Before m_int_counter 0
Thread #1: After m_int_counter 1
Thread #5: Before m_int_counter 0
Thread #5: After m_int_counter 1
Thread #3: Before m_int_counter 0
Thread #3: After m_int_counter 1

 
Disregard the affinity value for Windows, it’s not used. Anyhow, here’s the Xbox output.

Xbox 360

Started thread #0 Affinity=1
Thread #5: Before m_int_counter 3
Thread #6: Before m_int_counter 3
Thread #3: Before m_int_counter 3
Thread #2: Before m_int_counter 3
Thread #7: Before m_int_counter 3
Thread #9: Before m_int_counter 3
Thread #0: Before m_int_counter 3
Thread #5: After m_int_counter 4
Thread #6: After m_int_counter 5
Thread #3: After m_int_counter 6
Thread #2: After m_int_counter 7
Thread #7: After m_int_counter 8
Thread #9: After m_int_counter 9
Thread #0: After m_int_counter 10

The [ThreadStatic] variable is obviously not unique to each thread. It’s just behaving like a regular static. If I comment out the line which sets the thread affinity (which is Xbox-specific code), I get similar output.

What’s also concerning is that the “Debug.WriteLine()” function seems to be randomly failing. Each time I run it, I get different lines missed out from my expected output.

Perhaps on Xbox it’s implementation is not completely thread-safe? I swear it was on XNA 3.1′s Compact Framework, but maybe my memory is failing me.

Solutions?

I have an alternate way of implementing TLS with C#. It’s very fast as long as you’re not running a ridiculous number of threads (under a dozen is fine). When I last looked at the code under XNA 3.1 it outperformed the ‘GetData/SetData()’ method of TLS considerably. In fact when I tested the code under Windows and compared it to [ThreadStatic], it outperformed that too.

Will be the next article I post up, since I think I’ve encountered all the issues I can on the road to finishing it up.

References

• ThreadStaticAttribute Class
• Thread.SetProcessorAffinity Method

A few months back I was partway through an article about XNA. I was going to cover how to implement efficient “Thread Local Storage” on the Xbox 360. Back in Game Studio v3.1 I found that the [ThreadStatic] attribute was not supported on the Xbox, and the only workable methods out of the box were the Thread.GetData(), Thread.SetData() methods. Those methods aren’t good performance-wise, so I proceeded to look at more efficient alternatives.

That article was going to detail the solution I came up with, and currently used in my engine. However upon revisiting the work recently and upgrading to the new v4.0 XNA, I found that I had some compile errors. Specifically ‘System.Threading.LocalDataStoreSlot’ was now missing, it’s that slow method of TLS I was using to benchmark against. So I commented that code out. I then figured why not for the hell of it try to enable the [ThreadStatic] code, that I had #ifdef’d to the WINDOWS build. Lo and behold, it now compiled on Xbox 360!

No idea if it’s any better performing than the old ‘LocalDataStoreSlot’ though, or better than the alternate method I came up with. That’s an exercise for another day.

My Googling Escapades

The only article I could find back when I was working on 3.1 about [ThreadStatic] on Xbox was this one. Doing a search now, I couldn’t find anything else new. I also tried searching for information about “LocalDataStoreSlot”, the class which was removed in the 4.0 upgrade. Again, I found nothing about the removal.

I also had noticed in some other code I have that the ‘System.Diagnostics.Trace’ class, and static functions therein were also removed. Now I start to scratch my head a little and wonder what else has disappeared, and what other cool things may have been added. My searches weren’t very fruitful, I just found a handful of forum posts which mentioned briefly some sort of changes to the Compact Framework (I’ve included them in my references at the bottom of this page). Nothing complete and comprehensive which listed all the differences.

The MSDN documentation does denote on the page for each class, type, function… which platforms it is available on. There’s also a dropdown to switch the framework version. I confess I found it a bit unwieldy to check up on. For example ‘ThreadStaticAttribute’ shows a ‘Silverlight’ entry in the dropdown box. On that page it denotes that it works on the Xbox 360. Where ‘LocalDataStoreSlot’ (the one which was removed), does not have Silverlight in it’s list, but does show XNA support for the 360 on the 3.5 Framework. Which tells me that it used to work, but now is deprecated in the Silverlight-compliant Framework.

Anyhow, checking these things one-by-one isn’t exactly viable.

Taking a dump

Knowing that there was a number of changes, but not knowing exactly what those changes were; I was still a little curious. What I did was to use the ‘ildasm’ tool to dump the types which existed in the .NET Framework DLL files.

For those equally curious, they reside here:
C:\Program Files\Microsoft XNA\XNA Game Studio\v3.1\References\Xbox360
and here:
C:\Program Files\Microsoft XNA\XNA Game Studio\v4.0\References\Xbox360
Or wherever your “\Program Files” directory lies.

With a little tinkering I could collate that output and determine the changes which occurred between the two versions of the Framework. I omitted the “Microsoft.Xna.Framework.Input.Touch” library from the output though, since new Xbox 360 projects do not include it as a reference. I’m unsure though as to why there is a DLL sitting in the “Xbox 360″ directory. Possibly early Natal/Kinect support code? Heh, I doubt it. I’m guessing it’s just stub functionality for parity with the Windows 7 phones.

So anyhow, here’s the data I collected. Hopefully others may find this useful… Or possibly just point me to a website that had this information already.

System types removed from the Xbox 360 Compact Framework

These are types which used to exist in the Compact Framework for the Xbox 360, in the “XNA Game Studio” v3.1 release. But these are now missing from the new v4.0 release:

  Microsoft.Xna.Framework.BatteryChargeStatus
  Microsoft.Xna.Framework.Graphics.BasicDirectionalLight
  Microsoft.Xna.Framework.Graphics.ClipPlane
  Microsoft.Xna.Framework.Graphics.ClipPlaneCollection
  Microsoft.Xna.Framework.Graphics.Color
  Microsoft.Xna.Framework.Graphics.CompilerOptions
  Microsoft.Xna.Framework.Graphics.DepthStencilBuffer
  Microsoft.Xna.Framework.Graphics.DeviceNotSupportedException
  Microsoft.Xna.Framework.Graphics.DeviceStillDrawingException
  Microsoft.Xna.Framework.Graphics.DeviceType
  Microsoft.Xna.Framework.Graphics.DriverInternalErrorException
  Microsoft.Xna.Framework.Graphics.EffectFunction
  Microsoft.Xna.Framework.Graphics.EffectFunctionCollection
  Microsoft.Xna.Framework.Graphics.EffectParameterBlock
  Microsoft.Xna.Framework.Graphics.EffectPool
  Microsoft.Xna.Framework.Graphics.GammaRamp
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/AddressCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/BlendCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/CompareCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/CursorCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/DeclarationTypeCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/DeviceCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/DriverCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/FilterCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/LineCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/PixelShaderCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/PrimitiveCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/RasterCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/ShadingCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/StencilCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/TextureCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/VertexFormatCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/VertexProcessingCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCapabilities/VertexShaderCaps
  Microsoft.Xna.Framework.Graphics.GraphicsDeviceCreationParameters
  Microsoft.Xna.Framework.Graphics.MultiSampleType
  Microsoft.Xna.Framework.Graphics.OutOfVideoMemoryException
  Microsoft.Xna.Framework.Graphics.PackedVector.Normalized101010
  Microsoft.Xna.Framework.Graphics.PackedVector.Rgba32
  Microsoft.Xna.Framework.Graphics.PackedVector.UInt101010
  Microsoft.Xna.Framework.Graphics.PixelShader
  Microsoft.Xna.Framework.Graphics.PresentOptions
  Microsoft.Xna.Framework.Graphics.QueryUsages
  Microsoft.Xna.Framework.Graphics.RasterStatus
  Microsoft.Xna.Framework.Graphics.RenderState
  Microsoft.Xna.Framework.Graphics.RenderTarget
  Microsoft.Xna.Framework.Graphics.ResolveTexture2D
  Microsoft.Xna.Framework.Graphics.ResourceType
  Microsoft.Xna.Framework.Graphics.SaveStateMode
  Microsoft.Xna.Framework.Graphics.ShaderConstant
  Microsoft.Xna.Framework.Graphics.ShaderConstantCollection
  Microsoft.Xna.Framework.Graphics.ShaderConstantTable
  Microsoft.Xna.Framework.Graphics.ShaderProfile
  Microsoft.Xna.Framework.Graphics.ShaderRegisterSet
  Microsoft.Xna.Framework.Graphics.SpriteBlendMode
  Microsoft.Xna.Framework.Graphics.StateBlock
  Microsoft.Xna.Framework.Graphics.SwapEffect
  Microsoft.Xna.Framework.Graphics.TextureUsage
  Microsoft.Xna.Framework.Graphics.TextureWrapCoordinates
  Microsoft.Xna.Framework.Graphics.VertexElementMethod
  Microsoft.Xna.Framework.Graphics.VertexShader
  Microsoft.Xna.Framework.Graphics.VertexStream
  Microsoft.Xna.Framework.Graphics.VertexStreamCollection
  Microsoft.Xna.Framework.Net.InviteAcceptedEventArgs
  Microsoft.Xna.Framework.Net.NetworkException
  Microsoft.Xna.Framework.Net.NetworkNotAvailableException
  Microsoft.Xna.Framework.NoSuitableGraphicsDeviceException
  Microsoft.Xna.Framework.PowerLineStatus
  Microsoft.Xna.Framework.PowerStatus
  System.ApplicationException
  System.CharEnumerator
  System.Collections.ArrayList
  System.Collections.CaseInsensitiveComparer
  System.Collections.CaseInsensitiveHashCodeProvider
  System.Collections.CollectionBase
  System.Collections.Comparer
  System.Collections.Generic.SortedList`2
  System.Collections.Hashtable
  System.Collections.IHashCodeProvider
  System.Collections.Queue
  System.Collections.SortedList
  System.Collections.Specialized.BitVector32
  System.Collections.Specialized.BitVector32/Section
  System.Collections.Specialized.HybridDictionary
  System.Collections.Specialized.ListDictionary
  System.Collections.Specialized.NameObjectCollectionBase
  System.Collections.Specialized.NameObjectCollectionBase/KeysCollection
  System.Collections.Specialized.NameValueCollection
  System.Collections.Specialized.StringDictionary
  System.Collections.Stack
  System.ComponentModel.AddingNewEventArgs
  System.ComponentModel.AddingNewEventHandler
  System.ComponentModel.AttributeCollection
  System.ComponentModel.BindingList`1
  System.ComponentModel.CancelEventHandler
  System.ComponentModel.CollectionChangeAction
  System.ComponentModel.CollectionChangeEventArgs
  System.ComponentModel.CollectionChangeEventHandler
  System.ComponentModel.Component
  System.ComponentModel.ComponentResourceManager
  System.ComponentModel.Container
  System.ComponentModel.DataObjectAttribute
  System.ComponentModel.DataObjectMethodAttribute
  System.ComponentModel.DataObjectMethodType
  System.ComponentModel.DesignerCategoryAttribute
  System.ComponentModel.DesignTimeVisibleAttribute
  System.ComponentModel.EventDescriptor
  System.ComponentModel.EventDescriptorCollection
  System.ComponentModel.EventHandlerList
  System.ComponentModel.IBindingList
  System.ComponentModel.IBindingListView
  System.ComponentModel.ICancelAddNew
  System.ComponentModel.ICustomTypeDescriptor
  System.ComponentModel.IDataErrorInfo
  System.ComponentModel.IListSource
  System.ComponentModel.IRaiseItemChangedEvents
  System.ComponentModel.ISupportInitialize
  System.ComponentModel.ISupportInitializeNotification
  System.ComponentModel.ITypedList
  System.ComponentModel.ListChangedEventArgs
  System.ComponentModel.ListChangedEventHandler
  System.ComponentModel.ListChangedType
  System.ComponentModel.ListSortDescription
  System.ComponentModel.ListSortDescriptionCollection
  System.ComponentModel.MarshalByValueComponent
  System.ComponentModel.MemberDescriptor
  System.ComponentModel.PropertyDescriptorCollection
  System.ComponentModel.TypeDescriptor
  System.ComponentModel.Win32Exception
  System.Diagnostics.DefaultTraceListener
  System.Diagnostics.TextWriterTraceListener
  System.Diagnostics.Trace
  System.Diagnostics.TraceListener
  System.Diagnostics.TraceListenerCollection
  System.Globalization.CultureTypes
  System.Globalization.DaylightTime
  System.IO.Compression.CompressionMode
  System.IO.Compression.DeflateStream
  System.IO.Compression.GZipStream
  System.IO.InvalidDataException
  System.MarshalByRefObject
  System.Resources.ResourceReader
  System.Security.Policy.Evidence
  System.SerializableAttribute
  System.Text.ASCIIEncoding
  System.Text.RegularExpressions.RegexRunner
  System.Text.RegularExpressions.RegexRunnerFactory
  System.Text.UTF7Encoding
  System.Threading.EventResetMode
  System.Threading.Mutex
  System.Threading.ThreadPriority
  System.TimeZone
  System.TypedReference
  System.UriHostNameType
  System.UriPartial
  System.Xml.EntityHandling
  System.Xml.Formatting
  System.Xml.IHasXmlNode
  System.Xml.Schema.IXmlSchemaInfo
  System.Xml.Schema.ValidationEventArgs
  System.Xml.Schema.ValidationEventHandler
  System.Xml.Schema.XmlAtomicValue
  System.Xml.Schema.XmlSchemaAll
  System.Xml.Schema.XmlSchemaAnnotated
  System.Xml.Schema.XmlSchemaAnnotation
  System.Xml.Schema.XmlSchemaAny
  System.Xml.Schema.XmlSchemaAnyAttribute
  System.Xml.Schema.XmlSchemaAppInfo
  System.Xml.Schema.XmlSchemaAttribute
  System.Xml.Schema.XmlSchemaAttributeGroup
  System.Xml.Schema.XmlSchemaAttributeGroupRef
  System.Xml.Schema.XmlSchemaChoice
  System.Xml.Schema.XmlSchemaCompilationSettings
  System.Xml.Schema.XmlSchemaComplexContent
  System.Xml.Schema.XmlSchemaComplexContentExtension
  System.Xml.Schema.XmlSchemaComplexContentRestriction
  System.Xml.Schema.XmlSchemaComplexType
  System.Xml.Schema.XmlSchemaContent
  System.Xml.Schema.XmlSchemaContentModel
  System.Xml.Schema.XmlSchemaContentProcessing
  System.Xml.Schema.XmlSchemaContentType
  System.Xml.Schema.XmlSchemaDatatype
  System.Xml.Schema.XmlSchemaDatatypeVariety
  System.Xml.Schema.XmlSchemaDerivationMethod
  System.Xml.Schema.XmlSchemaDocumentation
  System.Xml.Schema.XmlSchemaElement
  System.Xml.Schema.XmlSchemaEnumerationFacet
  System.Xml.Schema.XmlSchemaException
  System.Xml.Schema.XmlSchemaExternal
  System.Xml.Schema.XmlSchemaFacet
  System.Xml.Schema.XmlSchemaFractionDigitsFacet
  System.Xml.Schema.XmlSchemaGroup
  System.Xml.Schema.XmlSchemaGroupBase
  System.Xml.Schema.XmlSchemaGroupRef
  System.Xml.Schema.XmlSchemaIdentityConstraint
  System.Xml.Schema.XmlSchemaImport
  System.Xml.Schema.XmlSchemaInclude
  System.Xml.Schema.XmlSchemaInference
  System.Xml.Schema.XmlSchemaInference/InferenceOption
  System.Xml.Schema.XmlSchemaInferenceException
  System.Xml.Schema.XmlSchemaInfo
  System.Xml.Schema.XmlSchemaKey
  System.Xml.Schema.XmlSchemaKeyref
  System.Xml.Schema.XmlSchemaLengthFacet
  System.Xml.Schema.XmlSchemaMaxExclusiveFacet
  System.Xml.Schema.XmlSchemaMaxInclusiveFacet
  System.Xml.Schema.XmlSchemaMaxLengthFacet
  System.Xml.Schema.XmlSchemaMinExclusiveFacet
  System.Xml.Schema.XmlSchemaMinInclusiveFacet
  System.Xml.Schema.XmlSchemaMinLengthFacet
  System.Xml.Schema.XmlSchemaNotation
  System.Xml.Schema.XmlSchemaNumericFacet
  System.Xml.Schema.XmlSchemaObjectCollection
  System.Xml.Schema.XmlSchemaObjectEnumerator
  System.Xml.Schema.XmlSchemaObjectTable
  System.Xml.Schema.XmlSchemaParticle
  System.Xml.Schema.XmlSchemaPatternFacet
  System.Xml.Schema.XmlSchemaRedefine
  System.Xml.Schema.XmlSchemaSequence
  System.Xml.Schema.XmlSchemaSimpleContent
  System.Xml.Schema.XmlSchemaSimpleContentExtension
  System.Xml.Schema.XmlSchemaSimpleContentRestriction
  System.Xml.Schema.XmlSchemaSimpleType
  System.Xml.Schema.XmlSchemaSimpleTypeContent
  System.Xml.Schema.XmlSchemaSimpleTypeList
  System.Xml.Schema.XmlSchemaSimpleTypeRestriction
  System.Xml.Schema.XmlSchemaSimpleTypeUnion
  System.Xml.Schema.XmlSchemaTotalDigitsFacet
  System.Xml.Schema.XmlSchemaType
  System.Xml.Schema.XmlSchemaUnique
  System.Xml.Schema.XmlSchemaUse
  System.Xml.Schema.XmlSchemaValidationException
  System.Xml.Schema.XmlSchemaValidationFlags
  System.Xml.Schema.XmlSchemaValidator
  System.Xml.Schema.XmlSchemaValidity
  System.Xml.Schema.XmlSchemaWhiteSpaceFacet
  System.Xml.Schema.XmlSchemaXPath
  System.Xml.Schema.XmlValueGetter
  System.Xml.Serialization.IXmlTextParser
  System.Xml.Serialization.SoapAttributeAttribute
  System.Xml.Serialization.SoapAttributeOverrides
  System.Xml.Serialization.SoapAttributes
  System.Xml.Serialization.SoapElementAttribute
  System.Xml.Serialization.SoapEnumAttribute
  System.Xml.Serialization.SoapIgnoreAttribute
  System.Xml.Serialization.SoapIncludeAttribute
  System.Xml.Serialization.SoapReflectionImporter
  System.Xml.Serialization.SoapTypeAttribute
  System.Xml.Serialization.UnreferencedObjectEventArgs
  System.Xml.Serialization.UnreferencedObjectEventHandler
  System.Xml.Serialization.XmlAnyAttributeAttribute
  System.Xml.Serialization.XmlAttributeEventArgs
  System.Xml.Serialization.XmlAttributeEventHandler
  System.Xml.Serialization.XmlElementEventArgs
  System.Xml.Serialization.XmlElementEventHandler
  System.Xml.Serialization.XmlNodeEventArgs
  System.Xml.Serialization.XmlNodeEventHandler
  System.Xml.ValidationType
  System.Xml.WhitespaceHandling
  System.Xml.XmlAttribute
  System.Xml.XmlAttributeCollection
  System.Xml.XmlCDataSection
  System.Xml.XmlCharacterData
  System.Xml.XmlComment
  System.Xml.XmlDeclaration
  System.Xml.XmlDocument
  System.Xml.XmlDocumentFragment
  System.Xml.XmlElement
  System.Xml.XmlEntityReference
  System.Xml.XmlImplementation
  System.Xml.XmlLinkedNode
  System.Xml.XmlNamedNodeMap
  System.Xml.XmlNode
  System.Xml.XmlNodeChangedAction
  System.Xml.XmlNodeChangedEventArgs
  System.Xml.XmlNodeChangedEventHandler
  System.Xml.XmlNodeList
  System.Xml.XmlNodeOrder
  System.Xml.XmlNodeReader
  System.Xml.XmlOutputMethod
  System.Xml.XmlProcessingInstruction
  System.Xml.XmlSignificantWhitespace
  System.Xml.XmlText
  System.Xml.XmlTextReader
  System.Xml.XmlTextWriter
  System.Xml.XmlUrlResolver
  System.Xml.XmlWhitespace
  System.Xml.XPath.XPathException
  System.Xml.XPath.XPathItem
  System.Xml.Xsl.XsltException

System types added to the Xbox 360 Compact Framework

These are types which did not exist in the Compact Framework for the Xbox 360, in the “XNA Game Studio” v3.1 release. But these are now present in the new v4.0 release:

  Microsoft.Win32.SafeHandles.SafeHandleMinusOneIsInvalid
  Microsoft.Win32.SafeHandles.SafeHandleZeroOrMinusOneIsInvalid
  Microsoft.Win32.SafeHandles.SafeWaitHandle
  Microsoft.Xna.Framework.Audio.AudioChannels
  Microsoft.Xna.Framework.Audio.DynamicSoundEffectInstance
  Microsoft.Xna.Framework.Audio.Microphone
  Microsoft.Xna.Framework.Audio.MicrophoneState
  Microsoft.Xna.Framework.Audio.NoMicrophoneConnectedException
  Microsoft.Xna.Framework.Color
  Microsoft.Xna.Framework.DisplayOrientation
  Microsoft.Xna.Framework.GamerServices.Achievement
  Microsoft.Xna.Framework.GamerServices.AchievementCollection
  Microsoft.Xna.Framework.GamerServices.AvatarRendererState
  Microsoft.Xna.Framework.GamerServices.GameUpdateRequiredException
  Microsoft.Xna.Framework.GamerServices.IAvatarAnimation
  Microsoft.Xna.Framework.GamerServices.InviteAcceptedEventArgs
  Microsoft.Xna.Framework.GamerServices.LeaderboardEntry
  Microsoft.Xna.Framework.GamerServices.LeaderboardIdentity
  Microsoft.Xna.Framework.GamerServices.LeaderboardKey
  Microsoft.Xna.Framework.GamerServices.LeaderboardOutcome
  Microsoft.Xna.Framework.GamerServices.LeaderboardReader
  Microsoft.Xna.Framework.GamerServices.LeaderboardWriter
  Microsoft.Xna.Framework.GamerServices.NetworkException
  Microsoft.Xna.Framework.GamerServices.NetworkNotAvailableException
  Microsoft.Xna.Framework.GamerServices.PropertyDictionary
  Microsoft.Xna.Framework.Graphics.AlphaTestEffect
  Microsoft.Xna.Framework.Graphics.BlendState
  Microsoft.Xna.Framework.Graphics.DepthStencilState
  Microsoft.Xna.Framework.Graphics.DirectionalLight
  Microsoft.Xna.Framework.Graphics.DualTextureEffect
  Microsoft.Xna.Framework.Graphics.EffectMaterial
  Microsoft.Xna.Framework.Graphics.EnvironmentMapEffect
  Microsoft.Xna.Framework.Graphics.GraphicsProfile
  Microsoft.Xna.Framework.Graphics.IEffectFog
  Microsoft.Xna.Framework.Graphics.IEffectLights
  Microsoft.Xna.Framework.Graphics.IEffectMatrices
  Microsoft.Xna.Framework.Graphics.IVertexType
  Microsoft.Xna.Framework.Graphics.NoSuitableGraphicsDeviceException
  Microsoft.Xna.Framework.Graphics.PackedVector.Bgra4444
  Microsoft.Xna.Framework.Graphics.RasterizerState
  Microsoft.Xna.Framework.Graphics.RenderTargetBinding
  Microsoft.Xna.Framework.Graphics.SkinnedEffect
  Microsoft.Xna.Framework.Graphics.VertexBufferBinding
  Microsoft.Xna.Framework.LaunchParameters
  Microsoft.Xna.Framework.Net.WriteLeaderboardsEventArgs
  Microsoft.Xna.Framework.TitleContainer
  System.AccessViolationException
  System.AppDomainManager
  System.AppDomainSetup
  System.ComponentModel.AsyncOperation
  System.ComponentModel.AsyncOperationManager
  System.ComponentModel.BackgroundWorker
  System.ComponentModel.CategoryAttribute
  System.ComponentModel.ComponentCollection
  System.ComponentModel.DescriptionAttribute
  System.ComponentModel.DoWorkEventArgs
  System.ComponentModel.DoWorkEventHandler
  System.ComponentModel.ITypeDescriptorContext
  System.ComponentModel.ProgressChangedEventArgs
  System.ComponentModel.ProgressChangedEventHandler
  System.ComponentModel.ReadOnlyAttribute
  System.ComponentModel.RunWorkerCompletedEventArgs
  System.ComponentModel.RunWorkerCompletedEventHandler
  System.ComponentModel.TypeConverterAttribute
  System.ContextStaticAttribute
  System.DataMisalignedException
  System.DateTimeOffset
  System.Diagnostics.DebuggerBrowsableAttribute
  System.Diagnostics.DebuggerBrowsableState
  System.Diagnostics.DebuggerDisplayAttribute
  System.Diagnostics.DebuggerTypeProxyAttribute
  System.Diagnostics.StackFrame
  System.Diagnostics.StackTrace
  System.DllNotFoundException
  System.Environment/SpecialFolder
  System.ExecutionEngineException
  System.FieldAccessException
  System.FileStyleUriParser
  System.FtpStyleUriParser
  System.GenericUriParser
  System.GenericUriParserOptions
  System.Globalization.HebrewCalendar
  System.Globalization.HijriCalendar
  System.Globalization.UmAlQuraCalendar
  System.HttpStyleUriParser
  System.InvalidTimeZoneException
  System.IO.FileLoadException
  System.IO.IsolatedStorage.IsolatedStorageException
  System.IO.IsolatedStorage.IsolatedStorageFile
  System.IO.IsolatedStorage.IsolatedStorageFileStream
  System.IO.UnmanagedMemoryStream
  System.LoaderOptimization
  System.MissingFieldException
  System.NotFiniteNumberException
  System.Reflection.AssemblyFileVersionAttribute
  System.Reflection.GenericParameterAttributes
  System.Reflection.InterfaceMapping
  System.Reflection.IReflect
  System.Reflection.LocalVariableInfo
  System.Reflection.MemberFilter
  System.Reflection.ReflectionTypeLoadException
  System.Reflection.ResourceAttributes
  System.Reflection.TargetException
  System.Reflection.TypeDelegator
  System.ResolveEventArgs
  System.ResolveEventHandler
  System.Resources.UltimateResourceFallbackLocation
  System.Runtime.CompilerServices.CompilationRelaxations
  System.Runtime.CompilerServices.CompilationRelaxationsAttribute
  System.Runtime.CompilerServices.RuntimeCompatibilityAttribute
  System.Runtime.InteropServices.AllowReversePInvokeCallsAttribute
  System.Runtime.InteropServices.ClassInterfaceAttribute
  System.Runtime.InteropServices.ClassInterfaceType
  System.Runtime.InteropServices.CoClassAttribute
  System.Runtime.InteropServices.ComDefaultInterfaceAttribute
  System.Runtime.InteropServices.COMException
  System.Runtime.InteropServices.ComImportAttribute
  System.Runtime.InteropServices.ComInterfaceType
  System.Runtime.InteropServices.DispIdAttribute
  System.Runtime.InteropServices.InterfaceTypeAttribute
  System.Runtime.InteropServices.SafeHandle
  System.Runtime.Serialization.OnDeserializedAttribute
  System.Runtime.Serialization.OnDeserializingAttribute
  System.Runtime.Serialization.OnSerializedAttribute
  System.Runtime.Serialization.OnSerializingAttribute
  System.Runtime.Serialization.StreamingContext
  System.Runtime.Serialization.StreamingContextStates
  System.Runtime.Versioning.TargetFrameworkAttribute
  System.RuntimeArgumentHandle
  System.Security.Permissions.CodeAccessSecurityAttribute
  System.Security.Permissions.SecurityAttribute
  System.Security.Permissions.SecurityPermissionAttribute
  System.Security.Permissions.SecurityPermissionFlag
  System.Security.Principal.IIdentity
  System.Security.Principal.IPrincipal
  System.Security.SecurityCriticalAttribute
  System.Security.SecuritySafeCriticalAttribute
  System.Security.SecurityState
  System.Security.SecurityTransparentAttribute
  System.Security.UnverifiableCodeAttribute
  System.ServiceModel.XmlSerializerFormatAttribute
  System.StringSplitOptions
  System.Text.DecoderFallbackException
  System.Text.EncoderFallbackException
  System.Threading.IOCompletionCallback
  System.Threading.NativeOverlapped
  System.Threading.Overlapped
  System.Threading.ParameterizedThreadStart
  System.Threading.RegisteredWaitHandle
  System.Threading.SendOrPostCallback
  System.Threading.SynchronizationContext
  System.Threading.SynchronizationLockException
  System.Threading.ThreadStartException
  System.Threading.ThreadState
  System.Threading.WaitOrTimerCallback
  System.ThreadStaticAttribute
  System.TimeZoneInfo
  System.TypeInitializationException
  System.UriBuilder
  System.UriIdnScope
  System.UriParser
  System.UriTypeConverter
  System.Xml.DtdProcessing
  System.Xml.Linq.ReaderOptions
  System.Xml.NamespaceHandling
  System.Xml.Serialization.XmlMappingAccess

Notables?

A number of deprecated collections are no longer available, the classic “ArrayList” from C# 1.0 among several others. Most of these would cause big performance problems with the Xbox garbage collector, so makes total sense that they have been trimmed out.

Additions which piqued my interest included “DebuggerDisplayAttribute”, and “StackTrace”/ “StackFrame”. The former I have a vague recollection trying to use before, it helps nicely format how a type is viewed in the Debugger. You can achieve the same functionality with ToString(), but if you need to use ToString() for runtime purposes and want something different shown in the debugger, it’s useful. “StackTrace” I was interested in because I believe the crash handler I’d written had a slightly different pathway between 360 and Windows. For 360 I had to throw an exception, and handle it. In the exception handling code I could pull out the callstack. With the StackTrace class you can just grab the callstack right off in the crash handler.

“System.Threading.ThreadState” too, I think I’d had to do a different workaround for the Xbox in the past. Nice to see that it’s now available.

Strangely I noticed “System.LocalDataStoreSlot” is not included in my removal list. Not sure why this is the case. Maybe I missed some key parameter in ‘ildasm’?

The Zune, or Windows 7 Phone?

Sorry, but I haven’t been developing for those platforms. Given some comments about the v4.0 release wanting to ‘standardize’ the Compact Framework to some kind of “Silverlight” API-set, I’d imagine the ones I’ve mentioned were added will probably exist for these platforms too. No idea though whether all the removed types I list apply to the Zune, many may not have even existed in the first place.

Er… So yeah, this is just covering the Xbox. Your mileage may vary with the other platforms.

Edit: Ah, Zune isn’t even supported by XNA 4.0!

References

• MSDN: Differences Between the .NET Compact Framework and the .NET Framework
• MSDN: Namespaces, Types, and Members in the .NET Compact Framework for Xbox 360
• Forums: XNA 4.0 RTM – Xbox Project – Serializable doesnt work
• Forums: XNA 4.0 .Net reference version for XBox
• Forums: Thread.Priority property deprecated on 360/XNA 4.0?
• Shawn’s Blog: Breaking changes in XNA Game Studio 4.0