Cubes All The Way Down @ MIGS

Back in November 2011, I gave a talk at the Montréal International Game Summit in the Technology track called “Cubes All The Way Down”, where I talked about how FEZ was built, what’s the big modules, the challenges and intricacies of making a tech-heavy indie game from scratch.

It went okay.
I was really stressed, a bit unprepared due to FEZ crunch time, and just generally uncomfortable speaking in front of an audience.
I spoke so fast that I finished 15 minutes early and had 30 minutes for questions, which worked great for me because the relaxed setting of a Q&A session meant better flow, better information delivery, I really liked that part. Also I had friends in the front row that kept asking good questions and were generally supportive, so all in all a good experience. :)

I was asked about giving the slides out, so here they are! Unedited.

It’s Cubes All The Way Down (Powerpoint 2007 PPTX format) (PDF format)

Enjoy!

Encoding boolean flags into a float in HLSL

(this applies to Shader Model 3 and lower)

Hey! I’m still alive!

So, imagine you’re writing a shader instancing shader (sounds redundant, but that’s actually what they are) and you’re trying to pack a lot of data into a float4 or a float4x4 in order to maximize the amount of instances you can render in a single draw call.

My instances had many boolean flags that changed per-instance and that defined how they were lit or rendered. Things like whether or not they are fullbright (100% emissive), texture transform flags (repeating on x or y, more efficient to rebuild the texture matrix than pass it), etc.
Using one float out of your instance data matrix for each boolean is doable, but highly wasteful. A natural way to fit in many flags into an integer is to use a bitfield, but there’s no integer arithmetic in HLSL, and they’re floating point values… how does one proceed?

Here’s how I did it.

Application side

First, this is how I pack my data into floats from the application side (setting the effect parameter) :

int flags = (fullbright ? 1 : 0) | 
	(clampTexture ? 2 : 0) | 
	(xTextureRepeat ? 4 : 0) | 
	(yTextureRepeat ? 8 : 0);

Geometry.Instances[InstanceIndex] = new Matrix(
	p.X, Rotation.X, Scale.X, color.X,
	p.Y, Rotation.Y, Scale.Y, color.Y,
	p.Z, Rotation.Z, Scale.Z, color.Z,
	Animated ? Timing.Step : 0, Rotation.W, flags, Opacity);

Just putting an OR operator between the flags you wanna put, and keep the flag bits powers of two.
Ignore the rest of the matrix contents, they’re just here for show. (in my case : position, rotation, scale, color, opacity, animation frame and the flag collection).

A note on floating point : in a single-precision floating point number as defined by the IEEE, you’ve got 23 bits for the significand. That means you can theoretically put 23 flags in there! That’s a lot of data.
(also, considering the decimal point is floating, you can effectively put much more than 23 bits if some of them are mutually exclusive…!)

Vertex shader

Now in the vertex shader, they get passed to an effect parameter through vertex shader constants, and here’s now the decoding works :

int flags = data[2][3];

bool fullbright = fmod(flags, 2) == 1;
bool clampTexture = fmod(flags, 4) >= 2;
bool xTextureRepeat = fmod(flags, 8) >= 4;
bool yTextureRepeat = fmod(flags, 16) >= 8;

I know my flags reside in the 3rd row, 4th column of my matrix, so I grab ‘em from that. Might as well cast them to an integer right now since I won’t be using decimals.

Then I can test for values by testing the remainder of the division of each power-of-two. There is no integer modulo intrinsic function in HLSL for Shader Models 3 and lesser, but the floating-point version works fine.

If I set the first (least significant) bit of a number and divide it by two, the remainder will be 1 if that bit is set. Basically, we test if that number is odd or even; odd means the bit is set.

For every other test, we can test whether the remainder is greater or equal to half the divisor. Effectively, we’re masking the bits greater than the one we’re testing, and testing remaining bits for the presence of the one we’re looking for. Here, if we test for the 3rd bit (from the LSB), so masking with 8 (1000 in binary) and testing against 4 (0100 in binary) :

0000 % 1000 = 0000 // 0 < 4, bit not set
0100 % 1000 = 0100 // 4 >= 4, bit set
1011 % 1000 = 0011 // 3 < 4, bit not set
1110 % 1000 = 0110 // 6 >= 4, bit set
1101 % 1000 = 0101 // 5 >= 4, bit set

Enjoy!

Pax Britannica

Even though the game’s been out for a long while and even has an official page, I’ve never made a post about it, and now that we’ve been elected #3 Free Indie Game of 2010 on Bytejacker (!!!), now seems like a good time to spread the word!

Multiplayer underwater mayhem!

We originally made the game for Gamma IV earlier this year, and released a Windows version when the competition ended, but recently we went back and finished up the Mac and Linux versions. It’s great that porting the game didn’t involve touching any of the game code (which is in Lua, and MIT-licensed).

So whatever platform you’re running on, try the game out! It’s great fun in multiplayer, and the learning curve is almost inexistent.

All the latest versions are downloadable on the official site, http://paxbritannica.henk.ca/.

Pico³

Pico³ is a game I made with Aliceffekt (as Les Collégiennes) over the course of a month, and that we presented at the Prince of Arcade party on November 9th.

Download

Pico³ – Windows Version [6.3 Mb]
Pico³ – Mac OS X Version [11.6 Mb] (Nov. 15th Edit : Fixed the mac build, it runs now!)

Aliceffekt designed the game mechanics, levels and visuals, while I took care of all programming and procedural animations.

How to play

The game is fairly simple on the surface :

  • Emitters emit cells of a primary color (red, green or blue).
  • Receptors expect cells of a certain color, or color (ordered) sequence.
  • You can place Projectors that redirect cells or combine them, if different cells hit the projector simultaneously.

The challenge is to combine colors at the right time, with the given resources and world layout. It becomes an intricate resource management/puzzle game, and even the simplest-looking puzzle can prove almost impossible!

There is only 13 levels in this version, which was made for a party setting. The difficulty curve proved to be very harsh for new players, and even seasoned players (like me) can’t reach the end. It’s a hard game — Aliceffekt’s trademark game design. ^_^

Science! (shot by Aliceffekt at the Prince of Arcade)

 
It is played with mouse+keyboard on all platforms, but also supports the Xbox 360 gamepad (either wired or wireless with an USB receiver) on Windows by using Rémi Gillig’s XInput.NET for Unity. I made my own wrapper over it to detect press/hold/down, actually the code was ripped out of my XNA code. That’s the fun part of using C# scripting, I can just share code between projects even if it’s not the same technology!

Controls

If you’re too lazy to read the tutorials :

  • Right click and drag to rotate the camera round the world, scrollwheel to zoom in/out
  • Left click to create a Projector, and left click on a face to select its direction
  • Z to undo the last Projector (or hover any Projector and hit Z to undo that one)
  • R to restart the level
  • Escape to return to the first Level
  • ALT+F4 or Command+Q to quit

Hope you like it, it was a a lot of fun to make and I’m already looking forward to my next Unity creation… It’s a great work environment.

Tea-Time Quarrel

Goats!

The game I made with Matthew Gallant and Henk Boom (us three being the “No Fun” collective) at ToJam earlier this year is finally out!

Matthew already wrote a super thorough write-up about it, so I won’t bother. But here’s the direct link to the game itself :

P.S. : Did you notice the new Games page on the top menu-bar? It keeps track of all the games I worked on, with links to blog posts and download links. Check it out!

The Cloud is a Lie

The Cloud is a Lie is a game that me and Aliceffekt (our team name was “Les Collégiennes”) made at a 36-hour gamejam/competition in Québec City called Bivouac Urbain. (Last year I also attended the event, and made Stimergy with Heather Kelley).

Download

The game is available for Windows or Mac OS X. [10 and 15Mb respectively]

Controls

WASD or Arrows keys move. (you need to hold them)
Spacebar shoots.
M toggles the music.
Escape quits. (there may be a 2-5 seconds pause when quitting the game on Windows)

Gameplay

The Cloud is a Lie is a LAN multiplayer game for any number of players, but best suited for 2 to 4 players. It can be played alone too, it’s just less fun.

Otherwise, it’s like Tower Defense meets Tron.

  • There is a constant flow of enemies (the spinning cyan cylinders) coming in from the end of the map, and their goal is to reach your home base.
  • By moving around, the players create walls that eventually fade out, but that can slow down the enemies’ movement.
  • You can shoot lasers using your headpiece/weapon, and one shot kills an enemy (but there’s a cooldown time).
  • You gain experience points by killing enemies, and experience points levels up your weapon (faster kills, up to 3 attacks per shot).
  • The more experience you and other players gain, the more enemies are produced.
  • If an enemy makes it through your defenses to your base, every player loses a level.
  • The game never ends, but eventually there’s just too many enemies, everyone falls back to level 1 and the game falls back to its initial state.

Concept

The idea was to make a multiplayer game without a server, that happens “on the wire” and free for any one to drop in or out.
So it kinda goes against the idea of cloud computing, where the client is as thin as possible and the server (or server farm) does all the work and keeps all the state. In our game, the clients are as heavy as can be, and try to maintain the state of the whole game based on the messages that are passed around.

Also, the theme of this year’s Bivouac Urbain was the song Dan Dan by Misteur Valaire. The song that is played in-game is Aliceffekt’s remix of that song, and it complements the gameplay nicely.

Post-Mortem

This year’s jam was very different from last year’s. Aliceffekt is a super-productive multi-talented artist/designer/hacker and he was quite an asset at a gamejam.
The game’s design was already decided before the jam started, so that saved us a lot of time. And 12-hours in, most of the art was already done, and I was really lagging behind. :P

Concept art for the player model

It was a huge challenge to me for two main reasons :

  1. It was my first real game in Unity (I only skimmed through tutorials before, but I use C# daily)
  2. It also was my first network-multiplayer game, and a weird one at that.

Most of my time at the jam was spent debugging the network code, and just messing around trying not to crash Unity. Even at the end of the 36-hours, the game had pretty bad synchronization issues between clients, different enemies on-screen and lasers shooting in the void. People seemed to like it nonetheless. :D

I can’t say I regret decisions that we’ve made, or that I/we should have done things differently. It was a great learning experience, stressful but gratifying, and jumping into unknown territory like that is one of the best ways to kick yourself in the butt and learn a technology for good. I feel infinitely more confident in using Unity now than before the jam.

About the initial concept (a headless LAN multiplayer game), I’d say that it’s feasible, but keeping the clients in sync is a big challenge. I’d also say that my implementation is pretty poor and doesn’t play well with lost packets (and it uses UDP, so no guarantee) or an unstable connection. I didn’t do any kind of client-side prediction either, but I literally flood the network with packets (every client sending 10 packets a second) so interpolation isn’t really necessary if the network can stand it.

A couple of things annoyed me with Unity, but I was at such an early learning stage that I probably did things all wrong, so they’re not worth pointing out. The general thing that stood out is that Unity is fantastic when you use the built-in stuff, and less fantastic when you want to hack it your way… but everything’s possible if you have sufficient google-fu and patience. :)

Hope you like the game!

Behind Fez : Collision and Physics

Here’s the third part of the “Behind Fez” series (part 1, part 2), in which I’ll try to explain how collision detection and physics work in Fez.

And yes, Fez is still actively developed in all areas. Making a game on your own : IT’S HARD.

Collision Engine as of Early 2008

Back when we made the IGF 2008 build, we had at least two massive limitations that made culling and collision detection very simple :

  1. The world was completely static. No moving platforms, no physics except for the player sprite, you can assume that if something is at one place at level-loading time, it’ll stay there until you quit the game. (as a matter of fact there WAS only a single level, but that’s another topic)
  2. Everything was aligned to the world grid. Everything took up a full cell worth of collision boundaries, nothing bigger or smaller than 16x16x16 trixels – exactly one trile.

The super-blocky world of Fez circa 2008

This allowed me to only calculate collision detection of the player’s collision rectangle (for each of its 4 vertices, point-to-line) whenever it traversed a world grid “line”, and since this was done very rarely, optimization was not an issue, and I went with the most intuitive and naive way possible.

Consider the world as a 3D array (or whatever indexed data structure you can think of) with filled or empty spaces, and each filled space containing visual and physical information. Visual information consists of the polygonal mesh, the textures, etc. Physical information defines if that trile should collide with the player, and from which of its 2D boundaries.
We decided early on that the three possible “collision types” are : no collision, top-only collision (for fall-through/climbable platforms) and all-sides collision (for blocking level boundaries or obstacles).

The three different collision types as seen in Fezzer

This way of mapping world entities with their collision information is elegant because the level designer doesn’t need to paint a separate collision map, or add invisible objects that act as colliders. It also means that any change you make to the level visually is propagated physically to how it plays.

Fez is obviously played from a 2D perspective. The collision results must match what the player sees, and visibility works front-to-back, with only the top-most layer being visible and active.
Knowing the collision type of each and every space (if filled), it’s easy to find the 1D “row” of possible colliders if you have the 2D screen coordinates in hand. Then you just traverse front-to-back, and the first hit is kept, at which point you can early-out from the loop.

Depth “Collision”

So now I know what’s blocking the player in 2D. But we had to make additional rules for the Z position or depth of the player, so that the game would behave like a 2D platformer AND still make sense in the 3D world :

  1. Gomez should stay visible. He should stay on-top of the world geometry as long as he doesn’t rotate the viewpoint. This is done by correcting the depth such that Gomez stands right in front of the geometry.
  2. Gomez should never walk in mid-air. In 2D this is solved by the collision detection, but in the remaining axis it needs to be enforced, such that Gomez stands on the platform nearest to the camera (this is an arbitrary rule-of-thumb that we chose).
  3. Otherwise, don’t change Gomez’s depth for no reason. The player expects it not to change. It’s really easy to get lost in Fez, and if the engine messes up the little spacial perception you’ve got left, it’s not fair anymore.

A hilarious mock-up that shows rules 1 & 2 of depth adjustment

The player will never see that Gomez moves around in the Z axis because the view is flattened and it has absolutely no depth perception, so we can do all we want to ensure that rules 1 & 2 are enforced.

Breaking the Grid (Late 2008 to Late 2009)

So that was good until we decided to implement crate physics, moving platforms, offset triles and variable trile size. Then, this happened :

  1. Every rule defined above has to be tested every time the player moves. If the triles aren’t aligned to the world grid, the “only test when a grid line is traversed” trick won’t work anymore.
  2. For both culling and collision, the world grid stops being an exact reference of how the world appears/behaves, and more of a helper structure where more than one trile can be in a cell, and some triles overlap many cells.
  3. Collision stops being specific to the player, it needs to be generalized in order to support particles and other objects that should have all the same 2D/3D tricks.

The many joys of offset and oddly shaped triles

None of these problems is trivial, but the hardest by far to implement was #2. Thing is, I didn’t want to throw away everything I did and start over. So I made small, incremental changes until the new features were supported. And just by then I had ~1.5 years worth of C# code to maintain…

To explain my final approach, I need to specify that “variable trile size” does not mean that a trile can be bigger than 16x16x16, only that its collision volume can be smaller.
With that in mind, here’s what I did :

  • A trile always has a majority of its volume within a single world cell, even if it’s oddly shaped or positioned arbitrarily. In other words, a single cell holds the center of a trile. This world cell is where it’s stored.
  • When colliding a vertex of a collision rectangle to the world, look up the 4 nearest 1D “rows” (in 2D screen space) of possible collider cells from the world grid. Traverse front-to-back each of those rows, and test if one of the triles contained at each level ACTUALLY collides with the point, taking in consideration the trile’s positional offset and size. The 4 neighbour rows need to be tested because triles within these rows may exceed the cell boundaries by up to 50%!
  • When triles move, update their location within the world grid only if the center changed to a new cell.

So everything’s covered, we’re good! Right?

Optimization

But it was slow as molasses. I do many, many more checks than I did before, and especially on the Xbox where the JIT compiler is less efficient, all those random accesses killed the game’s performance. Truly a case of CPU/Memory bottlenecking.

This section is a work-in-progress… As long as I’ll be maintaining/developing the game, I’ll worry about it going too slow. But here’s the steps I’ve taken up to now :

  • After every camera rotation, cache the nearest and farthest trile for each screen-space world grid location. This way, I don’t have to loop through the entire level boundaries and test for trile presence, I know that within these cached bounds, I have data. Parts of the cache need to be invalidated every time an object moves. The caching process of the whole level has to be done in another thread while the rotation happens, else it pauses the game for ~250ms… And threading is a headache.
  • Simplify the algorithm for particles and other small objects. The player won’t notice if particles physics aren’t 100% accurate; I can reduce the collision points to a single centered one, and ignore some rules.
  • All the standard optimization techniques… Avoid dynamic memory allocations. Ensure cache spacial locality (still struggling with this one). Start up ANTS Profiler, find a bottleneck, eliminate it, rinse, repeat.

The problem with the world being so dynamic is that I can’t precache everything. I certainly can’t precache the collision result of every pixel of the screen everytime a viewpoint rotation occurs.
Separating dynamic objects from static objects and treating their collisions separately is something I’d like to try if necessary. But it means so many changes to the current system that it scares me a little bit.

Exceptions

In Mid 2008, we decided to implement something called Big Art Objects, which are like triles but bigger than 16³ (they can go up to 128³). They are sculpted like triles, but they don’t have any collision information attached to them, because they stick out of the “world grid” system.

A tree art object, with its collision triles faded in. (T = top-only, π = no-collide)

To make them look like they’re standard world objects, we fill them with invisible collision triles. (yes, I said we wouldn’t need those, but that’s a special case :P)
It’s worth it in the end, because they look fantastic and break the mold of lego-like blocky structures.

Another common exception is what we call immaterial triles. They’re no-collide triles that ALSO don’t make Gomez go in front of them. Strands of grass can pass in front of Gomez, it just looks better that way.

I could go on about the other exceptions, but then I’d reveal features that we haven’t announced or shown. So I’ll just stop now and let your imagination do the rest. :)

Last.fm Scrobble Fetcher & Mapper

Update August 26th 2010 : New v2.1 build on google code that fixes the “Invalid XML Characters” issue! Try that one if you had errors when fetching scrobbles.

Updated June 26th 2010 : This project is now hosted on Google Code! I’m new to this so bear with me, but I’ll try to make it nice so that people can contribute.

Updated March 5th 2010 : You shouldn’t need to install both iTunes and WMP for it work, just the one you want to use! Finally.

Original March 15th 2009 post follows.

Downloads

Source & Binaries : See the google code project.
Binaries Only : scrobblemapper_v2.1_bin.zip (271 kb, .NET 3.5 SP1 Framework needed)

Description

The Last.fm Scrobble Fetcher & Mapper does exactly that. It fetches all your scrobbles from your (or anyone’s, really) Last.fm account, assembles them in terms of Play Count and Date Last Played for each music track, and then exports this data to either Windows Media Player 11 or iTunes.

I’ve been working on this application on and off for some time now and I think it’s ready for deployment. I built it because I have quite an extensive music library, I like my playlists smart and automatic, and I have poor luck with music players, hardware failure and vendor lock-in.

It seems that the “Play Count” and “Date Last Played” metadata fields are not part of a file’s ID3 tags, but rather is stored in the player’s local library. This is usually fine, and probably more efficient than writing to files all the time, but it means that if your player’s database gets corrupted (as happened to me with Windows Media Player) or that you decide/are forced to use another player such as iTunes because your iPhone doesn’t want to sync with anything else, then you’re screwed. And same thing of course if you reinstall your machine or get a new one.

I feel that this metadata is important because I like to have automatic “best-of” playlists that are based upon it, and sometimes it’s nice to listen to a comfortable, time-proven playlist.

Thankfully, if like me you’ve been using the Last.fm services since they were called Audioscrobbler, you’ve gathered an impressive amount of playback information in your account over the years. And using their web services, and my application, now you can take it back home!

Technology

I wanted this application to be my cleanest, most error-tolerant and most efficient piece of work yet in application design. I also tried to exploit all C# 3.5 features, having accumulated some months of experience with LINQ, lambda expressions, etc.

Here’s a rundown of its tech features :

  • Multi-threaded

    • Uses the Parallel Extensions for .NET June 2008 CTP and a little home-made framework for progress-reporting asynchronous foreground tasks.
    • Most operations will use multiple cores thanks to the Task Parallel Library’s capabilities, and will seldom lock because of its lockless concurrent data structures.
  • WCF-enabled

    • Uses the Windows Communication Foundation classes in .NET 3.5 to communicate with Last.fm’s RESTful API.
    • All the response data objects are deserialized automatically from XML using the built-in XmlSerializer.
  • Error tolerant

    • Since Last.fm track data will not always match your library’s perfectly, I used the Levenshtein distance string metric and “neutralized” strings (removes all accentuated characters, symbols, whitespace, etc.) to get accurate matches.
    • Will (should!) recover gracefully from errors, both caused by user input or unexpected conditions. It also allows graceful cancelation of long-running operations.

Screenshots

scrobblemapper9 scrobblemapper8scrobblemapper5 scrobblemapper3

Closing notes

- This code uses a Community Technical Preview version of the Parallel Extensions for .NET, and one that is almost a year old… So it’s delivered as-is, and you cannot (and should not) use that library in commercial products. Although I have functionally tested the program pretty extensively, I cannot guarantee it’s not going to corrupt your library or overwrite some information. Use at your own risk!

- I noticed that iTunes checks if the file is writable before setting any metadata concerning it. So you should make your files writables unless you’ll get a ton of errors in the error log after the scrobble-mapping.

- You need to keep the iTunes instance running for the iTunes mapping to work. The COM interface requires it to be open, but you can minimize it to the tray.

- The code is released under the Creative Commons Attribution-Share Alike 2.5 license.

If you encounter anything unusual, if you have ideas for extensions to this program or have comments/questions about how it works, do ask! That said I will probably blog some more about parts of this program, like the WCF usage or the asynchronous task classes. Enjoy!

Smart Screen-Space Blurring for Shadows

I’ve been working on a screen-space solution for blurring shadows that has the following features :

  • Variable-sized kernel based on surface distance and view angle
  • Rejection of samples if they lie on a non-contiguous surface, by identifying depth and normal discontinuities
  • Bilateral, two-pass blur filter to maximize the possible kernel size
  • Vertex and pixel shader 3.0, so no worry about 64 instructions in the PS

…but also works with the following constraints, to simplify matters :

  • Flat surfaces only, no curves! (to simplify the sample rejection process)
  • Full control over the rendering pipeline, so the “main render” pass can output weird values in order to be properly blurred

I quickly realized that it’s very hard or impossible to do a variable-sized Gaussian filter in real-time. If you change the kernel size, you change the standard deviation, and you need to recalculate the weights… this is too heavy for a pixel shader. So I chose to use a box filter with uniformly-spaced samples.
I totally would’ve liked to use Poisson-Disk distribution, but it’s not doable in a two-pass bilateral scenario. And you can’t achieve big kernels in real-time without separating the process in two passes.

My XNA3 implementation currently uses no additional render targets (!) but just a “resolve texture”, and resolves from the main buffer quite a lot. A R32F (Single) render target is used for the shadowmapping process itself, but otherwise everything’s done with a A8R8G8B8 (Color) main buffer.
The shadowmapping solution is standard orthographic/directional depth testing, but I’m using Exponential Shadow Maps (ESM) to simplify the depth biasing problems. I could never get my hands on a really good way to do Slope-Scale Biasing for standard depth testing, so ESM saves the day here. Otherwise, there’s no fancy cascading, splitting or projection tricks.

Here’s how it looks at different distances. The blur kernel stays approximately the same size in world-space even if the whole process is screen-space!

I’ll keep working on a clean sample to show, and I’ll definitely release the HLSL code if I can’t release the source to the whole thing. Stay tuned!

Common Kanji Character Ranges for XNA SpriteFont Rendering

Note : This sample is practically useless, because the XNA Localization sample has a much better alternative using the Content Pipeline and character detection from resource files, which works for any language (Chinese, Korean, Japanese…). But I guess if you wanted to get the ranges of common Kanji, here’s how.

While working on Japanese language support in XNA, I realized a couple of things about Japanese writing (some of which may seem obvious, but wasn’t for me) :

  • There’s two broad character sets : Kana (syllabic) and Kanji (logographic)
  • Kana has two modern subsystems or components, Hiragana and Katakana, each with two distinct Unicode regions of respectively 92 and 95 different glyphs (187 total)
  • Kanji originate from Chinese “Han” characters, and are stored within the CJK (Chinese, Japanese, Korean) portion of Unicode. But CJK characters don’t uniquely reference Japanese logographs, and it contains over 20000 glyphs!
  • There’s over 10000 actual Japanese kanji, but only about 2000 of which every high-school grade Japanese person should know

In XNA, the SpriteFont class and its associated content pipeline use bitmap fonts internally to cache and render text strings. It becomes obvious that generating a bitmap font with 20000 Han characters would take a very long time, and is also very irresponsible memory usage. Even 10000 characters seems ridiculous.
I wanted to keep using SpriteFonts, so switching to a realtime font rendering option like FreeType was out of question. So how does one make bitmap fonts usable in Japanese?

While researching the subject, I stumbled upon a whitepaper called “Unicode and Japanese Kanji” by Tony Pottier, in which he discusses how to isolate Japanese Kanji from the CJK characters, and even dresses a list of all 1946 unique characters that are learned in Japanese education up to grade 7 (sorted by Unicode point, or by learning grade). Even if it’s a large amount of glyphs, it’s a lot more reasonable than 10k.

So the only remaining step is to make this table into a a list of XML CharacterRegion elements so that we can use them in an XNA SpriteFont declaration.
I made a little C#3 program that takes a list of Kanji, one per line, and blurts out the expected XML; it also joins the succeeding characters into regions to save space.

using System;
using System.IO;

namespace KanjiFinder
{
    static class Program
    {
        static void Main()
        {
            var output = new StreamWriter("regions.xml");
            var input = File.ReadAllLines("kanjis.txt");
            var writeCount = 0;
            var intervalsCount = 0;

            var start = (int)char.Parse(input[0]);
            var end = start;
            foreach (var line in input)
            {
                var cur = (int)char.Parse(line);
                if (cur - start > 1)
                {
                    output.WriteLine(string.Format("<CharacterRegion><Start>&#x{0:X4};</Start><End>&#x{1:X4};</End></CharacterRegion>", start, end));
                    writeCount += end - start + 1;
                    start = cur;
                    intervalsCount++;
                }
                end = cur;
            }
            output.WriteLine(string.Format("<CharacterRegion><Start>&#x{0:X4};</Start><End>&#x{1:X4};</End></CharacterRegion>", start, end));
            writeCount += end - start + 1;

            output.Close();

            if (writeCount != input.Length)
                throw new InvalidOperationException();
            Console.WriteLine(intervalsCount);
        }
    }
}

Here’s its input kanjis.txt (in Unicode format), and its result is regions.xml.

I chose to go up to Grade 7, but one may choose to ignore Grade 7 characters and just do 1-6. I don’t know whether Grade 7 characters are useful in game menus and usual dialogue.

All that’s left is to put those regions in a <CharacterRegions> tag inside a .spritefont file, and supply a valid Japanese font! Thankfully, Windows 7 comes with a bundle of these (MS Gothic, Kozuka, Mieryo and Mincho) and the M+ Fonts offer a public domain alternative.

Apart from the Kanji regions list, a couple more regions you’ll probably need :

<!-- Ideographic Symbols and Punctuation -->
<CharacterRegion><Start>&#x3000;</Start><End>&#x303F;</End></CharacterRegion>
      
<!-- Hiragana -->
<CharacterRegion><Start>&#x3040;</Start><End>&#x309F;</End></CharacterRegion>   
      
<!-- Katakana -->
<CharacterRegion><Start>&#x30A0;</Start><End>&#x30FF;</End></CharacterRegion>   
      
<!-- Fullwidth Latin -->
<CharacterRegion><Start>&#xFF01;</Start><End>&#xFFE6;</End></CharacterRegion>   
<!-- and/or the standard Latin set... -->
<CharacterRegion><Start>&#32;</Start><End>&#126;</End></CharacterRegion>

That’s it! Hope it helped.

Tech talk and game programming by Renaud Bédard