So, when is the supernova going to explode?

With all the progress made on the supernova engine in ScummVM since the last post, I decided it was time to post an update. And in case you can't wait to know the answer to the question asked above, watch the video at the end of this post. But it would be a shame not to read first about the amazing work done lately ;)

Translations

Thanks to the many suggestions from JenniBee, and some help from Joefish who reviewed all the translations for which I had a doubt, the first pass of the translation for the first Mission Supernova game is now completed. And I even had time to do a second pass for the first half of the game while testing the engine, and so improve the translation with the help of the context.

Now what would help the most to get the translation completed is for native English speakers to play the first half of the game and suggest improvements on our translations web site.

We also already have a lot of suggestions for the second game, but it is a bit early for me to review those as I would like to focus on completing the engine for the first game.

As a reminder, the game is available for free on the original developer web site and the source code for the scummvm engine is on GitHub. To play the game you will also need to get the supernova.dat file that contains the game text (both German and English). I have uploaded a version on my dropbox, but you can also compile the devtools/create_supernova tool yourself (available from the same repository as the game engine) and execute it to generate the data file.

Engine

With Strangerke we made a lot of progress on the engine, getting to a point where more than half of the game is fully playable without any major issues, and this includes the supernova explosion! To get to this point, we needed to:

• Implement the dialog system. In the early game there is no dialog (you will understand why when you play the game), so the dialog system had not been implemented yet. But it was required to progress beyond the first scene where the protagonist meets a fellow creature capable of speech.
• Implement the event callback mechanism. There are a several events in the game which are on a timer, the supernova explosion being the first of those. I am not a fan of timed event, but it kind of make sense in the context of the game. And we are not going to change the game anyway ;)
• Fix the graphics pipeline. There was some minor graphic issues, so major ones, and some catastrophic ones (causing a random crash due to out of bound access when trying to access the image at index -1 in the image array).
• Fix a palette brightness glitch that caused some rooms to be entirely black. This made it quite hard to progress.

The graphics is an interesting aspect, so I will now give a bit more details about that point. It will also give me a good opportunity to add some much needed illustrations to this post. In the supernova engine, images contain multiple layers. The first layer always contain an image for the full screen (minus the inventory area, except for a few images that really are fullscreen) and then additional layers can be displayed or not for example to make animations, or depending on the context (door open or closed, object taken on not by player). Each room has an associated image and an array of boolean that indicates which layers are visible. When we render the room we thus iterates on all the layers and render the visible ones.

During animations though, we only render a layer and all those above it or "unrender them". To unrender a layer, we simply render the room starting from the first layer, for the area previously occupied by the now hidden layer.

The first graphical glitch found was due to setting the layer visibility flag in the wrong place when showing a layer, which caused the flag to be set not only for this layer, but also for all the layers above it. And we ended up with layers drawn in cases where they should not be.

What is this brown thing in the toilets?

Cleaned version.

Another minor issue with similar circumstances is that there was a logic bug that caused a single layer to be drawn in some places without redrawing layers above it.

I also introduced graphic glitches myself when rewriting the code to reduce memory usage. Initially the code had been implemented by using a surface of the full screen size for each layer. But most layers are a lot smaller and I rewrote the code to only use a surface only as big as needed for each layer. I got the surface pitch wrong however in the code bitting the background layer to screen when "unrendering" a layer. That resulted in some interesting animations...

I know he is ugly, but was that a reason to censor his face?

Also, two more hours before what?

Then more recently I stumbled upon a more serious issue that required rewriting a big chunk of the way images were rendered. The issue was that the wrong image was being rendered, and with the way the engine had initially been implemented in ScummVM there was no way to tell the rendering code to use the correct image. This caused the random crash mentioned above (when trying to render image -1), and when the crash did not occur (which happened more often than you might think), an interesting cutscene mixing layers from two images.

Flight cutscene, before and after fix

Rewriting the graphics code to fix this last bug gave me the opportunity to also easily change the way the graphics data are loaded. The initial engine implementation in ScummVM was loading all the graphics data when starting the game. Now they are loaded on demand, which should reduce considerably the memory usage (and speed up a bit engine startup on slow platforms).

Conclusion

And now, finally, enjoy the supernova explosion!

But I won't tell you when it occurs so that you have to watch the full video of me playing the game (but at least I recorded it with the English text and not the original German one, and I slightly edited it to make it a couple of minutes shorter).

Supernova - translation to English

As you may be aware I was a mentor for the ScummVM organisation for the Google Summer of Code this year. I was supervising the work by Joefish on the Mission Supernova game. The engine has not yet been merged into the main repository, but work is continuing in Joefish's GitHub fork.

At the end of the GSoC coding period the first chapter of the game was completable. However the game is only available in German. One of our aim is not only to make it possible to play this game again on all platforms supported by ScummVM, but also to play it in English. And this is what I have been focussing on in the past week.

I have actually been working on the translation for a while with the help of Joefish. We are using the ScummVM translations website to handle the translation. With previous games when working on a translation we would use more rudimentary tools, such as a shared Google Doc spreadsheet. Thus using our translations website to translate a game is an experiment, but so far I have found it a positive one. And I expect it will also allow participation from the community to complete or improve the translation (this has not been the case yet as we did not advertise this translation until now).

What I was focussing on this week is writing a tool to generate an engine data file, as we do with other engines. Until a few days ago all the text was hardcoded in the engine. But the idea is to have the text in this engine data file instead and to have the engine load all the strings when it starts. This has two main advantages:

• The strings being into a separate file instead of being in the engine, the ScummVM executable ends up being smaller and using less memory when the engine is not running (for example when playing another game).
• We can store the strings in several languages in this engine data file and the engine will load the strings for the language selected by the user when the game was added to ScummVM. This means we can provide translations for the game.

Format of the engine data file

The data file starts with a 3 byte identifier ('MSN') that can be used to identify the file, and a 1 byte version number so that we can change the format in the future without breaking everything.

The rest of the file is a collection of blocks, with each block having a 12 byte header and a variable size data section:
 byte 1-4   marker, for example 'TEXT' for the game text
 byte 5-8   language code, for example 'de' for German
 byte 9-12  block data size in byte
 byte 13-N  block data

This file structure means that we can easily add other types of data in the future if needed, such as the font data (that is currently hardcoded in the engine) or the mouse cursors (also currently hardcoded in the engine). The marker tells us what type of data we will find in a block. Currently the markers used are 'TEXT' (game strings), 'IMG1' and 'IMG2' (for newspaper bitmaps).

With this file format we can also easily add translations to more languages.  Currently we have data for German and English.

Generating the engine data file

The German text is hardcoded in the tool that generates the engine data file. All other data however are provided with supporting files. The tool has a list of languages for which it will look for those supporting files. Currently for each language we can have 3 files:

• strings-xx.po
• img1-xx.pbm
• img2-xx.pbm

(where 'xx' is a language code)

For each file found a corresponding data block will be added in the engine data file.

The strings-xx.po file is typically generated and updated by our translations web site and contains the translation in a given language for each German string. The German strings are defined in a specific order, and when generating the TEXT block for another language the tool iterates on the German strings, for each string looks for a translation in the po file for this language, and if found write this translation and otherwise write the original German string. This means that untranslated text will appear in German when playing.

The ppm files are bitmap images in portable bitmap format for the two newspaper article images in the game. They can for example be generated with gimp. Other than the header, the ppm format happens to store the data in exactly the same format as the game does for the original bitmaps in German. So the tool simply reads the header to do some sanity checks on  the format and image size and write the rest of the file to the engine data file.

So adding a new language simply means adding its code to the language array in the tool source code, and dropping some pbm and po files in the directory where we execute the tool.

I didn't paste any source code in this post, but you can see the source code for the tool here: https://github.com/Joefish/scummvm/tree/supernova/devtools/create_supernova

Status

The game contains 655 strings. Currently 281 of those have been moved to the engine data file, and this includes the name and description for all the objects. A lot of strings are still hardcoded in the game engine though, such as all the dialogs, and they will be moved to the engine data file as well eventually.

The progress on the translation to English is similar, as 279 strings are currently translated, although some of those are for strings that have not yet been moved to the engine data file, and can thus not yet been seen translated in the game.

Mission Supernova - A look at the music

Earlier this month we announced two projects for this year Google Summer of Code to add support for the Sludge engine and for the Mission Supernova games in ScummVM. I am a co-mentor for the Mission Supernova project (the other mentor being Strangerke). We were lucky enough to be provided with the original source code for Mission Supernova (for which we have to thank the rights owner). With the coding period for GSoC starting officially tomorrow we spent the last month looking at this original source code. Interestingly, in addition to the source code for the game we were also given the source code for some tools. One of those converts a MOD music file to a game data file. And I though it would be interesting as a side project to reimplement it so that it works on modern computers, and to then extend it to perform the reverse conversion from game file to the original MOD file (which we don't have).

I thus spent a few days working on this in the past two weeks.

We have been asked not to share the original source code (and anyway you would have to be a bit of a masochist if you want to see C code from over 20 years ago). But I will show a small extract to give you an idea of the work involved. The original source code is in C and for DOS.

The source code for the mod conversion tool is very compact and starts with these two functions:


For anyone familiar with supporting both big endian and little endian platforms, what they do is obvious. They swap bytes to convert between big endian and little endian representations.

If you are wondering what big and little endians are, go read my first post in this blog on adding support for the mac version of Broken Sword (you can also read the third post in that series about fixing speech for some mac versions of Broken Sword).

Conversion between big endian and little endian should come as no surprise. The MOD format was originally developed for Amiga, which are (or at least were at the time) big endians computers. Looking at the specifications of the MOD format shows that it is indeed using the big endian convention. On the other hand the DOS operating system was working on little endian computers. Using a little endian format for the music in Mission Supernova thus made sense to avoid having to swap bytes during runtime. Every little helped at the time to get good performances...

Another thing visible in the code above and that should come as no surprise (at least for developers dealing with old platforms) is that an unsigned int is coded on two bytes (and not on 4 bytes as you would expect nowadays) and a long int uses 4 bytes.

The last point we can note is that functions and variables have German names. Fortunately for me I did study German at school and could understand most of the code straightaway without having to ask Google translate (or my German sister in law) for help.

The first step of my work was to rewrite the code of that tools so that it works on modern computers, whether they are using big endian or little endian conventions, and can be understandable by others.

• I replaced the byte swap function from the original source code with code we already have in ScummVM that handles byte swapping depending on the platform on which the code is run (so that for example reading a MOD file would only swap bytes when the code is run on a little endian computer).
• I replaced data types such as unsigned and long with types provided by ScummVM such as uint16 and int32.
• I rewrote the code to use ScummVM Common::File API instead of the low level DOS file access code.
• I translated variable and function names to English.
• I objectified the code a bit adding a ModReader class.

At this point, without the original MOD file, I had no way to know if the code I wrote was correct. Writing this code however helped me understand the differences between the MOD format and the format used by the Mission Supernova game.

The two formats are very similar, but besides the different endianness, there are a few other differences. Actually the format for the two parts of Mission Supernova  is slightly different.

Here is a description of the MOD file header:

And one of the Mission Supernova part 1 data file header:

For the Mission Supernova part 2, there are only 15 instruments stored and not 22.

Note how some information is missing in the Mission Supernova data file. That means that we have to guess what that information should be when converting that data file back to a MOD file. Fortunately none of that missing information is really important. For example for the song name I just decided to use the name of the MOD file that was hardcoded in the original source code.

Some other information is just formatted in a different way, such as the Mission Supernova instruments data having a loop start and loop end instead of a loop start and loop length.

Also the Mission Supernova data file stores explicitly the number of patterns and the offsets of the samples data. Those have to be computed from other informations in the MOD format.

The other difference not seen above between the two formats is in the pattern data. Both are using 32 bit values, but they are not coded in exactly the same way. For details on the differences just look at the source code and comments in the rewritten tool.

This knowledge of the MSN music data file might be useful when we have to work on supporting the music in the game engine reimplementation. For now I used it to write some code to do the conversion the other way around: from the game data file to a MOD file.

This allowed me to check that the code is correct:

• By checking that the converted MOD file I am getting is played correctly in a player supporting that format.
• By doing a round trip conversion: converting from MSN data file to MOD and then back to MSN data and checking that I get back the original file.

My first round trip test actually resulted in the original and converted MSN data file having a one byte difference (every bytes were identical except one). The offset of that bytes indicated it was the second byte of the order list length value, coded on two bytes in the Mission Supernova format. And then I realised that  I was using a char variable  (that uses one byte) since in the MOD format the order list length is coded on one byte. Writing that variable on two bytes meant the second byte was garbage.

The final source code is available at https://github.com/criezy/scummvm-tools/tree/supernova/engines/supernova. At some point I might merge it in the main ScummVM repository.

Implementing this reverse conversion also allowed me to listen to the music without waiting for the games to be supported in ScummVM. And to let you enjoyed that music as well, here are recordings for the music of the first and second parts of Mission Supernova converted to MOD and played back in an OpenSource ProTracker clone.

Mission Supernova part 1 music

Mission Supernova part 2 music