Big chiptune code project

[vampirefrog]

Hi guys. Just thought I'd write down my thoughts on this. I have in mind a project that would handle many needs of the vgm community - players, input plug-ins (winamp, fb2k), web players and so on. The main idea is that, the chip emulation code being the biggest chunk of code, if we write once, in one language, then by way of automated tools convert the code to other languages (JS, AS, Java etc.), it would make the project very versatile. The code could be reused over and over, in as many projects as you can think. For example, for the web-based VGM player, we just have to compile to JS, then write the web-specific code that would use the chip code.

The second biggest chunk of code would be the input file code. Code to read not just VGM, but many other formats (which is the reason why I've been keeping a list of common formats for chiptunes). Once we have that, we'll be able to play any file in any output medium, including the browser. That means that we can make mirrors of existing sites (which is why I've been keeping a list of sites with large music libraries), and just allow browsing of those mirrors with a browser player, so people can just browse our site forever - pick a chosen or random game and just click play, possibly minimize the browser, and rock out to some chiptunes. Another scenario would be an application (desktop or mobile) that would interface with the site and allow the same actions (browsing and listening) so you can run it in the background or the system tray or something. Furthermore, if people want to have the collection (which will get quite large once we import other formats) locally, and play it, they would have a single plugin they need to install, for winamp or fb2k (or other players), which would support all of the formats and all of the chips.

I think this sounds like an awesome project, but it requires a lot of work. Firstly, we need to figure out what language we'll write all the code in. I've been searching for a tool or language for a couple of months now and I haven't found anything satisfactory. There is haxe, ci, and others. I'm not satisfied with any of them, so I figured I could write my own, using the classic compiler tools flex and bison. These have been used for decades to write compilers, and there's lots of info, however, it's still difficult and I'm still struggling with it. I am trying to keep in mind practicality, so the search here continues.

Once we've settled on a language, we'll have to begin porting all the chip code to that language. Hopefully it will be object oriented and that will make things easier. There are dozens of chips to rewrite, and usually rewriting involves a cleanup as well, so the final code will be nice and clean. Each chip will extend a parent class, which will be easily reusable.

After that, we'll need to write support for the various file formats, starting with VGM. We can order these depending on the sizes of the libraries we find. Some formats require CPU emulation, but I think we can avoid that for starters. We can write that later, to have complete support for all important formats.

So, once we have both of those big code chunks, we can focus on the different platforms the code will run. Of course, for testing, we'll need to test the code when we write it, so we'll probably have the C++ and JS targets working by this step, but this is when we start completing the list of targets. Like Java for Android, for example (although I think you can use C++ on Android too).

At some point we'll have the player implemented on our site, being able to play all the files, or most of them, so we'll have that fun feature which will bring in users, and possibly contributors. I consider it an exciting project, so the extra contributors might be a side benefit. At least we'll have bug reporters, if nothing more.

So, by this point we have a pretty big project, a pretty nice website, and hopefully a lot of contributors. You may ask - why are we doing all this? It takes a lot of time to code this, and nobody's paying us. Should we monetize this somehow? Well, to tell you the truth, I'm not against monetization, and some income might pay for a better server, but that is far from now, so let's worry about this later. Personally, I hate ads, so we'll need some sort of sponsorship plan. Not sure who would sponsor this, seeing as there might be legal issues, after all, this music is extracted from commercial games. This remains to be seen. Maybe someone can do some research? The question is - how can we monetize a website dedicated to chiptunes - extracting, storing, playing, listening a large number of old video game songs. A very enjoyable site.

[vampirefrog]

So, for now I'm still figuring out what language to use to write all the chip code.

[vampirefrog]

Just an update for those interested - for the past weeks I've been trying to figure out how flex and bison work. They are tools specifically written for writing parsers. The parser I'm writing is a C++ parser, so I can parse C++ files and generate JS and AS files later on. I've finally started to understand how bison works (flex is easy compared to bison), and I've begun writing the parser, and it now parses the basic stuff like class declarations and whatnot.

That's all fine and dandy, but the issue is now how do I translate some of the C++ features to JS, such as pointers and whatnot. It should be possible, but eventually I might have to choose another language as the source language, simply because of the incompatibilities between C++ and JS.

So now, I think, is time for some research into what language would be best for the source language, and this is done by taking some code from the chip cores, for example ym2151.c, and converting that to C++, then to JS, then to AS, and extract some conclusions from the experience.

[vampirefrog]

Update: I've started a github project here https://github.com/vampirefrog/chiptune

If you take a look in the cpp folder, you'll see a few C++ cores, like the YM2151, M6258, (Hu)C6280, PSG and QSound. I'm still working on the vgm player (vgmplay.cpp), but soon it will generate wav files from VGM, and that will be good for testing the cores.

I've also found a C++ lexer/parser project by Ed Willink (http://www.edwillink.plus.com/projects/fog/), which I am trying to use to generate the parse tree. It contains the lexer + the bison parser, and some basic classes defined, so all I have to do is plug my code into those classes. I'm still trying to figure out how to do that.

[vampirefrog]

Updates:

In the meanwhile, I've decided to use clang to parse the C++. It's quite good, and so far I'm generating some JS and AS3 output, but still incomplete. For a complete translation, the code has to be iterated more than once, with a few analysis passes and an output pass. It's quite involved.

Also, I've discovered that there is a C++ to SWF compiler called FlasCC or CrossBridge, so I might end up using that if I lose my patience with the translator code, and emscripten for the JS part.

Also, as far as the C++ code is concerned, I have a VGM to wav "player" which supports HuC6280, OKI M6258, OKI M6295, PSG, QSound, SN76489 and YM2151.
Those chips are written in C++, and they should be fairly easy to translate to AS3 by hand. I'm saying this because I have a flash-based SN76489 player in mind for smspower.

[denisk]

Hi!
Your ideas seem to be very interesting, but probably a bit too ambitious. Parsing one programming language into another one is hard. I have browsed through your project and seems that you have already outlined several drawbacks of porting C++ to javascript/actionscript - pointers and memory management. There probably will be much more than that. Data types will be a big problem I think... And keep in mind that you need different javascript for different web browsers sometimes. Google has been working many years on GWT, which is essentially java->javascript parsing engine. And it still has bugs...
If your goal is basically to let more people use VGM, which I believe is the case, then why not re-use what's already there. VolleyBell's VGMPlay seems very interesting, though it refuses to compile on Ubuntu (haven't tried it on Mac yet). There is a plugin for audiacious: https://github.com/audacious-media-play ... readme.txt
If you really want to create something new, why not create in in Java? Java runs everywhere and as a bonus you'll get a codebase for Android (However, I would probably invest time in making VGMPlay cross-platform). I don't really advocate for Java (although being Java developer for like 7 years).
Actually I don't listen music in the browser, I (and, probably many people too) listen it in my favorite player (which is Amarok for Linux and Vox for Mac). But they're unable to play VGMs. I would like to have a queue of VGMs and MP3 in a single player. So I'd like to create a plugin for Amarok for playing VGMs (there as an opened defect for that: https://bugs.kde.org/show_bug.cgi?id=167626 ). (Amarok itself needs to be adjusted to be able to run on Mac, but that's a different story...)

Thanks,
Denis

[vampirefrog]

Hi!
Your ideas seem to be very interesting, but probably a bit too ambitious. Parsing one programming language into another one is hard. I have browsed through your project and seems that you have already outlined several drawbacks of porting C++ to javascript/actionscript - pointers and memory management. There probably will be much more than that. Data types will be a big problem I think... And keep in mind that you need different javascript for different web browsers sometimes. Google has been working many years on GWT, which is essentially java->javascript parsing engine. And it still has bugs...

It already works, just gotta iron out some bugs. Don't worry about it.

VolleyBell
VolleyBell
VolleyBell

Don't you mean VolleyBall?

If your goal is basically to let more people use VGM, which I believe is the case, then why not re-use what's already there. VolleyBell's VGMPlay seems very interesting, though it refuses to compile on Ubuntu (haven't tried it on Mac yet). There is a plugin for audiacious: https://github.com/audacious-media-play ... readme.txt

My goal is to make the libraries of game music as easily available as possible.
There's also a gstreamer plugin that uses libgme to play VGMs up to 1.50, I think.

If you really want to create something new, why not create in in Java? Java runs everywhere and as a bonus you'll get a codebase for Android (However, I would probably invest time in making VGMPlay cross-platform). I don't really advocate for Java (although being Java developer for like 7 years).

No Java, please. Personal preference.

Actually I don't listen music in the browser, I (and, probably many people too) listen it in my favorite player (which is Amarok for Linux and Vox for Mac). But they're unable to play VGMs. I would like to have a queue of VGMs and MP3 in a single player. So I'd like to create a plugin for Amarok for playing VGMs (there as an opened defect for that: https://bugs.kde.org/show_bug.cgi?id=167626 ). (Amarok itself needs to be adjusted to be able to run on Mac, but that's a different story...)

Sure, many people use the packs pages to just preview packs they're going to download. Personally, I use it to browse using the metadata, such as composer name and so on. I could also make it into a web service later on, and make some sort of desktop app that just uses that web service, or perhaps a player plugin.

Thanks,
Denis

Thank you.

[denisk]

If it works then it's great, really curious to see where it will get to.
I would suggest adding some sort of roadmap to the github repository so that it is easier for the folks to to see where do you want to bring the project to. A list of issues/tasks would also help to bring some community efforts.
Keep it going!

[MaliceX]

If you really want to create something new, why not create in in Java? Java runs everywhere and as a bonus you'll get a codebase for Android (However, I would probably invest time in making VGMPlay cross-platform). I don't really advocate for Java (although being Java developer for like 7 years).

No Java, please. Personal preference.

Already been done

+1 for cross-platform.

[neologix]

In all fairness, all the C cores should have been modernized at least to C++03 years ago but likely weren't due to laziness. Vampi, please continue your modernization efforts in C++ first, as that will benefit all future desktop players and even some mobile ones and makes long-term maintenance way easier than it currently is.

[vampirefrog]

Well, as far as my endeavors go, I'm going to focus on some more practical solutions for the 'webmaster' part of the deal, leaving ValleyBell to do the VGM emulation core stuff. That is why I'd like to just get a flash-based VGM player up and running (not long now), and try to convince VB to start moving the chip cores to C++. Most chip cores are simple, except for the YM stuff, which is kind of complex.

Also, as webmaster, once I create a proper interface to upload, approve and edit packs, VB's job will be easier, perhaps leaving more time for him.

[neologix]

Most chip cores are simple, except for the YM stuff, which is kind of complex.

Don't I know it

[vampirefrog]

Just an update: I am now focusing on cxx2as3. Only a few things left to fix, most importantly the ways pointers are handled. This includes some code analysis in the converter tool, and that's the hardest part, but definitely possible.

I've created an issue list here: https://github.com/vampirefrog/chiptune/issues

In the meantime I've also added some features, like per-class file output - one file is created per class, named after the class. There is also a package name setting. This integrates nicely with the flex package structure.

Also, I've found a way to make the resulting swf file much smaller, first try was 600kb -> 9kb. It eliminates the preloader and probably a bunch of other stuff that's not needed.

I plan on adding some graphical display to the swf file too, a spectrum analyzer and maybe a piano-roll effect, if the chip code can figure out which notes it's playing.

[vampirefrog]

Update: I've had the idea of using the flash object only for the sound callback, which means the rest of the project can be written in JS. I've inspected some "audio sink" type projects, such as XAudioJS, and they seem to have the same idea - use WebAudio where possible, or use a flash fallback, which itself uses ExternalInterface to communicate with JS. There are other options as well, which a smart sink-type backend knows how to use.

I'd use XAudioJS, but it's poorly implemented. The inactive tab bug that you can see in neologix' VGM player is due to XAudioJS.

So what I'd like to do is simply implement my own audio sink thing, and continue to work on cxx2js.

As far as cxx2js goes, I'd like to support regular functions, which means it will be possible to translate the vgmplay nearly directly.

Also, I'd like to retry with emscripten, to see what results we can get.

To tell you the truth, I don't see the point in fiddling around with stuff that isn't 100% necessary. The point is to get VGM playing in the browser, and I'd like to take the most practical path there.