NuVGM file format, to fix VGM problems

[LTVA1]

Hi, I have been working on a new format specification which aims to fix the issues of VGM format, like limited number of chips supported, lack of SID support, lack of more chip-specific settings etc. The spec is not finished, however, it was made with some feedback from ValleyBell himself.

The main thing is to get rid of "opcode-per-chip" structure and put chip signature after the opcode. It makes file size bigger, but allows for virtually unlimited number of chips and their types (see spec, at this point 65536 chip types are possible). To reduce file size, there are special shortened opcodes for first eight chips (almost nobody would put in more than 8 anyway, even in Furnace). If only shortened opcodes are used, the file size would be almost the same with VGM.

PCM ("DAC") streams support loop points. In chip settings you can declare RAM/ROM units for each chip and specify their size. You can write to ROM as you can in RAM (so you can initialize only the parts of ROM that have actual sample data and not store the entire image).

Format supports GD3 (maybe it will be changed to Vorbis) tags.

Some parts of the spec are yet to be polished, and if there would be any interest from the community, I will try to make a real player and logger for this format, e.g. for Sega Genesis and C64. Furnace support is also planned at some point.

Here is the spec: https://github.com/LTVA1/NuVGM

P.S. I know I am not the first one who suggests a new format, and I know that it would be impossible to entirely move to new format, because there are many, many tools that are made for or use VGM format. However, if the format is thoroughly planned and is flexible enough, there wouldn't be any need to abandon it at some point. And, if whatever, NuVGM and VGM can be converted into each other (NuVGM->VGM would work only if NuVGM fits into current VGM limitations, of course).

[Kaminari]

Very interesting and good luck to you. The previous effort about VGM 2.0 seems to have died many years ago.

I'd love to finally see support for SID and MSM5205.

[LTVA1]

Yeah, I want it to support all the known chips, with the ability to specify preferred emulator in player or in file (and the player would have preferences to allow to either choose the emulator according to file data or the player's own settings)

[GTheGuardian]

It would be nice to feature some generic way of handling banking, if only to deal with cases like MSM6295 and it's hundreds of games using banking.

While on topic of panning, It might be a good idea to add options for channel specific panning (and ability to change it on the fly). This is mostly needed for games like Darius and Xexex.

DSP support should also be considered, just so we can handle things like TMS57002 (Needed for Konami GX and also eventual ZSG-2 support).

[LTVA1]

You can either declare one long ROM/RAM unit that has the size of all the banks combined or use the 0x02 opcode which I made just now to support banking.

Channel panning yes, if you mean output channels (like ES5506 has the damn 12 output channels!). I will think how it can be handled.

Wdym by DSP support? If it's DSP effect executed by the player and not by the chip emulator, then most probably no. If DSP effect is executed by the sound chip emulator itself, then it comes down to writing some data to some registers, as far as I know. I don't know what TMS57002 and two other things you mentioned are...

[Soundshock]

Why not just define the chip idents in the header and then use 0, 1, 2, etc rather than using hardcoded chip identifiers in the first place. That'd support 255 minus some reserved codes for waits and data blocks and multiple banks without bloating the filesizes (which are already large enough to be problematic in vintage players as-is)

[LTVA1]

Okay, lemme explain why the 16-bit chips ID are used only in the header...

Well, I need to make it clearer in the spec probably. You use chip indices to reference chips. For example, you have 3 chips with signatures 0x0050, 0x0032, 0x0067. These signatures indicate that they are three different chips because signature determines chip type. The chip with signature 0x0050 is referenced by index 0x0 because it's the first in the list. The chip with signature 0x0032 is referenced by index 0x1 because it's second in the list, etc. You have shortened opcodes for the first 8 chips so file size stays almost identical with vgm here.

[LTVA1]

By the way, I may add subroutines at some later point. This requires

Code: Select all

jsr

and

Code: Select all

rts

opcodes. Subroutines would be useful, obviously, when you have repeating sequence of opcodes with the same operands, e.g. when you load FM patch into registers. Of course, using these is justified only when they really reduce file size. However, if file size reduction is achieved, we benefit from having these. But I don't really know if it would increase real hardware player CPU load.

Generally speaking about real hardware playback, I think the file may be specially prepared for it. For example, the entire file header block and, of course, sample ROM data entries may be removed because they are not usually needed for hardware playback. This can be done at the point of making the final ROM image which is later executed on hardware, or you can produce a condensed NuVGM file which is loaded by the player from floppy disk or whatever you have on your real hardware machine. The player would just go to the start of the actual data then, on some systems parsing the PCM streams / RAM writes sections before.

[blitzlunar]

I think this is a good idea. Maybe not necessary to reinvent the wheel for the thousands of sets already ripped, but a nice fallback option for games not supported by VGM, or for homebrew scene music.

Format supports GD3 (maybe it will be changed to Vorbis) tags and multiple loop points inside the file.

I've been using the External Tags component in foobar2000 to sidestep tag limitations of game music formats for a number of years, as well as to reduce pressure on my backup strategy, so perhaps "the horse has bolted" at least as far as I am concerned, but I do welcome a more robust tagging solution - it was certainly a bugbear prior to External Tags that I couldn't write track/disc/genre/album artist/replaygain etc. to VGM.

GD3/Vorbis also allow for arbitrary fields, if I'm not mistaken?

[LTVA1]

To be honest, I don't know much about tags standards flexibility too (but looking just now - Vorbis seems to support arbitrary fields), so I will just choose the most flexible system out there, if it ever comes to real player, logger and other format uses. Tags would be placed in their own block or just at the very end of the file.

[GTheGuardian]

By DSP support I was talking about DSP chips (Like the TMS57002). The reason I'm mostly requesting it is because of Konami GX games (which we already have some packs from), which used said TMS57002 DSP for echo/reverb effects. (Said effects are currently missing in the VGMs from those games, and is the reason I avoid logging GX games, as well as games using GX's sound hardware, like Crypt Killer)

DSP support will be also needed for Taito games using the ES5505 (which also used ES5510 DSP) and if also one were to add the Zoom ZSG-2 sound chip (Used in Taito FX-1B and G-Net boards) as that was also paired with the TMS57002 DSP.

(Of course, I imagine this will be tough to implement as one will need a way to route the sound chips' outputs to the DSPs, not to mention also the hurdles of implementing CPU emulation in players).

[LTVA1]

If DSP thing is controlled with register writes as chips are or has some fixed settings it may be possible actually

[(Jazz) Jackalope]

I'm not sure if it is, but is https://vgmrips.net/wiki/ChipTuneSoundFormat useful to the discussion? There were never any implementations, and I'm not equipped to judge if there's transferable ideas in the spec, but just in case, I wanted to bring it up.

[LTVA1]

This chiptune sound format you posted a link to is kinda less generic. It seems closer to Furnace tracker hardware playback, I think. It specifies things like macros and envelopes which aren't always present in generic command stream (which e.g. comes from DAW or MML editor). That's why I try to keep my format as generic as possible, while also making it sort of real hardware playback friendly (as it is the case with VGM). The only abstractions I allow are generic subroutines that can be nested and PCM streams (for e.g. YM2612 sample playback, as in VGM). Subroutines could be used when you have repeating initialization (e.g. FM patch/some other "tracker instrument" type writes which repeat over and over). The same can be applied to tracker patterns if the register writes happening during each pattern repitition are exactly the same.

[vampirefrog]

If all the vgm tools used libvgm, and if libvgm had all the capabilities to support the tools, then adding support for a new logging format would be trivial, and all (or most) of the existing tools and players would work with the new format automatically.