32X Progress

[Jaguarandine]

Nice explanation of the 32X's abilities, Notaz. Bus sharing seems to be a huge limitation, as was memory which must've been very cost-prohibitive back then. What you're saying about VR/VF as pushing the 32X makes sense. I'd put Afterburner Complete in there too because it's the only game to use almost all 32X PWM for sound, not including all the scaling and rotation effects.

I wonder if the 32X was made to be paired with the Sega CD, as doing so seems to fix a lot of its shortcomings. Plus with a RAM cart, that would plug up some memory issues as well. I'm going off-topic though, this subject is probably more suited for the Sega-16 forums

Playing Phantasy Star 1 on SMS with FM sound would be great. Sounds like another good step for Picodrive.

 

[craigix]

I always wished they had just put the scaling hardware from the system16 arcade machines in to the original MD.

I bet they seriously considered doing that, but they wanted to keep the cost down, if they had put it in I doubt there would have been as much need to make the SegaCD, 32X etc. to try to keep up with other consoles, and because of the scaling hardware it would have saved cartridge memory and saved Sega money in the long run.

 

[shinneri]

This is very exciting news! Playing a portable version of Kolibri would be wonderful. Thanks and good luck, notaz!

 

[Tag5]

This is very exciting news! Playing a portable version of Kolibri would be wonderful.

Thanks and good luck, notaz!

I 2nd that!

 

[namco]

I wonder if the 32X was made to be paired with the Sega CD

Funny you should say that as there were games that specifically required you to have the 32x and sega cd. IIRC they were called Mega-CD 32X.

From wikipedia:

In addition to regular cartridge-based 32X games, there were also a very small number of CD-ROM games for the 32X. These games were labeled with Sega Mega-CD 32X (Sega CD 32X in North America). As the name suggests, these required both the 32X and Mega-CD/Sega CD add-ons. The lack of a significant userbase due to the high cost of purchasing all three necessary components saw only five games released, only one of those developed by Sega.[4] The most notable of these was a new version of the infamous Night Trap with 32,768 onscreen colors instead of the 64 found on the regular Mega-CD/Sega CD version.

Those games were:
Corpse Killer

Fahrenheight

Night Trap

Supreme Warrior

and

Slam City

 

[slaanesh]

One of them is slow framebuffer access (3 cycles), which makes it impossible to even clear whole framebuffer in one NTSC frame.

I don't understand why Sega would cripple the machine to this degree? This seems like a ludicrous limitation.
I suppose most 3D type games ran at less than 60FPS - more like 10-15FPS?

 

[Exophase]

One of them is slow framebuffer access (3 cycles), which makes it impossible to even clear whole framebuffer in one NTSC frame.

I don't understand why Sega would cripple the machine to this degree? This seems like a ludicrous limitation.
I suppose most 3D type games ran at less than 60FPS - more like 10-15FPS?

I don't think that making hundreds of KB of RAM that ran without wait states would have been that feasible back in 1994, or at least it would have been prohibitively expensive. It's probably DRAM, not SRAM, and the latency will have only went so low on processes back then. If this weren't the case then there wouldn't have been any reason to need cache. Having some kind of writebuffer would have done a good deal to help the situation. Then the latency would be moot because it's not like you can render a meaningful 3D pixel in under 3 cycles - unless it's flat shaded, but then you'd may as well just use the RLE capabilities of the framebuffer.

Very few PS1 and even N64 3D games were 60fps, most were 30fps or under. It wouldn't be until Dreamcast where it'd become more of the norm.

 

[notaz]

I suppose most 3D type games ran at less than 60FPS - more like 10-15FPS?

VR seems to be doing 15-20 and VF 15-30 depending on the scene.

Having some kind of writebuffer would have done a good deal to help the situation.

There is one 4 word FIFO, writing to it continuously while it's not full causes 3 wait states, when it's full it becomes 5, occasional write is only 1 wait state.

 

[Exophase]

There is one 4 word FIFO, writing to it continuously while it's not full causes 3 wait states, when it's full it becomes 5, occasional write is only 1 wait state.

A 4-word FIFO that's not zero-wait state? Yeah, that's just bad design. Under what conditions is it 1 wait state?

 

[notaz]

There is one 4 word FIFO, writing to it continuously while it's not full causes 3 wait states, when it's full it becomes 5, occasional write is only 1 wait state.

A 4-word FIFO that's not zero-wait state? Yeah, that's just bad design. Under what conditions is it 1 wait state?

If FIFO is not full and last write was more that 3 cycles ago. 32x hardware register access is slow, in some cases it's not fast enough to service 68k i/o and may halt it.

There is also a question what exactly is "more 32bit" on 32x then on base MegaDrive unit - all buses are 16 bit (SH2 upper address pins are not connected), and both 68k and SH2s have 32bit registers, so..

 

[Exophase]

If FIFO is not full and last write was more that 3 cycles ago. 32x hardware register access is slow, in some cases it's not fast enough to service 68k i/o and may halt it.

That's awful. It's pretty obvious what the problem is, they should have made the FIFO and probably other I/O regs internal to the chip.

There is also a question what exactly is "more 32bit" on 32x then on base MegaDrive unit - all buses are 16 bit (SH2 upper address pins are not connected), and both 68k and SH2s have 32bit registers, so..

Let's face it, bit ratings are a bunch of crap a good deal of the time and were marketing fodder. If Genesis is 16bit based on bus width alone then SNES can fairly be considered 8bit. But to answer your question, one thing that's different is that SH2 has a genuine 32bit ALU that can do 32bit operations in one cycle. Also, the cache is 32bit, which actually makes a substantial difference since more of the "normal" memory accesses will hit the cache instead of the external bus. This is sort of nullified by the fact that SH2 shares cache with code and data, and has to rely on overfetching 2 instruction words at a time to give the data accesses slots, but it's still 32bits wide.

 

[notaz]

Got some basic bock generating code working with most opcode handlers calling back the interpreter. The good news is that many games work without self-modifying code handling of any kind, however some don't (VR and VF are among them). I hope they're just swapping code modules, not doing something nasty. Not sure what strategy to use for this, probably will hook special memhandlers for regions that have code compiled from them.

 

[Ravnos]

Kind of off-topic, but is there a possibility of an x86 Linux release when you've finished adding the 32X support and whatnot? It would make a great addition to an HTPC setup, and the existing Linux Genesis emulators lack in features by comparison.

 

[notaz]

Kind of off-topic, but is there a possibility of an x86 Linux release when you've finished adding the 32X support and whatnot? It would make a great addition to an HTPC setup, and the existing Linux Genesis emulators lack in features by comparison.

Fusion has recently been released for Linux, should have everything you need.
http://www.eidolons-inn.net/tiki-index.php?page=Kega

 

[Rivroner]

Hi Notaz, do you think is possible to get almost fullspeed in 32x games in Wiz at 800 mhz?

Thank you.

 

[Ravnos]

Kind of off-topic, but is there a possibility of an x86 Linux release when you've finished adding the 32X support and whatnot? It would make a great addition to an HTPC setup, and the existing Linux Genesis emulators lack in features by comparison.

Fusion has recently been released for Linux, should have everything you need.
http://www.eidolons-...x.php?page=Kega

Gadzooks! I didn't even know there was a Linux version of Fusion! I stopped paying attention to it back in the day when it was Windows only. Thanks!

 

[Exophase]

Hey notaz, what kind of register allocation are you going to use? I'm curious to see if static or combined static/dynamic is chosen over full dynamic per-block in cases where the number of emulated registers isn't that much more than the number of available registers. Or if you'll be going for something more ambitious altogether.

About self modifying code strategy: what's the granularity on your memory handler lookup? Are you going to have some kind of fallback strategy for false-positives like Ari64 does? Actually, for that matter, what is your strategy for memory to begin with? Conventionally people like to use function LUTs, but I prefer pointer LUTs, especially since on platforms like ARM9 function calls and returns have a fixed multi-cycle cost. Since the RAM is only 256KB you should probably fall back on a per-halfword table of code information for it.

 

[notaz]

Hey notaz, what kind of register allocation are you going to use?

Thinking about making it static to keep it all simple. Not going for some crazy 8-pass solution like Ari64 has. The most likely target is Pandora anyway, which shouldn't need something super efficient.

About self modifying code strategy: what's the granularity on your memory handler lookup? Are you going to have some kind of fallback strategy for false-positives like Ari64 does? Actually, for that matter, what is your strategy for memory to begin with?

The memory map is a bit unfortunate:

00000000 - 000007ff - BIOS
00004000 - 000043ff - system regs, VDP, palette
02000000 - 023fffff - ROM
04000000 - 0401ffff - framebuffer
04020000 - 0403ffff - framebuffer, overwrite mode
06000000 - 0603ffff - SDRAM
c0000000 - c0000fff - SH2 on-chip RAM, can be used if cache disabled
fffffe00 - ffffffff - SH2 on-chip registers

There is also lots of mirroring going on (checked on the real hardware), but I haven't seen any games relying on that. For now I'm using MS 7 bits and function LUTs (actually just implemented that yesterday), should be trivial to convert to pointer LUTs later. Functions are doing masking as needed.

Since the RAM is only 256KB you should probably fall back on a per-halfword table of code information for it.

Not sure what do you mean here? 128kB array of flags for every halfword?

Today I've added an array of flags which tell if any particular 4kB chunk of RAM has code compiled from it, and check for them in write handler. That allows RAM changing games to work but I get lots of false hits, obviously 4kB chunks are too large (right now it runs through all block descriptors to verify if it really hit a block).

That on-chip RAM functionality is annoying, Virtua Racing is rewriting it every 1-2 frames and running code from there, which means I will have to keep recompiling it. Actually this is what SVP version does too, but since it was a recompiler for a single game I was able to enumerate all IRAM states and avoid constant recompiling. This ain't going to work here as VF also seems to do the same (haven't properly verified though, pretty tired already).

 

[Exophase]

Thinking about making it static to keep it all simple. Not going for some crazy 8-pass solution like Ari64 has. The most likely target is Pandora anyway, which shouldn't need something super efficient.

I think this will work fine, especially if you can keep 10+ registers available for allocation. Merging local data with the stack is a good way to get another register. The dynamic hybrid I mentioned would just involve using register caching (not allocation) in the remaining registers used for temporaries, just to avoid reloading/storing non-allocated registers used consecutively over short periods.

Naturally you should profile games to see which registers they use the most, possibly with game specific setups for the most demanding ones if it varies much.

Not sure what do you mean here? 128kB array of flags for every halfword?

Essentially yes. What I do in gpSP is have an array offset from the RAM writable regions that's the size of the region, and stores first check to see if there's something at address + offset. If there is they go to an SMC handling routine. The actual array contains block offset indexes, so it has two purposes. A 0 in either byte means that there's no block there, otherwise it identifies which block you're overwriting. There's also a bit that determines if the block starts there or not. This means there are less than 32K entries, but you shouldn't end up with so many blocks in RAM anyway.

Unless you can practically mark entire memory pointer LUT regions then I think this is a good option, because you have to do the additional check per-write anyway. This check is relatively fast and has no false positives. It is however harder on cache.

Today I've added an array of flags which tell if any particular 4kB chunk of RAM has code compiled from it, and check for them in write handler. That allows RAM changing games to work but I get lots of false hits, obviously 4kB chunks are too large (right now it runs through all block descriptors to verify if it really hit a block).

Yes, you definitely want an array per RAM area for blocks rather than that. You could of course use a bitmap as well, so it's really not that much space. But keeping it the same size as the RAM areas lets you use a constant offset and store block indexes.

If you do decide to stay with a first check using larger blocks then you can do what Ari64 does, which is to remove the modified blocks from whatever it is that's getting caught per write and patch them with self-checking code that determines upon block start if the block has been modified. It's pretty easy to see how this can horribly backfire, though.

That on-chip RAM functionality is annoying, Virtua Racing is rewriting it every 1-2 frames and running code from there, which means I will have to keep recompiling it. Actually this is what SVP version does too, but since it was a recompiler for a single game I was able to enumerate all IRAM states and avoid constant recompiling. This ain't going to work here as VF also seems to do the same (haven't properly verified though, pretty tired already).

Is it rewriting the entire thing or do some functions stay resident? Either way, 2KB doesn't span that much code so I think you should be OK even recompiling the entire thing every frame. It's not like some GBA games I know that modify bytes in code hundreds or even thousands of times per frame.

Do games use the overwrite feature of the VDP a lot? If they do then the function pointer LUT approach might remain a better idea. There's actually something you can do which is sort of inbetween (credit to hlide for this approach, gpSP targeting MIPS uses it). For each memory access check the region and if it's wrong call a function to patch it to check the new region and use a new handler. Then have an instruction to either perform the access or call a function to do it. This shouldn't be much worse than function pointer LUTs in the function calling cases (subtract, compare, conditional call, function call vs shift, load, indirect function call) and would be better for the direct cases. It doesn't handle mirroring as well, but you indicated that's not a problem.

 

[notaz]

Essentially yes. What I do in gpSP is have an array offset from the RAM writable regions that's the size of the region, and stores first check to see if there's something at address + offset.

Sounds like a good idea and easy to implement. Will see if there is really a cache thrashing problem with this when I have it working (by simply turning this on/off on non-smc games).

That on-chip RAM functionality is annoying, Virtua Racing is rewriting it every 1-2 frames and running code from there, which means I will have to keep recompiling it. Actually this is what SVP version does too, but since it was a recompiler for a single game I was able to enumerate all IRAM states and avoid constant recompiling. This ain't going to work here as VF also seems to do the same (haven't properly verified though, pretty tired already).

Is it rewriting the entire thing or do some functions stay resident?

It looks like it overwrites it whole. Will use separate translation cache buffer and throw it all away when it does this (right now everything is shared for simplicity).

Do games use the overwrite feature of the VDP a lot? If they do then the function pointer LUT approach might remain a better idea. There's actually something you can do which is sort of inbetween (credit to hlide for this approach, gpSP targeting MIPS uses it).

Some of them do, but I'll leave that as-is (function LUTs) for now. May be something for the future.

Thanks for the ideas, now just need to sit down and and write some code.