Topic: VICE testbench

yo!

since some time now, the VICE testprograms repository contains a testbench script that would allow to run a lot of the tests (over 1000 individual programs, runtime over 5 hours) automatically - which makes it a lot more interesting and useful for regression testing, or simply to find out what the current state of you emulation is smile

so far, i have added support for a few emulators (not only VICE), chameleon C64 and VIC20 cores, and soon gideons U64 core. i am offering my support to add the required things to test the fpga arcade cores too.

however, that means one of you guys needs to implement some essential features to the core (a debug register and remote memory access to name the most crucial ones). some things are described in this readme: https://sourceforge.net/p/vice-emu/code … readme.txt

let me know what you think, if you need additional info, etc

Re: VICE testbench

Oh, really interesting!
I'll take a look thanks.

Re: VICE testbench

From what I experienced, the C64 core still has a lot of issues. I hardly managed to run any late C64 games at all. Nothing has changes about this in the last years, so I do not have much hope here.

A test bench could probably help if it points to bugs which are easily locatable, but my feeling is that most issues are with the floppy code (games usually hang up while loading), and this test bench does not seem to have a focus on floppy emulation.

Re: VICE testbench

there are quite some drive tests as well, more than *any* existing emulator passes right now smile should be good enough for a start smile

(and from my experience... drive isnt very critical either, you can get away with a lot of broken things in the drive - however in the c64 itself not so much)

Re: VICE testbench

I'm surprised, other people have said it was quite good. Ok, then we need to look a this.
I have some local friends who are C64 demo coders, and I have all the real chips to hand. I'll crack on with this.

Re: VICE testbench

mm @phluxx could you name a few late games as you pointed out for example that you found did not work c64?? might help the fix process.

Re: VICE testbench

One I can remember was R-Type. This was rather in the game itself than in the loading phase. The system was completely freezing, somehow related to joystick input. Also I never got Katakis to run, this was rather a loading issue.

To be honest, I should be a bit more careful here: I used an SD-Card which had no approved SanDisk branding. (But it did work well with Amiga cores for long time, until a certain Firmware Upgrade.) Maybe the wrong sd card can even trigger in-game load errors?

However, I tried around 20 of my favourite C64 games and the ratio of working titles was way below 50%. Somehow I lost interest in the whole thing, as there was never progress for any core except Amiga.

I reported some issues here:
http://www.fpgaarcade.com/punbb/viewtop … 5708#p5708


Is this just my exerience? Did anyone try, lets say 10-20 more complex C64 games (in-game loading, tricky VIC usage..) and found the majority of them working?

8 (edited by gpz 2018-06-03 12:22:03)

Re: VICE testbench

i can only (again and again) advice against using games or demos for testing the emulation. its a pointless exercise mostly - using the test programs and debug systematically is a much better idea. when using games or demos its very likely to miss subtle errors, and even when you see them its often impossible to reproduce reliably. ie pointless smile eg once we had a subtle CIA related problem in chameleon which caused giana sisters to crash - somewhere mid game, requiring an hour of playing before it happens. in another old game (slurpy) we had the most weird behaviour in VICE, which you wouldnt even notice if you dont know the game already - and it turned out to be a yet unemulated TOD glitch. i could go on forever smile

Re: VICE testbench

@gpz: You are absolutely right. But you really need to do actual debugging and reverse engineering on such kits to know or even slightly imagine what pain in the a* it can be to find nasty bugs and what is needed for that. I also see the actual T65 with all my late fixes spreading around already, so this proofs it shouldn't be that bad anymore what I gave back to the community here... But of course still not perfect and I would like to go on with it if possible.   ;-)

@all:
I am sorry that I was/am quite absent, unfortunately have a lot to do. It is not that I don't like to work for the Replay anymore (or bought in by any other project out there) or I got upset about anything or anyone here. No, it is all cool...

Simply not a lot of spare time for many hobbies or a longer vacation.  That's why I don't follow all the changes you lately did, but it is great that there is something happening. So I won't yet follow the GIT thing you are planning, I would need to change my setup here, which is quite automated and I don't want to spend my time for anything not directly helping improving the core. I made a SVN dump and reinstalled it to my local SVN, because it is important for me for development - one who uses any DM for more than a web drive to deploy some files to a community will understand me.

At least I try to stay a bit in contact with Mike and support where I can - but I am aware it is by far not as much as I would do, which I am really sorry. Let's see - when my situation gets better again, I may join more deeply in the development again.

Nevertheless, coming to my point:

I still have my working Replay FPGA simulation environment where I can boot up a C64+C1541, download a PRG file and let it run, while tracing nearly every signal in the design. This works for any tests lasting a few seconds realtime with a reasonable turn-around time in development (like one single test of the Lorenz suite, to give an example - exactly this I did to iron a lot of issues out of the T65 core for the undocumented instructions). For this I also do not directly need the most actual Replay framework either.

So if one can test/set up such similar small test cases and sends me a _very_ concrete bug report via PM (the small PRG I need to run plus short description what the expected behavior should be), I promise to find the time to push it through my simulation setup and try to fix this specific issue, at least to give it a try. Please do not reply here in the forum for this, I won't read it regularly and I don't think noone else than me can really use it further. I also don't expect I get now 10 reports a day flooding my mailbox, so this should work out. And as said, finding bugs in games directly is not reasonable, not even if the emulation can run significantly faster than realtime.

And please don't just send me a link with a comment "look here and there", I will simply ignore these mails. I know as well all the C64 emulator/test resources and most of them are not suitable for a simulation approach needed here - beside a lot are not even properly documented to even know what the result should be, so they are useless for me.  So, if you want a better core for the community, please spend also a bit of your time to code some simple PRG or check out small existing tests and help out here with some pre-work making not only my live easier as well. Many thanks in advance!

Everything more is out of range for me at least for now, but I hope this offer helps a bit.

/WoS

10

Re: VICE testbench

"But you really need to do actual debugging and reverse engineering on such kits to know or even slightly imagine what pain in the a* it can be to find nasty bugs and what is needed for that."

sure thing. it was my primary job for a decade now - that testsuite (and the respective fixes in chameleon and/or VICE) didnt come from nowhere smile