Topic: Disk file support for 1541 core (d64, g64, ...)

Thread to collect topics around supporting different file formats with the C64 + 1541 core.

I hope I moved the g64 topic from the core discussion w/o destroying the thread, honestly I didn't read all in detail big_smile

No, seriously: as summary, my strategy for now is (order of some items may change):
- change d64 support from my RAM-based approach to the replay_lib fileio setup (direct read from sdcard)
- add write support for d64 (directly back to sdcard)
- add g64 read support (basic, gcr tracks only)
- add g64 write support (basic, gcr tracks only)
- extend g64 support to full spec (if it is feasible using two channels for one file with the fileio setup,
    I already got some G64 test files for experiments)
- add a "real" flux-based media format which is independent of the core but just describes the media
   and allows to be used with any core (there are plans I discuss with Mike, too early to state/discuss here)
- additional fast loader 1541/HW support beside jiffy (which one: tbd)
- additional 1541/HW extension support (tbd, actually about to fix Jims wish for RAM/ROM extension properly)

There are enough other things to work on (VIC, SID, ... and not to forget tape and cartridge file support), so progress on this roadmap will depend on these topics as well.

I'd also like to show my respect to all the developers out there on C64 side, on emulation side and on media archiving, they did some amazing work which will be hard to reach (if ever) as I am sure they continue to work as well. But I can hope to give back a little with this development wink

/WoS

Re: Disk file support for 1541 core (d64, g64, ...)

Just need to check if they just use ordinary dumps or special file formats.

just plain linear memory dumps for georam and reu (although georam dumps are very uncommon, i only know of one release that comes with a georam image)

G64 support is the number one reason why people want to use the 1541 Ultimate 2 cartridge - number 2 is the 16MB of REU.

actually the number one reason is using D64 files. G64 is infact not supported very well at all (much worse than in VICE, which by itself isnt perfect either)

Re: Disk file support for 1541 core (d64, g64, ...)

I absolutely do NOT agree.  If you ask any U2 owner, you will find it is not for the .d64 support.  There are tons of devices that give you that capability, but NOTHING else (until I release the u1541) gives you G64 support - and that is something that people use all of the time with copy protected disk images and for running custom demos.  The G64 support is not perfect, but it works in the majority of cases.

Re: Disk file support for 1541 core (d64, g64, ...)

I have all of the info on the G64 format if you need it.  If you plan to implement G64 support, please consider supporting a density level for each and every byte of data.  The G64 format allows for this, but it is not supported by ANY emulation system ever made.  There are many programs where they change the density level for a string of bytes (typically a full sector), and those do not work under WinVICE, Hoxs64, etc. such because they only support a single density for each track.  I can create G64 files with proper data using SuperCard Pro... I haven't done it yet because no emulation supports it.  FULL G64 support would be one more thing up on every emulator.  smile

Let me know if you have any questions, need test code, etc.  I spent more than two decades working on the PET/C64/Amiga stuff as my daily job, and now I am back into it again.  I would be happy to help in anyway I can.

Re: Disk file support for 1541 core (d64, g64, ...)

There are no fewer than 10 different devices that support d64 images, perhaps you are living in a cave!

i am well aware of that. however, only 1541u and chameleon offer actual drive emulation, and thus work with custom drive code and (fast)loaders - which, as said, is the whole point of using them. as a matter of fact that was the reason for the 1541u project - to be able to run demos (which often use custom loaders and drive code) and cracks (which often have builtin fastloaders). supporting G64s was basically a byproduct, since once you emulate the drive, you can do that very easily anyway.

The Chamelon's drive emulation is worse than WinVICE.

   
aha. if you say so smile the test programs are out there for everyone to try. in reality, it passes numerous tests that every other emulation fails on, including VICE :=)

There are some g64 images of the various programs that require multiple densities per track, and they aren't magically fixed by dumping them with KryoFlux.  The KryoFlux team and WinVICE team have stated that they don't care about these programs, at first stating that they didn't really exist... clearly that is not the case. I don't want to see another C64 emulation with half-ass support for important features.

the difference between you, and the kryoflux team and the VICE people is, that we actually cared about testing, comparing, and hard facts. ALL mentioned titles from the previously linked thread work just fine in VICE once the right data has been put into the images (which is not the case with a lot of G64s that are floating around, hence we had to redump), because none of them actually requires the level of emulation you think they do. if you'd actually be able to generate proper images as you say, you could just try it yourself. (we did actually create images containing speedzone entries - which VICE would ignore, and the images still work fine)

wolfgang: the georam image is, as said, plain linear... no special block order (and it does use custom code, not the "ramdrive" stuff). and yes, neoram and georam is the same thing (except neoram is battery buffered)

my favourite monitor back then was SMON (from 64er magazine)... however you really want a cartridge based one, IMHO (action replay v5 has a very good one, or super snapshot). no monitor shows register info in "clear text" (as eg VICE does in its "io" command) i think - if you find one that does, i want to know smile

6 (edited by JimDrew 2014-11-01 16:52:43)

Re: Disk file support for 1541 core (d64, g64, ...)

I am certain that I have tested and compared more disks than the WinVICE and Kryoflux teams combined.  I did it on a daily basis as my job (not a hobby) for more than 20 years, and I am doing it again now (and with the most advanced hardware available). There are many C64 disks that will NOT (and can not) work under WinVICE or any other emulation that does not support the complete g64 image format - which includes density information for every single byte so that mid-track/mid-sector density changes can be emulated.  It's as simple as that.  Re-dumping them without supporting the full g64 format yields a waste of time, and a non-working image.

Wolfgang, if you want a simple assembler/editor/memory viewer that will show the status register values in simple terms, then something like HESMON64 is good way to go.  I have a license for distribution of this program still.  I have versions that will load into different locations in memory ($8000, $C000, $2000, etc.)  If this helps, let me know and I can put together something for you.  I don't know of anything that will "watch" registers or explain the contents of I/O registers ($Dxxxx), etc.

Re: Disk file support for 1541 core (d64, g64, ...)

There are many C64 disks that will NOT (and can not) work under WinVICE or any other emulation that does not support the complete g64 image format - which includes density information for every single byte so that mid-track/mid-sector density changes can be emulated.  It's as simple as that.  Re-dumping them without supporting the full g64 format yields a waste of time, and a non-working image.

as said, we already fixed a bunch of such images (all that were mentioned here).
there are a few more that will still not work - but those will never work in g64 anyway due to other problems with the format (like those who read data while stepping). other than that you are - yet again - exaggerating the problem a lot, there are not "many" titles that will not work, you'll have a hard time to find even a dozen, and half of them will be obscure copy programs noone with his right mind wants to run on emulation anyway.

8 (edited by JimDrew 2014-11-01 22:17:59)

Re: Disk file support for 1541 core (d64, g64, ...)

I am not exaggerating anything.  The thread mentioned contains just a few titles that are affected.  Apparently you do not know how many titles are actually affected by the lack of the complete g64 support.  There are dozens of programs that use multiple densities on a single track.   The Kwik series is a good example, and they are not "fixed", so they do not work under WinVICE in g64 format because of the lack of density information.  ALL of these titles, and many more, do certainly work when the density is changed for each byte.  That is easy to simulate by putting break points in the drive code and manually altering the register values.  You can also deliberately alter the GCR so that the values are for the expected density (even though no change is made to the data separator clock).

Re: Disk file support for 1541 core (d64, g64, ...)

The Kwik series is a good example, and they are not "fixed", so they do not work under WinVICE in g64 format because of the lack of density information.

once you put the right GCR data into the G64, the programs from the Kwik series work just fine. apparently you didnt even try and/or couldnt create proper images.

attached image uses all 4 speed zones on track 15. works just fine in recent VICE (not sure about 2.4 release)

10 (edited by JimDrew 2014-11-02 00:59:22)

Re: Disk file support for 1541 core (d64, g64, ...)

This is not part of any release of VICE available.  It has been publicly stated by the VICE team that they refuse to add this support (refer to the thread you pointed out).  I can certainly create images with every single byte having a different density, and optionally check the track to see if all of the bytes stay within a certain density and use the fixed single density for the entire track.

Wolfgang, that image is an example of how the g64 format allows you to use a different density for every single byte of data that passes through the data separator.

I tried that image with 2.4a WinVICE and it works... which made me chuckle.  I then went and changed the density table for track 15 to be 03 (instead of pointing to the ficticious data at the end of the file).  Not surprisingly that also worked.  I then went and looked at the GCR data for track 15 and see that it was altered so that it matches the correct result for each change of the density.  So, this shows that although it looks to be "speed matched",  it was just patched with the data needed.

11 (edited by gpz 2014-11-02 10:55:11)

Re: Disk file support for 1541 core (d64, g64, ...)

So, this shows that although it looks to be "speed matched",  it was just patched with the data needed.

if anything, your blurb shows that you did not understand what kind of data should be present in the G64 file - which is valid GCR data, plus the corrosponding speed zone used to read it - and thats exactly what is contained in this image.
removing the speedzone entries (or ignoring them in the emulation) works, simply because this (and most other) protections only ever check if certain data can be read at a certain speedzone, and the emulator will put the (valid) data into the data seperator regardless of selected speedzone. the case that will not work is when the protection would check if at another speed it would read different (invalid) data from the same area - however noone was able to name or supply such protection until now, so it has little to no relevance in reality. (here are recent VICE builds)

Re: Disk file support for 1541 core (d64, g64, ...)

JimDrew wrote:

I am certain that I have tested and compared more disks than the WinVICE and Kryoflux teams combined.  I did it on a daily basis as my job (not a hobby) for more than 20 years, and I am doing it again now (and with the most advanced hardware available).

I wonder if that (you being back) is good or bad.
<removed>
Brutaldeluxe, I have edited your post (and it is rare I do so), not because I agree or disagree with you, but because it is
a) off topic
b) borderline abusive.

Aggressive technical debate is great, but nothing personal please.
Thanks,
MikeJ

13 (edited by JimDrew 2014-11-03 03:52:04)

Re: Disk file support for 1541 core (d64, g64, ...)

gpz wrote:

So, this shows that although it looks to be "speed matched",  it was just patched with the data needed.

if anything, your blurb shows that you did not understand what kind of data should be present in the G64 file - which is valid GCR data, plus the corrosponding speed zone used to read it - and thats exactly what is contained in this image.

Supplying a G64 image with a fictitious speed zone setting and claiming that is what makes it work is a bit odd, don't you think?  To use a g64 image with real hardware requires supporting the clocking of the GCR data at the data rate specified, be it at a byte or track level.  This is the only way to have 100% compatibility.

14

Re: Disk file support for 1541 core (d64, g64, ...)

Supplying a G64 image with a fictitious speed zone setting and claiming that is what makes it work is a bit odd, don't you think?

 
what makes you think its fictitious? the image represents exactly what is on the original disk. (minus some minor irrelevant details that the G64 format can not represent, like the exact length of individual bit cells)

To use a g64 image with real hardware requires supporting the clocking of the GCR data at the data rate specified, be it at a byte or track level.

what makes you think you could not do that using that image? however, as mentioned before, just stuffing the expected data into the data seperator regardless of selected speed zone is good enough for pretty much everything, since the protection would only check if data can be read, and never if it can not be read.

Re: Disk file support for 1541 core (d64, g64, ...)

good enough for pretty much everything

...is not what I am after.   I guess I have higher standards.

If the G64 speed information is correct, you should be able to use that image for anything that changes densities between bytes.

16 (edited by gpz 2014-11-03 20:19:24)

Re: Disk file support for 1541 core (d64, g64, ...)

...is not what I am after.   I guess I have higher standards.

you just like to ignore the fact that even when the emulator would support G64 fully, it can never accurately represent whats really on the disk, and a bunch of titles will still not work. (things like bitcell size, ie actual data rate, are lost) if you really need this level of accuracy, then you will need a flux based format anyway.

If the G64 speed information is correct, you should be able to use that image for anything that changes densities between bytes.

no. to support anything, you need a flux based format (and be prepared for not being able to properly emulate evrything with a track based format either, for titles that read while stepping you will need to keep a representation of the entire surface)
that said, VICE _can_ use any (properly made) image that changes densities between bytes on a track. really the only thing you can not do is writing such image from within the emulation (because creating the speedzone entries is not supported) - and since there is no point in doing so, noone bothered implementing it. if you really want to play with the drive on this level for whatever reason, use a p64 image and it will work (much better than it ever could using G64)

Re: Disk file support for 1541 core (d64, g64, ...)

Would anybody mind if I split this thread into several sub-threads - including a lively G64 one?
<ducks>

Re: Disk file support for 1541 core (d64, g64, ...)

Yeah, sorry Mike.  smile

Re: Disk file support for 1541 core (d64, g64, ...)

Just split but don't stop the thread. It is indeed a lively and educational discussion. It comes down to the classic observation that complete perfection for the emulation will come at a higher price of excess time/resources spent achieving the goal or what is necessary at a minimum to make it work for almost everything you throw at it. As long as the initial design is sound improved perfection can be added later on. In this case purism versus practicality may not be too far from each other and understand the points of both sides. Once you settle for less it is easier to lower the bar for something else ..... speaking of   purism - just replaced the internal floppy of my A600 with a Gotek drive. I am already missing that clunky old sound.

20

Re: Disk file support for 1541 core (d64, g64, ...)

indeed. one should realize that making the C64 core work (near) perfectly, ie the CIAs, the VICII, the CPU ... is a _LOT_ more important than having a perfect drive emulation. because at the end of the day you want to run games and/or demos, and if they crash because of a quirk in the C64 core, a perfect drive emulation wont help you at all smile and it will take quite a while until the point is reached that bugs in the C64 core are less of a problem than a handful obscure drive related things. putting a lot of energy into the drive emulation just doesnt make sense in this early stage of development.

21

Re: Disk file support for 1541 core (d64, g64, ...)

Right, at the end of the day it is not so important how a game/demo got on the core as long as it runs properly wink

Nevertheless a proper range of different files to be supported helps to use all the stuff in the first place - currently it is still quite limited (only read of d64, beside disk support I'd like to add tape support as well and improve crt support). And it is still a little about the technical challenge to get everything right.

Speaking of that:

Fixed the proper speed handling for reading d64 files in the c64 setup, just disconnected the table quite some time ago, didn't know why...  Now Jims disk scanner shows the right (default) speed for each zone on the disk (d64 can't handle any other speeds, will become interesting for g64 again).

Will continue now on the vic-20 for a while, as I did the 1541 separation already there (e.g. d64 write works there as well) and will put everything together for the c64 again later on (basically removing the local 1541 files and taking a single source for both with the c1541 tree I set up). Using the replay fileio lib will help simplifying the interfaces further. By that it is then quite easy to add multiple drives internally and externally.

/WoS

Re: Disk file support for 1541 core (d64, g64, ...)

Good news on your progress!

23

Re: Disk file support for 1541 core (d64, g64, ...)

Good news: checked in a new c64 version including the latest 1541 core. Now vic20 and c64 uses a single source found in the c1541 project directory. It should be close to the original 1541, and supports d64 read/write directly from sdcard.

Just played a little ghost'n'goblins, high score will be saved on disk again big_smile

I have not yet included the 1541 quick hack with the additional ram/rom, will follow with proper OSD configuration after some more testing with the base setup. I checked the 1541 test suite from The Dreams, looks quite good but a very few opcodes fail.

By the way: there is a separate d64 write protect feature in the OSD, by default it is enabled to avoid destroying files. So please set up some backups of your favourite d64 files when testing the brand new write feature of the c64/1541 on the Replay ...

/WoS

Re: Disk file support for 1541 core (d64, g64, ...)

Where is this test suite located?

25 (edited by WoS 2014-11-25 17:38:57)

Re: Disk file support for 1541 core (d64, g64, ...)

http://csdb.dk/release/?id=73533&show=summary  It was on berlios.de but this project page seems to be down (or the URL is wrong), just google for 1541-Testsuite_r192.zip - better than nothing for a start...  wink

Edit: found the project again on SF:
http://sourceforge.net/projects/rrtools/

/WoS