Jump to content

Getting data to/from the X16


m00dawg
 Share

Recommended Posts

Here’s the use case I keep thinking about: by sheer luck I actually finish my Traveller Trader game, and want to distribute it free to others.  Or someone writes a very nice Ultima-IV-styled game.  Or David Murray writes Planet X16.  Etc.

There’s at least 512k of banked RAM on this beast.  My game has a 270k map, and other files.  In other words it reads files into 40 banks.  The game loads all that using KERNAL calls, easy peasy, when it starts up.

 

1. I have to use a KERNAL solution.  That means IEC or the new device 1 (SD), right?  I can’t use an expansion card or the user port unless it’s got a modular boot loader that patches the KERNAL and makes it transparent to programs using it.  AND that our games are clever enough to use the “current” device, instead of defaulting to device 1 or device 8.  Or asking which device to use.  Or something.

So while I can hope for an expansion card, what we have now is an SD socket that I will want to buy an extension cable for, and the old Commodore IEC bus (which may not survive to the smaller X16 models).

 

2. Speed.  It is super fast on the emulator, surprise.  But, it won’t be as fast on the X16.  Will it be as fast as a 1581?  Or will it chug along like the 1541? I remember this is LGR’s big criticism of 80s nostalgia in one of 8BG’s videos about his Dream Computer.

How fast do you want 512 kilobytes transferred to the X16 by a game someone wrote?  Answer: probably at fast-loader speeds.  Maybe that just means we load a wedge at boot time.  Ok.

 

Edited by rje
Link to comment
Share on other sites

On 3/19/2021 at 12:50 PM, rje said:

2. Speed.  It is super fast on the emulator, surprise.  But, it won’t be as fast on the X16.  Will it be as fast as a 1581? 

The SD card will certainly be much faster than a 1581 ... except when the slow clock mode is set, the SPI clock is 12.5MHz, which is substantially faster than the maximum bits per second that a 1MHz or 2MHz CIA can transfer data over a clocked hardware serial line, and the processing speed & sector seek speeds for the 1581 pulls maximum throughput well below the maximum throughput of the CIA serial shift register (which is in turn faster than 1571 fast serial mode, which is in turn much faster than VIC-20 IEC mode).

If you are worrying about the SD card failing to be many multiples of the speed of the 1581, you may be worrying over nothing.

As far as getting files onto the SD card without sneakernetting, an important resource constraint for different people's choices will be different quantities of patience on offer.

 

Edited by BruceMcF
Link to comment
Share on other sites

It sounds like the concern comes down to:

  1. The SD card is fast, but:
    1. It is inconveniently placed.
    2. It has limited insertions before failure.
  2. The IEC port is slow.

I'm still partial to architecturally defining a storage expansion card interface that KERNAL supports and letting 3rd parties build cards that conform to the spec. If it were possible to flash ROM after the unit has shipped, then this could be turned around and a 3rd party could create a solution that is compelling enough that it gets into post-release KERNAL update, but it isn't clear that field updates to the ROM will be supported. The flash IC it looks like they're using (SST39SF040) seems to have a straightforward host-controlled rewriting process, but allowing field updates also means defending against malicious and accidental bricking of the device.

Link to comment
Share on other sites

1 hour ago, Wavicle said:

It has limited insertions before failure.

Is there any real evidence that the SD2IEC is any more durable than the SD slot on the VERA? This seems to be a lot of worrying for nothing. How many times a day is anyone likely to insert and eject the SD card? My guess is that for most folks it will be far less than once a day. Maybe a bit when getting it first set up, but after that when you get new software for it, which is going to be what, twice a week? Like others have suggested, get an SD extension and mount it wherever you like, if you are really concerned. If you wear that out, buy a new one for <$10.

  • Like 1
Link to comment
Share on other sites

Matt (and Bruce), I *think* the SD route is reasonable and (like with the cable) wear can be avoided.

I'm thinking aloud about this, because that's what I'm trained to do at work -- (1) ask questions and (2) think about use cases and (3) suggest worst-case game-stopping scenarios, if it's possible that they can be avoided upstream.

 

How often will we swap that card out?

First, think about the process for new resources.

1. Pop it out of its slot and connect it to your computer.
2. Make a directory for the new thing (e.g. like /planetX16).
3. Copy the files into that directory.
4. BACK UP THE IMAGE (maybe not always).
5. Put the SD card back into the X16.

At what point does that become a bit tedious?  I don't actually think I'll be doing this more than once a week, or even once a month.  So maybe it's a moot point.

I just want to make sure it's a moot point.

Edited by rje
Link to comment
Share on other sites

1 minute ago, rje said:

I know someone mentioned that they couldn't believe that a program would take up all available banked RAM.

I tell you, if it's there, it will be used.

 

Maybe not a program, per se, but definitely a program + data. Here's how the current state of Cavy's Quest works:

Load #1: Engine code + Main config + Title screen data = 129kB

Load #2: Zone 0 data: 333 kB (nearly complete)

Load #3: Zone 1 data: 320 kB (likely to grow significantly)

Then more zones are to come, but only one needs to be loaded at a time, and the game will be structured to minimize zone transitions.

Then there is other data generated on the fly in three other banks, so an XCI game can absolutely use all 63 user banks at once and pretty much all of main RAM. Maybe not all banks at 100%, but close to it.

  • Like 3
Link to comment
Share on other sites

1 hour ago, SlithyMatt said:

Maybe not a program, per se, but definitely a program + data. ...

Yes, it's program+data that is the source of "biggish" size sets of files to be loaded onto the SD card.

And one thing to bear in mind is how much some of the C64 game companies would CRUNCH their data down to get it loaded onto a 1541 disk. Given that this is more hobbyist or niche programming, there will be sometimes be something like the US revolutionary war soldier who was writing home, "Dear sister, I apologize for the long letter, but I didn't have the time to write a short one."

  • Like 1
Link to comment
Share on other sites

11 hours ago, rje said:

Matt (and Bruce), I *think* the SD route is reasonable and (like with the cable) wear can be avoided.

If the sd slot location is inconveniently located and/or wear is a concern, why not 'make' an extension adapter that plugs into the micro and ribbon-cable's (v) it out to a standard size SD slot on the front/under the front of the X16.  Of course, I'm talking about an add-on product that some enterprising youngster can sell for $25 with just $10 worth of parts.  Limited market yes; mother-of-invention type solution, also 'yes'.

Sort of like the TFW8Bit Pet 'extension' Brought the card to the front: https://www.thefuturewas8bit.com/sd2pet.html. The aesthetics of battle-zone style braked metal 70's CBM that incorporates the color matched boxy 2031 style won't fit the Commander's motif but I'm sure something can be worked out.  (if only it were the 70's again, my riveting, shear, brake, punch, spot welding skills could come in handy and I could be a millionaire.  Come to think of it, the 'blue' that Dave uses in his cartridges is probably very close and he already has the mould for the SD slot... hmmm...)

Link to comment
Share on other sites

4 hours ago, EMwhite said:

If the sd slot location is inconveniently located and/or wear is a concern, why not 'make' an extension adapter that plugs into the micro and ribbon-cable's (v) it out to a standard size SD slot on the front/under the front of the X16.  Of course, I'm talking about an add-on product that some enterprising youngster can sell for $25 with just $10 worth of parts.  Limited market yes; mother-of-invention type solution, also 'yes'.

Such a cable has quite a few markets which is why you can find them on Amazon for $6. I have a couple of them used to reduce wear on the micro SD socket on my 3D. They didn't turn out to be as necessary as I'd thought though because most of my prints are handled via Octoprint which enables me to start a print from any computer on my network.

I think the concept here is somewhat the same: we (well, some of us anyway) don't want to move around physical storage media in order to move bits. On my Mister, I don't move the SD card to a computer and copy the ROMs I want onto the card; I use FileZilla to copy them using FTP over the network.

  • Thanks 1
Link to comment
Share on other sites

On 3/21/2021 at 12:05 PM, rje said:

Matt (and Bruce), I *think* the SD route is reasonable and (like with the cable) wear can be avoided.

It will be fine with a lot of the potential market. For others in the potential market, they will find it annoying. But if there is a range of under $10-$20 solutions that cover the range of different ways that people prefer to handle it - eg, sneakernet, WiFi onto SD, USB flash drive, cheap cable to existing PC - then I think it's OK.

  • Like 1
Link to comment
Share on other sites

  • 1 month later...
On 3/13/2021 at 9:37 PM, BruceMcF said:

There is, however, Kernel support for installing device drivers that can use the numbered device API, so as long as an autostart program is an option, there doesn't have to be a lot of difference between hardware that has Kernel support and hardware that has loadable Kernel API drivers.

...

And one thing to bear in mind is how much some of the C64 game companies would CRUNCH their data down to get it loaded onto a 1541 disk.

This is my preferred solution -- thing that patches into the KERNAL and acts like a Device, nice and sneaky proper.

And as for crunching data: I'm already doing this with my Traveller-Trader program.  I already have TOO MUCH data to put on the X16 without file-swapping, and I want to minimize that, so I pare down the data.  Remove excess bytes.  Mash two fields together into one byte.  Bitfields and flags.  Even Z-compressing labels (the texts aren't long enough to use the built-in compression library).  It'll still be more data than I can load.  So I drop all the non-essential data... and now I can fill up the RAM banks with a bit of space to spare.  But I had to crunch it.

Quote

How fast do you want 512 kilobytes transferred to the X16 by a game someone wrote?  Answer: probably at fast-loader speeds.  Maybe that just means we load a wedge at boot time.  Ok.

 

Here's my back-of-the-envelope load speeds.  I made what I *assume* are reasonable assumptions, but of course the devil's in the details.

 

ROB'S LOAD SPEEDS TABLE

CLASS  THROUGHPUT  ~BAUD    32K RAM   ONE BANK   256 BANKS  SUBJECTIVE RATING
-----  ----------  -----   --------   --------   ---------  -----------------
Lower      1 kb/s   9600     32 sec      8 sec      33 min  Slow
Meh      2.4 kb/s  19200     13 sec      3 sec      14 min  1541 + Fastloader
OK         5 kb/s  38400      6 sec      2 sec       7 min  Okay
Good       6 kb/s  48000      5 sec     ~1 sec       6 min  Nice (C128 + 1571)
Better     7 kb/s  56000     <5 sec     ~1 sec      <5 min  Nicer
NIC       16 kb/s  128 K      2 sec     <1 sec       2 min  Luxurious
Upper     30 kb/s  huuuge    ~1 sec    0.3 sec       1 min  Bliss




	

LOWEST NON-PAINFUL SPEED

If we can load 1 kilobyte per second, I think we'd be OK.  You'd have to be careful about how you load your banked RAM, but loading the main RAM is not painful.
 

"REASONABLE" SPEED

7 seconds is roughly the half-life of your short-term memory.  Call it 4 to 8 kb/s.  That would feel like luxury to 8-bit people. 

At this speed range (plus or minus), main RAM loads in a few heartbeats, and that's a good place to be in.

If I read right, this is the speed of the C128 with a 1571.

 

UPPER END

Speeds more than 6 kb/s start to approach Luxury. 

8 kb/s isn't a whole lot more noticeable.  12 kb/s is noticeable.

But after that, returns diminish.  Yes, you can load larger and larger chunks of RAM banking without caring.  And that may be important to the demoscene, so OK, there's value.

At what point does that value drop off, though?

30 kb/s is probably as fast as you'd need for file loading...

...and I suspect that 16 kb/s will typically look nearly as fast as 30 kb/s... but that depends on your data...

...unless I'm missing something.  

 

 

Edited by rje
Link to comment
Share on other sites

2 hours ago, rje said:

 

 

...unless I'm missing something.  

 

 

VRAM loads way faster than RAM, because data loads through VERA first. I'm working on a game right now that needs to switch between four 64kb lookup tables on the fly, and I needed to have the indices to those tables be only two bytes, so they'll go into the low bank of VRAM and all my tile maps etc will be in the high bank of VRAM.  I'm estimating that loading to VRAM will be about 3 times as fast as loading to RAM.

  • Like 1
Link to comment
Share on other sites

3 hours ago, Elektron72 said:

According to the release notes in the x16-rom repo, some recent changes to the load routine (specifically, the addition of block-loading capabilities) have made it possible to load files from the SD card at 140 KB/s.

Yeah, and because it loads through VERA, the load speeds directly to VRAM should be 2-3 times faster; I've been estimating 300kb/s.

Link to comment
Share on other sites

48 minutes ago, Ed Minchau said:

Yeah, and because it loads through VERA, the load speeds directly to VRAM should be 2-3 times faster; I've been estimating 300kb/s.

Could you explain why loading to VRAM would be faster? As far as I understand, the data would still need to pass through the CPU before being transferred to VRAM.

  • Like 1
Link to comment
Share on other sites

1 hour ago, Elektron72 said:

Could you explain why loading to VRAM would be faster? As far as I understand, the data would still need to pass through the CPU before being transferred to VRAM.

It's the other way around. The SD card goes to an IEC2SD implemented on VERA, which then has to push it through the CPU to RAM. Just loading directly to VRAM skips about eleven 8MHz cpu cycles per byte that would be required to push it to RAM and increment the RAM address, and VERA operates at 25MHz and can autoincrement. A 300kbps estimate is conservative, about eighty 25MHz cycles per byte. 

Edited by Ed Minchau
Link to comment
Share on other sites

48 minutes ago, Ed Minchau said:

It's the other way around. The SD card goes to an IEC2SD implemented on VERA, which then has to push it through the CPU to RAM. Just loading directly to VRAM skips about eleven 8MHz cpu cycles per byte that would be required to push it to RAM and increment the RAM address, and VERA operates at 25MHz and can autoincrement. A 300kbps estimate is conservative, about eighty 25MHz cycles per byte. 

While both VRAM and the SPI port are on VERA, the VERA documentation does not describe a mechanism by which a direct transfer between the SD card and VRAM could be performed. Looking at the kernal load routine (in kernal/cbm/channel/load.s), loading to VRAM involves the CPU manually loading bytes into VERA's DATA0 port. This doesn't use the block-loading API, so loading to VRAM will likely be slower than loading to RAM.

Link to comment
Share on other sites

12 hours ago, Ed Minchau said:

It's the other way around. The SD card goes to an IEC2SD implemented on VERA, which then has to push it through the CPU to RAM.

Last time I read the Vera doc, it appeared to just be an SPI port with a 12.5MHz serial clock normally, fitting within the regular maximum 20MHz serial clock for accessing the SD card in SPI mode, and a slower serial clock for the phase of starting an SD card into SPI mode that is specified at a lower top speed limit.

It's not clear why it would be necessary to implement an IEC2SD on Vera when the byte to/from the SPI is directly addressed in the memory mapped I/O space, and the 65c02 is perfectly capable of doing SPI mode SD card operations, but in any event, I don't recall any mention of the coprocessor on Vera that would be required to go that route.

Link to comment
Share on other sites

14 hours ago, Elektron72 said:

While both VRAM and the SPI port are on VERA, the VERA documentation does not describe a mechanism by which a direct transfer between the SD card and VRAM could be performed. Looking at the kernal load routine (in kernal/cbm/channel/load.s), loading to VRAM involves the CPU manually loading bytes into VERA's DATA0 port. This doesn't use the block-loading API, so loading to VRAM will likely be slower than loading to RAM.

That's definitely not what the emulator is doing. Data loaded directly to VRAM doesn't touch the CPU at all.

Link to comment
Share on other sites

40 minutes ago, SlithyMatt said:

Is that the case when using an SD card image, or just when using the host file system?

I usually use the emulator with an SD card image so I can access features that are not supported when using the host filesystem. I have used the debugger to step through the routine that handles loading to VRAM, and can confirm that when an SD card image is used, the CPU has to process each byte. On the other hand, loading to VRAM (or anywhere else) does not involve the CPU when the host filesystem is used, as the emulator itself handles calls to the load routine.

  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...

Important Information

Please review our Terms of Use