Adventures in File IO and Commander X16

[ZeroByte]

I have to say that working with the filesystem is one of the most challenging aspects of the Commander X16 that I've personally run across. There are tons of permutations of "Host FS vs SD image", emulator/ROM version, bugs, compiler/system library behavior, etc.

One thing that's been updated since R39 is that the hyper load/save hooks have changed a bit, if my understanding is correct - and now instead of just intercepting LOAD and SAVE, the emulator(s) now intercept calls to MACPTR and ACPTR (or some other similar choke point) in order to allow more operations to work with the host FS.

It appears that whatever the current methodology is, it doesn't catch everything, because I've recently started working with sequential access to files instead of just bulk loading them into (V)RAM/HIRAM. I've found that this doesn't work unless you're using an SD image. I find SD images to be very inconvenient to work with but hey, I guess that's just the nature of working with something like the X16 in pre-release state (life on the frontier - there's no Dominos and no running water).

One thing I've run across is cc65's cbm.h library, which provides what it calls "BASIC-like" interfaces to the low-level kernal file handling routines:

• cbm_open(LFN, DEVICE, SA, filename)
• cbm_close(LFN)
• cbm_read(LFN, buffer* , num_bytes)
• cbm_write(LFN, buffer*, num_bytes)

Where LFN = logical file number, DEVICE = 8 (for disk drive), and SA = that bizarre, multi-purpose byte that is essentially an "argument" for various low-level routines like LOAD or OPEN....

So here we are in cc65 space. The include file also provides three defines to use for the SA in cbm_open():

Quote

/* Constants to use with cbm_open() for openning a file for reading or
** writing without the need to append ",r" or ",w" to the filename.
**
** e.g., cbm_open(2, 8, CBM_READ, "0:data,s");
*/
#define CBM_READ        0       /* default is ",p" */
#define CBM_WRITE       1       /* ditto */
#define CBM_SEQ         2       /* default is ",r" -- or ",s" when writing */

 

This leads me to believe that cbm_open() might be doing a little bit of spying on the SA value before calling the underlying Kernal routines.

In BASIC, you can use any combo of LFN and SA as long as LFN >= 2 as 0 and 1 have special meaning. I'm going to dig a little deeper into the cc65 sources to see if it's intercepting and interpreting the SA and LFN or not.

But the interesting thing I've discovered is that in order for sequential access to work properly, you must use the same value for LFN and SA - and furthermore, these must be 2, 6, or 10 in order for file seeking to work. (I notice that these result in bit 1 of the SA/LFN value being set - hmm)

Oh, and seeking - stdio.h implements fopen() fseek() and fclose() - but fseek() won't build on X16 as there's no underlying code for it in cx16.lib - and there's no cbm_seek() function because CBM machines don't naturally support seek (well, without add-ons like SD2IEC or DOS extenders) so the library doesn't do it.

To do a seek, I wrote my own function with some advice from @TomXP411 - if you open the command channel to device 8 and send a "P" command (position) while the file is open, Commander-DOS will do a seek.
The P command epects binary values to follow it, so here's how to do it in cc65:

int8_t cx16_fseek(uint8_t channel, uint32_t offset) {
  #define SETNAM 0xFFBD
  static struct cmd {
    char p;
    uint8_t lfn;
    uint32_t offset;
  } cmd;

  // open command channel to DOS and send P command.
  // P u8 u32 (no spaces) u8 is LFN to seek(), and u32 = offset.
  cmd.p='p';
  cmd.lfn=channel;
  cmd.offset=offset;
  // can't call cbm_open because the P command is binary and may
  // contain zeros, which C would interpret as null terminator.
  //
  // Roll-your-own inline asm call to SETNAM:
  __asm__ ("lda #6");
  __asm__ ("ldx #<%v",cmd);
  __asm__ ("ldy #>%v",cmd);
  __asm__ ("jsr %w",SETNAM);
  cbm_k_setlfs(15,8,15);
  cbm_k_open(); // this sends the CMD bytes..
  cbm_k_close(15); // close the command channel
  return 0;
  // TODO: ERROR HANDLING!!!!!
}

Note that this code uses cbm_k_open() and not cbm_open() - the k version is a direct Kernal call, so it's not monkeying around with anything like cbm_open might be.

 

Anyway, if anyone has any tips, insights, or whatever regarding file IO, I thought that would be a useful topic for people to be able to hit upon.

[rje]

Thank you!

I've shied away from sequential files so far because of the not-knowing.  Your function will be helpful when I finally have the time and inclination to wrap these sorts of functions myself.

 

BUT ALSO... even after the X16 is released, I will still be writing cc65 code on my Mac.  This means SD card access will STILL be inconvenient.  Which makes me think....

 

Edited June 20 by rje

[TomXP411]

This is also a good lesson in dealing with the quirks of C programming. This custom code was only necessary because of C's use of null-terminated strings and the fact that the Position command uses zeros as part of its data. ZeroByte is getting around this problem by forcing the data length to 6 (lda #6) and then pointing to a binary version of his Position parameters. 

Once this has been thoroughly tested and has something in there to trap any errors, this definitely needs to end up as part of the CX16 library for both cc65 and KickC. 

 

[desertfish]

I'm gonna snatch the essence of it for prog8 as well I think ! (can't reuse the C code, but can reuse the meaning of it! )

[ZeroByte]

The saga continues:

I fully expected this result, but let it be known that cbm_read() has no concept of banked RAM at this time.
If you perform a read that runs off the end of a bank, then it will happily throw your data at the ROM region after running off the end of the bank window, and I presume then wrap back around to 0000 after passing FFFF

Fortunately, my current experiment is going to involve streaming music from a data file on the fly (chip music, not PCM - no WAY to keep up with that!!!) and so it's not going to be an issue for me down the road with this particular program, but I thought folks should be made aware that while LOAD works in BASIC, and Kernal's LOAD routine works for HiRam, cbm_read() just runs off the cliff. (Goofy yell: Yaaaa-hoo-hoo-hoooey!)

One of the items I'm going to experiment with is a cx16_macptr() frontend routine similar to the cx16_seek() routine above. Kernal MACPTR will be bank safe, and run much faster than cbm_read() {which doesn't know MACPTR exists - it's a CX16 exclusive!

For those who don't know, MACPTR is the Kernal routine to perform a block read from disk as opposed to a byte-by-byte read using ACPTR.

Also for those who don't know: VLOAD only uses ACPTR, which is hella-slow compared to MACPTR, so if your VLOADs are taking too much time, try LOAD-ing them into HiRam and then blitting them into VRAM yourself. You'll do that before VLOAD gets to the halfway marker with any "large" amounts of data.

 

[svenvandevelde]

This is my sequential file read routine ...

/**
 * @brief Load a file to banked ram located between address 0xA000 and 0xBFFF incrementing the banks.
 *
 * @param channel Input channel.
 * @param device Input device.
 * @param secondary Secondary channel.
 * @param filename Name of the file to be loaded.
 * @param bank The bank in banked ram to where the data of the file needs to be loaded.
 * @param sptr The pointer between 0xA000 and 0xBFFF in banked ram.
 * @return bram_ptr_t
 *  - 0x0000: Something is wrong! Kernal Error Code (https://commodore.ca/manuals/pdfs/commodore_error_messages.pdf)
 *  - other: OK! The last pointer between 0xA000 and 0xBFFF is returned. Note that the last pointer is indicating the first free byte.
 */
unsigned int load_file_bram(char channel, char device, char secondary, bram_bank_t dbank, bram_ptr_t dptr, size_t size) 
{
    #ifdef __FILE
        printf("load file, c=%u, d=%u, s=%u, b=%x, p=%p, si=%u", channel, device, secondary, dbank, dptr, size);
    #endif

    byte status = 0;
    char ch = 0;
    unsigned int read = 0;

    byte bank_old = bank_get_bram();
    bank_set_bram(dbank);

    status = cbm_k_chkin(channel);
    status = cbm_k_readst();
    if(status) return 0;

    byte* ptr = dptr;

    // gotoxy(0,10);
    // printf("load: ptr = %p, bank = %x", ptr, bank);
    while ((size && read < size) || !size) {

        ch = cbm_k_chrin();

        // OK, character read and not end of input
        read++;

        // check if pointer is between bram boundaries
        if(ptr == 0xC000) {
            dbank++;
            // printf(", %x", (word)bank);
            bank_set_bram(dbank); // select the bank
            ptr = (byte*)0xA000;
        }

        // put the read character into the pointer location
        *ptr = ch;

        // increase the pointer
        ptr++;

        status = cbm_k_readst();
        if(status) break;
    }

    #ifdef __FILE
        printf(", r=%u, status=%u\n", read, status);
    #endif

    bank_set_bram(bank_old);

    #ifdef __FILE
        // cbm_k_chkin(0);
        // while(!getin());
    #endif

    return read;
}

Note that the routine and the description still requires some love from my side ... it's just a version at the moment to get my files loaded into bram in a sequential manner, in chunks as I use a bram heap manager that allocates blocks of memory for me. 

Edited June 21 by svenvandevelde

[ZeroByte]

I just had some initial success using MACPTR in cc65. It’s TONS faster and already bank-aware. I’m going to play with it some more but I think it could be even faster if it really will blit in 512 bytes per pass.

CHRIN has tons of overhead per byte.

will post example code later.

[ZeroByte]

No example code tonight, as it's late and I've just hit a good stopping point.

So here is a screenshot of my Wolf3d music demo program, but running in a test mode playing with SEQ access. I post it here because the debug output gives a very good view of how MACPTR does its thing, and the screen includes a ticks counter benchmark to show load speeds. It loads 8K in about 12 frames' time (and would be faster if not having to stop and write all of this debug output to the screen on every pass)

[ZeroByte]

As promised, here's the code that I'm using to harness MACPTR. It's incomplete in that it doesn't fall back on ACPTR (cbm_load() in this case)  if MACPTR returns -1. I'm not happy with the return type being signed, and will probably fix this by either making it a signed log, or else just returning zero on failure and requiring that the calling code check status byte to know if it's zero because EOF or because of unsupported device.

Most likely, though, this is a case that will never happen with files, as the OS supports block reads from disk - but if you pass it an LFN that's connected to the screen or keyboard or whatever, then that would fail....

extern int __fastcall__ macptr(unsigned char numbytes, void* buffer);

int cx16_read(unsigned char lfn, void* buffer, unsigned int size) {
  int error = 0;
  char* buf = (char*)buffer;
  static unsigned int bytesread;
  static int tmp;

  /* if we can't change to the inputchannel #lfn then return an error */
  if (_oserror = cbm_k_chkin(lfn)) return -1;

  bytesread = 0;
  printf("\n");
  while (size > 0 && !cbm_k_readst()) {
    if (size>=512)
      tmp = macptr(0,buf);  // let MACPTR read as much as it wants
    else if (size>=256)
      tmp = macptr(255,buf); // If size 256..512, unlimited MACPTR would read past desired stopping point.
    else
      tmp = macptr((size), buf);
    if (tmp == -1) return -1;
    bytesread += tmp;
    size -= tmp;
    // wrap the buffer pointer back into the bank window
    // if it advances above 0xbfff. Note that MACPTR did
    // wrap banks correctly, but our calculation must match.
    // also note that MACPTR incremented the active bank,
    // so there is no need to do RAM_BANK++ here.
    if (buf >= (char*)0xc000) buf -= 0x2000;
    buf += tmp;
    if (cbm_k_readst() & 0xBF) break;
    if (tmp == 0) break;
  }

And the assembly is short:

MACPTR := $FF44

.import popa
.export _macptr ; the _ is important - C symbols all start with _

.proc _macptr: near
    phx
    tax
    jsr popa
    ply
    jsr MACPTR
    bcs macptr_unsupported
macptr_supported:
    txa ; C wants int in .AX not .XY
    phy
    plx
    rts
macptr_unsupported: ; return -1 on error
    ldx #$ff
    lda #$ff
    rts
.endproc

Note that the MACPTR function itself returning a signed int is fine, since MACPTR will return 512 bytes at most.

[Greg King]

On 6/21/2022 at 2:10 PM, ZeroByte said:

if (buf >= (char*)0xc000) buf -= 0x2000;
buf += tmp;

buf should be increased before the test.  Otherwise, the current wrap-around won't be seen.  But, if tmp is zero, then there's no point in doing any increases/decreases.

Edited Wednesday at 02:05 AM by Greg King

[ZeroByte]

On 6/21/2022 at 8:55 PM, Greg King said:

buf should be increased before the test.  Otherwise, the current wrap-around won't be seen.  But, if tmp is zero, then there's no point in doing any increases/decreases.

Good catch - that is a bug indeed.

Right now, I'm wrestling with why VIA IRQs aren't behaving as I expect.... will fix that line right now tho.

[ZeroByte]

I've uploaded a new version of my OPL music on X16 demo (VOPLDEMO) to the downloads section.

Now, you can play _all_ the songs from the shareware version, and it loads each one into memory dynamically as you switch between them (using the left/right arrow keys)

[TomXP411]

Ok. color me impressed. Well done!

 

[ZeroByte]

On 6/21/2022 at 8:55 PM, Greg King said:

But, if tmp is zero, then there's no point in doing any increases/decreases.

Actually, there is a reason - I made my routine report results via a global struct containing the next-available memory location just like the Kernal's LOAD routine does. (only it's written into a struct and not passed back directly)

Also, since all the values are right there in the routine's hot little hands, I decided to save the caller the effort of calculating offsets, etc., and the struct contains total bytes transferred and a success/fail boolean. (which I suppose could be the final status byte, come to think of it)