Jump to content

Binary data for the X16


rje
 Share

Recommended Posts

My space game, ever in slow-time development, uses binary data for ship data, character data, map data, and game state data.  I load these into high RAM and use them as needed, pulling them into main RAM long enough for a display or whatever.

Perl is really useful for creating binary files -- more useful than C, in that these sorts of tools are scripts, not worthy of dedicated development time with a proper language (like C).

Here's the easiest way to create a one-record binary file.

use strict;

EVERY Perl script should start with that line there.  That enforces some discipline in both the developer and the Perl interpreter.

Now let's declare our data and shove it into Perl variables.  This gives the data some tweakability, decoupled from the binary file's structure... although it also gives us an idea as to what lengths we want our data to fit into.

my $header = "TEMPLE OF APSHAI GAME STATE";   # fits into 31 characters
my $name   = "MATT HEFFERNAN";                # fits into 15 characters
my $gx     = 502;                             # global X, looks like 2 bytes
my $gy     = 500;                             # global Y, looks like 2 bytes
my $sx     = 3;                               # local X, 1 byte
my $sy     = 10;                              # local Y, 1 byte
my $life   = 10;                              # 1 byte

So we have seven pieces of data up there.  Now let's write them.

open my $fp, '>', 'GAMESTATE.BIN';    # fopen for writing ('>')

print $fp pack 'AC', 'X', 16;   # two byte throwaway 'header'
print $fp pack 'A31x', $header;      
print $fp pack 'A15x', $name;
print $fp pack 'vv', $gx, $gy;
print $fp pack 'CC', $sx, $sy;
print $fp pack 'C', $life;
print $fp pack 'x16';

close $fp;

 


The 'pack' command puts our data into a binary format.  We specify that exact format with the string that follows each pack... that string is a template telling the command how to pack in the data.

 

For example,

pack 'A5', 'HELLO';

tells Perl to pack in 5 ASCII characters from the source string 'HELLO'.

Add a zero value with 'x'.  For example, 

pack 'A15x', $name;

tells Perl to pack $name into 15 ASCII characters, then add a zero byte to the end of that.  Or this line, which simply adds 16 zero bytes to the file:

pack 'x16';

'v' encodes an integer in little-endian format.  'CC' encodes two unsigned char values.  And so on.

The trickiest example in my code is

pack 'AC', 'X', 16;

which first encodes a single ASCII character ('A'), an unsigned char value (16).  This is the throwaway two bytes that the X16 will ignore when you load the file to a specific address.  Since I load the file to a specific address, these two bytes are completely irrelevant.  Just on a whim, I put "X16" into them.

The resulting binary file is exactly 73 bytes long, and looks like this:

00000000  58 10 54 45 4d 50 4c 45  20 4f 46 20 41 50 53 48  |X.TEMPLE OF APSH|
00000010  41 49 20 47 41 4d 45 20  53 54 41 54 45 20 20 20  |AI GAME STATE   
00000020  20 00 4d 41 54 54 20 48  45 46 46 45 52 4d 41 4e  | .MATT HEFFERMAN|
00000030  20 00 f6 01 f4 01 03 0a  0a 00 00 00 00 00 00 00  | ...............|
00000040  00 00 00 00 00 00 00 00  00                       |.........|
Edited by rje
  • Like 1
Link to comment
Share on other sites

  • 1 year later...

While Perl is useful for these types of tasks, perhaps now the tools for creating graphics/sound binaries have developed enough on the X16? I've phrased it as a question since I'm not there yet in coding for the X16 platform, nor have I investigated yet.

The other thought is that it becomes another tool one has to learn to do this type of work, something that requires a little background knowledge and investment of time. While many may already have this skillset, consider the users who just don't have the experience or the bandwidth to take on another thing. Right now my feeling is the X16 platform requires one to know or learn too many things from too many places that really aren't in coding up an application, rather just tasks to get at the business end of coding. I think one of the successes of the VIC-20 was that with the user's guide and the programmers guide one had all one needed to get going with no side tasks that required building software tools. 

That being said,  I'm no Mozart or Picasso. A sound editor or sprite composer in my hands would only allow me to easily create something bad like a smashed bug on a blue Triscuit. 😜 But I'd like to have the ready built tools rather than learn the bare minimum on another complex technology that I wouldn't have to know/use otherwise. 

PS - I have learnt and used Perl in the past, and might use it in the future someday, but that effort came with some $ involved, and the X16 is a hobby platform.

 

  • Like 1
Link to comment
Share on other sites

I think everyone can use whatever they know for this kind of task. Albeit admittedly some tools allow for more comfort than a Hex editor. I am using Python for generating binary data. It would look like this

struct_1 = [12, 13, 14]
 

struct_2 = [0, 80, 0, 90, 0, 100]
 

data = struct_1 + struct_2 + struct_1 # appending data is as easy as this.
 

data_bytes = bytes(data)
 

f = open("DATA.BIN","wb") # "wb" stands for write binary
 

f.write(data_bytes)
 

f.close()

 

 

 

Edited by kliepatsch
  • Like 2
Link to comment
Share on other sites

On 9/25/2020 at 10:09 AM, rje said:

My space game, ever in slow-time development, uses binary data for ship data, character data, map data, and game state data.  I load these into high RAM and use them as needed, pulling them into main RAM long enough for a display or whatever.

Perl is really useful for creating binary files -- more useful than C, in that these sorts of tools are scripts, not worthy of dedicated development time with a proper language (like C).

Thanks for that ... as a long time AWK user, Perl is surely an easier learning curve for me than Python.

  • Like 1
Link to comment
Share on other sites

I note that his Python code looks familiar:  if I was writing raw bytes, that's what the Perl code would look like.

Note that he has to first break down larger numbers, and encode strings into bytes.  He might have to do some custom coding if endianness is a problem.

 

... I want AWK on the X16.

 

 

 

Edited by rje
Link to comment
Share on other sites

As much as open flexible general purpose tools are useful, I think for the X16 world that specific purpose tools would serve the users in the community better. 

In the examples given above, each binary file needs a source file to create it. While source code version control and storage have come a long way, it's still maintenance work. Each resource is going to need its own generating script. Then there is a need to have a tool that can present the data in the file for human inspection. 

For specific resources, such as game state data, that's going to be tied to the game itself and most likely requires the approach of general purpose tools.

But for a sprite resource a well designed sprite editor would eliminate a per-sprite development tool by serving to work on any sprite file. It also eliminates the need for a separate tool to look at the resource files. While it would be a stand alone specific purpose application, if it was an open community project then improvements could come from those who have the time, motivation, and the skill set. Everyone would benefit. 

I see the x16 as a "fun/simple" development platform, but for the serious/commercial work being done on a modern computer that overcomes most of the limitations.

Which comes to the question - should X16 development be on a modern platform, or limited to tools on the X16 itself?

Link to comment
Share on other sites

On 9/29/2021 at 1:34 PM, rje said:

... I want AWK on the X16.

AWK on the X16 is the thing most likely to make me learn the ins and outs of CC65. Programming in C is too much like working for a living for me to want to do it on a regular basis, but to get an AWK implemented, I'd make an exception.

However, it might be just System 7 AWK, leaving development toward a One True Awk compatible interpreter to a smarter and/or more ambitious person than I.

I realized that since the System 7 sources are available, the YACC and {that other one which escapes me for a minute} could be compiled and run on their own AWK sources to generate an all-K&R-C set of sources for the System 7 AWK. Then it is a matter of porting to the heavily modified Small C that CC65 uses and to build in replacements for the Unix functions the System 7 assumed ... I imagine a menu based system for loading and saving the AWK script and input and output into their own strings of HighRAM segments could make for a fairly nimble system.

Link to comment
Share on other sites

On 9/29/2021 at 2:35 AM, kliepatsch said:

 

struct_1 = [12, 13, 14]
 

struct_2 = [0, 80, 0, 90, 0, 100]
 

data = struct_1 + struct_2 + struct_1 # appending data is as easy as this.
 

data_bytes = bytes(data)
 

f = open("DATA.BIN","wb") # "wb" stands for write binary
 

f.write(data_bytes)
 

f.close()

 

Hey, that doesn't write bytes to the output channel, it creates a text file containing the Python expression "[12, 13, 14, 0, 80, 0, 90, 0, 10, 0, 12, 13, 14]". The binary file should have 12 bytes in it, since there are 12 byte values in the concatted array. Maybe I'm doing something wrong?

 

And, by the way, the Perl version looks very similar, and also creates a text file, although just the digits and not the Python expression.

@struct_1 = ( 12, 13, 14 );
@struct_2 = ( 0, 80, 0, 90, 0, 100 );
@data = (@struct_1, @struct_2, @struct_1);
open my $fh, ">:raw", "DATA.BIN";
print $fh @data;
close $fh;

To write actual bytes, I still need pack():

@struct_1 = ( 12, 13, 14 );
@struct_2 = ( 0, 80, 0, 90, 0, 100 );
@data = (@struct_1, @struct_2, @struct_1);
open my $fh, ">:raw", "DATA.BIN";
print $fh pack "C*", @data;
close $fh;

 

Edited by rje
  • Like 1
Link to comment
Share on other sites

On 9/29/2021 at 1:40 PM, BruceMcF said:

AWK on the X16 is the thing most likely to make me learn the ins and outs of CC65. Programming in C is too much like working for a living for me to want to do it on a regular basis, but to get an AWK implemented, I'd make an exception.

My online friend Alex had written a very very very low C heap manager, for the purpose of implementing a CSH variant on the C64.  I've got his code, but I didn't use it.  I'm having enough trouble writing a compiler down to interpretable bytecode.  

I attach them here for your amusement.

heap.c heap.h testcsh.c testcsh.notes

Edited by rje
Link to comment
Share on other sites

As for AWK, I search for "practical examples" instead of tutorials that show me how to do uninteresting things.

    <CONDITION> [<ACTION>] <SETTINGS> <SOURCE FILE NAME>

^ This looks like the generic template for AWK statements.  Note I'm using [brackets] because PETSCII doesn't have {braces}.  No worries.

First, let's posit that this particular AWK is only for one-liners. No reason to think so, except for simplicity maybe.

All the settings/field separators etc are fine.

 

 

    1 MY_FILE

Cats the file to the "terminal" (does it pause at the bottom of the screen?)

    'NR>1 && NR < 5' MY_FILE

Cats lines 2 through 4.  Note that the initial condition, if complex, is in single quotes.

    NF MY_FILE

Prints lines that have non-space characters on them.  I.E. ignores lines which only have whitespace.

    1  RS=''  file

Ignores lines that only have the newline character.  Note that the Record Separator assignment is outside of the action, so isn't in brackets.

   '{ print $1, $3}' FS=, OFS=, file

Prints columns 1 and 3 of every line.  Note the input and output field separators are both the comma.

   'NF { print $1, $3 }' FS=, OFS=, file

If the line has fields, then print the columns, otherwise ignore the line.

 

 

* * *

Now.

Can we have multi-line scripts, complete with BEGIN and END options?

I think variables would be no problem, so we could have variable assignment in the action blurbs.

Associative arrays are, I think, just out of reach, without some shoehorning.  I'd defer that until the rest of it works beautifully.

The most interesting parts of regular expressions will probably never work here.  Too big.

 

 

Edited by rje
Link to comment
Share on other sites

On 9/30/2021 at 4:44 AM, rje said:

Hey, that doesn't write bytes to the output channel, it creates a text file containing the Python expression "[12, 13, 14, 0, 80, 0, 90, 0, 10, 0, 12, 13, 14]". The binary file should have 12 bytes in it, since there are 12 byte values in the concatted array. Maybe I'm doing something wrong?

Just tested it, and that's what happens in Python 2. In Python 3 it does indeed write a binary file. In Python 2 it is "bytearray()" instead of "bytes()", then it works as intended. Not sure why "bytes()" doesn't throw an error in Python 2.
 

 

On 9/29/2021 at 7:34 PM, rje said:

Note that he has to first break down larger numbers, and encode strings into bytes.  He might have to do some custom coding if endianness is a problem.

Partially true. You can convert other things into bytearrays. E.g. appending a 16-bit integer to a bytearray is

existing_bytearray += (65535).to_bytes(2,"little")

where the argument 2 states that it should become 2 bytes, and "little" or "big" endian can be used. There's also an option for signed integers. Strings can be dealt with as

existing_bytearray += ("hello world").encode("utf-8")

Although encoding strings to PETSCII may require some extra code...

  • Thanks 1
Link to comment
Share on other sites

On 9/29/2021 at 11:48 PM, rje said:

...

* * *

Now.

Can we have multi-line scripts, complete with BEGIN and END options?

I think variables would be no problem, so we could have variable assignment in the action blurbs.

Associative arrays are, I think, just out of reach, without some shoehorning.  I'd defer that until the rest of it works beautifully.

The most interesting parts of regular expressions will probably never work here.  Too big.

AFAIR, a bit thing V7 awk did not have was user definable functions.

It had constant string regular expressions, but couldn't get a regular expression from a variable.

The differences between the V7 awk and the "One True Awk" is described here.

  • Thanks 1
Link to comment
Share on other sites

On 9/30/2021 at 1:43 PM, BruceMcF said:

AFAIR, a bit thing V7 awk did not have was user definable functions.

It had constant string regular expressions, but couldn't get a regular expression from a variable.

The differences between the V7 awk and the "One True Awk" is described here.

Thank you for the link.  I'm reading it now.

I'm also writing a throw-away thing to do some very careless things in a vaguely AWKlike manner.

  • Like 1
Link to comment
Share on other sites

On 9/30/2021 at 4:56 PM, rje said:

Thank you for the link.  I'm reading it now.

I'm also writing a throw-away thing to do some very careless things in a vaguely AWKlike manner.

The other big thing for some things ... like fully correct csv parsing ... is that since regular expressions cannot be assigned to variables, they cannot be assigned to parsing input fields. So I'd likely have a separate csv-lint program to make sure that the "," separator works.

  • Like 1
Link to comment
Share on other sites

Posted (edited)

Now that I'm thinking about it, I'm also thinking about the reasonable scope for the X16.  These sorts of UNIX-like operations aren't very common on Commodore 8 bit machines.

I've already found the plain-old catenate pattern useful.  I suspect one that matches a substring and prints the matching line and line number would be useful, so a condition to match a string, and an action for printing NR and the line itself would be nice.  I wonder if vprintf() is as easy as it sounds?

 

Edited by rje
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...

Important Information

Please review our Terms of Use