Jump to content

External RS-232 Interface, storage, and second screen.


TomXP411
 Share

Recommended Posts

  • 3 weeks later...
Posted (edited)

Hi, I was also looking for possible options of network connectivity for X16 and stumbled upon this thread.

I read the design doc and very liked it.

I'm not very knowledgeable with hardware, but I would want to share some of my thoughts on the software part of it:

I would make the transport layer between x16 and the external card as abstract and general purpose as possible:
* I would make port numbers purely logical (e.g. 0-255), not bound to any particular physical ports on the external card (e.g. Arduino).
* I would not require to provide parameters like transfer rate from X16, if it's not strictly required for establishing connection between the X16 and the card.
* x16 should not need to know how actual transport between the external card and outside world is implemented (HTTP, USB, COM etc.).

As a programmer I would want:

To have an external card (Arduino or Raspberry) with modern Linux and modern networking stack (Wi-Fi, Ethernet, HTTPS, OpenSSL etc.), and ability to write a server app using any modern programming language (C++, Go, Python etc.) which would listen for incoming connections from the X16, and make requests to outside services using modern protocols (HTTP, SSH, FTP, SMB etc.) with all encryption, authentication, compression etc.

The choice of logical ports between X16 and the server implementation on external card would not be predefined and up to the server implementation, application level protocol also could depend on particular server implementation.

Examples of use cases:

Raspberry Pi with Wi-Fi, connected to internet.

A server application which listens to logical port 100 from X16, as input it accepts URL string, as output it returns binary stream of downloaded file.
On X16, code just opens the port, sends URL strings, reads the incoming byte stream. X16 doesn't know how the actual protocol implemented.

A server application on port 110 provides a lightweight binary protocol for remote file management to X16 (list files, read file, write file, copy file, delete file etc.), and works as a client for SMB or FTP protocols, and makes it possible to access network files from X16.

A server application on port 112 provides primitive telnet protocol to X16 which is being converted to secure SSH connection to outside system.

A server application on port 115 could bridge a game in X16 with the game server in internet.

Edited by kostrse
Link to comment
Share on other sites

Posted (edited)

Those are some good notes. I'll address them:

13 hours ago, kostrse said:

I would make the transport layer between x16 and the external card as abstract and general purpose as possible

The connection is currently broken down into an Address and a Data channel. The Address channel is used to select the active port and to control serial parameters. The Data channel is completely transparent, so you can shove through whatever data you want - text, binary data, or even data meant to be UDP or TCP payloads. 

13 hours ago, kostrse said:

* I would make port numbers purely logical (e.g. 0-255), not bound to any particular physical ports on the external card (e.g. Arduino).

Port numbers are entirely at the discretion of the external hardware (ie Arduino). There's nothing in the design or specification that requires port 1 to be the USB port or port 2 to be a serial port or anything like that. If you use an Arduino Mega, there is plenty of room to have several ports of different types, including USB, RS-232, SPI, and TTL serial to a network interface. 

i do NOT plan to directly support a network transceiver, so this should not be used for TCP/UDP data. Instead, you would need to build a smart device externally to handle that. The preferred method at this point would be an ESP32 or ESP8266 running an AT firmware (aka a modem emulator.)

13 hours ago, kostrse said:

* I would not require to provide parameters like transfer rate from X16, if it's not strictly required for establishing connection between the X16 and the card

The purpose of this interface is to connect to RS-232 devices. Other layer 1 protocols, such as I2C or SPI are compatible enough at the data layer, but you should still send the RS-232 initialization sequence, to ensure that if the connection is to an RS-232 device, the UART is configured correctly. Having said that, I expect there to be a default connection speed, which will be 9600,n,8,1 in my reference driver. 

Having said that - SPI connections will simply ignore the parameters that don't make sense for it. SPI does not have a "baud rate", but it does have a maximum speed cap, so I plan to add some basic SPI parameters into the protocol later. I'll probably also work in some way to set the I2C address to allow sub-addressing on an I2C connection. 

13 hours ago, kostrse said:

To have an external card (Arduino or Raspberry) with modern Linux and modern networking stack (Wi-Fi, Ethernet, HTTPS, OpenSSL etc.), and ability to write a server app using any modern programming language (C++, Go, Python etc.) which would listen for incoming connections from the X16, and make requests to outside services using modern protocols (HTTP, SSH, FTP, SMB etc.) with all encryption, authentication, compression etc.

The choice of logical ports between X16 and the server implementation on external card would not be predefined and up to the server implementation, application level protocol also could depend on particular server implementation.

That's a perfectly reasonable use case. In fact, it's one of my expected use cases - using a Pi as an external UART is exactly the kind of this this standard is designed to for.

However, the reference firmware will run on an Arduino Mega, which has 4 serial ports. What you do with it is up to you, but 0-3 should generally be used for RS-232. Use other port numbers as you wish. The important thing is that your software should allow the user to set any port number, and the software should always have the ability to set all of the standard parameters, to allow for any Layer 1 transport to the communication device. (ie: a terminal program needs to set not only baud rate, but should also be able to set SPI speed and I2C channel, in the event someone makes a network modem that runs over SPI.) Never expect that a communication device will run on just the Layer 1 you expect.

If you want to support additional standard protocols and port numbers, it's reasonable to create a list of suggested port numbers for specific purposes. For now, I want to keep that loose, and if you always allow the user to pick a port number and communication parameters, there should be no conflicts with this use case.

 

Edited by TomXP411
  • Like 1
Link to comment
Share on other sites

As a side note, I'm moving away from the idea of patching the KERNAL to use device 2, and I plan to remove all of that from the documentation (or add it as an addendum.) 

Instead, the reference driver will be accessed via a SYS command, with a jump table at $410. 

$410: Read status from device (updates the buffer counts)
$413: Send a byte 
$416: Read a byte
$419: Send a parameter to the device (parameter in $403, data in $404-40F)
$41C: Read a parameter from the device
$41F: Send a BASIC string
$422: Receive data into a BASIC string

This routine should be fairly small, and I expect to provide binaries assembled for various places in the $400-800 block and at $A000. 

 

Link to comment
Share on other sites

58 minutes ago, TomXP411 said:

As a side note, I'm moving away from the idea of patching the KERNAL to use device 2, and I plan to remove all of that from the documentation (or add it as an addendum.) 

Instead, the reference driver will be accessed via a SYS command, with a jump table at $410. 

$410: Read status from device (updates the buffer counts)
$413: Send a byte 
$416: Read a byte
$419: Send a parameter to the device (parameter in $403, data in $404-40F)
$41C: Read a parameter from the device
$41F: Send a BASIC string
$422: Receive data into a BASIC string

This routine should be fairly small, and I expect to provide binaries assembled for various places in the $400-800 block and at $A000.

Why $403-$424? Has somebody yelled dibs on $400-$401?

Link to comment
Share on other sites

7 hours ago, BruceMcF said:

Why $403-$424? Has somebody yelled dibs on $400-$401?

I’m using $400-40F for data and control structures. To use the interface from BASIC, you poke values into the $400-40F range, then SYS the routine. 

  • Like 2
Link to comment
Share on other sites

So out of curiosity, why not just design an expansion card and stick it straight on the data / address bus? There is plenty of space in the IO area for the chip and something like the SC28L92 only has 4 registers lines so technically it could fit into half of one of the available expansion ranges. 

This is the chip I chose for my project and I intend on using it with the X16 as well once I get mine 🙂

Link to comment
Share on other sites

Because I want something anyone can assemble with off the shelf parts in a few minutes. My design uses no proprietary parts and can use just about any commonly available microcontroller board. 

  • Like 2
Link to comment
Share on other sites

Good news! I just found a GIthub repo of various chip emulators, including 6502 and 6522 emulations

https://github.com/floooh/chips

I'll probably load this up on a Raspberry Pi or two, which should give me a good starting point for testing. 

In fact, the more I think about it, the more I like the Raspberry Pi for this kind of interface. The Pi Zero is cheap, and it can support wireless and even act as an external storage device and second screen! So with a simple file manager, you could copy files onto the computer from USB or the network.

 

Link to comment
Share on other sites

Posted (edited)

Update: I played around with two Pis today, and I have got them communicating. I started out using Python to test communication and make sure I could get data moving back and forth. That works fairly well, although I'm limited to around 100 characters per second that way. Python is just too slow to do a reliable job. 

I've also done some simulated SIO testing on the CX16 emulator, and I get roughly 50KB/sec that way. (kilobytes, not kilobits.) Quite frankly, that's faster than the internal SD card reader, and I'm giving more thought to turning this into a general purpose I/O module, with disk storage, network connectivity, and serial port access all in one unit. 

 

 

 

Edited by TomXP411
  • Thanks 1
Link to comment
Share on other sites

Well, some more experimentation and research leaves me disappointed. I can only seem to get a few KB/sec out of the Pi using the GPIO pins in C. I can reliably read the pins at low data rates, but as soon as I get  the rates up to where I want them to be (50KBbytes/s or more), the software just won't drive the pins fast enough on the transmit side, and I keep dropping clocks on the receive side. 

So I'm going to go back to using an Arduino or a Pi Pico as a server/SPI slave. Fortunately, I've got MCU boards coming out my ears right now: Adruino Mega, Grand Central Teensy 3.5, Teensy 4.1, and Pi Pico.

The Pico is promising, as it has multiple channels that can be used for hardware UARTS, including SPI modes. If that works reliably as a slave and client at the same time, I can still use it as an external SD card reader and comm hub...

Link to comment
Share on other sites

I've not been thinking about this as long as you have, but it seems to me (now that I'm doing the math for the first time) that 50 KB/sec is 400000 bps, which means one bit every 20 clock cycles. I'm assuming this isn't interrupt driven. One bit per 20 cycles seems doable if all you want to do is read bits, but assuming you want to do something useful with the bits, that just seems (again, based on intuition and not necessarily logic) that is just not achievable.

I know, that's basically what you said (that it isn't achievable by saying it isn't working). I'm just trying to do the math to see if I understand.

With a real 16550A or some such, with a 16 byte buffer that can signal an interrupt at some point so that you only have to process a batch of data (say 8 bytes) you would only have to read every 1280 cycles, but having to bit bang puts a lot more demands on the system.

 

Link to comment
Share on other sites

4 hours ago, TomXP411 said:

I can only seem to get a few KB/sec out of the Pi using the GPIO pins in C. I can reliably read the pins at low data rates, but as soon as I get  the rates up to where I want them to be (50KBbytes/s or more), the software just won't drive the pins fast enough on the transmit side, and I keep dropping clocks on the receive side. 

Yeah, I believe @Lorin Millsap mentioned something about that limitation of using the high-level libraries quite some time ago.

Link to comment
Share on other sites

19 minutes ago, Scott Robison said:

I've not been thinking about this as long as you have, but it seems to me (now that I'm doing the math for the first time) that 50 KB/sec is 400000 bps, which means one bit every 20 clock cycles. I'm assuming this isn't interrupt driven. One bit per 20 cycles seems doable if all you want to do is read bits, but assuming you want to do something useful with the bits, that just seems (again, based on intuition and not necessarily logic) that is just not achievable.

I know, that's basically what you said (that it isn't achievable by saying it isn't working). I'm just trying to do the math to see if I understand.

With a real 16550A or some such, with a 16 byte buffer that can signal an interrupt at some point so that you only have to process a batch of data (say 8 bytes) you would only have to read every 1280 cycles, but having to bit bang puts a lot more demands on the system.

 

50kBps is easily achievable with an 8 bit parallel SPI interface. Remember this interface is parallel, so that’s one baud unit per 160 clock cycles, or roughly 32 instructions per baud unit. That should be eminently achievable on the Commander side when the Commander is driving the clock. 

the issue seems to be that the software that multiplexes the GPIO pins on the Pi is just not optimized for speed, and since the hardware transceiver can’t act as an SPI slave, the Pi is no longer in consideration for this use. 

Link to comment
Share on other sites

1 hour ago, Scott Robison said:

I've not been thinking about this as long as you have, but it seems to me (now that I'm doing the math for the first time) that 50 KB/sec is 400000 bps, which means one bit every 20 clock cycles. I'm assuming this isn't interrupt driven. One bit per 20 cycles seems doable if all you want to do is read bits, but assuming you want to do something useful with the bits, that just seems (again, based on intuition and not necessarily logic) that is just not achievable.

Also, what you’re describing is exactly the reason I thought to build an external UART in the first place. 
The reason I chose to do this with a microcontroller is that anyone can build this with just DuPont wires and an Arduino or Raspberry Pi. But I may yet end up switching to a solution using a hardware bit shifter. That will require PCB or breadboard work, though, which defeats the purpose of this design. 

Link to comment
Share on other sites

Maybe I'm just not thinking about it. I understand why you are using an external device as the UART, but to get the X16 to read the data, it still has to process the data at some point. Is it trying to bit bang, or is it trying to do a byte at a time? My timing assumed one bit at a time, but if the interface you were attempting could do 8 bits parallel between the pi and the X16, then suddenly we're talking about something different. I was assuming (probalby due to lack of critical reading) that this serial interface was bit oriented, thus needing to read a bit every 20 cycles, but if the bits are bunched up into a byte that you read as one atomic unit on the X16 then you're up to 160 cycles per byte.

Anyway, not trying to be critical, just jumping late into a discussion trying to understand. Even when I worked for the PCBoard company back in the day, I didn't worry about serial interfaces, that was another programmers job. When it comes to the X16, I'm thinking of Commodore software bitbanged serial port or IEC, bit at a time style serial which is probably not the method you're using.

Link to comment
Share on other sites

Posted (edited)
1 hour ago, Scott Robison said:

Maybe I'm just not thinking about it. I understand why you are using an external device as the UART, but to get the X16 to read the data, it still has to process the data at some point. Is it trying to bit bang, or is it trying to do a byte at a time? My timing assumed one bit at a time, but if the interface you were attempting could do 8 bits parallel between the pi and the X16, then suddenly we're talking about something different. I was assuming (probalby due to lack of critical reading) that this serial interface was bit oriented, thus needing to read a bit every 20 cycles, but if the bits are bunched up into a byte that you read as one atomic unit on the X16 then you're up to 160 cycles per byte.

Anyway, not trying to be critical, just jumping late into a discussion trying to understand. Even when I worked for the PCBoard company back in the day, I didn't worry about serial interfaces, that was another programmers job. When it comes to the X16, I'm thinking of Commodore software bitbanged serial port or IEC, bit at a time style serial which is probably not the method you're using.

I will check the OP again and make sure it’s clear, but my interface is an 8 bit parallel interface, loosely based around bidirectional SPI (which uses the same wire for data in both directions). With additional control wires for clock, data direction, signaling the buffer status, and for invoking command mode. 

** edit: clarified in the post, but please visit the link in the OP and read the draft specification. 

Edited by TomXP411
  • Like 1
Link to comment
Share on other sites

13 minutes ago, TomXP411 said:

I will check the OP again and make sure it’s clear, but my interface is an 8 bit parallel interface, loosely based around bidirectional SPI (which uses the same wire for data in both directions). With additional control wires for clock, data direction, signaling the buffer status, and for invoking command mode. 

Forgive me ... these threads stretch out over such a period of time that I missed earlier context. When I think Commodore I just naturally go to bit banging serial bit delivery and don't do my due diligence.

Link to comment
Share on other sites

Just now, Scott Robison said:

Forgive me ... these threads stretch out over such a period of time that I missed earlier context. When I think Commodore I just naturally go to bit banging serial bit delivery and don't do my due diligence.

🙂  No problem. It's understandable. I did add some specifics to the original post, but there's also a link to a Google Doc that explains the whole thing in more detail. 

 

Link to comment
Share on other sites

5 hours ago, Scott Robison said:

Forgive me ... these threads stretch out over such a period of time that I missed earlier context. When I think Commodore I just naturally go to bit banging serial bit delivery and don't do my due diligence.

Notice that the CX16 User Port has all of one VIA parallel port A and most of it's parallel port B, so there's a lot more flexibility in setting up a half duplex parallel port on the User Port.

Link to comment
Share on other sites

Hi there! I'm very interested in connecting the X16 to internet, expecially 1) using cheap off the shelf parts (esp32 or similar) and 2) with the most ease in code.

Also, i think it sould be important to have some sort of "unified interface" so that programs can take advantage of whatever kind of device is available without knowing the details. If a channel works as a modem, then it could have its own custom AT commands and behaviours, probably breaking the abstraction one way or another.

So i was wondering if it would be possible to create a "tcp socket" device, where the filename is an IP address. It would need to be managed by the microcontroller, but it shouldn't be hard at all. It would work like this:

OPEN 1, 20, 0, "142.250.184.100:80"

OPEN 2, 20, 0, "api.server.com:555"

The microcontroller will open the socket connection and associate it with the file number. Subsequent read or write will be routed to the socket, and errors can be mapped to the usual FILE_NOT_FOUND, DEVICE NOT PRESENT etc.

The only problem is that filenames are 16 character long AFAIK, so they can hold any ip address (15 chars top) but not the port. Using the secondary address as port is a problem becouse i think it's at most 128. Now an address+port is 6 bytes so it CAN be fitted in 16 char, for example using hex (C345A100:00C1), but it's pretty unelegant. It would require a conversion routine which is annoying. Perhaps someone has some suggestion (if the whole proposal makes any sense :P). 

What i'm afraid most is that different hardwares will require different setups, forcin programs to have vendor-specific code to deal with all the possible modems. That's one aspect of the 8 bit era that i'm NOT eager to revive 🙂

 

(I hope it's not OT but this was the most fitting post i could find)

Link to comment
Share on other sites

I think a better way would be to have a channel for socket creation / teardown, and then a file for each open socket with some arbitrary string as the socket identifier.

so in the control channel, you send some string like cmd:proto:host:port (e.g. O:T:www.example.com:80 or O:U:tftp.example.com:69)

The device would reply with the "filename" of the socket, or an error code if unable to create the socket.

commands could be O, C, L for Open, Close, and Listen. Listen would be a way to run a server on the X16 locally. Each new incoming connection would generate a new socket file and an IRQ.

 

Link to comment
Share on other sites

Posted (edited)

You’re really talking about new firmware for an ESP32 or perhaps an Ethernet controller. That’s probably a topic for a new thread, since this thread is specifically about developing a serial port multiplexer, not a dedicated network interface. 

Edited by TomXP411
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...

Important Information

Please review our Terms of Use