(SOLVED) How to create an SD card image for X16?

[StephenHorn]

Right now I'm struggling to recall whether the emulator even supports (in a meaningful way) mounting an SD card image to use with the X16. But setting that aside for the moment, I have an even more basic question:

How are folks creating and maintaining that image in the first place? In my own Googling, nobody seems to tackle the subject of how to create an image file for an SD card without first having a physical SD card, chock full of data they want to convert into an image file. They all start with some variation of "in order to create a backup of your SD card..."

And I'm perfectly willing to concede that >99% of cases want to create a backup of an existing SD card. That's entirely reasonable, and I get it. But what if, say, I'm developing software for the X16 and will be repeatedly generating new data files? Do I leave a USB stick with a mostly-blank SD card inserted into my PC, copy my stuff onto the stick, invoke a third-party application to generate a new image file from the card, and then specify that file to the emulator to mount it on the X16?

Surely, there's a better way that doesn't involve the hardware acting as a middle man, and I'm just not finding it. Any ideas?

 

Edit: Solutions! Thank you to everyone who helped out!

 I can't favorite/upvote all of them, so I'm going to link and briefly explain them here:

For my peculiar purposes, (Linux and WSL2 on Windows, no superuser permissions required), @Michael Parson came up with the gold standard answer, by ultimately sleuthing out how to build the image and then wrote a bash script to automate the process:

Special thanks to @lamb-duh for helping us get onto the right track, as far as command-line tools were concerned. With that, my Makefile-based workflow is back to being "make && make run".

If you're not a fan of Linux, @TomXP411 describes a way to create and mount images purely through the Windows 10 VHD feature:

 

Edited January 9, 2021 by StephenHorn
Solved!

[Michael Parson]

OK, worked up a script for creating a 32MB image that can be used with the emulator, no root needed.

$ cat bin/mkcard
#!/bin/bash
printusage() {
cat - <<EOF
$0 [-f filename] [-s size-in-megabytes]

To create a 64 MB file:

$0 -f card.img -s 64
EOF
}
# print usage if we don't have the right number of arguments
if (($# != 2 )); then
    printusage
    exit 0
fi

while getopts "f:s:" opts; do
    case $opts in
        f )  diskimg="$OPTARG"
             ;;
        s )  size="$OPTARG"
             ;;
        * )  printusage
             exit 0
             ;;
    esac
done

alignment=1048576
size=$((size *(1<<20)))
size=$(( (size + alignment - 1)/alignment * alignment))

# create the file
truncate -s $((size + 2*alignment)) "${diskimg}"

# partition the file
echo ',,c;' | sfdisk ${diskimg}

# create the filesystem
mformat -F -i ${diskimg}@@$alignment

For extra fun, have multiple images available to mtools:

$ cat ~/.mtoolsrc
mtools_skip_check=1
drive s: file="~/scratch.img" exclusive partition=1
drive x: file="~/basic-progs.img"    exclusive partition=1
drive y: file="~/demos.img"    exclusive partition=1

And now you can use the mtools to move files back and forth from the different virtual cards, just make sure your UPPERCASE your filenames when you copy files into the images.

************

Post accepted solution edit:

I've uploaded a "polished" version of the script to github.

Edited January 9, 2021 by Michael Parson
updated the script & added more info about .mtoolsrc

[JimmyDansbo]

Have a look at the Readme for the emulator:

https://github.com/commanderx16/x16-emulator#sd-card-images

edit: It seems the link I have provided does not actually go to the SD card section of the readme, you have to scroll down and find it your self.

Edited January 6, 2021 by JimmyDansbo
Link inaccurate

[desertfish]

I'm on Linux, and use the loopback device mounting as pointed out in that emulator readme.  I'm just copying files into a mounted folder then.
I started with the downloaded img file from github, although it should be possible to create a blank file of the appropriate size and then actually formatting it on the loopback device with FAT32 but I haven't tried that yet.

The only snag I ran into was when I had it open in the emulator at the same time as having it mounted via loopback and then writing some data corrupted the directory.

 

[Michael Parson]

First, you need a file of appropriate size, for a 1G file:

$ dd if=/dev/zero of=card.img bs=10240 count=102400
102400+0 records in
102400+0 records out
1048576000 bytes (1.0 GB, 1000 MiB) copied, 9.53099 s, 110 MB/s

$ ls -lh card.img
-rw-rw-r-- 1 mparson mparson 1000M Jan  6 08:17 card.img

Now, it needs partitions and a filesystem, but first, it needs to be setup as a loopback device:

$ sudo losetup /dev/loop0 card.img
[sudo] password for mparson:

Now we can create the partition:

$ sudo fdisk /dev/loop0

Welcome to fdisk (util-linux 2.35.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xc071d876.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-2047999, default 2048):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-2047999, default 2047999):

Created a new partition 1 of type 'Linux' and of size 999 MiB.

Command (m for help): t
Selected partition 1
Hex code (type L to list all codes): b
Changed type of partition 'Linux' to 'W95 FAT32'.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.

For good measure, remove and re-create the loopback device (might not be necessary, but it does force the kernel to re-read the partition table):

$ sudo losetup -D
$ sudo losetup /dev/loop0 card.img

Now, put a FAT32 filesystem on the new partition:

$ sudo mkfs.fat -F32 /dev/loop0p1
mkfs.fat 4.1 (2017-01-24)
$ sudo mount /dev/loop0p1 /mnt
$ df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0p1    998M  4.0K  998M   1% /mnt
 

This method requires that you copy the files to the /mnt mount point as root, best practice would be to use 'sudo cp MYFILE.PRG /mnt'

 

As others have noted, you'll want to unmount it before using it with the emulator:

$ sudo umount /mnt

Edited January 9, 2021 by Michael Parson

[lamb-duh]

17 hours ago, Michael Parson said:

Now, it needs partitions and a filesystem, but first, it needs to be setup as a loopback device:

If you don't create a partition table, you can skip a step manually creating the loopback device--- `mkfs.vfat` will ~~partition~~ format a regular file, and then when you mount it you can pass `-o loop` to automatically create the loopback device (which will automatically be removed when the filesystem is unmounted).

Also, a very useful option when mounting a fat filesystem: `-o uid=your-username` so that your regular user account is the owner of the mounted filesystem, instead of root.

 

There's also a tool that can copy file into/out of fat images without mounting it (viz. without root permissions) but I'm completely blanking on the name. if no one else knows what I'm talking about, I'll be back when I wake up in the middle of the night suddenly remembering..

Edited January 7, 2021 by lamb-duh

[StephenHorn]

8 hours ago, lamb-duh said:

If you don't create a partition table, you can skip a step manually creating the loopback device--- `mkfs.vfat` will partition a regular file, and then when you mount it you can pass `-o loop` to automatically create the loopback device (which will automatically be removed when the filesystem is unmounted).

Also, a very useful option when mounting a fat filesystem: `-o uid=your-username` so that your regular user account is the owner of the mounted filesystem, instead of root.

 

There's also a tool that can copy file into/out of fat images without mounting it (viz. without root permissions) but I'm completely blanking on the name. if no one else knows what I'm talking about, I'll be back when I wake up in the middle of the night suddenly remembering..

I believe I found the tool you were talking about: mcopy.

So it looks like I needed to install dosfstools and mtools:

$ sudo apt-get install dosfstools
$ sudo apt-get install mtools

From there, dd can create the image file, mkfs.vfat can initialize it, and mcopy can copy files into it:

$ dd if=/dev/zero of=sdcard.img bs=1K count=10K
$ /sbin/mkfs.vfat sdcard.img
$ mcopy -i sdcard.img build/* ::

mdir (part of the mtools package containing mcopy) can verify that the files are copied:

$ mdir -i sdcard.img

<list of all the files and directories at the root of the image>

I can additionally verify that the SD card image was created correctly by mounting it to some location:

$ mkdir sdcard
$ sudo mount -o loop -o uid=myuser sdcard.img ./sdcard
$ cd sdcard
$ ls -la

<list of all the files>

Now I'm just struggling with the dumbest part, which may be a case-sensitivity issue or something, but when I launched the emulator and it said "SD card attached", I couldn't subsequently load my PRG. And when I attempted LOAD"$",8 the kernal froze up and I got a warning message from the emulator's stdout that read "Warning: short read!"

So I've done something dumb, I'm sure.

Edit: And yes, I remembered to unmount the file before opening it with the emulator.

Edit2: I suppose one question worth asking is whether the emulator/kernal can handle arbitrary VFAT image sizes. You can see from my commands above that the image I created was only 10MiB, which is presently ample but obviously I could recreate the image later with a larger size if I needed to (in fact, my intent is to put the meat of the work into a makefile that just re-creates the whole SD card image from scratch on each rebuild, so it's definitely a bonus to avoid any commands that require sudo). If the emulator or kernal require some minimum image size, that would explain the "Warning: short read!"

Edited January 7, 2021 by StephenHorn

[Ender]

53 minutes ago, StephenHorn said:

And when I attempted LOAD"$",8 the kernal froze up and I got a warning message from the emulator's stdout that read "Warning: short read!"

I think there's currently a bug with the FAT32 driver in the kernal that LOADing "$" causes it to freeze.  DOS"$" should work though.

Edited January 7, 2021 by Ender

[Michael Parson]

Yeah, I was playing around with that too.  Seems that the emulator wants there to be a partition on the (virtual) card, so, to use a partitioned image file, you'll need to set up a ~/.mtoolsrc like this:

$ cat .mtoolsrc
mtools_skip_check=1
drive x: file="~/card.img" exclusive partition=1

Now you can copy files to/from your virtual "X:" drive:

$ mcopy myfile.prg x:

And to see the files on the new 'X' drive:

$ mdir x:
	Volume in drive X has no label
Volume Serial Number is 1F4D-A765
Directory for X:/
MAD-CX16          9977 2021-01-07    2:29
00001820 ~1~        32 1980-00-00    0:00 HELLO
        2 files              10 009 bytes
                      1 070 559 232 bytes free

"HELLO" is a short BASIC program I did as a test save in the emulator.

Edited January 7, 2021 by Michael Parson

[StephenHorn]

For some reason, mkfs.vfat doesn't want to respect the partition table written to the file by sfdisk, it just clobbers it and turns the whole image into a vfat filesystem. I take it that whenever you've run mkfs.vfat, you had the image mounted to the OS via the 'mount' command, and mkfs was pointed to the mounted directory?

[lamb-duh]

2 hours ago, StephenHorn said:

I believe I found the tool you were talking about: mcopy.

So it looks like I needed to install dosfstools and mtools:

that's the one! I remember now why I wasn't able to find it.. it has a useless name and is packaged with a completely different name.

 

1 hour ago, StephenHorn said:

For some reason, mkfs.vfat doesn't want to respect the partition table written to the file by sfdisk

yeah, I think that's just not a feature of mkfs.vfat. I'm not sure of a particularly good way to get a partition table without creating the loopback device. (I guess you could create a standalone filesystem, then use dd to attach a premade partition table to it)

[StephenHorn]

40 minutes ago, lamb-duh said:

yeah, I think that's just not a feature of mkfs.vfat. I'm not sure of a particularly good way to get a partition table without creating the loopback device. (I guess you could create a standalone filesystem, then use dd to attach a premade partition table to it)

Yeah, looks like that's what I'm going to have to do. So for instance, if I wanted to create a 10MB SD Card image, I would do the following:

dd myself an 10 MB file and a 9MB file.
sfdisk the 10MB file to have a single partition.
mkfs.vfat the 9MB file into a VFAT image.
mcopy everything into the 9MB file.
dd the 9MB into the 10MB file, starting at the 1MB offset.

And I've found some scripting that folks say does pretty much that, only they're working with ext2 partitions. But in principle, it should work the same way, right?

I'll give it a shot sometime tomorrow.

[lamb-duh]

if the filesystems you want to create are really that small, you can go as small as 9MB+512 bytes. The MBR partition table takes up exactly one sector on a disk (the actual partition table is, iirc, 8 bytes long, you can fill the rest of the sector with anything you want).

I remember reading somewhere that commander dos works with fat-32 specifically. If you create a 9MB filesystem with mkfs.vfat, you will get fat-8 by default. You might have to do `-F 32` when you create the filesystem.

The difference you'll find between the fat and the ext2 tools is that whereas the fat tools require you to create a filesystem of a particular size and then copy files onto it, with the ext2 tools, you can create a filesystem with the files you want in one step.

Also, mkfs.ext2 accepts a command line argument to start the filesystem at a specific offset, a feature missing from mkfs.vfat. However, the script you have probably uses genext2fs which doesn't seem to have the option. (so it may or may not have a solution that will work with the fat tools)

Edited January 7, 2021 by lamb-duh

[StephenHorn]

Yeah, I wondered about that and when I made sure to be creating FAT32 I quickly ran into the problem that my partition was, in fact, too small. I needed to pad it out to 40MB, which is quite a bit more than I expect to need for a long while, but at least it's not gigabytes.

However, there was no joy in Mudville, in any case. I no longer get the warning about short reads, but I also can't find any files in the image. LOAD"$":LIST gives me nothing, DOS"$" gives me nothing, OPEN 8,8,0,"$" leaves STATUS at 2.

And since my DOS partition is now a separate file that I'm dd'ing into my SD Card image, I've tried mounting both in different attempts and neither works. The DOS image directly gives me file-not-found errors.

I wonder if I'm not placing the DOS image at the proper location in the image file. Or what I might be doing wrong from here.

In any event, I'm super tired now so I'm definitely calling it a night.

Edited January 7, 2021 by StephenHorn

[lamb-duh]

I can take a look at your script and/or image files, if you want

[JimmyDansbo]

Quote from emulator readme:

"Images must be greater than 32 MB in size and contain an MBR partition table and a FAT32 filesystem"

You can not just format an image file, you do need to partition it.

[TomXP411]

On 1/5/2021 at 9:38 PM, StephenHorn said:

Right now I'm struggling to recall whether the emulator even supports (in a meaningful way) mounting an SD card image to use with the X16. But setting that aside for the moment, I have an even more basic question:

How are folks creating and maintaining that image in the first place? In my own Googling, nobody seems to tackle the subject of how to create an image file for an SD card without first having a physical SD card, chock full of data they want to convert into an image file. They all start with some variation of "in order to create a backup of your SD card..."

And I'm perfectly willing to concede that >99% of cases want to create a backup of an existing SD card. That's entirely reasonable, and I get it. But what if, say, I'm developing software for the X16 and will be repeatedly generating new data files? Do I leave a USB stick with a mostly-blank SD card inserted into my PC, copy my stuff onto the stick, invoke a third-party application to generate a new image file from the card, and then specify that file to the emulator to mount it on the X16?

Surely, there's a better way that doesn't involve the hardware acting as a middle man, and I'm just not finding it. Any ideas?

 

Edit: There is a caveat. I'm not using LOAD because I'm wanting to hold open a file for an extended period of time and only read data out of it on-demand, in a piecemeal fashion. So I quite specifically don't want to have the whole thing brought into banked memory or VRAM all at once.

Are you on Windows? Here's how you do it: 

• Open Computer Management (click start and type the name)
• Select Disk Management on the left side
• Pull down Action and select Create VHD
• Enter the location and filename in the box. 
• Select VHD (not VHDX) and Fixed Size
• Make the file at least 40MB. 
• This will create the VHD and attach it as a drive in the list on the bottom half of the Disk Management window.
• Right click in the box on the left column for that new disk.
• Select Initialize. 
• Select MBR and click OK
• Now right click on the RIGHT side (the 40MB Unallocated side) and select New Simple Volume
• Use the defaults on the first page and second page of the wizard (click Next twice)
• In the third page, select Format the volume with the following settings
  • File System: FAT32
  • Allocation unit size: Default
  • Volume label: whatever you want
  • Perform a quick format
• Click Next

You should now have a new drive letter. Copy files in, then eject it by right-clicking the drive letter in File Explorer and selecting Eject. (You can't mount it in the emulator until you eject it from Windows.)

 

[Michael Parson]

Posted an updated & downloadable version of the above script to the dev tools section.

[StephenHorn]

Looks like one of the things I was doing wrong was incorrectly partitioning the image file as type 7 (exFAT) instead of type c (WIN95 FAT32). That's probably a critical oops unto itself. Also, being able to format the partition with mformat, as opposed to trying to create it with mkfs.fat, is a fantastic shortcut to avoid having to create a separate file.

[Michael Parson]

20 minutes ago, StephenHorn said:

Looks like one of the things I was doing wrong was incorrectly partitioning the image file as type 7 (exFAT) instead of type c (WIN95 FAT32). That's probably a critical oops unto itself. Also, being able to format the partition with mformat, as opposed to trying to create it with mkfs.fat, is a fantastic shortcut to avoid having to create a separate file.

Yeah, I was bumping against the fstype problem with mformat for a bit too.  Was getting ready to post the original version of the script to use losetup & mkfs.vfat, but I decided to take one more read of the mformat man page and found that magic '-F' flag.

[DusanStrakl]

19 hours ago, TomXP411 said:

Are you on Windows? Here's how you do it: 

• Open Computer Management (click start and type the name)
• Select Disk Management on the left side
• Pull down Action and select Create VHD
• Enter the location and filename in the box. 
• Select VHD (not VHDX) and Fixed Size
• Make the file at least 40MB. 
• This will create the VHD and attach it as a drive in the list on the bottom half of the Disk Management window.
• Right click in the box on the left column for that new disk.
• Select Initialize. 
• Select MBR and click OK
• Now right click on the RIGHT side (the 40MB Unallocated side) and select New Simple Volume
• Use the defaults on the first page and second page of the wizard (click Next twice)
• In the third page, select Format the volume with the following settings
  • File System: FAT32
  • Allocation unit size: Default
  • Volume label: whatever you want
  • Perform a quick format
• Click Next

You should now have a new drive letter. Copy files in, then eject it by right-clicking the drive letter in File Explorer and selecting Eject. (You can't mount it in the emulator until you eject it from Windows.)

 

Thanks for these instructions Tom, this works perfectly and creates an image that can be used by standard Windows tools and mounts to X16 Emulator.

 

[TomXP411]

18 hours ago, DusanStrakl said:

Thanks for these instructions Tom, this works perfectly and creates an image that can be used by standard Windows tools and mounts to X16 Emulator.

I also prefer that over the IMG format, since - as you said - mounting a VHD in Windows is simple and painless.

Edited January 8, 2021 by TomXP411

[StephenHorn]

@Michael Parson That script works beautifully, and as I mentioned in the latest edit to my original post, it's an absolute gold standard answer for what I was looking for. Many thanks!

Do you mind if I include a customized version of your script in my public Github repo, under an MIT license?

[Michael Parson]

I'm glad it solved your problem.  I'm a sysadmin, by trade, but now manage a team and don't have as much time for script as I used to, so this was fun.  I'm not much of a programmer, outside shell stuff needed to do my job and stupid BASIC tricks and just recently started playing around in C again, so, it's fun to be able to contribute however I can to this project.

I went ahead and stuck it up in my github repo under a CC-SA license.  Feel free to do whatever with it now.

Edited January 9, 2021 by Michael Parson