DrawBridge aka Amiga Floppy Disk Reader/Writer

This project continues the work from my previous disk reader project at https://create.arduino.cc/projecthub/projects/485582/

For more information visit http://amiga.robsmithdev.co.uk

• My Aim: To create a simple, cheap and open source way to recover and rewrite data from and to Amiga DD floppy disks from within Windows 10.

• My Solution: An Arduino sketch + a Windows application (which can be ported to other O/S) that actually works!

• Why: To preserve data from these disks for the future. Also, a normal PC can't read/write Amiga disks due to the way they are written.

Writing Data - Attempt 1

So after successfully being able to read disks, I figured if you want to keep the original physical medium, you might want to write disks back again. I figured I'd work this out in reverse, starting with the software (i.e.: converting the ADF disk files into MFM data for the interface to write somehow).

So I started by adding classes to read an ADF disk, and encode all the sectors as one track. Knowing I could potentially test the data I created by feeding it back into the decoding part, I started work on this. While working on this I decided to try to find out what was wrong with my Amiga. After all, I can't test any disks I create if I don't have anything real to test them on.

Taking my A500+ apart, I noticed it had suffered one of the most common problems, the clock battery had leaked everywhere. So I desoldered this from the board and set about cleaning the board up. Whilst at it, I pulled the entire machine out and set about cleaning up 20 years of dust and grime. I even took the floppy drive apart to clean it.

Whilst cleaning it, I decided it was time to get rid of the yellowing, so I followed the information about Retr0brite and tried it.

I then checked all of the joints on the main motherboard and found a loose connection by the power connector, a few touchups with the soldering iron and as good as new. I waited until I was happy with the Retr0brite process before reassembling the computer.

Meanwhile I continued working on the code for writing disks. I wanted to read the status of the write protect line, but no matter what I set it to it didn't seem to change voltage. So I pulled the drive apart and followed the traces from the little switches that detect the write protect status to a little IC. At this point I guessed that the output is probably only available when you actually want to write data.

After a lot of experimentation, I found that you needed to pull the /WRITE_GATE pin LOW before spinning up the drive to enable writing. At this point you could obtain the write protect status. I also noticed that while the /WRITE_GATE was low the drive didn't switch back off like it used to until that pin had returned to its default HIGH state.

The Amiga would write an entire track in one go. A track in memory is 11*512 bytes (5638 bytes), however, after MFM encoding and putting in correct AmigaDOS format, the track works out as 14848 bytes. Well, there's no way that can fit in the Arduino's 2k of memory, nor its 1k of EEPROM. I needed an alternative method.

I decided I would try to send the data 1 byte at a time in a high priority thread and wait for a response byte from the Arduino before sending the next. I changed the baud rate to 2M to reduce the lag between characters. This meant that it took roughly 5.5 uSec to send each character, and 5.5 uSec to receive one back. The Arduino would need to write out 8 bits, at 500khz, so it would need a new byte every 16 uSec. So there should be time, assuming the code loop is tight enough and the operating system doesn't delay the sending and receiving too much.

This was a complete, utter failure. The entire read/write cycle took far too long, well beyond one revolution of the disk. The Arduino side was probably fast enough, but the OS wasn't responsive enough. Reading disks works because the OS (Windows in my case) would buffer the data coming in, but writing, Windows would just send it all in one go, but because the rate I'm sending at is far faster than the Arduino needs it, data would be lost. This was why I decided on this two-way acknowledgement process.

Writing Data - Attempt 2

Software flow control for this application was just not fast enough. I decided to investigate hardware flow control. I noticed on the FTDI breakout board there are CTS and DTR pin. These stand for Clear To Send and Data Terminal Ready. I noticed that while the breakout board was connected, the Arduino board connected the CTS to GND.

I also didn't know which direction these pins were actually in, but after some experimentation, I found the CTS pin could be signaled from the Arduino and used by the PC to control the flow. Normally this is done using a circular buffer, but in my case I couldn't allow this, so I simply set it to '1' when I don't want data, and '0' while I do.

This now meant I could just ask the OS to bulk send the bytes as one chunk, and hope that it was all handled at the kernel level so it wouldn't get interrupted.

I had an inner loop that output each bit from the 8 bits but decided it was probably better timing wise to unravel it into 8 sets of commands instead.

This didn't work. If I allowed the code to run without actually running the disk writing part, then all bytes were received correctly, but with running the code, it didn't and bytes being received were being lost.

I suspected that changing the status of the CTX line didn't instantly stop the flow of data and the computer may still send a character or two. Possibly by the time I had signaled the CTX line, it was already in the process of sending the next character.

Writing Data - Attempt 3

I didn't want to have a serial interrupt as I didn't want any of the writing timings to be distorted. I realised that inbetween writing each bit to the floppy drive there would be a number of CPU cycles sitting in the next while loop. I decided to check between each bit write if another byte had been received since CTX went high and store it.

My theory was that when you raised CTX, the computer was probably already in the middle of transmitting the next byte and as you can't stop it mid-stream, then it would half after this one. This means I only need to check for one extra byte during the loop and use it if found instead of looking at the serial port again.

So this seemed to work, and the Arduino completed the write without losing any data from the computer. The only questions now were: has it actually written any data, and if so, is any of it valid?

At this point I had only encoded one track, so I decided to run the entire algorithm to encode all 80 tracks. Something strange was happening. The drive head wasn't moving at all. It still did when reading, but not when writing.

I found that in order to move the drive head back and forth you first had to raise the /WRITE GATE pin, I suspected this was required for changing the surface also. Once I added code to do this the drive head moved as expected. This did make sense and would prevent accidental writing of tracks while moving the head around.

So at this point I wrote a disk image out I had created previously, and then tried to read it back. Nothing could be detected! Either the data I had written was invalid, or the way I was writing it was wrong.

I decided to feed the encoded MFM sector data that I was creating into my sector decoding algorithm used by the reader to validate that what I was generating was correct and valid, and it was. Something was obviously wrong with how I was writing the data to the disk.

Writing Data - Attempt 4

As no data was being read correctly I decided try a few different approaches. I wasn't sure if the /WRITE DATA pin should be pulsed (and if so, by how long), toggled or just set to the raw data value. My current implementation pulsed the pin. I hadn't been able to find any information online about how the write pin was physically suppose to be manipulated when writing.

The read head would send us a pulse each time there is a flux reversal. I decided to change the implementation so that WRITE DATA was just set to the value of the bit. That didn't work either. So I changed the code to toggle the current state of the pin. Still no luck.

Clearly one of these approaches must have been the correct one. So I decided to get out the trusty oscilloscope again to have a look at what was going on. I decided to write the MFM pattern 0xAA to every byte on a track continuously. 0xAA in binary is B10101010, so this would give me a perfect square wave that I could monitor for the required frequency.

If it didn't see a perfect square wave at the desired frequency, then I knew there must be some kind of timing issue.

I hooked up the scope, but was surprised to see the timings were perfect. However, being an old scope I couldn't see more than a few pulses. The scope had this wonderful x10 "mag" mode. When pressed, it increased the timebase by 10, but more importantly allowed you to scroll through all of the data much like on a modern digital scope.

Something wasn't correct here. It looked like every 12 bits or so I ended up with a period of just "high".

Either the data I was sending was in some way invalid, or there was something causing a pause in the writing process every 12 bits or so. 12 being a strange number considering there are only 8 bits in a byte.

After thinking about this, I wondered if I was back with a flow control issue. The way I had designed the loop was to scoop up any stray extra bytes that were received after we had waited for one. But it wasn't intelligent enough to prevent the wait every other byte. I had two choices, move something into an interrupt, or patch the loop.

I decided to have a go at correcting the way the loop worked first. The issue was as a result of a delay caused by waiting for the next byte from the computer. If we lowered CTX and waited for a byte, by the time we raised CTX again, another byte was already on the way.

I change the loop so that when the second byte received had been used, the Arduino momentarily pulled CTS low and then high again to allow another character to be sent. This meant on the next loop we would have already received the next byte so no waiting was required.

Testing this produced a perfect square wave:

This meant all of the timing for writing a track was perfect, it was just down to the actual data that was being written. I decided to let this run for a few tracks and sides, and then read it back to see if it had written correctly. I was setting the /WRITE_DATA pin to the corresponding bit value from the data received.

When reading the data back it looked like nothing had been encoded, but then I skipped to the other side of the disk. Sure enough there was my pattern. I didn't know why it had only written to one side of the disk.

After some thinking I started to wonder if the /WRITE GATE pin didn't actually work the way I thought it did. It occurred to be that by pulling the pin low it may be enabling the erase head on the drive. If this was the case, then I should only do this when I was actually writing or I might end up with noise on the disk as it spins and erases.

I changed all of the code so that the /WRITE GATE was only used when first starting the drive, and later only literally during the write loop. That worked! I was now writing data to both sides of the disk!

So I tried again with a real ADF disk image and let it complete. I then used the reader portion to see if I could read it back. It worked! But for some reason it took quite some time to read this disk back. I wasn't getting any MFM errors but it was struggling to find all of the sectors.

There's were two possibilities for me to look at now: firstly, had the data actually written timely enough; and secondly, would the disk actually work in a real Amiga?

Too excited with the idea that I might have actually written a disk I booted up the now working A500+ and put the disk in. Moments later, the disk started booted and then displayed the famous checksum error message. So I was writing something valid, but it wasn't consistent.

I decided that unless I could read the data back at a much more accurate rate, writing a disk was pointless.

Reading Data (again)

I wanted to improve the reading quality as I wasn't happy with the current implementation. The current implementation didn't allow enough flexibility for the pulses to arrive at slightly odd times. I needed a new approach.

Firstly, I decided I was going to sync the reading to the /INDEX pulse. It's not required by the Amiga but may come in handy later on for me testing, writing and reading.

Several people in the comments to the first half of this project suggested that I should be recording the timing between pulses rather than the method that I had implemented. The only issue with this was getting this data to the PC fast enough. If I was to send a byte for each bit, then I could easily exceed the maximum 2M baud.

I decided that the best thing to do would be to try to make sense of the data a little. So I decided to let the counter I was using originally to free-run, right up to 255. I then put the code in a loop waiting for a pulse and that this point saw how much time had passed.

In an ideal situation, the lowest possible minimum value would be 32 (corresponding to 2 uSec). With MFM you could only ever have a maximum of three 0's in a row, so the maximum this value should reach was 128. This meant there were a maximum of 4 possible combinations in a row.

I sampled several disks to see where the majority of these frequencies lay, and the results can be seen below:

Looking at this, I find the majority of the points around a counter of 52, 89 and 120. However, these were somewhat specific to my drive and therefore not a good guideline. After some experimentation, I used the following formula: value = (COUNTER - 16) / 32. When clipped between 0 and 3 this gave me the output I required. Every 4 of these and I could write a byte out.

It occurred to me that because you couldn't have two '1's together in an MFM encoded bit stream I could safely assume anything for the first value was invalid and could be treated as another '01' sequence. The next part was to unpack this data once received by the PC and turn it back into MFM data. This was simple, since 00 couldn't happen, a 01 meant write '01', an 10 meant write '001' and a 11 meant write '0001'. I gave this a try and to my surprise my results were 100% successful. I tried with a few more disks too, 100%! I now had a very reliable disk reader.

With this new approach being a lot more tolerant on the data from the disk I no longer needed any phase analysis or as many retries. Most of my disks now read perfectly. Some required a few retries but got there in the end. The last part was to statistically analyse the data and see if it could be repaired, however, 99% of the time bad data coming in was completely unrecognizable and so was little help.

Writing Data - Attempt 5

Now that I could verify what I had written with high accuracy it meant testing the writer would be much easier.

I set about analysing the code to see what was going wrong. I wrote a 0x55 sequence to an entire track and then read it back in. From time to time a bit had shifted in the data coming back, meaning there was some kind of timing issue in writing.

It turned out that this was partly due to the way I was handling the serial port, and partly due to the use of the timer. I was waiting for the timer to reach the value 32, writing the bit, and then resetting it. I changed it so I didn't have to modify the timer counter value.

I would write the first bit when the counter reached 16, then the next when it reached 48 (16+32), and the next when it reached 80 (16+32+32) and so on. Timer2 being only 8-bit rolls over back to zero after the 8th bit, exactly when we needed it to. This meant that as long as we wrote the bit at the required timer value we would be at exactly 500kbps.

I also looked at how I was reading the data from the serial port. This was being read inbetween each bit, but this needed to be as short as possible too. After a little experimentation I achieved the shortest working block.

After modifying the Windows code to support verify, I was now ready to try again. This time I knew that if the disk verified properly, then it should work properly in the Amiga.

So I tried writing another disk out. With verify it took longer. With the new algorithm about 95% of the tracks passed verification on the first go, with only the remaining 5% having to be re-written once more. I was happy with this and popped the disk into the Amiga. It worked perfectly!

Writing Data - Attempt 6

After some feedback from some people who have been using this it was clear that even with verify on the drive wasn't always producing fully readable disks. The software could read them back perfectly, but the Amiga computers would report of a few checksum errors here and there.

I had another look at the code, wondered if it was a timing issue and looked to see if it could be made to be interrupt driven, but sadly with the small amount of time between each bit there simply isn't enough time with interrupts to achieve this with preserving the registers you modify etc.

I then looked back at the writing code. There is a small chance that after a full byte has written, the code could have looped back to start writing the next byte before the timer had overflowed back to 0, allowing the first bit to be written early.

I added a small loop to ensure this couldn't happen which hopefully will fix this for anyone having this issue.

Writing Data - Attempt 7

After getting a lot of reports of of checksum errors for written disks I started to investigate. I thought at first I was going to have to get down to looking at the MFM data from the disk but the problem was actually much simpler

Looking at XCopy Pro to see the checksum errors, it reported codes 4 and 6 meaning checksum errors in the sector headers and data areas. If it had just been the data area then I would have assumed that it was purely something to do with writing the last few bits of the track, but it wasn't.

I started looking at the writing code and the padding I had around each track, wondering if I was overwriting the start of a track now and then, so I massivly reduced the post-track padding from 256 bytes to 8. To my suprise my verify then kicked out a tonne of errors.

This made me wonder if the actual issue is I'm not writing enough data. I set about adding a Track Erase command to the Arduino which would write the 0xAA pattern to the entire track and then write my track afterwards. To my surprise XCopy gave it a 100% thumbs up. So hopefully thats cleared that problem up.

Diagnostics

I have had lots of feedback from people who have successfully made this project, both fully working and not working. I decided I would build a diagnostics module into the code to help anyone who can't get theirs to work.

The diagnostics option consists of a few extra commands for the Arduino to process as well as a whole series of events that get ran through to ensure everything is wired correctly.

So What's Next?

The entire project free and open source under GNU General Public Licence V3. If we want to have any hope of preserving the Amiga, then we shouldn't be ripping each other off for the privilege. And besides, I want to give back to the best platform I ever worked on. I’m also hoping people will develop this and take it further and keep sharing.

The current writing solution isn't an option on the Arduino UNO unless you use a separate FTDI/serial breakout board, so my next tasks are to make it work on that (possibly using the 23K256 IC to buffer the track before writing it to the disk).

I still want to look at other formats. ADF files are good, but they only work for AmigaDOS formatted disks. There are lots of titles with custom copy protection and non-standard sector formats that simply cannot be supported by this format. I have received some very useful information on this but don't currently have many disks to test with.

According to Wikipedia, there’s another disk file format, the FDI format. A universal format, that's well documented. The advantage of this format is it tries to store the track data as close to the original as possible so hopefully will fix the above issues!

I also came across the Software Preservation Society, specifically CAPS (formally the Classic Amiga Preservation Society) and their IPF format. After a little bit of reading, I was very disappointed; it's all closed, and it felt like they were just using this format to sell their disk reading hardware.

So my focus will be on the FDI format. My only concern here is with data integrity. There won't be any checksums for me to check against to see if the read was valid, but I have a few ideas to resolve that!