I've spent my spare time this week learning about #Amiga filesystem formats, to find out why I had all those problems with file corruption in my effort last weekend to get an image file from Linux into Deluxe Paint on an emulator. (Previous thread: hachyderm.io/@simontatham/1140…)
The _root_ cause is: there are two flavours of the Amiga filesystem, and I was using the less well tested one.
The two flavours are usually called OFS ("Original File System") and FFS ("Fast File System", and let's just assume we've done the jokes already). FFS took over from OFS early in the Amiga's life. So only Amigas from the very dawn of time use OFS, and third-party implementations of the filesystem (like the one in the Linux kernel) have _mostly_ been tested against FFS.
But my emulated Amiga _is_ from the very dawn of time (on purpose, because that most closely matches the real Amiga I had as a child). So
... show more
I've spent my spare time this week learning about #Amiga filesystem formats, to find out why I had all those problems with file corruption in my effort last weekend to get an image file from Linux into Deluxe Paint on an emulator. (Previous thread: hachyderm.io/@simontatham/1140…)
The _root_ cause is: there are two flavours of the Amiga filesystem, and I was using the less well tested one.
The two flavours are usually called OFS ("Original File System") and FFS ("Fast File System", and let's just assume we've done the jokes already). FFS took over from OFS early in the Amiga's life. So only Amigas from the very dawn of time use OFS, and third-party implementations of the filesystem (like the one in the Linux kernel) have _mostly_ been tested against FFS.
But my emulated Amiga _is_ from the very dawn of time (on purpose, because that most closely matches the real Amiga I had as a child). So it _only_ speaks OFS, and so I exercised all the less tested code paths and ran into all the bugs.
Last month I posted a picture of a piece of needlework art in my dad's house, and remarked on its similarity to Amiga-era pixel art: hachyderm.io/@simontatham/1137…
I said in that post that it would be cool to digitise it back to a paletted image suitable for actually loading into Deluxe Paint. I didn't get round to doing it … but someone else did! kaberett.dreamwidth.org/ put in a *lot* of effort (more than I would have dreamed of asking for) and sent me a PNG.
I thought the only reasonable thing to do with that was to convert it back to IFF ILBM, transfer it on to an Amiga floppy image, and load it in _actual_ Deluxe Paint. Tada!
But even given a starting PNG, getting it into DPaint was a challenge. (1/8)
The key difference between OFS and FFS is that in OFS, each disk sector that contains file _data_ (as opposed to file headers, directories, etc) has a 24-byte header, and only 488 bytes of data. In FFS, they threw out the header, and just use all 512 bytes of the sector for data.
It was possible to make that design change because none of the data in the header is _necessary_. It's all redundant copies of things you could have worked out another way. So Linux implementations of the filesystem ignore it completely when reading, which makes their OFS and FFS reading code conveniently more similar.
Therefore they can get it very wrong indeed, without noticing any problem themselves!
The reason the Linux kernel created disk files that the Amiga just wouldn't read at all:
One of the fields in the OFS data block header is a sequence number: each data block identifies itself as "I am the Nth block in this file". The sequence numbers are supposed to start at 1. But the Linux kernel wrote them starting from 0. It did it consistently, so I guess that was just a misunderstanding.
When I found this problem I hoped that it might be specific to _creating_ files, so I worked around it by making a valid file using the Amiga, and overwriting its data. I guessed right (although wrong about the cause, I was expecting a bogus field in the _file_ header). Writing an existing file, the kernel finds the sequence numbers already set up and doesn't mess with them.
I made sure my starting file was at least the same size as the one I wanted, on general principles of caution. That was a good call too – otherwise the kernel would have had to add data blocks at the end, and got the sequence numbers wrong again!
Another field in the OFS data block header is 'data size': how much data in this block is part of the file? This _ought_ to take its maximum possible value (488 bytes) in every block of the file, except the last one, which might be short.
When I overwrote an existing file and read that back on the Amiga, I got bad data. Specifically, the Linux kernel was making some data-size fields _too long_. That was because the calculation of the new data size field was based on the assumption that you were writing a new file, therefore appending to the end of a block. If you wrote to a block _not_ starting at its current end-of-data, the calculation would add (new amount of data written) to (old block size) instead of to (start point of write), and deliver nonsense.
The Amiga took it at its word and returned extra data to the reading application. As best I can tell, that data came from whatever was in RAM beyond the end of the buffer the Amiga had read the disk sector into.
The Linux kernel accidentally exploited a Heartbleed-style buffer overflow in AmigaDOS! Those glitches
... show more
Another field in the OFS data block header is 'data size': how much data in this block is part of the file? This _ought_ to take its maximum possible value (488 bytes) in every block of the file, except the last one, which might be short.
When I overwrote an existing file and read that back on the Amiga, I got bad data. Specifically, the Linux kernel was making some data-size fields _too long_. That was because the calculation of the new data size field was based on the assumption that you were writing a new file, therefore appending to the end of a block. If you wrote to a block _not_ starting at its current end-of-data, the calculation would add (new amount of data written) to (old block size) instead of to (start point of write), and deliver nonsense.
The Amiga took it at its word and returned extra data to the reading application. As best I can tell, that data came from whatever was in RAM beyond the end of the buffer the Amiga had read the disk sector into.
The Linux kernel accidentally exploited a Heartbleed-style buffer overflow in AmigaDOS! Those glitches in my corrupted DPaint image are fragments of the Amiga's kernel data.
As well as the Linux kernel, I also tried an independent implementation of the Amiga filesystem: 'fuseadf', a FUSE wrapper around 'ADFlib'. That too had the problem that it seemed to have been mostly tested on FFS and neglected OFS.
But ADFlib went the other way. It wrote data blocks with a too _short_ size field, by forgetting to update the field at all when you appended to a block.
My file copy wrote 4096 bytes to the file in its first write(). That left a short data block at the end, because 4096 isn't a multiple of 488. Then it appended another 4096 bytes, so that that short data block was filled up and more blocks appeared after it – but the data block that had _temporarily_ been short still had its old data-size header value, claiming less than 488 bytes.
So when the Amiga read _that_ file back, it _deleted_ ranges of bytes from the file!
ADFlib has merged my fix now. The Linux kernel might take longer…
When appending data to an existing OFS block, the dataSize field in the block header wasn't being extended. So if you wrote a file via fuseadf, 4Kb at a time, then the first 4Kb write would ter...
@metin mine was an A500, acquired around 1988 (though I can't remember the exact date). I kept using it until I went off to university in 1994, when I replaced it with a boring old PC.
I never upgraded it to a more up-to-date actual computer, but I did end up giving it a hard disk, via a strange third-party upgrade Dad found somewhere which fitted into the slot on the bottom where the 512Kb RAM upgrade went, and was indeed a perfectly functional 512Kb RAM upgrade but _also_ an ST-506 interface. Dad's employer had a ton of disused ST-506 hard drives waiting to be landfill, and didn't mind employees helping themselves, so he brought an armful of them back from work one day and we tried them all until we found one that still worked.
@flippac ah, yes, in the same way that if you UTF-8-encode something twice, or encode the Unicode typographical quotes by UTF-8-ifying their Windows-1252 code points, you're writing WTF-8.
So far as I can remember, OFS remained the default for floppy disks even up to AmigaOS 3.1, while FFS was used on most hard disks. I think the reasoning was that the extra data validation and copying of data in OFS was worthwhile for slow and unreliable floppy disks but unnecessary and a performance bottleneck for hard disks.
Simon Tatham
in reply to Simon Tatham • • •The slightly more specific root cause:
The key difference between OFS and FFS is that in OFS, each disk sector that contains file _data_ (as opposed to file headers, directories, etc) has a 24-byte header, and only 488 bytes of data. In FFS, they threw out the header, and just use all 512 bytes of the sector for data.
It was possible to make that design change because none of the data in the header is _necessary_. It's all redundant copies of things you could have worked out another way. So Linux implementations of the filesystem ignore it completely when reading, which makes their OFS and FFS reading code conveniently more similar.
Therefore they can get it very wrong indeed, without noticing any problem themselves!
(2/5)
Simon Tatham
in reply to Simon Tatham • • •The reason the Linux kernel created disk files that the Amiga just wouldn't read at all:
One of the fields in the OFS data block header is a sequence number: each data block identifies itself as "I am the Nth block in this file". The sequence numbers are supposed to start at 1. But the Linux kernel wrote them starting from 0. It did it consistently, so I guess that was just a misunderstanding.
When I found this problem I hoped that it might be specific to _creating_ files, so I worked around it by making a valid file using the Amiga, and overwriting its data. I guessed right (although wrong about the cause, I was expecting a bogus field in the _file_ header). Writing an existing file, the kernel finds the sequence numbers already set up and doesn't mess with them.
I made sure my starting file was at least the same size as the one I wanted, on general principles of caution. That was a good call too – otherwise the kernel would have had to add data blocks at the end, and got the sequence numbers wrong again!
(3/5)
Simon Tatham
in reply to Simon Tatham • • •Another field in the OFS data block header is 'data size': how much data in this block is part of the file? This _ought_ to take its maximum possible value (488 bytes) in every block of the file, except the last one, which might be short.
When I overwrote an existing file and read that back on the Amiga, I got bad data. Specifically, the Linux kernel was making some data-size fields _too long_. That was because the calculation of the new data size field was based on the assumption that you were writing a new file, therefore appending to the end of a block. If you wrote to a block _not_ starting at its current end-of-data, the calculation would add (new amount of data written) to (old block size) instead of to (start point of write), and deliver nonsense.
The Amiga took it at its word and returned extra data to the reading application. As best I can tell, that data came from whatever was in RAM beyond the end of the buffer the Amiga had read the disk sector into.
The Linux kernel accidentally exploited a Heartbleed-style buffer overflow in AmigaDOS! Those glitches
... show moreAnother field in the OFS data block header is 'data size': how much data in this block is part of the file? This _ought_ to take its maximum possible value (488 bytes) in every block of the file, except the last one, which might be short.
When I overwrote an existing file and read that back on the Amiga, I got bad data. Specifically, the Linux kernel was making some data-size fields _too long_. That was because the calculation of the new data size field was based on the assumption that you were writing a new file, therefore appending to the end of a block. If you wrote to a block _not_ starting at its current end-of-data, the calculation would add (new amount of data written) to (old block size) instead of to (start point of write), and deliver nonsense.
The Amiga took it at its word and returned extra data to the reading application. As best I can tell, that data came from whatever was in RAM beyond the end of the buffer the Amiga had read the disk sector into.
The Linux kernel accidentally exploited a Heartbleed-style buffer overflow in AmigaDOS! Those glitches in my corrupted DPaint image are fragments of the Amiga's kernel data.
(4/5)
Simon Tatham
in reply to Simon Tatham • • •As well as the Linux kernel, I also tried an independent implementation of the Amiga filesystem: 'fuseadf', a FUSE wrapper around 'ADFlib'. That too had the problem that it seemed to have been mostly tested on FFS and neglected OFS.
But ADFlib went the other way. It wrote data blocks with a too _short_ size field, by forgetting to update the field at all when you appended to a block.
My file copy wrote 4096 bytes to the file in its first write(). That left a short data block at the end, because 4096 isn't a multiple of 488. Then it appended another 4096 bytes, so that that short data block was filled up and more blocks appeared after it – but the data block that had _temporarily_ been short still had its old data-size header value, claiming less than 488 bytes.
So when the Amiga read _that_ file back, it _deleted_ ranges of bytes from the file!
ADFlib has merged my fix now. The Linux kernel might take longer…
(5/5)
Adam Sjøgren
in reply to Simon Tatham • • •Nice story!
Do you have links to the fixes? Could be fun to see...
Simon Tatham
in reply to Adam Sjøgren • • •@asjo two kernel patches: lore.kernel.org/linux-fsdevel/…
ADFlib PR: github.com/lclevy/ADFlib/pull/…
adfFileWrite: update dataSize when extending an OFS block. by sgtatham · Pull Request #83 · lclevy/ADFlib
GitHubMetin Seven 🎨
in reply to Simon Tatham • • •Simon Tatham
in reply to Metin Seven 🎨 • • •Metin Seven 🎨
in reply to Simon Tatham • • •Simon Tatham
in reply to Metin Seven 🎨 • • •@metin mine was an A500, acquired around 1988 (though I can't remember the exact date). I kept using it until I went off to university in 1994, when I replaced it with a boring old PC.
I never upgraded it to a more up-to-date actual computer, but I did end up giving it a hard disk, via a strange third-party upgrade Dad found somewhere which fitted into the slot on the bottom where the 512Kb RAM upgrade went, and was indeed a perfectly functional 512Kb RAM upgrade but _also_ an ST-506 interface. Dad's employer had a ton of disused ST-506 hard drives waiting to be landfill, and didn't mind employees helping themselves, so he brought an armful of them back from work one day and we tried them all until we found one that still worked.
Metin Seven 🎨
in reply to Simon Tatham • • •Philippa Cowderoy
in reply to Simon Tatham • • •Simon Tatham
in reply to Philippa Cowderoy • • •Ben Hutchings
in reply to Simon Tatham • • •