As originally conceived, Atari computers were going to be a platform where the technical secrets were held closely, allowing Atari to reap easy profits by providing software that third parties would be hard to match. At least, that was the theory. But the technical secrets were not going to remain secret forever, and as Atari was slow to provide the software to drive sales of the computers, it made more and more sense to abandon that strategy and open the platform to more developers. Atari eventually did this in late 1981, publishing the technical notes and then more broadly with the publication of De Re Atari in 1982, a more approachable presentation of the material.
But before the technical notes became widely available, Atari used the popular computer magazines of the day to publish some of the technical information. Chris Crawford and Lane Winner at Atari wrote two articles in late 1980, one for Byte magazine and one for Compute!, which explained display lists and player/missile graphics respectively. Both appeared in the January 1981 issues of the magazines, and were later endlessly rehashed in books and magazines over the years.
The Byte magazine article, An Introduction to Atari Graphics describes some of the functionality of display lists and then proceeds to use a modified display list and a modified character set to generate a Byte magazine splash page. Interestingly, the page references the August 1980 issue, perhaps publication was delayed for unknown reasons. In any event, it’s probably the first published article on display lists, and is quite interesting both for what it reveals and what it does not. It shows the 14 graphics modes, revealing that there are modes that aren’t available from the GRAPHICS command. Since it modifies an existing display list, it omits mentioning details important to writing a display list from scratch. Nonetheless, there was enough information here to help software authors start producing better graphics on the computers.
![[Title Screen]](byte.png)
0 IF FRE(0)<5825 THEN PRINT "NOT ENOUGH MEMORY!":END 20 RAMTOP=PEEK(106):POKE 106,RAMTOP-16:GRAPHICS 0 30 DIM SML$(32):SML$(1,32)=" the small systems journal " 40 DIM MGH$(41):MGH$(1,41)="@@@A@McGRAW@HILL@@@@@@@@PUBLICATION" 50 GRAPHICS 7+16:COLOR 2:POKE 765,255 POKE 708,128:POKE 709,40:POKE 710,128:POKE 712,128 60 START=PEEK(560)+PEEK(561)*256+4 70 POKE START+10,2 80 FOR X=0 TO 3:POKE START+46+X,6:NEXT X 90 FOR X=0 TO 3:POKE START+50+X,7:NEXT X 95 POKE START+54,65:POKE START+55,PEEK(560):POKE START+56,PEEK(561) 110 A=0:READ B,C:IF B>-1 THEN GOSUB 800:GOTO 110 120 READ A,B,C:IF A>-1 THEN GOSUB 800:GOTO 120 200 CHBAS=RAMTOP-4:ADDR=CHBAS*256 210 FOR X=0 TO 1023:POKE ADDR+X,PEEK(57344+X):NEXT X 220 POKE 756,CHBAS+2 230 FOR X=0 TO 255:POKE ADDR+512+X,PEEK(ADDR+256+X):NEXT X 240 FOR X=0 TO 7:POKE ADDR+512+X,0:NEXT X 250 FOR X=0 TO 7:READ A:POKE ADDR+99*8+X,A:NEXT X 290 POKE 755,0:POKE 87,0 300 POSITION 4,9:? #6;"AUGUST 1980 Volume 5, Number 8"; 310 MEMST=PEEK(START)+PEEK(START+1)*256:CHRPOS=MEMST+46*40 320 FOR X=1 TO LEN(SML$):POKE CHRPOS+X-1,ASC(SML$(X,X))+128:NEXT X 330 CHRPOS=CHRPOS+60 340 FOR X=1 TO LEN(MGH$):POKE CHRPOS+X-1,ASC(MGH$(X,X))-64:NEXT X 350 POKE 708,200 360 GOTO 360 800 ON A+1 GOTO 810,820,830 810 PLOT B,C:RETURN 820 DRAWTO B,C:RETURN 830 POSITION B,C:XIO 18,#6,0,0,"S:":RETURN 900 DATA 111,20,111,21,110,21,109,21,108,22,107,23,107,24 905 DATA 106,25,106,26,107,27,107,28,108,29,109,30,110,30,111,30 910 DATA 111,31,110,31,109,31,108,32,107,33,107,34,106,35 915 DATA 106,36,107,37,107,38,108,39,109,40,110,40,111,40,111,41 920 DATA -1,-1,1,97,41,2,96,40,1,96,40,2,96,21 925 DATA 2,97,20,0,93,21,1,92,21,1,91,21,1,90,22,1,89,23,1,89,24 930 DATA 1,88,25,1,88,40,1,87,41,1,80,41,2,79,40,0,79,40 935 DATA 2,79,25,0,79,25,2,78,24,0,78,24,2,78,23,0,78,23,2,77,22 940 DATA 0,77,22,2,76,21,1,74,21,1,74,20,1,93,20,0,71,20 945 DATA 1,71,36,1,70,20,1,70,36,1,69,33,1,69,36,1,68,34,1,68,36 950 DATA 1,67,34,1,67,40,1,66,41,1,59,41,2,58,40,0,58,40 955 DATA 2,58,36,1,54,36,2,54,34,1,64,33,1,63,32,1,63,21,1,62,20 960 DATA 1,55,20,2,54,21,0,54,21,2,54,33,0,51,21 965 DATA 1,51,29,0,51,32,1,51,40,1,50,41,1,50,32,0,50,29,1,50,20 970 DATA 1,49,20,1,49,22,0,49,28,1,49,33,0,49,39 975 DATA 1,49,41,1,48,41,1,48,40,0,48,32,1,48,29,0,48,21,1,48,20 980 DATA 1,46,22,0,46,28,0,46,33,0,46,39,1,45,38,1,45,33 985 DATA 0,45,28,1,45,23,0,47,41,1,36,41,2,35,40,0,35,40,2,35,21 990 DATA 1,36,20,1,49,20,-1,0,0 999 DATA 0,60,96,96,96,60,0,0
We’ve now reached a compact bit of code in the Dealer Demo that provides an assembler to Forth. And the assembler in Forth is a thing of beauty indeed. Written by Bill Ragsdale (as was most of the Forth kernel), it provides an assembler that can produce surprisingly readable code without the use of labels. In essence it implements high-level assembler:
Recall TRAVERSE. In traditional assembler, it was written as:
154A: 4C 15 TRAV .WORD *+2 154C: B4 00 LDY 0,X 154E: 98 TRAV1 TYA 154F: 18 CLC 1550: 75 02 ADC 2,X 1552: 95 02 STA 2,X 1554: B5 01 LDA 1,X 1556: 75 03 ADC 3,X 1558: 95 03 STA 3,X 155A: A1 02 LDA (2,X) 155C: 10 F0 BPL TRAV1 155E: 4C 4D 0E JMP POP
In Forth assembler, it might be written as:
CODE TRAV ,X 0 LDY, BEGIN,
TYA, CLC, ,X 2 ADC, ,X 2 STA,
,X 1 LDA, ,X 3 ADC, ,X 3 STA,
X) 2 LDA, 0< NOT UNTIL,
POP JMP, ;CODE
Notice that we didn’t need a label we simply used BEGIN, to mark the beginning of a block and UNTIL, to mark the end. The Forth assembler also has IF, ELSE, THEN, which behave as expected. This is flexible enough to handle most assembly constructs without ever resorting to labels. It also uses a Forth-like syntax, with RPN ordering for the assembler codes, and the opcodes always ending in comma, since the effect is similar to the comma operator which puts data into memory in Forth. While I wouldn’t want to write a large program this way, this is much more convenient than the USR function in BASIC.
Another interesting implementation detail is that all opcodes are split into two sets. The first set, encoded using the word CPU, are most of the traditional one-byte ‘immediate’ mode opcodes, and have the opcode byte value attached to them. The remaining opcodes, encoded using the word M/CPU, use a word and a byte for each of these to select the appropriate opcode depending on the current mode and whether to emit 0, 1 or 2 additional bytes taken from the stack. This is remarkably compact, thanks to the <BUILD ... DOES> construct in Forth which we discussed in an earlier post.
You can read about this assembler in an article by Bill Ragsdale himself. It appeared in the September 1981 issue of Dr. Dobbs Journal (a issue dedicated to Forth), as well as the January 1982 issue of Forth Dimensions (v.3, number 5). The implementation itself was available earlier, since the Dealer Demo dates from late 1980, and the screens for the assembler date to June 1979 and earlier. It opens with:
This article should further polarize the attitudes of those outside the growing community of FORTH users. Some will be fascinated by a label-less macro-assembler whose source code is only 96 lines long! Others will be repelled by reverse Polish syntax and the absence of labels.
The author immodestly claims that this is the best FORTH assembler ever distributed. It is the only such assembler that detects all errors in op-code generation and conditional structuring. It is released to the public domain as a defense mechanism: three good 6502 assemblers were submitted to the FORTH Interest Group but each had some lack. Rather than merge and edit for publication, I chose to publish mine with all the submitted features plus several more.
If you compare the published Ragsdale implementation with the assembler in Dealer Demo, only a few minor differences appear. The word VS was added, to give symbolic access to the overflow branches (BVS and BVC, an oversight in the original assembler), and the word END-CODE was renamed to C;.
In Dealer Demo, the assembler occupies $25a0 - $2be4, or about 1.6k. So the Forth kernel and its assembler are about the same size as the Atari Assembler/Editor cartridge. Atari Forth on cartridge, that would have been an interesting product to see!
OK, let’s finish off the kernel of Dealer Demo.
The next chunk is the COLD start routine, starting at $1C84, which is implemented as a primitive. Following that are a variety of standard Forth words dealing with mathematical operations and buffer and screen management. This leads up to TCIOV, a Dealer Demo primitive to invoke CIO routines. After the implementations of EMIT and KEY (labeled XEMIT and XKEY), we add a primitive BLKIO for invoking the disk handler, some SIO DCB constants (DCDNO, DCCOM, DCADD, DCSEC) and then the word RSLW which implements reading and writing to disk. After that we find the implementation of TYPE, and then more standard Forth words for implementing loops and conditions, digit formatting, listing words and screens.
That finishes off the core Forth kernel. To check our work, we should strip the left hand byte data from the listing and send it through an assembler. MADS is a popular assembler, but its syntax is a little different from the syntax we’ve been using, so we need to massage the output slightly, specifically we need to convert *= to ORG and drop the A from the shift accumulator operators. We should also not emit lines that are included just to list all the bytes for a line that generates more than three bytes of output. Those requirements yield this little bit of code.
sub lst2mads {
my $fh = open_lst();
while (<$fh>) {
chomp;
my ($output, $rest) = (/^(.{0,16})(.*)/);
next if $output !~ /^\s+$/ && $rest eq '';
my ($comment) = ($rest =~ /(\s*(?<!');.*)$/);
$rest =~ s/\s*(?<!');.*//; # truncate comments
$rest =~ s/\*=/ORG /;
$rest =~ s/(\s+(?:ASL|LSR|ROR|ROL)\s+)A(\s+|$)/\1\2/;
print "$rest$comment\n";
}
}
And sure enough, it assembles without error. We could also do some data validation of the $output data, but let’s postpone that and instead focus on how the words we just decompiled differ from the published fig-Forth listing.
So what’s different:
COLD word checks if the WARMST flag is set, in which case it skips to doing a WARM start only.USE and PREV start out as different values rather than the same in fig-Forth.FLUSH is more complicated in Dealer Demo. It doesn’t use BUFFER, so it needs to duplicate some of its logic.DR0 and DR1 have to set both OFFSET and PHYSOFF, rather than just OFFSET.XKEY, XEMIT and XTYPE are implemented as Forth colon definitions, instead of assembly primitives. The primitive is instead TCIOV. Of the remaining words the fig-Forth documentation says to provide implementations, CR is implemented in terms of EMIT, QTERM is always zero and RSLW is implemented around the BLKIO primitive.FORGET is different, using LESS instead of ULESS in some places (in fig-Forth they are equivalent though due to a bug), and starting with CURR @ CON @ - $18 ?ERR, and never invoking the FORTH word. This word was significantly modified between the 1979 and 1980 publications of the fig-Forth Installation manual, replacing the Ragsdale version with a ‘smart’ one by David Kilbridge. The version in Dealer Demo seems to be some combination of the two.INDEX doesn’t EMIT a form feed character when done (oddly TRIAD still does, which is likely a bug).The interesting new primitives are TCIOV and BLKIO.
2025: 27 20 TCIOV .WORD *+2 2027: 86 FF STX XSAVE 2029: B5 00 LDA 0,X 202B: A2 00 LDX #0 202D: 20 56 E4 JSR CIOV 2030: A9 00 LDA #0 2032: C0 80 CPY #$80 2034: D0 01 BNE TCIOV1 2036: 98 TYA 2037: 48 TCIOV1 PHA 2038: A6 FF LDX XSAVE 203A: 4C 3B 0D JMP PUSH
This regularizes entering into CIO, moving the top of the data stack to the accumulator and pushing the return status to the data stack.
20C9: CB 20 BLKIO .WORD *+2 20CB: 86 FF STX XSAVE 20CD: 20 53 E4 BLKIO1 JSR DSKINV 20D0: AD 03 03 LDA DSTATS 20D3: 10 06 BPL BLKIO2 20D5: A6 FF LDX XSAVE 20D7: 48 PHA 20D8: 4C 3D 0D JMP PUT 20DB: EE 0A 03 BLKIO2 INC DAUX1 20DE: D0 03 BNE BLKIO3 20E0: EE 0B 03 INC DAUX2 20E3: A9 80 BLKIO3 LDA #$80 20E5: 4D 04 03 EOR DBUFLO 20E8: 8D 04 03 STA DBUFLO 20EB: 30 03 BMI BLKIO4 20ED: EE 05 03 INC DBUFHI 20F0: A6 FF BLKIO4 LDX XSAVE 20F2: D6 00 DEC 0,X 20F4: D0 D7 BNE BLKIO1 20F6: 4C 42 0D JMP NEXT
This likewise provides a Forth vector into the disk handler, which RSLW and related words need to interact with the disk.
So there you have it, the Forth kernel in Dealer Demo is about 90% the same as the traditional kernel published by fig-Forth and William Ragsdale. A few changes here and there, but this is recognizably a fig-Forth derived kernel.
In our next post, we’ll talk about the Forth assembler, which is the next bit of code in the Dealer Demo.
We continue with decompiling Dealer Demo at $175D, seeing -TRAILING, (.") (PDOTQ), and then a handful of words leading to the word ERROR which decompiles incorrectly.
Looking closely, we see that the .WORD PDOTQ precedes strings in the listing, which are represented as a count, followed by the string contents. The code to decompile such a string manually is easy to implement, since we built most of the infrastructure already to decompile Forth name fields, namely:
sub cstr_buf {
my ($buff, $addr, $size) = @_;
my $count = unpack "C", substr($buff, 0, 1);
my $string = get_string(substr($buff, 1, $count));
$string = sprintf "%s.BYTE %d,%s", get_label($addr), $count, $string;
$count += 1; multi_buf($buff, $addr, $count, $string);
$count;
}
sub cstr {
my ($buff, $addr, $size) = read_img(@_);
cstr_buf($buff, $addr, $size);
}
To decompile automatically, we can insert this snippet into forth_buf:
if (get_cstrq($val)) {
$i += cstr_buf(substr($buff, $i), $addr + $i, $size - $i);
}
sub get_cstrq {
return $_[0] == 0x179c; # PDOTQ
}
We can apply this to the ERROR word to test that the code works as advertised, e.g. invoking dealerdemo.pl 1a1c yields:
1A1C: 9C 17 .WORD PDOTQ 1A1E: 04 20 20 .BYTE 4,' ? ' 1A21: 3F 20
Let’s keep decompiling up through the word FORTH, which has .WORD DODOE, which we discussed last time, and then up through ABORT.
Among these words, what differences do we see?
EXPECT is implemented significantly differently. The Dealer Demo version is much shorter and simpler than the fig-Forth assembly version. The fig-Forth version has extra code to handle back spaces and carriage returns which are done elsewhere in the Dealer Demo kernel.BSCR 1 - AND instead of 0 BSCR U/ DROP. The fig-Forth screen listing uses 7 AND unconditionally and has BSCR (blocks per screen) equal to 8, so it more closely follows the Dealer Demo listing. However, in the Dealer Demo, BSCR is one, so these gymnastics to figure out where to read the character from isn’t really needed.UPPER is dropped from Dealer Demo. This kernel (as are all Atari Forth kernels) is case sensitive so it isn’t used.ERROR is modified slightly before calling QUIT. Instead of always leaving IN and BLK on the data stack, we omit IN if reading from disk. Since QUIT doesn’t use this data, this presumably is left for debugging reasons.ABORT and QUIT use slightly different strings than the fig-Forth listing. ABORT ends with SEMIS, which isn’t really needed since QUIT doesn’t return. ABORT also calls some future word at $863A instead of CR, probably to run some Dealer Demo specific initialization.Invoking -refs in our tool shows CREAT, ERROR, WORD, ABORT and QUIT forward references can now be fixed up. CREATE was the missing word used in colon and constant definitions, so we now have largely filled in all the words that implement compilation.
I think that’s enough for today. Our decompilation tool is now complete, we just need to keep applying it until we’ve cranked through the rest of the disk. The next post (or maybe two) should complete the kernel (which ends at byte $259f), and the post after that will describe the assembler (which ends at $2be4).
We pick up decompiling again at $148C, where we find 1+ (ONEP). It is followed by 2+ (TWOP), HERE, ALLOT, , (COMMA), C, (CCOMM), - (SUB), = (EQUAL), > (GREAT), ROT, SPACE and -DUP (DDUP). All of these are identical to the fig-Forth listing, which implements these as colon-definitions, which is expected. Although SUB, EQUAL and ROT are sometimes rewritten as primitives to speed up Forth (cf. val-Forth), optimizing these was not done in this Forth.
TRAVERSE is the next word in the listing and it is implemented completed differently. It’s used to traverse a name field forwards or backwards to find start or end of the name. Here’s the version from fig-Forth:
0A84: D6 07 TRAV .WORD DOCOL 0A86: 22 07 .WORD SWAP 0A88: 08 07 L1312 .WORD OVER 0A8A: 99 06 .WORD PLUS 0A8C: 67 02 .WORD CLIT 0A8E: 7F .BYTE $7F 0A8F: 08 07 .WORD OVER 0A91: 8E 07 .WORD CAT 0A93: 1F 0A .WORD LESS 0A95: 56 03 .WORD ZBRAN 0A97: F1 FF L1320 .WORD $FFF1 0A99: 22 07 .WORD SWAP 0A9B: 19 07 .WORD DROP 0A9D: 0C 06 .WORD SEMIS
154A: 4C 15 TRAV .WORD *+2 154C: B4 00 LDY 0,X 154E: 98 TRAV1 TYA 154F: 18 CLC 1550: 75 02 ADC 2,X 1552: 95 02 STA 2,X 1554: B5 01 LDA 1,X 1556: 75 03 ADC 3,X 1558: 95 03 STA 3,X 155A: A1 02 LDA (2,X) 155C: 10 F0 BPL TRAV1 155E: 4C 4D 0E JMP POP
The Dealer Demo version is a nice improvement, perhaps motivated to speed up Forth compilation.
From here on in our decompilation we’ll largely omit details where the implementation follows the fig-Forth implementation exactly.
The next words, LATEST, LFA, CFA, NFA, PFA, !CSP, ?ERR, ?COMP, ?EXEC, ?PAIR, ?CSP, ?LOAD, COMP, LBRAC, RBRAC and SMUDGE are colon definitions to support the compiler. HEX and DECIMAL follow, and then more compiler/runtime support with (;CODE), ;CODE and then <BUILD, DOES> followed by some assembly code DODOE to support <BUILD ... DOES> and then the word COUNT.
If we run -refs now, we’ll see that (;CODE), labeled PSCOD, is the routine that finishes all the various dictionary adding definitions we’ve already seen (e.g. CONSTANT, VARIABLE, colon). I would have expected this routine to be a primitive, but it in fact is a colon definition:
16D4: 4C 12 PSCOD .WORD DOCOL 16D6: 85 10 .WORD RFROM 16D8: 6A 15 .WORD LATES 16DA: AC 15 .WORD PFA 16DC: 89 15 .WORD CFA 16DE: 13 12 .WORD STORE 16E0: 45 10 .WORD SEMIS
This just sets the code field of the latest word to the code that follows the PSCOD address. It’s used in all definitions because all those definitions finish adding to the dictionary in the same way, so it might as well be shared code.
DODOE deserves a little study:
171E: A5 F9 DODOE LDA IP+1 1720: 48 PHA 1721: A5 F8 LDA IP 1723: 48 PHA 1724: A0 02 LDY #2 1726: B1 FB LDA (W),Y 1728: 85 F8 STA IP 172A: C8 INY 172B: B1 FB LDA (W),Y 172D: 85 F9 STA IP+1 172F: 18 CLC 1730: A5 FB LDA W 1732: 69 04 ADC #4 1734: 48 PHA 1735: A5 FC LDA W+1 1737: 69 00 ADC #0 1739: 4C 3B 0D JMP PUSH
This routine configures Forth to resume execution at the word after DODOE, but first puts the current IP on the return stack, and the address of the subsequent word on the data stack. This is similar to how DOCOL works, but in practice it means we restart execution inside another DOCOL definition (the address after a DODOE is not the start of a definition) with the top of the stack set to a particular value. In modern terms, we’d probably call this a continuation, since it allows us to continue another routine with a modified environment. In the Forth kernel, it is used only to implement particular vocabularies (e.g. FORTH, ASSEMBLER), which can configure the set of words in the dictionary available to the compiler, but the assembler itself will use this extensively to create a table driven assembler from a handful of words.
The next word that is different is TYPE. In place of a real implementation, a stub implementation is here (.WORD DOCOL, XTYPE, SEMIS), deferring the implementation to later in the listing. This is partially understandable, TYPE depends on being able to use the Atari CIO routines which have yet to be defined. But if Dealer Demo was bootstrapped using an assembler, this isn’t really necessary, the assembler can handle the backward references just fine. Perhaps the implementation started by following the original fig-Forth, where XTYPE would have been a primitive defined later in the listing, but at some point it was converted to a colon definition and the original forward definition adjusted. As a result, we have two definitions of TYPE which are equivalent, which wastes some memory but otherwise works fine.
I think that’s enough decompiling for this post. Let’s fix up the values -refs show can now be replaced with labels, and pick up again later.
Thus far in disassembling the Dealer Demo, we’ve run across primitive definitions that have a PFA of .WORD *+2 and then some 6502 assembly code. There were two exceptions I didn’t draw attention to at the time, the word I and the word DROP. DROP’s PFA was simply “.WORD POP”, in other words it took advantage of the fact the POP falls into NEXT, so the semantics of DROP could use that routine directly. I’s PFA was similarly “.WORD R+2”, in other words it had the same implementation as the word R, i.e. the current loop counter is at the top of the return stack.
What’s more interesting about the word I is its implementation is later in the listing. The Forth compiler can’t generally deal with forward references. Words are built out of existing words, not words that have yet to be defined, but an assembler doesn’t have that limitation, so the Forth kernel which is bootstrapped with an assembler can put the kernel words in almost any order. However, our decompiler does have that limitation. We can’t emit a name we don’t yet know, so generally our tool will just have to emit a hex address instead and we’ll have to remember to go back and fix it later when we discover the label for that address. In the case of I, I filled it in without comment at the time, but that was actually when I should have thought about how to address that problem.
So let’s revert that bit of future knowledge and write some code that tries to detect when that happens:
sub fwdrefs {
my $fh = open_lst();
my $data = {};
while (<$fh>) {
if (/^([0-9A-F]{4}): .*(?:\.WORD|JMP|JSR|BNE|BEQ|BC[CS]|BMI|BPL|BV[CS]) \$([0-9A-F]{4})$/) {
my ($addr, $val) = (hex $1, hex $2);
push @{$data->{$val}}, $addr if exists $names->{$val};
}
}
foreach my $key (sort { $data->{$a} <=> $data->{$b} } keys %$data) {
my $addrs = join ',', map { sprintf "%04x", $_ } @{$data->{$key}};
printf "\$%04x: %s: %s\n", $key, $names->{$key}, $addrs;
}
}
We also need to add a little stub in main to invoke this with -refs.
elsif ($opt =~ /^\-refs/) {
fwdrefs(@_);
}
What this code does is record every time an unresolved reference is made in a .WORD, JMP, JSR, or branch that is also in our name table. We then output the address locations and the name we should have used.
If we run this now, we get:
$0490: XBOOT: 0488 $1098: R+2: 0e56
Sure enough it detected the forward reference we reverted, and another one in the bootloader where I added a label for a branch but then forgot to use the label (oops). So let’s fix those now.
We can periodically run this after disassembling to detect when we have put labels in that cover previous forward references.
The reason we’re writing this reference code now is we’re about to hit a cluster of forward references in the next word to decompile, the implementation of ‘:’. Running our tool with -def 1234 yields:
1234: C1 BA .BYTE $C1,$BA 1236: 26 12 .WORD L813 1238: 4C 12 .WORD $124C 123A: 04 16 .WORD $1604 123C: C0 15 .WORD $15C0 123E: 31 14 .WORD $1431 1240: EF 11 .WORD AT 1242: 24 14 .WORD $1424 1244: 13 12 .WORD STORE 1246: 75 1A .WORD $1A75 1248: 85 16 .WORD $1685 124A: D4 16 .WORD $16D4 124C: A5 F9 .WORD $F9A5
COLON is the first word we’ve run into that isn’t a primitive. It is a sequence of other words. We recognize two of them in the listing as primitives we’ve already seen, AT and STORE, but most of the other addresses are yet to come. The PFA here starts with .WORD $124C, which is just at the end of the above block. Even without looking at the fig-Forth listing, you can guess it is 6502 code, so let’s disassemble that and give it the name DOCOL we find for it the fig-Forth listing.
124C: A5 F9 DOCOL LDA IP+1 124E: 48 PHA 124F: A5 F8 LDA IP 1251: 48 PHA 1252: 18 CLC 1253: A5 FB LDA W 1255: 69 02 ADC #2 1257: 85 F8 STA IP 1259: 98 TYA 125A: 65 FC ADC W+1 125C: 85 F9 STA IP+1 125E: 4C 42 0D JMP NEXT
This routine is found at the start of every Forth colon definition (thus the name DOCOL). It takes the current interpreter pointer and pushes it on the stack. It then takes the code field pointer W (which is the location of this DOCOL word, since the we just came from the NEXT routine), adds two and makes that the interpreter pointer. In other words, when you enter a colon definition, you record where to return to and then start interpreting the word after DOCOL. This is very similar to how JSR works, except that the callee pushes the IP, not the caller.
COLON is also the first word we’ve hit where the length byte has bit 6 set as well as bit 7. This bit is called the precedence bit, and is used during compilation. We won’t likely discuss how compilation works in Forth for some time, so for now we can simply note that there’s a bit in the definition that marks words that need to be handled differently.
If we put this into our listing and rerun -refs it picks up that COLON starts with DOCOL (as we noted before), but the rest of the implementation is unknown. Most colon definitions end with “.WORD SEMIS”, which we noted before restores IP from the stack and calls NEXT. But a handful of words like COLON handle it internally; we’ll have to postpone that discussion for COLON till we get to those words, specifically (;CODE).
The next word at $1261 is the word ‘;’ (distinct from ‘;S’), followed by CONSTANT at $1273. Like COLON, CONSTANT contains some code after a few Forth definitions at $1288, so we need to manually disassemble that and add a DOCON label to it.
1288: A0 02 DOCON LDY #2 128A: B1 FB LDA (W),Y 128C: 48 PHA 128D: C8 INY 128E: B1 FB LDA (W),Y 1290: 4C 3B 0D JMP PUSH
Constant takes the value after the current “.WORD DOCON” and pushes it onto the stack.
Next comes VARIABLE at $1293. It also is followed by a short 6502 routine called DOVAR at $12A4.
12A4: 18 DOVAR CLC 12A5: A5 FB LDA W 12A7: 69 02 ADC #2 12A9: 48 PHA 12AA: 98 TYA 12AB: 65 FC ADC W+1 12AD: 4C 3B 0D JMP PUSH
It is similar to DOCON, except it pushes the address after the DOCON word, not its value. So a CONSTANT contains an invariant word, a VARIABLE contains a variant block of data.
The next word is USER, at $12B0, with a block of code called DOUSE at 12BD.
12BD: A0 02 DOUSE LDY #2 12BF: 18 CLC 12C0: B1 FB LDA (W),Y 12C2: 65 FD ADC UP 12C4: 48 PHA 12C5: A9 00 LDA #0 12C7: 65 FE ADC UP+1 12C9: 4C 3B 0D JMP PUSH
The routine takes the byte after the “.WORD DOUSE” and adds it to UP, pushing the result onto the stack. In other words, it computes the address of some data in the user block.
We now come to a whole bunch of CONSTANT and USER definitions. Since those have a fixed size, and are then followed by the next definition, we should modify our decompiler to handle that more gracefully for us.
We start by adding some helpers before forth_buf.
sub print_byte {
my ($buff, $addr, $i) = @_;
my $b1 = unpack "C", substr($buff, $i, 1);
my $sval = sprintf "\$%02X", $b1;
$sval = sprintf "\$%01X", $b1 if $b1 < 16;
$sval = $b1 if $b1 < 10;
my $string = sprintf "%s.BYTE %s\n", get_label($addr + $i), $sval;
print sb($addr + $i, $b1), $string;
}
sub get_byteq {
return $_[0] == 0x0d5b || $_[0] == 0x12bd; # CLIT, DOUSE
}
sub get_defq {
my ($val, $val1) = @_;
return $val == 0x1045 || $val1 == 0x1288 || # SEMIS, DOCON
$val == 0x12bd || $val1 == 0x12a4; # DOUSE, DOVAR
}
And then add the following into the inner loop of forth_buf.
if (get_byteq($val)) {
print_byte($buff, $addr, $i);
$i += 1;
}
if (get_defq($val, $val1)) {
print "\n";
def_buf(substr($buff, $i), $addr + $i, $size - $i);
last;
}
This is code deserves some explanation. Up to now, forth_buf, has just dumped words of data to add to the listing, switching to disassembling code when the word happened to be *+2. Now it’s going to look for a couple of other conditions. If the current word is CLIT or DOUSE, the next value is a BYTE, not a WORD, so emit that next using print_byte. If the current word is SEMIS or DOUSE, or if the last word was DOCON or DOVAR, let’s switch to decompiling a definition by calling def_buf. Occasionally for DOVAR this is incorrect as variables can be more than one word, but most of the time, this heuristic works well.
In other words, we’ve changed forth_buf to determine when to start decompiling a definition automatically in a lot of cases.
Running this new code at $12CC leads to a much nicer output, the definitions for 0, 1, 2, and 3.
12CC: 81 B0 .BYTE $81,$B0 12CE: B0 12 .WORD L902 12D0: 88 12 .WORD DOCON 12D2: 00 00 .WORD 0 12D4: 81 B1 .BYTE $81,$B1 12D6: CC 12 .WORD $12CC 12D8: 88 12 .WORD DOCON 12DA: 01 00 .WORD 1 12DC: 81 B2 .BYTE $81,$B2 12DE: D4 12 .WORD $12D4 12E0: 88 12 .WORD DOCON 12E2: 02 00 .WORD 2 12E4: 81 B3 .BYTE $81,$B3 12E6: DC 12 .WORD $12DC 12E8: 88 12 .WORD DOCON 12EA: 03 00 .WORD 3
We still need to go back and fill in the labels, but we don’t have to restart the decompilation for every word as we had to do for the primitives.
Note that we could have done something similar in the primitives, assuming when we saw a JMP NEXT or JMP PUT we’re at the end of a definition, but that heuristic isn’t as reliable as the one we’ve just added for non-primitive definitions.
So let’s crank through definitions using our newly augmented decompiler. After 1, 2, 3, and 4 we find BL, C/L, FIRST, LIMIT, B/BUF, B/SCR and then HIMEM. HIMEM isn’t in the fig-Forth listing, it’s the first such word we’ve seen, so we’ll need some conventions for labeling its NFA and PFA. All the NFA’s so far have been labeled with L and the line number in the original listing, which while not very descriptive was adopted to follow the original source code as closely as possible. For new words, we instead are going to prefix them with NF, so HIMEM’s NFA is simply NFHIMEM, and it’s PFA is simply HIMEM.
After HIMEM is +ORIGIN, which is the first traditional colon definition in the listing. By traditional, I mean it starts with DOCOL and ends with SEMIS.
After that are some more words original to the Dealer Demo, ICCM, ICCL, ICCAL, which are CIO constants. Then comes the fig-Forth words EMIT, CR, and ?TERMINAL and KEY.
We then hit the definitions of all the fig-Forth USER variables: TIB, WIDTH, WARNING, FENCE, DP, VOC-LINK, BLK, IN, OUT, SCR, OFFSET, CONTEXT, CURRENT, STATE, BASE, DPL, FLD, CSP, R#, HLD. Each reserves 2 bytes in the USER block, starting at $0A and ending at $31. Dealer Demo then adds two of its own USER variables, INPT and PHYSOFF.
If we now run -refs, we find that we’ve found two missing forward references:
$1431: CURR: 123e $1424: CON: 1242
So going back and fixing those, we now have a fairly good listing up to address $148B. The next post will pick up from there, I think we’ve covered enough territory for today.
We resume our disassembly of the Dealer Demo and find the next four Forth words, 0= and 0<, and then unexpectedly U< and <.
-def WORD NFA PFA same as fig-Forth? 10A7 0= L605 ZEQU Yes 10BC 0< L619 ZLESS Yes 10CE U< L1246 ULESS No 10EB < L1254 LESS No
I say unexpectedly, because U< and < in fig-Forth are defined much later, and U< is not a primitive, it is a combination of - (subtraction) and 0<. That said, the motivation to make it a primitive is sound, because U< in fig-Forth is quite frankly broken. You can see this for instance in X-Forth, which uses the fig-Forth definition:
X-FORTH 1.1b 030105/cs 0 65535 U< . 0 OK
fig-FORTH 1.4V 0 65535 U< . 1 oK
The problem with defining U< with “: U< - 0< ;” is that when the difference is more than $7FFF, 0< is going to give the wrong answer, as it expresses a signed comparison. So it makes sense to make U< a primitive to avoid having to write a complicated implementation in Forth itself (as APX Forth did, which is probably 20x slower, and consumes more space to implement to boot).
10CE: 82 55 BC L1246 .BYTE $82,'U',$BC 10D1: BC 10 .WORD L619 10D3: D5 10 ULESS .WORD *+2 10D5: B5 02 LDA 2,X 10D7: D5 00 CMP 0,X 10D9: B5 03 LDA 3,X 10DB: F5 01 SBC 1,X 10DD: B0 07 BCS ULESS1 10DF: 98 TYA 10E0: 48 PHA 10E1: A9 01 LDA #1 10E3: 4C DC 0F JMP BINARY 10E6: 98 ULESS1 TYA 10E7: 48 PHA 10E8: 4C DC 0F JMP BINARY
The implementation of LESS is also different, but not exactly better. Here’s the fig-Forth implementation:
0A1F: 21 0A LESS .WORD *+2 0A21: 38 SEC 0A22: B5 02 LDA 2,X 0A24: F5 00 SBC 0,X 0A26: B5 03 LDA 3,X 0A28: F5 01 SBC 1,X 0A2A: 94 03 STY 3,X 0A2C: 50 02 BVC L1258 0A2E: 49 80 EOR #$80 0A30: 10 01 L1258 BPL L1260 0A32: C8 INY 0A33: 94 02 L1260 STY 2,X 0A35: 4C EE 03 JMP POP
Now the Dealer Demo version:
10EF: F1 10 LESS .WORD *+2 10F1: B5 02 LDA 2,X 10F3: D5 00 CMP 0,X 10F5: B5 03 LDA 3,X 10F7: F5 01 SBC 1,X 10F9: 50 02 BVC L1258 10FB: 49 80 EOR #$80 10FD: 10 07 L1258 BPL L1260 10FF: 98 TYA 1100: 48 PHA 1101: A9 01 LDA #1 1103: 4C DC 0F JMP BINARY 1106: 98 L1260 TYA 1107: 48 PHA 1108: 4C DC 0F JMP BINARY
The fig-Forth version frankly has a simpler/faster way of putting the result (1 or 0) into the top of the stack. The Dealer Demo version uses the trick of using CMP instead of SBC thus obviating the need to call SEC, but uses the same mechanism as ULESS to put the result on the stack. I think the fig-Forth version is the better code overall, and the same trick for setting the result could be applied to U< as well. But both work, and that’s the most important part. An excellent discussion of various approaches to signed and unsigned compare can be found at http://www.6502.org/tutorials/compare_beyond.html. Section 4.3 in particular discusses the technique used here.
The next dozen words or so exactly match the implementation in the fig-Forth listing, so they don’t merit much detailed discussion. They implement some arithmetic (+, -, D+, D-), some stack primitives (OVER, DROP, SWAP, DUP), some read/write primitives (@, C@, !, C!) and a couple of miscellaneous commands (+! and TOGGLE).
-def WORD NFA PFA same as fig-Forth? 110B + L632 PLUS Yes 1123 D+ L649 DPLUS Yes 1146 - L670 MINUS Yes 115E D- L685 DMINU Yes 1177 OVER L700 OVER Yes 1188 DROP L711 DROP Yes 1191 SWAP L718 SWAP Yes 11AA DUP L733 DUP Yes 11BA +! L744 PSTOR Yes 11D7 TOGGLE L762 TOGGL Yes 11EB @ L773 AT Yes 11FF C@ L787 CAT Yes 120F ! L798 STORE Yes 1226 C! L813 CSTOR Yes
In our next post we’ll hit colon definitions, CONSTANTs and VARIABLEs, which will encourage us to make some small enhancements to our tooling to accelerate our reverse engineering. Stay tuned.
In the last post I explained how the NEXT routine uses IP and W to JMP from word to word, but otherwise neglected to explain the Forth environment. Let’s remedy that.
Forth contains two stacks, a data stack and a return stack.
The 6502 stack pointer indexes the return stack in page one. Saving and restoring the interpreter pointer is done here when Forth ‘subroutines’ are entered and exited, in a fashion analogous to JSR and RTS for the 6502. The routine SEMIS which we’ll disassemble shortly is similar to RTS.
The 6502 X register indexes the data stack in page zero. The stack grow downwards, so 0,X and 1,X is the entry at the top of the stack, 2,X and 3,X is the next entry in the stack, et cetera. To add a word to the stack, decrement the X register twice (e.g. PUSH in the listing). To remove a word from the stack, increment the X register twice (e.g. ZBRAN and POP in the listing).
The 6502 Y register is always zero when a word is entered as a side effect of the implementation of the NEXT routine. Some routines take advantage of this, using TYA instead of LDA #0 (e.g. DIGIT).
The N label references 9 bytes of scratch space for the interpreter, from N-1 to N+7. Forth routines can use this space on page zero as needed. The SETUP routine for instance uses this space to transfer up to four words from the data stack for routines to work with.
The UP label is a pointer to the user area. It is set to $480 on startup and hold $80 bytes of interpreter state (this area was also used for the boot loader code). For instance, at offset 8 is RP0, the initial return stack value. Before the interpreter starts the stack pointer is set to that value by calling RPSTO, a routine we’ll be disassembling shortly.
The XSAVE label provide a byte for saving and restoring the X register. The Forth environment requires that the X register always hold the data stack index, so routines that need to use the X register for other purposes have to save and restore it before returning to the Forth NEXT routine. For instance, PLOOP transfers the return stack pointer to the X register (TSX) so it can manipulate the return stack directly, so it needs to use XSAVE to save and restore the data stack pointer.
And that for now completes our discussion of the Forth interpreter environment. On to our regularly scheduled disassembly.
The next four words in the listing match what we find in the fig-Forth listing:
-def WORD NFA PFA same as fig-Forth? 0e52 I L207 I Yes 0e58 DIGIT L214 DIGIT Yes 0e88 (FIND) L243 PFIND Yes 0eed ENCLOSE L301 ENCL Yes
After that the fig-Forth listing has words for EMIT, KEY, ?TERMINAL, and CR, which are all implemented as forward refences (the actual code appears much later in the listing). In the Dealer Demo, these words don’t appear here, they appear later with their implementation.
The next words are all identical to the fig-Forth listings.
0f2f CMOVE L365 CMOVE Yes 0f57 U* L386 USTAR Yes 0f8f U/ L418 USLAS Yes 0fcb AND L453 ANDD Yes 0fe1 OR L469 OR Yes 0ff6 XOR L484 XOR Yes 100c SP@ L499 SPAT Yes 101b SP! L511 SPSTO Yes 102b RP! L522 RPSTO Yes 1040 ;S L536 SEMIS Yes 1050 LEAVE L548 LEAVE Yes 106E >R L563 TOR Yes 1080 R> L577 RFROM Yes 1092 R L591 R Yes
U* and U/ are unsigned multiply and divide. It’s peculiar they appear so early in the listing, as they aren’t really needed to implement anything right away. Of note is that both are incorrectly implemented in almost every 6502 Forth, so that some inputs generate the wrong values. The U* bug is simply a missed overflow condition on adding the carry. It can be fixed by replacing:
LDA #0
ADC 0,X
STA 0,X
L411 DEY
BCC L411
INC 0,X
BNE L411
INC 1,X
L411 DEY
The USLAS bug I don’t think can be fixed as simply. See the discussion as http://6502.org/source/integers/ummodfix/ummodfix.htm
OK, I think that’s enough progress for today. I’m attaching WIP dealerdemo.lst, which is more than twice the length of the last update. Although we have a long way to go, progress will be faster once we’re done with the ‘primitive’ words.
So we spent the last two posts working on tooling to reverse engineer the Dealer Demo, and got the bootloader disassembled for study. Now we can start disassembling the Forth kernel. Let’s start with “-dis d00 8”:
0D00: EA ORIG NOP 0D01: 4C 8D 1C JMP $1C8D 0D04: EA NOP 0D05: 4C A1 1C JMP $1CA1
Looking at the fig-Forth listing, this looks like the start of the kernel. 1C8D will be COLD+2, the cold start routine, and 1CA1 will be WARM, the warm start routine.
The next 13 words should be data words according to the fig-Forth listing, configuration parameters used where initializing the kernel:
0D08: 00 00 .WORD 0 0D0A: 01 00 .WORD 1 0D0C: 00 00 .WORD 0 0D0E: 7F 00 .WORD $7F 0D10: 80 04 .WORD $0480 0D12: EC 00 .WORD $EC 0D14: FF 01 .WORD $01FF 0D16: 00 01 .WORD $0100 0D18: 1F 00 .WORD $1F 0D1A: 00 00 .WORD 0 0D1C: FB 86 .WORD $86FB 0D1E: FB 86 .WORD $86FB 0D20: 01 1C .WORD $1C01
Of these, we can identify $EC as the TOS (top of stack), $86FB is the pointer to the top of memory (TOP), $1C01 should be the initial vocabulary pointer (VL0).
The next bit of data should be the first definition, the LIT word. In fig-Forth, all definitions start with a byte declaring the length of the word in the bottom 5 bits, then the word itself with the high bit set on the last letter, although in rare cases there will be an extra byte to avoid having the definition fall on a $..FF boundary (why will become clear later). This is called the name field, and its address is referenced as the NFA. There are then two words, the first is the link field which points to the last definition (and is called the LFA). The next word is the parameter field (at the PFA), which points to the code to run for this word (this is the value that must not be at $..FF). For primitive words, it is usually just .WORD *+2, meaning that the code immediately follows the definition.
Knowing this we could slowly disassemble the code definition by definition by hand, but we want our tools to do as much of this as possible for use, so let’s implement a definition decompiler.
The first thing we need is a get_string method from a buffer. It should output something that can be appended to a .BYTE directive. In most cases, surrounding the values with single quotes should be adequate, but recall that the last byte in a definition string will have it’s high bit set and thus be unprintable, so let’s handle that case in the most general fashion possible so we can use the code for other purposes later:
sub get_string {
my ($buff) = @_;
my ($string, $inquote) = ('', 0);
foreach my $val (unpack "C*", $buff) {
my $printable = $val >= 0x20 && $val <= 0x7e;
($inquote = 1, $string .= "'") if ($printable && !$inquote);
($inquote = 0, $string .= "',") if (!$printable && $inquote);
$string .= sprintf "%c", $val if $printable && $val != 0x27;
$string .= sprintf "''", $val if $val == 0x27;
$string .= sprintf "\$%02X", $val if $val >= 10 && !$printable;
$string .= $val if $val < 10;
$string .= ',' if !$printable;
}
$string .= "'" if $inquote;
$string =~ s/,$//; $string;
}
OK, now let’s define def_buf, the routine that will parse a definition:
sub def_buf {
my ($buff, $addr, $size) = @_;
my $first = unpack "C", substr($buff, 0, 1);
my $nfaSize = $first & 0x3f;
if (($first & 0x80) == 0 || $nfaSize == 0 || $nfaSize > 0x1f) {
forth_buf($buff, $addr, $size);
return;
}
$nfaSize += 1 if (($addr + $nfaSize + 3) & 0xff) == 0xff; # handle CFA = ..ff
my $string = get_string(substr($buff, 1, $nfaSize));
my $count = $nfaSize + 1;
$string = sprintf "%s.BYTE \$%02X,%s", get_label($addr), $first, $string;
multi_buf($buff, $addr, $count, $string);
forth_buf(substr($buff, $count), $addr + $count, $size - $count);
}
sub def {
my ($buff, $addr, $size) = read_img(@_);
def_buf($buff, $addr, $size);
}
This looks for the length byte first, if the value found doesn’t look like a length byte, we go to our general parser called forth_buf (which mostly just dumps .WORDs of data, more on that shortly). It then adjusts the size if we’re at the $..FF boundary, gets the string, passes it to multi_buf to output, and then resumes in the general forth_buf parser.
So what does forth_buf look like:
sub print_word {
my ($buff, $addr, $i) = @_;
my ($b1, $b2) = unpack "C*", substr($buff, $i, 2);
my $val = $b1 | ($b2 << ;
my $sval = sprintf "\$%04X", $val;
$sval = sprintf "\$%02X", $val if $val < 256;
$sval = $names->{$val} if exists $names->{$val};
$sval = sprintf "\$%01X", $val if $val < 16;
$sval = $val if $val < 10;
$sval = '*+2' if $addr + $i + 2 == $val;
my $string = sprintf "%s.WORD %s\n", get_label($addr + $i), $sval;
print sb($addr + $i, $b1, $b2), $string; $val;
}
sub forth_buf {
my ($buff, $addr, $size) = @_;
my ($val, $val1) = (-1, -1);
for (my $i = 0; $i + 2 < $size; ) {
$val1 = $val;
$val = print_word($buff, $addr, $i);
$i += 2;
if ($addr + $i == $val) {
disasm_buf(substr($buff, $i), $addr + $i, $size - $i);
last;
}
}
}
sub forth {
my ($buff, $addr, $size) = read_img(@_);
forth_buf($buff, $addr, $size);
}
It calls print_word to print the word sensibly. When the word is *+2, we switch to disasm_buf, otherwise we call print_word again on the next word.
Now to hook it up into main:
elsif ($opt =~ /^\-def/) {
def(@_);
}
else {
forth(@_);
}
So now let’s run “-def d22”
0D22: 83 4C 49 .BYTE $83,'LI',$D4 0D25: D4 0D26: 00 00 .WORD 0 0D28: 2A 0D .WORD *+2 0D2A: B1 F8 LDA ($F8),Y 0D2C: 48 PHA 0D2D: E6 F8 INC $F8 ...
This exactly matches up to the fig-Forth listing, but with some missing constants. It looks like $F8 is the interpreter pointer (IP), so we need to add the following to the listing:
=00EC TOS = $EC
=00F0 N = $F0
=00F8 IP = N+8
=00FB W = IP+3
=00FD UP = W+2
=00FF XSAVE = UP+2
We can now start disassembling and inserting the labels as we go along from the fig-Forth listing to get a more readable listing.
Continuing disassembling we see the PUSH, PUT, NEXT, CLIT and SETUP routines. The first difference between Dealer Demo and the published fig-Forth can now be seen. The NEXT routine doesn’t have a JSR TRACE call and all the associated tracing code has been omitted. This is to be expected, since that code is there for the initial debugging of the kernel during bootstrapping, there’s no need to keep it once the Forth is up an running.
There is another small difference, the single byte literal command CLIT is implemented, but the name and link for it have been dropped so you can’t use it. Nonetheless we’ll see it used in the code later, so how it got removed is a bit of a mystery.
NEXT is the most important routine in all of Forth, so it’s helpful to study it for a moment.
0D42: A0 01 NEXT LDY #1 0D44: B1 F8 LDA (IP),Y 0D46: 85 FC STA W+1 0D48: 88 DEY 0D49: B1 F8 LDA (IP),Y 0D4B: 85 FB STA W 0D4D: 18 CLC 0D4E: A5 F8 LDA IP 0D50: 69 02 ADC #2 0D52: 85 F8 STA IP 0D54: 90 02 BCC L54 0D56: E6 F9 INC IP+1 0D58: 4C FA 00 L54 JMP W-1
How does this work? First, we move the value pointed to by the interpreter pointer (IP) into the W register. Then we increment IP by 2 and then JMP to W-1. W-1 was setup with the indirect jump opcode at startup (6C), so this immediately jumps again to the value at the address we just picked up from the instruction pointer. So Forth basically uses this NEXT routine to automatically jump to addresses held in memory one after the other. In a way it executes code a bit like the 6502, except there are no opcodes, every operation is a 2-byte pointer to an address in memory, which itself points to the code we need to execute. This implementation style for Forth is known as indirect threaded code.
Let’s work through an example. Suppose the IP is currently pointing to an address defined with .WORD LIT (i.e. the code for a literal value). LIT equals $0D28, and the value at that address is $0D2A. So when NEXT is called, it copies the value $0D28 into the W register, increments IP by 2, and then calls JMP W-1. The next instruction is therefore JMP ($0D28), which sets the program counter to $0D2A and continues running LDA (IP),Y, et cetera. The code for LIT itself consumes the next word and puts it on the data stack, and increments IP by 2, then falls through to NEXT to fetch the next word.
It’s now hopefully obvious why no definition’s parameter field can be on address $..FF. JMP (xxxx) on the 6502 has a bug where it will incorrectly fetch the destination when the address crosses the page boundary, and the NEXT routine relies on JMP (xxxx) to function. This isn’t a problem though, we just need to detect when this is about to happen and move the definition by one byte, which is why the name string sometime has an extra byte.
Continuing our disassembly, we find the next few words:
-def WORD NFA PFA same as fig-Forth? d74 EXECUTE L75 EXEC Yes d8d BRANCH L89 BRAN Yes dab 0BRANCH L107 ZBRAN Yes dcd (LOOP) L127 PLOOP Yes dfc (+LOOP) L154 PPLOO Yes e36 (DO) L185 PDO Yes
In each case we have to add at least a couple of labels to the list (one at the NFA, and then one two lines later at the PFA), more if there are branches in the code. In a few cases, we need to revert a symbol inserted to better match the fig-Forth listing, but largely the tool we have is doing much of the work once we recognize the start of a definition.
In all of these cases, the listing matches the implementation in the original fig-Forth listing.
As we’ve been disassembling, we have also been modifying set_name to cover more cases. In particular W-1, N-1, N+1, N+2, ..., N+7, NEXT+2, and BRAN+2 should be recognized, so we’ve add the following lines to set_name:
$names->{$val+2} = "$label+2" if grep { $_ eq $label } qw/NEXT BRAN/;
$names->{$val-1} = "$label-1" if grep { $_ eq $label } qw/N W/;
if ($label eq 'N') {
$names->{$val+$_} = "$label+$_" for (2..7);
}
That’s enough for now, I’m attaching our progress and we’ll pick it up again next time.
The last post walked through how to implement the basics of a disassembler, but we omitted several features to simplify the initial presentation. So let’s fill in the omissions.
The main omission is the lack of a symbol table. I included a $names hash to hold the symbols and used it where needed, but there was no code to populate it. So let’s fix that.
sub open_lst {
open my $fh, '<', 'dealerdemo.lst' or die;
$fh;
}
sub set_name {
my ($label, $val, $type) = @_;
return if $val < 10;
$names->{$val} = $label;
$names->{$val+1} = "$label+1" if $type != 1;
}
sub populate_names {
my $fh = open_lst();
while (<$fh>) {
if (/^([0-9A-F]{4}): .{10}(\w+)\s+\.WORD/) {
my ($label, $val) = ($2, hex $1);
set_name($label, $val, 2);
}
elsif (/^([0-9A-F]{4}): .{10}(\w+)\s+(.*)/) {
my ($label, $val) = ($2, hex $1);
set_name($label, $val, 1);
}
elsif (/^ {5}=([0-9A-F]{4}) {6}(\w+)\s*=\s*(\S+)/) {
my ($label, $val, $rest) = ($2, hex $1, $3);
set_name($label, $val, 2);
}
}
}
This code says we can get our symbols from the file we are slowly building up, 'dealerdemo.lst'. It looks for assignments (e.g., ' =0000 FOO = $00'), and labels and put them into the name table, ignoring small values.
Now we need to hook this up.
sub main {
populate_names();
disasm(@_);
}
main(@ARGV);
And add some symbols for it to consume. Since the disassembly showed values in the low $03xx range, we add the SIO symbols to the top of our listing:
=0301 DUNIT = $0301
=0302 DCOMND = $0302
=0303 DSTATS = $0303
=0304 DBUFLO = $0304
=0305 DBUFHI = $0305
=0308 DBYTLO = $0308
=0309 DBYTHI = $0309
=030A DAUX1 = $030A
=030B DAUX2 = $030B
;
=E459 SIOV = $E459
And rerun the disassembler to start seeing symbols appear.
... 0490: 85 F0 STA $F0 0492: A9 52 LDA #$52 0494: 8D 02 03 STA DCOMND 0497: AD 8C 04 LDA $048C 049A: 8D 0A 03 STA DAUX1 049D: AD 8D 04 LDA $048D 04A0: 8D 0B 03 STA DAUX2 04A3: A9 01 LDA #1 04A5: 8D 01 03 STA DUNIT ...
Now in addition to disassembling, we also need to just plain dump data out. To do that first we need to direct read_img to specific areas rather than hard code it. Here’s our initial replacement.
sub read_img {
my ($addr, $size) = @_; $addr = 0 if !defined $addr;
$size = 0x30 if !defined $size;
$addr = hex $addr;
my ($buff, $offset) = ('', 0);
$offset = 0x10 - 0x480 if $addr >= 0x480 && $addr < 0x500;
$buff = read_file('../../../atr/dealerdemo.atr', $size, $offset + $addr) if $offset != 0;
($buff, $addr, length($buff));
}
So we translate addresses between 0x480 and 0x500 to offsets into the atr file, rather than hard coding the offset, address and size as in v1.
Next, we need to add code to dump data.
sub multi_buf {
my ($buff, $addr, $size, $def) = @_;
my $i = 0;
for (;;) {
my $count = $size - $i;
$count = 3 if $count > 3;
last if $count == 0;
print sb($addr + $i, unpack "C*", substr($buff, $i, $count)), "$def\n";
$def = '';
$i += $count;
}
}
sub get_label {
my $label = $names->{$_[0]} || '';
$label = '' if $label =~ /\+\d+$/;
sprintf "%-8s", $label;
}
sub bytes {
my $count = shift;
my ($buff, $addr, $size) = read_img(@_);
for (my $i = 0; $i + $count - 1 < $size; $i += $count) {
my @vals = unpack "C*", substr($buff, $i, $count);
my $string = sprintf "%s.BYTE ", get_label($addr + $i);
$string .= join ',', map { sprintf "\$%02X", $_ } @vals;
multi_buf(substr($buff, $i, $count), $addr + $i, $count, $string);
}
}
sub words {
my $count = shift;
my ($buff, $addr, $size) = read_img(@_);
$count *= 2;
for (my $i = 0; $i + $count - 1 < $size; $i += $count) {
my @vals = unpack "v*", substr($buff, $i, $count);
my $string = sprintf "%s.WORD ", get_label($addr + $i);
$string .= join ',', map { sprintf "\$%04X", $_ } @vals;
multi_buf(substr($buff, $i, $count), $addr + $i, $count, $string);
}
}
The routines bytes and words chunks up bytes and words into groups of $count, and uses multi_buf to output them in listing format. The routine get_label is an oversight from the last post, the disassembler should be able to put a label instead of 8 spaces before the opcode/directive if it has one handy. Since we didn’t populate the symbol table in the last post, the code wasn’t yet needed, but now we should replace this line in print_op:
my $string = sprintf "%s%s%s\n", ' ', $def->{name}, $sval;
my $string = sprintf "%s%s%s\n", get_label($addr + $i), $def->{name}, $sval;
Now let’s hook up all the pieces.
sub main {
populate_names();
my $opt = shift;
if ($opt =~ /\-dis/) {
disasm(@_); }
elsif ($opt =~ /^\-b\d+$/) {
bytes(substr($opt, 2), @_);
}
elsif ($opt =~ /^\-w\d+$/) {
words(substr($opt, 2), @_);
}
}
main(@ARGV);
Now we’re in a position to make sense of the boot loader. Since the first six bytes are 2 bytes and 2 words, we can use the arguments “-b1 480 2” and “-w1 482 4” to dump that data. The word at 0x484 is $E4C0, which is ERRTN, so let’s add a declaration for that and change it to .WORD ERRTN in the listing. We can then disassemble the code at $0486, which does a load and then an immediate branch over the next six bytes. Those six bytes appear to be grouped as 3 words, $0D00, 1, and $80. Let’s label them XDST, XSECT, and XSIZE, add ORIG = $0D00 for the first value and continue disassembly (the names for those locations will make sense in a minute). If we now disassemble using “-dis 0x490” we should see:
0490: 85 F0 STA $F0 0492: A9 52 LDA #$52 0494: 8D 02 03 STA DCOMND 0497: AD 8C 04 LDA XSECT 049A: 8D 0A 03 STA DAUX1 049D: AD 8D 04 LDA XSECT+1 04A0: 8D 0B 03 STA DAUX2 04A3: A9 01 LDA #1 ...
We need to add branch labels at locations 0490 (XBOOT), 4C0 (XBOOT1) 04C8 (XBOOT2), 04D7 (XBOOT3). The last six bytes in the sector look like leftover data, so lets just list those as groups of three bytes using “-b3 4fa 6”
We can now study the code and see that it simply calls SIOV with the READ ($52) command $F4 times, starting the buffer at ORIG, and incrementing it by $80 each time. So it loads sectors 2 up into 0xd00 to 0x8700, so let’s add the following to read_img.
$offset = 0x90 - 0xd00 if $addr >= 0xd00 && $addr < 0x8700;
Having now doubled the size of tool, we now are in a position to start disassembling the Forth kernel, but I’ll leave that for the next post. Here’s the progress so far.
So to decompile the Dealer Demo, we need to start by peeking at the boot sectors to see how it starts.
000010: SECTOR: 001: FILE:
0 ff 01 80 04 c0 e4 a9 f4-d0 06 00 0d 01 00 80 00 ................
10 85 f0 a9 52 8d 02 03 ad-8c 04 8d 0a 03 ad 8d 04 ...R............
The first six bytes of the boot sector describes the boot sequence. The first byte is a flag byte (0xff) and the second byte is the number of sectors to load (DBSECT), in this case one. The third and fourth byte are where to start copying the sectors to on boot ($0480). The fifth and sixth bytes is the boot address (BOOTAD), in this case ($e4c0) a pointer to an RTS instruction in 800A/B O.S. Execution resumes after the boot address at the seventh byte.
So the Dealer Demo boot is pretty spartan. It loads a single sector and then that sector has to load the rest of the program. So we need to disassemble the rest of sector 1.
When I first disassembled this, I used Altirra’s debugger to do this (after booting the disk, break into the debugger and type “u 480”), but that had a few drawbacks. By default, Altirra will disassemble both valid and “invalid” opcodes, so when the disassembly is going through a mixture of data and code, it will generate a lot of weird disassembly that is really just data. I’d rather when it hit invalid opcodes to just emit the data as .BYTE statements, since the invalid opcodes are rarely used. Another drawback is that it is disassembling what’s in memory, and some of the memory by the time we get around to disassembling it has changed. So I decided it was time to move away from Altirra and write my own disassembler.
Writing a disassembler is pretty easy. You need an array containing a definition of each of the 256 possible opcodes. Most of them will just map to .BYTE <value>, but the valid ones will have a name and a mode. The mode determines how much additional data to read (0, 1 or 2 bytes) and how to format the line. Move the pointer forward that many bytes and repeat the process. In other words, something like so:
sub disasm_buf {
my ($buff, $addr, $size) = @_;
my $opcodes = opcodes();
my $i = 0; for (my $i = 0; $i + 1 < $size;) {
my $code = unpack "C", substr($buff, $i, 1);
my $def = $opcodes->[$code];
my $count = print_op($buff, $addr, $i, $def);
$i += $count;
}
}
Here all the heavy lifting will be done by print_op, which relies on the definitions from the opcodes function to know what to print.
What does opcodes look like?
sub opcodes {
my $opcodes = [];
$opcodes->[$_] = { name => (sprintf ".BYTE \$%02X", $_), mode => 'ILL' } for (0..255);
my $modes = {
'IMP' => [
'00BRK', '40RTI', '60RTS',
'08PHP', '18CLC', '28PLP', '38SEC', '48PHA', '58CLI', '68PLA', '78SEI',
'88DEY', '98TYA', 'a8TAY', 'b8CLV', 'c8INY', 'd8CLD', 'e8INX', 'f8SED',
'8aTXA', '9aTXS', 'aaTAX', 'baTSX', 'caDEX', 'eaNOP' ],
'ZP' => [
'24BIT', '84STY', 'a4LDY', 'c4CPY', 'e4CPX',
'05ORA', '25AND', '45EOR', '65ADC', '85STA', 'a5LDA', 'c5CMP', 'e5SBC',
'06ASL', '26ROL', '46LSR', '66ROR', '86STX', 'a6LDX', 'c6DEC', 'e6INC' ],
'ZP,X' => [
'94STY', 'b4LDY',
'15ORA', '35AND', '55EOR', '75ADC', '95STA', 'b5LDA', 'd5CMP', 'f5SBC',
'16ASL', '36ROL', '56LSR', '76ROR', 'd6DEC', 'f6INC' ],
'IMM' => [
'a0LDY', 'c0CPY', 'e0CPX', 'a2LDX',
'09ORA', '29AND', '49EOR', '69ADC',
'a9LDA', 'c9CMP', 'e9SBC' ],
'REL' => [
'10BPL', '30BMI', '50BVC', '70BVS', '90BCC', 'b0BCS', 'd0BNE', 'f0BEQ' ],
'ABS' => [
'20JSR', '2cBIT', '4cJMP', '8cSTY', 'acLDY', 'ccCPY', 'ecCPX',
'0dORA', '2dAND', '4dEOR', '6dADC', '8dSTA', 'adLDA', 'cdCMP', 'edSBC',
'0eASL', '2eROL', '4eLSR', '6eROR', '8eSTX', 'aeLDX', 'ceDEC', 'eeINC' ],
'ABS,X' => [
'bcLDY', '1dORA', '3dAND', '5dEOR', '7dADC',
'9dSTA', 'bdLDA', 'ddCMP', 'fdSBC', '1eASL', '3eROL', '5eLSR', '7eROR',
'deDEC', 'feINC' ],
'ABS,Y' => [
'19ORA', '39AND', '59EOR', '79ADC', '99STA', 'b9LDA', 'd9CMP', 'f9SBC', 'beLDX' ],
')Y' => [
'11ORA', '31AND', '51EOR', '71ADC', '91STA', 'b1LDA', 'd1CMP', 'f1SBC' ],
'X)' => [
'01ORA', '21AND', '41EOR', '61ADC', '81STA', 'a1LDA', 'c1CMP', 'e1SBC' ],
'ACC' => [ '0aASL', '2aROL', '4aLSR', '6aROR' ],
'IND' => [ '6cJMP' ],
};
foreach my $mode (keys %$modes) {
$opcodes->[hex substr($_, 0, 2)] = { name => substr($_, 2), mode => $mode } for (@{$modes->{$mode}});
}
$opcodes;
}
Although it looks formidable, it really isn’t. The function creates an array of 256 definitions, with a name of “.BYTE $xx” and a mode of 'ILL', for illegal. It then takes data from the $modes hash table to fill in the valid opcodes. For example in the key 'ACC' there is a value '0aASL'. So that expands to $opcodes->[0x0a] = { name => 'ASL', mode => 'ACC' }, or opcode 0x0a is “ASL” in accumulator mode (i.e. ASL A). In other words, I’ve represented the valid opcode information in a compact form, and used a little code to expand it into a more useful representation.
When I first did this of course I missed a few opcodes, and reversed SEI and SED (which went undetected for a long time because they are rare opcodes) but I’ve used this routine for several months now and am fairly certain it’s accurate now.
print_op is a bit more complicated, but not too horrible. To implement it, let’s first write a helper function.
sub sb {
if (scalar(@_) == 2) {
sprintf "%04X: %02X ", @_;
}
elsif (scalar(@_) == 3) {
sprintf "%04X: %02X %02X ", @_;
}
elsif (scalar(@_) == 4) {
sprintf "%04X: %02X %02X %02X ", @_;
}
}
This routine is designed to show bytes (thus the name sb) in the first 16 characters of a line: sb(0x480, 0xff, 0x01, 0x80) for instance would show:
0480: FF 01 80
Ok, we’re now ready for print_op.
our $names = {};
sub print_op {
my ($buff, $addr, $i, $def) = @_;
my $mode = $def->{mode};
my $sval = '';
my $count = 1;
if ($mode eq 'IMM') {
my $val = unpack "C", substr($buff, $i + 1, 1);
$sval = sprintf "\$%02X", $val;
$sval = $val if $val < 10;
$sval = " #$sval";
$count = 2;
}
elsif ($mode eq 'ZP' || $mode eq 'ZP,X' || $mode eq 'ZP,Y' ||
$mode eq 'IMM' || $mode eq ')Y' || $mode eq 'X)') {
my $val = unpack "C", substr($buff, $i + 1, 1);
$sval = $names->{$val} || sprintf "\$%02X", $val;
$sval = $val if $val < 10; $sval = " $sval" if $mode eq 'ZP';
$sval = " $sval,X" if $mode eq 'ZP,X';
$sval = " $sval,Y" if $mode eq 'ZP,Y';
$sval = " ($sval,X)" if $mode eq 'X)';
$sval = " ($sval),Y" if $mode eq ')Y';
$count = 2;
}
elsif ($mode eq 'REL') {
my ($b1) = unpack "c", substr($buff, $i + 1, 1);
my $val = $addr + $i + 2 + $b1;
$sval = $names->{$val} || sprintf "\$%04X", $val;
$sval = " $sval"; $count = 2;
}
elsif ($mode eq 'ABS' || $mode eq 'ABS,Y' || $mode eq 'ABS,X' || $mode eq 'IND') {
my $val = unpack "v", substr($buff, $i + 1, 2);
$sval = $names->{$val} || sprintf "\$%04X", $val;
$sval = " $sval" if $mode eq 'ABS';
$sval = " $sval,X" if $mode eq 'ABS,X';
$sval = " $sval,Y" if $mode eq 'ABS,Y';
$sval = " ($sval)" if $mode eq 'IND';
$count = 3;
}
elsif ($mode eq 'ACC') {
$sval = ' A';
}
elsif ($mode eq 'IMP' || $mode eq 'ILL') {
}
else {
die "$mode";
}
my $string = sprintf "%s%s%s\n", ' ', $def->{name}, $sval;
print sb($addr + $i, unpack "C*", substr($buff, $i, $count)), $string;
}
So for each mode, we compute the value to append after the definition name. For instance, for 'ACC' we set $sval = ' A', and for 'REL' (a relative branch) we compute the destination and use that for $sval.
In a few spots there is reference to the $names hash. This is an empty hash currently, but eventually we will put well known symbols into it so that we don’t always output numbers.
OK, we are almost there. We need a read_file routine of course. We also need a read_img routine that will help us translate addresses to offsets into the file. For now, we’ll just hard code that the first sector is 0x480 to 0x4ff and change this in the future.
sub read_file {
my ($file, $size, $pos) = @_;
$size = -s $file if !defined $size || $size == 0;
open my $fh, '<', $file or die "open: $file\n";
binmode $fh; seek $fh, $pos, 0 if defined $pos;
my $buff; read $fh, $buff, $size; close $fh;
return $buff;
}
sub read_img {
my $buff = read_file('../../atr/dealerdemo.atr', 0x80, 0x10);
($buff, 0x480, length($buff));
}
The path in read_img is for my particular setup, you may need to change it. OK, now let’s hook it all up.
sub disasm {
my ($buff, $addr, $size) = read_img(@_);
disasm_buf($buff, $addr, $size);
}
disasm(@ARGV);
And run it to get a listing:
0480: FF .BYTE $FF 0481: 01 80 ORA ($80,X) 0483: 04 .BYTE $04 0484: C0 E4 CPY #$E4 0486: A9 F4 LDA #$F4 0488: D0 06 BNE $0490 048A: 00 BRK 048B: 0D 01 00 ORA $0001 048E: 80 .BYTE $80 048F: 00 BRK 0490: 85 F0 STA $F0 0492: A9 52 LDA #$52 0494: 8D 02 03 STA $0302 0497: AD 8C 04 LDA $048C 049A: 8D 0A 03 STA $030A 049D: AD 8D 04 LDA $048D 04A0: 8D 0B 03 STA $030B 04A3: A9 01 LDA #1 04A5: 8D 01 03 STA $0301 04A8: AD 8A 04 LDA $048A 04AB: 8D 04 03 STA $0304 ...
Not perfect, but a great start for only ~160 lines of code. We need to start populating the $names array to make this a bit more readable, and improve read_img so we can target where the disassembly takes place, but we’ll talk about how we’re going to do that next time.
I’m going to take a break from fixing disks images for now and instead talk about the language Forth.
Forth grew out of the programming work by Charles Moore in the late 1960s. By the late 1970s interest in it was such that a group of enthusiasts banded together to program it for the microcomputers that were then becoming available. They formed the Forth Interest Group (or FIG) and placed several implementations of the language in the public domain. The implementation for the 6502 was done primarily by William Ragsdale, and was complete in 1980, and it is from this implementation it seems all the various Atari versions derived. That 6502 fig-Forth listing can be found at https://archive.org/details/fig_FORTH_6502_Assembly_Source_Listing, and I’ve cleaned up the OCR to create a more readable listing at https://ksquiggle.neocities.org/ff6502.htm.
It was a significantly different language than the more commonly available BASIC, and due to its low cost and better performance it attracted many adherents. Since the code was available on such liberal terms (essentially public domain), many versions of it proliferated from user groups and some companies, and for a short time it was given modest coverage in the Atari-specific magazines such as Analog and Antic.
Forth is built around a small (~6k) kernel of core words which manipulate a pair of stacks and a handful of registers. The compiler converts the code to series of addresses which are interpreted by a tiny virtual machine, and the emphasis is placed on building up the code in small routines (called words). Almost all parts of the language can be rewritten if they are not to the programmer’s liking, and a clever inline assembler that let’s one rewrite timing critical routines.
Despite these strengths, the language failed to become especially popular. Large programs implemented using the fig-Forth model could consume all the memory available if care wasn’t taken, and the language itself was typically more difficult to learn than the BASIC’s of the day. Programmer’s who wanted to escape the limitations of BASIC were encouraged to investigate assembler more frequently, or perhaps a BASIC compiler. This wasn’t just a problem with Forth, all the languages for the Atari seemed to struggle to attract users.
The earliest version of Forth to appear on the Atari was almost certainly the Forth used to implement the Dealer Demo, most likely in 1980. Developed in-house at Atari, it was released to demonstrate the Atari computers capabilities. It later became the more sophisticated “Coin-Op” Forth. Antic Forth and the fig-Forth 1.4S distributed originally by SoftSide magazine all share this later kernel, but at the time the Dealer Demo was released, the Forth kernel in it was not significantly different than the original fig-Forth kernel, and with a copy of the demo disk you could construct a Forth kernel from it with a little bit of sector editing.
To show this, I’m going to decompile the Dealer Demo and extract its kernel for comparison with the original fig-Forth listing. That comparison is the goal, but I’m going to spend quite a bit of time developing the tooling to help do this, as we can leverage those tools in the future to examine other programs. Rather than simply drop all the code at once, I’m going to develop it up again like I did with the atr patching tools, hopefully giving some motivation for the choices I made in the code, and giving readers enough context to adapt the code for other purposes.
The Suburban Chicago Atarians (S.C.A.T.) is represented on the Pooldisk with a couple hundred disks from their public domain library. As with much of the Pooldisk, there is some identifiable bitrot in a small number of the disks. In particular, checking the file integrity found four disks with simple single empty sector in the chain.
The fixes are all a simple application of the tool we developed in earlier posts:
c:atr.pl -copy "..\..\mag\Antic\Antic Vol 3-04 1984-08 S1 Lucasfilm New Games.atr" 039.atr 499 217 1 c:atr.pl -copy 057_a.atr 056.atr 17 17 0 c:atr.pl -copy 095.atr 094_b.atr 595 634 1 c:atr.pl -copy 101.atr 101.atr 351 351 1
Notice in the last file we patched the sector links by copying the sector onto itself and then requesting to fix the links. Sneaky.
There are a few other disks that are clearly broken, but they aren’t as simple to fix, so that’s all we’re going to do today.
Recently I’ve been reading the Tyne & Wear Atari User’s Group Newsletters which I mostly got from atarimania.com. They are missing a couple of issues, so I was relieved to find another set of scans at https://www.strotmann.de/~cas/Infothek/ to fill in the gaps. Yay!
Anyhow, there is a long series in that newsletter by Keith Mayhew called “Cracking the Code”, which teaches assembly language. I’m always an advocate for more assembly tutorials, and this one apparently is a reprint from another newsletter I’d not heard of before, The U.K. Atari Computer Owners Club Newsletter, or Monitor. So I went looking for that newsletter and found http://www.page6.org/ukacoc/archive/archive.htm, a complete archive of the newsletter, and the newsletter appears to be professionally published (minus the first few issues). Yes!!
I followed the link, and landed on the site issuu.com, where the content is hosted. And started to weep... Issuu.com provides the newsletters, but in a web viewer that won’t export the data no matter how nice I ask it (it wants me there to show me endless ads of course). I was hoping to use these scans to patch up some of the listings which are difficult to read in the TWAUG scans, but if I can’t get them out of the browser, they’re useless to me.
Necessity being the mother of invention, I rolled my sleeves up and tried to figure out how issuu.com delivered the content.
First, let’s open a newsletter in any browser that monitors network traffic (I used Microsoft Edge), and look at the requests made while paging through the newsletter. This showed me that the pages of the newsletter were in files called page_1.bin, page_2.bin, all with a similar complex URL from layers.isu.pub for a given newsletter.
Now, let’s grab one of these binary files with wget and analyze it. The files start like so:
page_01.bin
0 1f 8b 08 00 00 00 00 00-00 ff 9c bb 7b 38 93 7f ............{8..
10 fc 3f 7e af c9 88 48 a9-c8 98 b2 52 92 1c 3a 6c .?~...H....R..:l
If you search for “1f 8b 08 ...” you’ll get hits indicating this is a gzip file. OK, let’s gunzip and dump it again:
page_01.xxx
0 08 01 10 02 1a 03 31 38-30 20 a5 08 28 dc 0b 30 ......180 ..(..0
10 01 4a 08 0a 06 10 a5 08-18 dc 0b 5a 90 90 1a 0a .J.........Z....
20 86 90 1a ff d8 ff e0 00-10 4a 46 49 46 00 01 01 .........JFIF...
30 00 00 01 00 01 00 00 ff-db 00 43 00 08 06 06 07 ..........C.....
OK, that JFIF signature is familiar, it means this is a JPEG file. Although the first 0x23 byte don’t look right to me, so I need to rip those off. Fortunately, I wrote a script for that ages ago.
sub cut_file {
my ($file, $offset, $size) = @_;
$offset = hex $offset if $offset =~ /^0x/;
$offset = -hex substr($offset, 1) if $offset =~ /^\-0x/;
$size = hex $size if $size =~ /^0x/;
my $buff;
my $fileSize = -s $file; $offset = $fileSize + $offset if $offset < 0;
open my $fh, '+<', $file or die;
binmode $fh;
seek $fh, $offset + $size, 0;
read $fh, $buff, $fileSize - $offset - $size;
seek $fh, $offset, 0; print $fh $buff;
truncate $fh, $fileSize - $size;
close $fh;
}
If we put that in a script called cut.pl, we can fix these files via cut.pl page_1.jpg 0 0x23.
And sure enough, once I remove the first 0x23 bytes, the file parses and displays as an ordinary JPEG file.
So for a given issue, you can generate a cbz (Comic Book viewer file) by doing something like so:
for /l %i in (1,1,60) do wget --no-check-certificate https://layers.isu.pub/<...>/page_%i.bin for /l %i in (1,1,9) do ren page_%i.bin page_0%i.bin ren *.bin *.jpg.gz gunzip *.gz for %i in (*.jpg) do cut.pl "%i" 0 0x23 7z -tzip a ukacoc.cbz *.jpg
Now to get to work transcribing some of these...
Another badly damaged Page 6 issue disk is 78B. I first noticed the VTOC and directory sectors had damage. A very peculiar damage, the 2nd byte of each sector was always 0x82. In fact, for the entire disk, the 2nd byte of every sector is 0x82. This disk is not in a happy place.
Still, we can (mostly?) fix this. Fixing the VTOC sector and the directory is easy enough. Fixing the DOS and DUP files is easy, we can find the correct bytes on almost any other Page 6 issue disk. Fixing some of the listings is possible by comparing the data against the listings in the magazine. Most of the disk is a game called Dragonlord, which is also on the OHAUG disk ohja96b.atr, so we can restore that game using that.
That leaves the file COMPRESS.TB, which is not listed in the magazine and which I can’t find another source. By parsing it and requiring the line numbers are increasing (as we did with the Rubik’s cube game) spots a few sectors we can fix. We can continue to parse the Turbo Basic file over and over adjusting the 0x82 values until we get something that’s pretty close to the original. It’s likely a few of the 0x82 values I changed are still incorrect, but given the extent of the damage, the repair job is a minor miracle.
BTW, I’ve found two TWAUG disks (2A and 2B) with similar damage. Someday I’ll fix those up, but given the amount of work involved, it won’t be anytime soon. ;-)
The Page 6 Issue Disks are an excellent collection of Atari software you can find at http://page6.org. The integrity of the disks is generally good, but I’ve found some disks with issues. Issue Disk 55A in particular is badly damaged, but we can try to at least partly fix it.
Sectors 232-235 and 642-706 are marked in the VTOC as free (and are empty on the disk), but they shouldn’t be, and this damages some of the files:
Restoring the MENU program is possible with some cleverness. Similar menu programs end with DATA statements description,filename, and it appears we’ve cut off the word “SPACE GAUNT”, so let’s finish the file off with “LET,SPACE.OBJ” and a line 32768 which we normally find. That means we need to put the following bytes:
015310: SECTOR: 679: FILE:
0 4c 45 54 2c 53 50 41 43-45 2e 4f 42 4a 9b 00 80 LET,SPACE.OBJ...
10 0e 0e 19 0f 06 44 3a 4d-45 4e 55 16 .. .. .. .. .....D:MENU.....
And sure enough, with this the MENU comes back to life on this disk (previously the program crashed).
So we lost 69 sectors and we’re able to restore 39 of them. INTRO.DAT remains damaged, and the MIDI files are completely lost, but at least the MENU works again.
So what’s the source of the damage here? My suspicion is that a disk copier believed the VTOC and simply omitted those sectors. We’ll never know for sure...
In looking over the ACE of Columbus disks, I originally thought disk acec027b.atr had some problems, but it turned out, it simply had something I hadn’t seen before, a DOS 1.0 file on a DOS 2.0 disk. You can see this by using the -dir command we wrote in the past.
00 42 039 004 DOS SYS 01 42 014 043 INFOBITSBAS 02 40 080 085 STING 03 42 150 165 MALAGUEN 04 42 092 315 FLIGHT 05 42 069 416 ELEPHANT 06 42 185 485 QUEST 32K 07 42 031 670 TARGETS BAS 08 42 017 701 ENEMY BAS 09 42 010 057 INFOBITSTXT 10 42 012 067 MAIL BAS
Notice that the file STING has a status byte of 0x40 instead of 0x42 like the rest of the files. The 0x02 bit indicates a DOS2.0 format file, so this file is in DOS 1.0 format.
What is DOS 1.0 format? Well, it’s very similar to DOS 2.0 format, except the byte count field is instead a sector index (0 in the first sector, 1 in the second, ...) in all but the last sector, where it is the traditional byte count again but with the 0x80 bit set as well.
For parsing a DOS 2.0 file we handled the last three bytes in the following way:
my ($nextSector, $nBytes) = link_data($sectorData);
$out .= substr($sectorData, 0, $nBytes);
$sector = $nextSector & 0x3ff;
last if $sector == 0;
We can support DOS 1.0 files as well by adding this after the first line:
$nBytes = ($nBytes & 0x80) == 0 ? 125 : ($nBytes & 0x7f) if $dos10;
Which is the Perl code to implement the following code you can find in DOS 2.0S:
104C 68 5040 PLA ;GET LEN 104D 3002 5041 BMI RDNS2 ;BR IF OLD SHORT SECTOR 104F A97D 5042 LDA #125 ;ELSE SET FULL SECTOR 1051 297F 5043 RDNS2 AND #$7F ;TURN OFF MSB 1053 48 5044 PHA ;BALANCE STACK
Notice that this design means the sector index has to wrap every 128 sectors, but that’s not a problem. The problem with this design (and likely why it was changed with DOS 2.0), is that the sector size can’t be larger than 128 bytes, since you only have 7 bits to store the length of the last sector. DOS 2.0S supports 256 byte sectors, so something had to change. There are many possible changes they could have made, making the last byte the actual byte count is just one solution.
In my initial examination of the ACE of Columbus disk archive, I was able to fix acec068b.atr by borrowing a couple of sectors from acec016a.atr. This restored a Rubik’s cube program, but if you compare the two files, they aren’t the same.
As discussed in the past, BASIC files start with 14 bytes of pointers, then the variable names, then the variable values, then the lines of code. In this case, the variable values are different, which is fine, those all get cleared when the program runs and whatever the last values were are what get SAVEd. But there was one byte in the middle of the lines of code that was different, which I didn’t expect.
So let’s write a simple BASIC line validator. Every line in a BASIC program starts with the line number and a byte count, and we expect the lines to be increasing in the file, so one simple validation might be to do the following.
sub bas {
my ($file) = @_;
my $program = read_file($file);
my $length = length($program);
my ($lomem, $vntp, $vntd, $vvtp, $stmtab, $stmcur, $starp)
= unpack "v7", substr($program, 0, 14);
my $offset = 14 + ($vntd - $vntp) + 1 + ($stmtab - $vvtp);
my $lastLineNumber = -1;
while ($offset + 3 < $length) {
last if $lastLineNumber == 32768;
my ($lineNumber, $lineLength) = unpack "vC", substr($program, $offset, 3);
die sprintf "%04x: $lineNumber <= $lastLineNumber", $offset
if $lineNumber <= $lastLineNumber;
$lastLineNumber = $lineNumber;
$offset += $lineLength;
}
}
This reads in a Basic file, parses the first 14 bytes, and then locates the beginning of the statement table for typical programs. It then parses all the lines in the file, terminating if we find a line number that goes backwards. If we run this on the two versions of the program, it doesn’t like the one from acec016a.atr. Clearly a byte on the disk has flipped, so let’s flip it back (from 0x20 to 0x7e at offset 0x44ea).
004490: SECTOR: 138: FILE:
0 1d 36 8e 2d 8e 25 0e 40-34 00 00 00 00 14 2b 36 .6.-.%.@4.....+6
10 8f 2d 8f 26 0e 40 64 00-00 00 00 16 74 04 2b 0f .-.&.@d.....t.+.
20 07 8d 21 0e 40 45 00 00-00 00 1b 1d 36 8e 2d 8e ..!.@E......6.-.
30 25 0e 40 17 00 00 00 00-14 2b 36 8f 2d 8f 26 0e %.@......+6.-.&.
40 40 64 00 00 00 00 16 79-04 0c 0c 36 8a 2d 81 38 @d.....y...6.-.8
50 8d 2c 16 7b 04 07 07 03-8a 16 20 04 83 09 2c 8e .,.{...... ...,.
60 12 8f 14 17 2f 8e 12 8f-25 0e 40 16 00 00 00 00 ..../...%.@.....
70 14 25 2c 8e 25 0e 40 01-00 00 00 00 12 10 8b 7d .%,.%.@........}
Back in the 1980’s, hundreds of Atari user groups formed throughout the United States. One of them, The Atari Computer Enthusiasts of Columbus, Ohio preserved the majority of their newsletters (which you can find on archive.org) and their “Disks of the Month” (342 images which you can find on the Pooldisk under the directory ACE). Just like the Bellcom archive I’ve been digging into, the preservation of some of the disks wasn’t perfect. So let’s spot a few corrupted images and see what we can do to fix them.
The first disk that caught my eye was acec111h.atr. This disk has the 850 Express program on it, but when I was validating the sector links, the DOS.SYS on the disk was incomplete. Dumping the sectors I noticed almost immediately that sectors 21-36 appeared to be backwards. We saw the exact same kind of damage on Bellcom disk 349 involving the same sectors, so we can fix it that same way (and continue to puzzle how the disk degrades in that particular way):
c:atr.pl -copy orig.atr acec111h.atr 36 21 0 c:atr.pl -copy orig.atr acec111h.atr 35 22 0 ... c:atr.pl -copy orig.atr acec111h.atr 21 36 0
Another interesting disk is acec069a.atr. All the sectors from 700 up are empty, but judging from the directory, sectors 700-713 should have data on them (parts of two documentation files, DOC.TXT and DRONE.TXT). By searching through the rest of the pooldisk, I find the same content on ANTIC\88_SEP_A.ATR. The only difference is the equivalent files on the ANTIC disk are one sector longer. Looking at those sectors, it’s likely the files were truncated a bit to fix them, as the ends of the files don’t quite look right. So to restore this disk, I take the equivalent sectors from the ANTIC disk, and manually decide where to end them.
c:atr.pl -copy E:\ANTIC\88_SEP_A.ATR acec069a.atr 230 713 0 + manually fix link to 14 00 5c c:atr.pl -copy E:\ANTIC\88_SEP_A.ATR acec069a.atr 229 712 1 c:atr.pl -copy E:\ANTIC\88_SEP_A.ATR acec069a.atr 228 711 1 c:atr.pl -copy E:\ANTIC\88_SEP_A.ATR acec069a.atr 270 710 0 + manually fix link to 4c 00 45 c:atr.pl -copy E:\ANTIC\88_SEP_A.ATR acec069a.atr 269 709 1 ... c:atr.pl -copy E:\ANTIC\88_SEP_A.ATR acec069a.atr 260 700 1
One last disk for today, acec068b.atr. Sectors 613 and 614 are empty, breaking RUBICUBE. A similar file exist on disk acec016a.atr, so we can restore the BASIC program via.
c:atr.pl -copy acec016a.atr acec068b.atr 206 614 1 c:atr.pl -copy acec016a.atr acec068b.atr 205 613 1
We’ll look at some other disks in the future.
Bellcom disk D068_A is the first disk in the Bellcom archive I’ve examined where the damage is substantial. This is the first disk of a four disk graphical adventure from Germany, “The Nephew”. The Bellcom catalog exclaims: “I had this unique game translated into English because I knew you’d want it.” All the sectors from 952 up are empty, except for 1024 (where the DOS 2.5 is partially stored), but the directory shows that at least two files (ROLLE, HINTS.BAS) should have data there. So let’s see what we can find to fix this.
I decided to use the string “TAKE, OPEN, ” which is on the disk to look for a donor disk, and I added all the TOSEC Atari 8-bit images from archive.org to increase my chances of finding matches. Sure enough, that TOSEC archive proved useful, as it has a four disk set for the Nephew. The third and fourth disks are exact matches of disk Bellcom disks D069_A and D069_B, and I can see that there is data from sectors 952 up on the first disk. There are about 75 sectors to copy, so I could invoke our given single sector tool that many times to fix it. But let’s extend the tool to copy sector ranges instead.
sub copy_sector {
my ($srcFile, $dstFile, $srcSector, $dstSector, $fixlink, $count) = @_;
$fixlink = 1 if !defined $fixlink; $count = 1 if !defined $count;
my $raw = $srcSector =~ /^0x/;
my $src = $raw ? { buff => read_file($srcFile) } : read_disk($srcFile);
my $dst = read_disk($dstFile);
for (; $count > 0; $count--) {
my $srcOffset = $raw ? hex $srcSector : (sector_offset($src, $srcSector))[0];
my ($dstOffset, $sectorSize) = sector_offset($dst, $dstSector);
my $sectorData = substr($src->{buff}, $srcOffset, $sectorSize);
substr($sectorData, -1, 1) = pack "C", $sectorSize - 3 if $raw && $fixlink;
substr($dst->{buff}, $dstOffset, $sectorSize) = $sectorData;
fix_disk_link($dst, $dstSector) if $fixlink;
$dstSector++;
$srcSector = $raw ? $srcSector + $sectorSize : $srcSector + 1;
}
write_file($dstFile, $dst->{buff});
}
This routine isn’t substantially different than the last version, except for the optional $count value you can specify. Using this, we can do:
atr.pl -copy src.atr D068_A.atr 951 951 0 75
And voila, we replace all those blank sectors with full ones from the TOSEC disk. Interestingly, the resulting disk isn’t an exact copy of the disk from the TOSEC archive. The TOSEC version has some changes to the Turbo BASIC runtime in sectors 42 and 43 so that it loads saying “The Nephew” instead of “TURBO-BASIC XL 1.6”. So the Bellcom version does differ slightly from the TOSEC version.
There may be problems with the 2nd disk as well, but I’ll revisit that another day.
At this point I’ve fixed a lot of Bellcom disks, and now find the remaining ones are going to have to wait until I find another source of the damaged content, or some manual work on my part.
Disk 397 is a good example of the latter. It is missing sector 228, which is the first sector of AUTORUN.BAS, which turns out to be the Autocrostics program, which is also archived on the Pooldisk as UMICH\CIS\CROSTI.ATR, sector 214. So using the tools developed so far, we copy over the sector and fix the link and all should be well.
006a90: SECTOR: 214: FILE:
0 00 00 00 01 ac 01 ad 01-25 03 1b 24 30 24 44 49 ........%..$0$DI
10 4d 45 4e 53 49 4f ce 4d-45 4e d5 4c 4f 41 44 5f MENSIO.MEN.LOAD_
20 44 41 54 c1 50 4c 41 d9-c1 52 45 54 55 52 ce 47 DAT.PLA..RETUR.G
30 4f 41 48 45 41 c4 47 4f-42 41 43 cb 44 49 53 4b OAHEA.GOBAC.DISK
40 53 41 56 c5 50 52 49 4e-d4 48 45 4c d0 46 4c 41 SAV.PRIN.HEL.FLA
50 53 c8 43 4c 45 41 d2 45-44 47 45 d3 d8 4c 49 4e S.CLEA.EDGE..LIN
60 4b a8 4c 49 4e cb 44 45-46 49 4e c5 44 d0 54 b3 K.LIN.DEFIN.D.T.
70 54 b4 42 cb 54 b1 44 45-46 a4 54 b2 54 0c d7 7d T.B.T.DEF.T.T..}
Well, it doesn’t work. Why? The two files are slightly different versions of the same BASIC program. Since slightly different isn’t exact, a blind copy of the first 125 bytes isn’t necessarily going to work. Some adjustment is going to need to be made, but what?
SAVE’d BASIC programs are all structured alike (this is actually a Turbo BASIC program, which follows the same design). They all start out with 7 words/pointers: lomem, vntp, vntd, vvtp, stmtab, stmcur, starp. This is followed by a variable name table (reflected by the vntp and vntd variables), a dummy byte, the variable value table (vvtp), followed by the list of statements (stmtab).
In the case of Autocrostics, byte 125 of the file is still in the variable name table and if you ignore the missing first 125 bytes, the files are mostly identical up to around offset 0x1b30. Around that point, a line in the program has been changed causing the rest of the file to shift. So we should be able to copy the first sector and edit the introductory words to restore the file.
A little thought suggests the first five words should be the same. The programs seem to have the same variables, so the name and value tables haven’t moved, and the first statement (stmtab) should be in the same place. So that suggests we only need to change stmcur and starp to restore the original file.
Let’s start with starp. It basically points to the end of the program (or the start of the string table). The UMICH file is actually smaller than this by 35 bytes, so naively you’d expect to have to increase starp by 35 = 0x23 bytes to compensate, and indeed, we need to change starp from 0x2430 to 0x2453.
Now let’s tackle stmcur. In normal cases, it will be smaller than starp by the size of the command to save the file, since SAVE "filename" was the last command run, and it will get stored as a hidden line 32768 in the file. In our case, the difference is 0x2430-0x241b = 0x15 bytes, and if we look at the end of the file, the last fifteen bytes are 00 80 15 15 19 0f 0d <D4:PLAYER.TBX> 16, which is indeed the last line of the program (notice that program lines start with the line number and then a length byte, followed by the line data).
In AUTORUN.BAS, the equivalent values are 00 80 15 15 19 0f 0d <D:AUTORUN.BAS> 16. By coincidence, although the names are different they are the same length, so the line is also 0x15 bytes. So we should set stmcur to 0x2453-0x15 = 0x243e.
So by patching in the sector and adjusting those two values we’ve restored the missing sector with fairly high confidence. And the disk boots again!
![[Autocrostics]](autocrostics.png)
Today’s blog entry is going to be a bit dull, since it’s just all the easy to fix Bellcom disks. No new Perl code is needed, these can all be fixed with the techniques described in past entries. In fact all of them involve just patching a single sector from another disk. The hardest part was finding the disks with equivalent content, but I’ve written before on how binary grep for content.
There are about ten more disks in the Bellcom collection with identifiable damage. I’ve yet to find exact matches for the sectors in my collection of disks for these, so I’m going to build up my collection of user group disks in the hope of finding some further easy fixes.
After the epic efforts to restore GIRL6 in the last two posts, I thought I’d tackle the remaining damaged pictures in the Bellcom archive. We start with disk 450, which has the title screen from Zaxxon ripped from the game, but unfortunately the first sector of the MicroPainter picture is gone. No problem, we can rip the 125 bytes from the game again.
I found similar byte patterns in the disk “Zaxxon (19xx)(Datasoft).atr” in my collection, so I just need to adapt the copy sector tools to pull a sector of data from a random offset in the file.
To do this, I rewrote copy_sector as follows.
sub fix_disk_link {
my ($disk, $sector) = @_;
my ($pos, $sectorSize) = sector_offset($disk, $sector);
my $sectorData = substr($disk->{buff}, $pos, $sectorSize);
substr($sectorData, -1, 1) = pack "C", $sectorSize - 3 if $sectorData =~ /^\0+$/;
my ($nextSector, $nBytes) = link_data($sectorData);
my $full = $nBytes == $sectorSize - 3;
my $offset = $full ? $sectorSize : -$sectorSize;
($nextSector, $nBytes) = link_data(substr($disk->{buff}, $pos + $offset, $sectorSize));
$nextSector = ($nextSector & 0xfc00) | ($full ? $sector + 1 : 0);
substr($sectorData, -3, 2) = pack "n", $nextSector;
substr($disk->{buff}, $pos, $sectorSize) = $sectorData;
}
sub copy_sector {
my ($srcFile, $dstFile, $srcSector, $dstSector, $fixlink) = @_;
$fixlink = 1 if !defined $fixlink;
my $raw = $srcSector =~ /^0x/;
my $src = $raw ? { buff => read_file($srcFile) } : read_disk($srcFile);
my $srcOffset = $raw ? hex $srcSector : sector_offset($src, $srcSector);
my $dst = read_disk($dstFile);
my ($dstOffset, $sectorSize) = sector_offset($dst, $dstSector);
my $sectorData = substr($src->{buff}, $srcOffset, $sectorSize);
substr($sectorData, -1, 1) = pack "C", $sectorSize - 3 if $raw && $fixlink;
substr($dst->{buff}, $dstOffset, $sectorSize) = $sectorData;
fix_disk_link($dst, $dstSector) if $fixlink;
write_file($dstFile, $dst->{buff});
}
This is clearly a more complex implementation than before, due to two reasons. Firstly, I isolated the fix_disk_link code into its own subroutine, and changed the logic to better handle empty sectors. If you hand it an empty sector to fix, it now patches it to be a full sectors of zeros linked the next sector. Secondly, I now treat a source sector starting with 0x as a direct offset into the file, so some adjustments had to be made to implement that.
Anyhow, with this updated routine, I can patch the Zaxxon picture like so:
atr.pl -copy "Zaxxon (19xx)(Datasoft).atr" 450.atr 0x296 320 1
![[Zaxxon]](zaxxon.png)
Two other disks also had damage, disks 655 and 656, both containing collections of CompuEyes black and white scanned images. In both cases sector 327 got dropped. Fortunately the DGS public domain archive has the same pictures on disk #19 (side 1, and 2), so single sector patching restores these old images.
Next time I’ll patch the remaining “easy to fix” disks in the archive, and discuss what other issues remain.
In the last post we partially restored the first sector of Girl6.pic by splicing in some data into the RLE stream to represent 160 zero bytes. But we have a slightly different version of the picture, so in principle we can completely restore the picture from that and some ingenuity.
First, let’s expand out the first column of GIRLJ.PIC (our more complete picture) and see what it looks like.
0 00 00 00 00 aa aa aa aa-aa 9a 7a ae a9 a9 a6 a9 ..........z.....
10 a5 a9 aa aa ab 9f af 6b-67 56 69 55 95 55 55 55 .......kgViU.UUU
20 55 55 95 55 65 a9 a5 55-59 55 fd f9 ff fe ff ff UU.Ue..UYU......
30 ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
40 ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
50 ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
60 00 00 00 00 aa a9 aa a9-a9 aa a9 aa aa 65 aa a6 .............e..
70 a9 aa aa 9a ab 9f af 6b-96 59 aa 95 55 55 55 55 .......k.Y..UUUU
80 59 55 99 69 a9 65 59 65-55 dd fd fe fb ff ff ff YU.i.eYeU.......
90 ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
a0 ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
b0 ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
I’m guessing each byte represents 4 horizontal pixels, and that the sequence of bytes is the entire column, but apparently not in linear order. We don’t really need to know the layout exactly, we simply need to assume we can infer the layout from another part of the file. Since the rightmost part of the picture is empty except for the border, we can expand out GIRL6.PIC, and look at the last 160 bytes to work out the layout. In those bytes, we only see the values 0x03, 0xff, and 0x00. Assuming the 0x00s are locations that are blank in all the 160 byte blocks, we can zero the same locations here, yielding the following.
0 00 00 00 00 aa aa aa aa-aa 9a 7a ae a9 a9 a6 a9 ..........z.....
10 a5 a9 aa aa ab 9f af 6b-67 56 69 55 95 55 55 55 .......kgViU.UUU
20 55 55 95 55 65 a9 a5 55-59 55 fd f9 ff fe ff ff UU.Ue..UYU......
30 ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
40 ff ff ff 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
50 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
60 00 00 00 00 aa a9 aa a9-a9 aa a9 aa aa 65 aa a6 .............e..
70 a9 aa aa 9a ab 9f af 6b-96 59 aa 95 55 55 55 55 .......k.Y..UUUU
80 59 55 99 69 a9 65 59 65-55 dd fd fe fb ff ff ff YU.i.eYeU.......
90 ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
a0 ff ff 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
b0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
The next thing we notice about the picture is that the entire outside of the picture has been set to lightest color, which turns out to be color #3. So we need to bitwise or every non-zero byte here with 0xc0 (since we want the left pixels to be changed to color 3). We also want to set the bytes at offset 0x64 and 0xa3 to 0xff, since in the last column those are 0xff and that should represent the first and last lines of the border in the picture. This yields:
0 00 00 00 00 ea ea ea ea-ea da fa ee e9 e9 e6 e9 ..........z.....
10 e5 e9 ea ea eb df ef eb-e7 d6 e9 d5 d5 d5 d5 d5 .......kgViU.UUU
20 d5 d5 d5 d5 e5 e9 e5 d5-d9 d5 fd f9 ff fe ff ff UU.Ue..UYU......
30 ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
40 ff ff ff 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
50 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
60 00 00 00 ff ea e9 ea e9-e9 ea e9 ea ea e5 ea e6 .............e..
70 e9 ea ea da eb df ef eb-d6 d9 ea d5 d5 d5 d5 d5 .......k.Y..UUUU
80 d9 d5 d9 e9 e9 e5 d9 e5-d5 dd fd fe fb ff ff ff YU.i.eYeU.......
90 ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
a0 ff ff ff 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
b0 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
Now we need to RLE encode these bytes into the file. It turns out the RLE for this is exactly the right size to fit into the 91 bytes we have available in the sector to contain these 160 bytes, strongly suggesting this is the right data restoration. So we end up with the following for sector 556.
011590: SECTOR: 556: FILE:
0 ff 80 c9 c7 1a 00 01 01-0e 00 28 00 c0 22 25 27 ..........(.."%'
10 0c 00 69 0a 00 00 9b 9b-9b 9b a2 04 00 05 ea 83 ..i.............
20 da fa ee 02 e9 84 e6 e9-e5 e9 02 ea 87 eb df ef ................
30 eb e7 d6 e9 09 d5 8a e5-e9 e5 d5 d9 d5 fd f9 ff ................
40 fe 15 ff 20 00 84 ff ea-e9 ea 02 e9 82 ea e9 02 ... ............
50 ea 84 e5 ea e6 e9 02 ea-88 da eb df ef eb d6 d9 ................
60 ea 05 d5 83 d9 d5 d9 02-e9 88 e5 d9 e5 d5 dd fd ................
70 fe db 16 ff 21 00 95 95-96 69 96 65 95 52 2d 7d ....!....i.e.R-}
And the picture even looks correct.
![[GIRL6]](girl6-2.png)
That was more work than I originally intended to expend on this disk, but it allowed me to learn at least a little bit about the MicroIllustrator compression format, so I think it was time well spent.
Poking around the Bellcom archive, we come across disk 481.atr, which contains a number of pictures. GIRL6.PIC is damaged, the first sector is all zeros, so let’s see how far we can get in fixing it today.
Usually, we find an undamaged version of the file and patch in the missing sector, but in this case, that plan goes immediately awry. Looking through the file I pick out the sequence UPDQTP to search for, and find a match in the PoolDisk archive \SEX\DIGIGIRL.ATR, file GIRLJ.PIC starting at sector 497. That file is larger, but clearly shares a lot of data with our picture file. So most likely GIRL6.PIC is a modified version of GIRLJ.PIC.
So let’s see if we can at least patch up the file well enough to load. To do this, we need to understand a little about how MicroIllustrator files are structured. These files are largely used for storing ANTIC mode E (160 × 192, 4 color) pictures in a compressed format. The file starts with a header, and then a sequence of run-length encoded bytes that should expand to 7680 bytes.
So let’s first write a stripped down decompressor for MicroIllustrator files to help us in our restoration.
sub decompress {
my $buff = read_file('out.bin');
die if substr($buff, 0, 4) ne "\xff\x80\xc9\xc7";
my ($len, $offset) = (length($buff), 0x1b);
my $out = '';
while ($offset < $len) {
my $val = unpack "C", substr($buff, $offset, 1);
my ($unique, $count) = ($val & 0x80, $val & 0x7f);
$offset++;
if ($count == 0) {
$count = unpack "n", substr($buff, $offset, 2);
$offset += 2;
}
if ($unique) {
$out .= substr($buff, $offset, $count);
$offset += $count;
}
else {
$out .= substr($buff, $offset, 1) x $count;
$offset++;
}
}
write_file('out.bin', $out);
}
This routine assumes we have a compressed MicroIllustrator file in ‘out.bin’. It verifies the header starts with ff 80 c9 c7 and then starts decompressing the data starting at offset 0x1b. The run-length encoding is pretty standard. Every run starts with a count. If the high bit is set, the low 7 bits are a count of the number of bytes that follow that are a unique sequence. If the low bit is clear, the low 7 bits is the repeat count on the next byte. If low 7 bits are all zero, the count is stored in the next two bytes.
The data at the start of sector 557 we expect to match the data in sector 498, and it does with an initial offset of three. Assuming that is part of the same run of bytes from the GIRLJ picture, the last few data bytes of sector 497 should be 95 95 96 69 95. Copying the header from GIRLJ onto the sector, fixing the length at offset 0x12 of the header, and then putting a 0xda (saying consume the next 0x5a bytes as a unique sequence) at offset 0x1b should yield a file that at least decompresses correctly. So we make those changes in a hex editor which yields:
011590: SECTOR: 556: FILE:
0 ff 80 c9 c7 1a 00 01 01-0e 00 28 00 c0 22 25 27 ..........(.."%'
10 0c 00 69 0a 00 00 9b 9b-9b 9b a2 da 00 00 00 00 ..i........l....
...
70 00 00 00 00 00 00 95 95-96 69 96 65 95 52 2d 7d .........i.e.R-}
Extracting this with the routine we wrote above yields 7574 bytes of uncompressed data, so we’re 106 = 0x6a bytes off (this is expected, the original data should have some compression). So replace da 00 00 00 ... with 6c 00 d8 00 ... fills in the missing data with zeros throughout, yielding:
011590: SECTOR: 556: FILE:
0 ff 80 c9 c7 1a 00 01 01-0e 00 28 00 c0 22 25 27 ..........(.."%'
10 0c 00 69 0a 00 00 9b 9b-9b 9b a2 6c 00 d8 00 00 ..i........l....
...
70 00 00 00 00 00 00 95 95-96 69 96 65 95 52 2d 7d .........i.e.R-}
So now we have a valid MicroIllustrator file where the missing data was simply replaced with zeros, and we get a picture that matches the original with a little bit of the left side removed (since the file used vertical compression, storing the data in vertical slices).
![[GIRL6]](girl6-1.png)
Comparing this to the picture on the other disk shows that the picture was cropped to make it less pornographic. ;-)
As a data restoration, this is good start, it restores enough of the sector to yield a valid file, but we only correctly restored 0x25 of the 0x80 bytes, splicing zeros into the rest of the RLE stream. In a future blog post, I hope to finish the job using more data from GIRLJ.PIC.
In earlier posts I explained how data is organized on a DOS 2.0/2.5 disk, but didn’t provide any code specific to the directories and files (other than link_data). This post remedies that, in part to fill in those gaps, and in part because I’m going to fix some disks that require a bit more effort than just copying a sector or two from another disk.
Let’s start with a helper function
sub toascii {
my ($val) = @_;
$val =~ tr/\x00-\x1f\x60\x7b\x7d\x7e\x80-\x9a\x9c-\x9f\xff/./;
$val =~ tr/\x7f\x9b\xa0-\xfe/\t\n\x20-\x7e/;
$val;
}
This routine will convert some ATASCII text to ASCII text. It translates unprintable characters to periods, inverse characters to their non-inverse equivalents, and 0x7F and 0x9B to tab and carriage return.
OK, now let’s write a function that dumps a DOS directory.
sub read_dos_dir {
my ($file, $sector) = @_; my $disk = read_disk($file);
$sector = 361 if !defined $sector;
my $dirData = '';
$dirData .= substr($disk->{buff}, (sector_offset($disk, $_))[0], 0x80) for ($sector .. $sector + 8);
for (my $i = 0; $i < 64; $i++) {
my $dirEntry = substr($dirData, $i * 16, 16);
my ($status, $sectorCount, $firstSector) = unpack "Cvv", substr($dirEntry, 0, 5);
last if $status == 0; my $name = toascii(substr($dirEntry, 5));
printf "%02d %02x %03d %03d %s\n", $i, $status, $sectorCount, $firstSector, $name;
}
}
Recall that the directory is in sectors 361 to 368, so this routine first reads in those sectors. Actually, it reads in 8 sectors from anywhere on the disk, since MYDOS can locate sub-directory information in other locations and we may want to reuse this code in the future. It then reads each 16-byte entry, getting the start, sector count, first sector and name (which it converts to ascii, thus the helper) and printing them. It stops when it gets to an entry that has a status byte set to zero.
Note that this routine works even with double density disks, by ignoring the 2nd half of every sector. sector_offset returns the size of the sectors, but we ignore that and always use 0x80 = 128.
OK, now let’s see if we can extract a file from a DOS disk:
sub read_dos_file {
my ($file, $sector, $ascii) = @_;
$ascii = 0 if !defined $ascii;
my $disk = read_disk($file);
my $out = '';
for (;;) {
my ($pos, $sectorSize) = sector_offset($disk, $sector);
my $sectorData = substr($disk->{buff}, $pos, $sectorSize);
my ($nextSector, $nBytes) = link_data($sectorData);
$out .= substr($sectorData, 0, $nBytes);
$sector = $nextSector & 0x3ff;
last if $sector == 0;
}
$out = toascii($out) if $ascii;
write_file('out.bin', $out);
}
In this routine, you provide the first sector of the file you want to extract. It then follows the sector links until the next sector is 0. It then writes the file out, optionally converting it to ascii first if requested.
Now to hook this up to our tool, we add the calls into our main subroutine as follows:
read_dos_dir(@_) if $opt eq '-dir'; read_dos_file(@_) if $opt eq '-file';
And voila, atr.pl has some rudimentary DOS inspection abilities. You can exercise them on Bellcom disk #1 like so.
C:> atr.pl -dir 001.atr
00 62 037 004 DOS SYS
01 62 042 041 DUP SYS
02 62 012 083 AUTORUN SYS
03 62 016 095 HELP DOC
04 62 132 111 ENTPRIS1BAS
05 62 235 243 ENTPRIS2BAS
06 62 047 487 STARTREKDOC
07 62 184 534 STARTREKBAS
C:> atr.pl -file 001.atr 95 1
C:> head out.bin
- B E L L C O M -
[Public Domain Library]
BOX 1043, PETERBOROUGH
ONTARIO, CANADA K9J 7A5
-----------------------
ATARI MENU INSTRUCTIONS
-----------------------
That’s enough for today, I’ve attached the current scripts to this post for anyone who wants to use them without constructing them from the snippets in the posts.
Now, that we have a tool to patch sectors and fix links we can incrementally improve the archive, usually by copying a single sector from a donor disk to the damaged disk. After fixing about a dozen disks I started looking at disk 349, in which the file DISCOM32.COM was shorter than the number of sectors in the directory.
What was going on? According to the directory the file started at sector 13 and was 105 sectors long. The next file started at sector 118, so you’d expect this file would just be a continuous chain of sectors from 13 to 117. When I mapped it out, the linkage had a gap. 13-21, 37-117. After studying the sectors from 21-36 for awhile, they looked backwards, 36 should have been 21, 35 => 22, et cetera. Somehow most of track 1 (sectors 19-36) had been dumped out of order.
So patting myself on the back I made a copy of the disk and then patched the 16 sectors in the “correct order” onto the copy which gave me a file that was the right length, but still wasn’t quite right. Even after reordering four of the sectors had file index numbers that didn’t match the rest of the file. Sectors 19, 23, 56 and 84 were marked as from file 2 whereas the rest of the sectors were marked as file 1. Comparing the data with another copy of the file from \UMICH\ARCHIVER\DISKCOMM.ATR suggested those sectors were from a different file/disk altogether. So using sectors 89, 93, 126, 154 from the umich disk I was able to complete fixing the file.
So in the final summary, I needed to move 15 sectors and copy 4 sectors from another disk. This resulted in a file that matched other versions of the file, so it’s highly likely it’s a better match for the original disk contents, but you have to wonder how the disk got damaged like that in the first place.
In my last post I fixed a couple of Bellcom disks by copying missing sectors from another disk, and then “fix[ing] the sector links”. This post is going to discuss that, and modify our copy tool to do it automatically.
As explained earlier, the last three bytes of a DOS 2.0/2.5 file contains file metadata, specifically the file index, the next sector, and the number of bytes. When a sector is dropped and you find a replacement on another disk, two of these three values are unlikely to be what the other disk expects, and so you need to change them to the correct values.
In almost all cases, the next sector value will be the current sector plus one, and the file index can be inferred from the next sector. So let’s add some code to do that to save us from having to do this manually. To do this, we’ll replace the copy_sector routine with the following two routines.
sub link_data {
unpack "nC", substr($_[0], -3, 3);
}
sub copy_sector {
my ($srcFile, $dstFile, $srcSector, $dstSector, $fixlink) = @_;
$fixlink = 1 if !defined $fixlink;
my ($dst, $src) = (read_disk($dstFile), read_disk($srcFile));
my ($dstBuff, $srcBuff) = ($dst->{buff}, $src->{buff});
my ($srcOffset, undef, $dstOffset, $sectorSize)
= (sector_offset($src, $srcSector), sector_offset($dst, $dstSector));
my $sectorData = substr($srcBuff, $srcOffset, $sectorSize);
substr($dstBuff, $dstOffset, $sectorSize) = $sectorData;
if ($fixlink) {
my ($nextSector, $nBytes) = link_data($sectorData);
my $fullSector = $nBytes == $sectorSize - 3;
my $offset = $fullSector ? $sectorSize : -$sectorSize;
($nextSector, $nBytes) = link_data(substr($dstBuff, $dstOffset + $offset, $sectorSize));
$nextSector = ($nextSector & 0xfc00) | ($fullSector ? $dstSector + 1 : 0);
substr($sectorData, - 3, 2) = pack "n", $nextSector;
}
substr($dstBuff, $dstOffset, $sectorSize) = $sectorData; write_file($dstFile, $dstBuff);
}
The code here assumes you want to fix the link, but you can pass a zero in if you want the code to be skipped. In the case that you do want to fix the link, it looks at the current sector. If the sector is a full sector, it will take the file index from the next sector and set the next sector to the current sector plus one, otherwise it will take the index from the previous sector and set the next sector to zero.
That’s it! In 95% of the cases, this works and avoids having to go in to the file with a hex editor to do this manually.
This post is about the Bellcom disks, a large collection of largely public domain software sold out of Canada by Don Bell back in the late 1980s and early 1990s. I became interested in this collection when I downloaded the ABBUC “Atari Pooldisk Two” (it’s on archive.org, e.g. https://archive.org/details/cdrom_PoolDisk_Too_disc2, although I believe that’s actually disk 1). It was the largest set of disks on that CD (967 images), and contained quite an assortment of software for the Atari. It was also clear when I started examining the disks that parts of the archive had bit rotted.
So let’s fight bit rot! The process is as follows:
The last post outlined the larger parts of steps 1+2 (but omitted the code for brevity). Step three is actually simpler than you’d think. Find a part of the file that isn’t bit rotted and is somewhat unique and grep for similar contents on the rest of the Pooldisk archive. There’s a number of ways to do that, but given my love of Perl, here’s what I’ve been using, a recursive binary grep routine, just add the read_file from the previous routines.
sub bgrep_dir {
my ($path, $pattern) = @_;
my (@dirs, @files);
opendir my $dh, $path or return;
for (readdir $dh) {
push @dirs, $_ if -d "$path/$_" && $_ ne "." && $_ ne "..";
push @files, $_ if /\.atr$/i;
}
foreach my $dir (@dirs) {
bgrep_dir("$path/$dir", $pattern);
}
foreach my $file (@files) {
my $buff = read_file("$path/$file");
print "Binary file $path/$file matches\n" if $buff =~ $pattern;
}
}
sub bgrep {
my $pattern = shift;
bgrep_dir('.', qr/$pattern/);
}
bgrep(@ARGV);
Let’s start with two disks that have zero sectors where there is another copy of the data in the Bellcom collection.
atr.pl -copy 642.atr 95 D112_B.atr 182 + fix the sector linksatr.pl -copy 417.atr 664 281.atr 327 + fix the sector linksI’ll append additional patched images over time to this post (at least 20 images have identifiable issues). But next time we’re going to change the patch routine so that we won’t usually need to manually fix the sector links.
I started this blog by fixing up a relatively obscure Atari title (The ABC of CPR). I could fix it because there were several different archives containing purportedly the same data, and since the most common problem with old disks is a bad sector which translates to an empty/dropped sector in an ATR archive, we could see the blank sectors and conclude fixes were needed.
So how do we find some (ideally all) disks with dropped sectors in a large archive. Well, it falls out of trying to build a comprehensive directory of all the files on those disk images.
The most common disk format for the Atari is the one used by DOS 2/2.5. It places its metadata in sectors 360-368 on the disk, plus sector 1024 in the case of enhanced density disks. For single density disks, sector 360 starts with the DOS code in the first byte (the value 2 in both cases), the amount of data sectors on the disk (usually 707), and then the amount of free sectors. Starting at offset 10 is the VTOC, 720 bits of data where each bit represent whether a sector is free (1) or used (0). Sector 361-368 then follows with individual directory entries, 16 bytes per entry. The first byte is the status byte, the next two bytes the sector count, then two bytes for the first sector of the file, with the remaining 11 bytes used for the filename.
So to find the contents of a file, you read from the directory entry the sector count and first sector, then read the first sector and decrement the sector count. That (and all subsequent sectors) contains 125 bytes of data, followed by 3 bytes (24 bits) containing the file entry number (the first 6 bits), the next sector number (the next 10 bits), and then finally the data byte count for that sector (the last 8 bits). If the sector count is non-zero, the byte count should be 125, and all 125 bytes of data belongs to the file. If the sector count is zero, the next sector field should be zero, and the byte count could be smaller than 125.
Given that description, detecting a dropped sector usually happens as a side-effect. As you walk all the sectors in a file, if one of them is empty, the next sector will be zero but the sector count will likely still be non-zero. If you’re on the last sector, the byte count should be non-zero (otherwise you would have ended the file in the previous sector).
If you think about this design, you realize it’s awful for random access to the data, but it’s great for quickly detecting integrity issues. The more common FAT (file allocation table) design doesn’t mix the file metadata with the file data, so if some of the data in the file is zeroed, the file maintains integrity and you might not realize the file is damaged. With the DOS design, the attempt to load the file itself will fail.
So one algorithm for finding damaged single density disks is to examine the first five bytes of sector 360, and if it starts with 2, 707, and a number less than 707, assume it’s a DOS 2.0/2.5 disk and walk the directory entries validating that the length of the chain of sectors matches the sector count in the directory.
I’ve written (in Perl of course) that algorithm (actually a slightly more permissive version) and started applying it to various archives to spot damaged disks that need repairs. So we’ll pick up the discussion next time with the results on a large collection of disks, specifically the Bellcom public domain archive.
So we’ve managed to write a simple tool that can dump and patch sectors for standard 90K disks. What about dual and double density disks?
A little bit of history first. The Atari computers first disk drive, the 810, was capable of supporting 18 128-byte sectors per track, leading to 720 sectors on a 40 track disk, or 90K. To get more data onto a disk, you can change the sector size, the number of sectors per track, or the number of tracks. When the 815 drive was designed, it changed the sector size to 256 bytes to increase the density, but because the OS boot routine was designed to read 128 byte sectors, the drive only returned 128 bytes for the first three sectors. The later 1050 drive took a different approach, instead putting 26 128-bytes sectors per track. So the 1050 could store less data than the 815 design. But third party drives initially followed the design the 815 implemented, returning truncated sectors for sectors 1-3.
The ATR disk format recognizes this distinction, and thus places the sector size in the file header, and always stores the first three sectors as 128-bytes.
So let’s write a tool that can handle all three formats. We can keep the read_file, write_file, dump_buffer, and dump_sector routines from previous posts unchanged. To that, we first add read_disk:
sub read_disk {
my ($file) = @_;
my $buff = read_file($file);
my ($sig, $imageSize, $sectorSize, $imageSizeHigh)
= unpack "vvvv", substr($buff, 0, 8);
die if $sig != 0x296;
my $disk = { buff => $buff, sectorSize => $sectorSize };
$disk;
}
Read disk reads the file, and examines the header by unpacking the data from the first 8 bytes. First it verifies the first two bytes are the the ATR signature. If then creates an “associative array”, or hash containing the buffer and the sector size and returns it.
We can now pass this around to allow our code to make different decisions depending on whether it has 128-byte or 256 byte sectors.
In fact, for our purposes, the main change now is to make sector_offset sector size aware.
sub sector_offset {
my ($disk, $sector) = @_;
if ($sector < 4) {
(0x10 + ($sector - 1) * 0x80, 0x80);
}
else {
my $sectorSize = $disk->{sectorSize};
(0x10 + 3 * 0x80 + ($sector - 4) * $sectorSize, $sectorSize);
}
}
The old sector_offset only returned the offset, sector offset now returns both the offset and the size of the sector requested.
Let’s rewrite dump_file now to handle the different style of disks.
sub dump_file {
my ($file) = @_; my $disk = read_disk($file);
my $buff = $disk->{buff};
for (my $sectorNum = 1; {
my ($pos, $sectorSize) = sector_offset($disk, $sectorNum);
last if $pos + $sectorSize > length($buff);
my $sector = substr($buff, $pos, $sectorSize);
print dump_sector($sector, $sectorNum++, $pos, '');
}
}
The old dump_file hard coded 128 bytes and 720 sectors. Now we just dump sectors until we run out of buffer, and we rely on the sector size value returned by sector_offset so that with double density disks we dump short sectors first and then the larger sectors.
The changes to copy_sector are simpler in fact, since it’s only assumption was the sector size.
sub copy_sector {
my ($srcFile, $srcSector, $dstFile, $dstSector) = @_;
my ($dst, $src) = (read_disk($dstFile), read_disk($srcFile));
my ($dstBuff, $srcBuff) = ($dst->{buff}, $src->{buff});
my ($srcOffset, undef, $dstOffset, $sectorSize)
= (sector_offset($src, $srcSector), sector_offset($dst, $dstSector));
substr($dstBuff, $dstOffset, $sectorSize) = substr($srcBuff, $srcOffset, $sectorSize);
write_file($dstFile, $dstBuff);
}
We use the destination disk sector size to determine how much data to copy. This utility is really only designed to handle copying when the sector sizes are the same, but if they are different the routine does do something sensible.
Anyhow, we know have a still fairly small utility (a little over a hundred lines) that we can use to examine and repair disk images when we have multiple copies of the data available. Over the coming months this will be our go to tool to fix disks, so let’s find some disks to repair!
In the last blog entry, I presented a little bit of Perl code to hex dump disk contents, and promised to extend that to allow disk patching. So here we go.
sub write_file {
my ($file, $buff) = @_;
open my $fhw, '>', $file or return;
binmode $fhw;
print $fhw $buff;
close $fhw;
}
The first routine we’ll need is something to write a file. Here’s a completely generic routine that will write a buffer to a file. So our strategy will be read_file, change the buffer, and then invoke this write_file command.
sub main {
my $opt = shift;
dump_file(@_) if $opt eq '-dump';
copy_sector(@_) if $opt eq '-copy';
}
main(@ARGV);
Our last utility invoked dump_file directly, but we want to have a single utility that both dumps and patches files. So we’ll replace the direct_call to dump_file with a new routine, called main, that will invoke dump_file or copy_sector as appropriate. Of course we haven’t written copy_sector yet.
sub sector_offset {
0x10 + ($_[0] - 1) * 0x80;
}
sub copy_sector {
my ($srcFile, $srcSector, $dstFile, $dstSector) = @_;
my ($dstBuff, $srcBuff) = (read_file($dstFile), read_file($srcFile));
my ($srcOffset, $dstOffset) = (sector_offset($srcSector), sector_offset($dstSector));
my $sectorSize = 0x80;
substr($dstBuff, $dstOffset, $sectorSize) = substr($srcBuff, $srcOffset, $sectorSize);
write_file($dstFile, $dstBuff);
}
Here’s copy_sector. It needs a little helper function (sector_offset), which computes the location of a sector in a buffer. It then reads both files, copies a sector from the source to the destination, and then writes it out.
Let’s try this out. The ABC of CPR Side B had a single corrupted sector, sector 96. The Page 6 disk which has a correct version of the sector is called c06_abccpr_b.atr, so to patch the sector:
atr.pl -copy c06_abccpr_b.atr 96 abc_b.atr 96
One drawback to this dump/copy utility is it only works on SS/SD disks. We’ll modify it next time to handle the three different disk types.
In my blog entry on the ABC of CPR, I found issues with all the (known to me) images of the disks by cross comparing the contents of the disk images. In this entry, I walk through how to generate such dumps using a utility in the Perl language.
Perl has fallen in popularity over the years, which is a bit sad, as it still has a lot to recommend it. It still probably has the best text handling facilities of any language, can handle binary data with ease, and has an simple object model that I’m still impressed by. Its critics generally argue it’s too easy to produce write-only code in it, but I find that’s true of pretty much every language, and a skilled programmer should not make that mistake.
So let’s write a sector dumping tool in Perl. For simplicity, it will dump whole 720 sector single density disks, but modifying it for more general cases will be something we’ll return to in the future.
To begin:
use strict;
Every Perl script needs to begin with use strict;. Perl historically was rather forgiving in its syntax, but experienced developers know it’s best to know early when you’ve wandered off the road.
sub read_file {
my ($file, $size, $pos) = @_;
$size = -s $file if !defined $size || $size == 0;
open my $fh, '<', $file or return;
binmode $fh;
seek $fh, $pos, 0 if defined $pos;
my $buff;
read $fh, $buff, $size;
close $fh;
$buff;
}
Ok, subroutines in Perl begin with sub, and this one is a general routine for reading a file. You only need to specify the file name, but if you also specify a size and or position, this routine will only read a slice of the file. @_ is a special variable in perl that contains the parameters passed to the routine. We don’t really need the position logic for this tool, but the extra line to have it in the future isn’t a great expense. The binmode statement says the file is binary, so just read the bytes. The routine ends with what we want to return to the caller, in this case the buffer containing the contents of the file.
This short routine uses what is one of my favorite features of Perl, statement modifiers. In most languages, an if statement is followed by a block of code. In Perl, you can also have a single statement followed by an if statement (or other statements). It behaves the same way as if it was written the traditional way, but the code reads more naturally. In the above code it is used to compute the size if it is omitted or zero, and to call seek if a position is provided.
sub dump_buffer {
my ($buff, $start, $length) = @_;
my $end = defined $length ? $length : length($buff);
$start = 0 if !defined $start;
my $result = '';
for (my $i = 0; $i < $end; $i += 16) {
my $string = substr($buff, $i, 16);
my $len = length($string);
$result .= sprintf "%6lx ", $i + $start;
$result .= join " ", unpack("H2" x $len, $string);
substr($result, -3 * ($len - , 1) = '-' if $len > 8;
$result .= " " x (16 - $len + 1);
$string =~ tr/\x20-\x7e/./c;
$result .= $string;
$result .= "\n";
}
$result;
}
This subroutine dumps a buffer so that it’s readable. Sixteen bytes at a time are dumped as hex codes, and the ASCII values are printed to the right. This routine packs a lot of power into a few short lines. I want to draw attention to the way Perl has made some of this quite easy. For instance, the line with the tr/// lets us convert all the non-printable characters to periods without having to write a loop individually checking if a character is printable or not.
sub dump_sector {
my ($sectorData, $sectorNum, $srcOffset, $fileName) = @_;
my $message = sprintf "%06lx: SECTOR: %03d: FILE: $fileName\n", $srcOffset, $sectorNum;
$message .= $sectorData =~ /^\0+$/ ? "EMPTY\n" : dump_buffer($sectorData, 0);
$message;
}
This routine dumps a sector by passing it to dump_buffer. Note that it checks if all the bytes are zero in which case it simply prints EMPTY rather than a hex dump.
sub dump_file {
my ($file) = @_;
my $buff = read_file($file);
my ($sectorSize, $sectorNum, $count) = (0x80, 1, 720);
while ($count--) {
my $pos = 0x10 + ($sectorNum - 1) * $sectorSize;
my $sector = substr($buff, $pos, $sectorSize);
print dump_sector($sector, $sectorNum++, $pos, '');
}
}
dump_file(@ARGV);
And now the main code to dump the file. It reads the entire file, then dumps each sector one by one. It assumes the disk is in atr format (a 0x10 = 16 byte header, and 0x80 = 128 byte sectors), but otherwise is fairly generic. The last line invokes this subroutine, passing the special variable @ARGV which contains the parameters passed to the script.
If you put all this code in a file named, say, atr.pl, you can dump an atr file by saying perl atr.pl [file.atr]. It will dump each sector one by one to the output, which you can redirect to a file, e.g. perl atr.pl file.atr > output.txt. Do this to two atrs of supposedly the same content and compare them using your favorite compare tool (I usually use windiff, but there are dozens of equivalent tools), and you can see where there are differences.
For instance, here’s the start of side A of The ABC of CPR as generated by this script.
000010: SECTOR: 001: FILE:
0 00 1a 00 06 e0 07 a9 00-8d 2f 02 8d 00 d4 a9 90 ........./......
10 8d 30 02 8d 02 d4 a9 06-8d 31 02 8d 03 d4 a9 94 .0.......1......
20 8d c8 02 8d 1a d0 a9 22-8d 2f 02 8d 00 d4 a9 fd ......."./......
30 8d 04 03 a9 9f 8d 05 03-ee 0a 03 a9 40 8d 03 03 ............@...
40 20 59 e4 30 1f 18 ad 04-03 69 80 8d 04 03 90 03 Y.0.....i......
50 ee 05 03 ce fd 9f d0 e0-ad fe 9f 85 0a ad ff 9f ................
60 85 0b 18 60 38 60 00 00-00 00 00 00 00 00 00 00 ...`8`..........
70 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
000090: SECTOR: 002: FILE:
Next we are going to expand this tool to allow us to replace a sector with another sector from a file, thus helping us to be able to repair images. But that’s for another day.
In this blog I hope to spend time dissecting and rebuilding various software artifacts for the 8-bit Atari.
My first entry is The ABC of CPR, an Edunetics title you can find at http://www.atarimania.com/game-atari-400-800-xl-xe-abc-of-cpr-_12807.html. I decided to look at this because I found the same/similar disks in the DGS and Page 6 public domain disk archives, and I was curious if there were any differences. All the images differ from each other in various ways (seen by hex dumping the disks and comparing), but the differences in the Atarimania copies suggest it is damaged:
The Page 6 and DGS copies are themselves not consistent, the DGS copies appear to have a couple of bit flips in sectors 193 and 256, and sector 334 is empty. The bytes in question are consistent between the Page 6 and Atarimania copies.
So given this, I’ve patched the 3 sectors mentioned above from the Page 6 disks to generate a “fixed” version of Atarimania images.