News You Can Bruise for 2012 March 19

On Friday I decoded a BCDIC punch card that my dad used to sign up for classes at UCLA in 1968. It says, "C 6088312496U40" What drove me to this? Well:

• Friday's BCDIC adventure inspired by C.E. Mackenzie, "Coded Character Sets, History and Development" (1979), a book ½ boring and ½ AMAZING.
• If I'm reading "About The Author" right, Charles Mackenzie was IBM's man on the committee that created ASCII.
• In celebration of his great book, today I broadcast the best bits that can be summarized in 140 characters. Why limit it like this?
• "How many readers of this book have experienced the aggravation of trying to squeeze 81 characters into an 80-column card?" (p269)
• Because I laughed at that quote the first time I read it, then realized I had in fact experienced that aggravation.
• Supplementary information from my friend @enf's paper "The Evolution of Character Codes, 1874-1968"
• Let's begin with [E]BCDIC tidbits! ¤ is "the international 'Currency Symbol'", a placeholder for your country-specific symbol. (EBCDIC 0x5B)
• ¤ is not to be confused with ⌑, the "lozenge". An IBM control character not intended for customer use. Spoiler: customers used it. (p67)
• If you had a German IBM keypunch, the hole patterns for #@$ were represented by ÄöÜ. Similarly for other European countries. (p90)
• If you had a "scientific" (FORTRAN) installation, the "commercial" characters %⌑&#@ showed up as ()+='. You couldn't have both in BCDICv2
• This meant FORTRAN code X=(A+B)*(C-D) might print as X#%A&B⌑*%C-D⌑. "Such program listings, though bizarre, were unambiguous." (p92)
• But COBOL "created a requirement for both the 'commercial' and 'scientific' graphics within a single application"! (p381) Thanks, COBOL.
• IBM unified "commercial" and "scientific" in BCDIC v3, but seemingly forgot to add the plus sign! BCDIC v3 was quickly replaced. (p100)
• BCDIC control characters were given symbols designed to discourage use by customers: ␢ Δ γ ⵐ ᚒ ✓, and one that looks like ᚒ rotated 90°
• "Graphics for control characters should have no intrisic meaning... How well the graphics meet the criterion the reader can judge." (p102)
• (I guess people who can suddenly use both + and % might not feel the need for Δ for a while.)
• Death of the lozenge: "Attempts to persuade the standards committees to assign [⌑] in ASCII were unavailing." Thus, omitted from EBCDIC too.
• But wait! Customers demanded ⌑ on their System/360 printers! "It was provided... although it ostensibly did not exist in EBCDIC." (p152)
• "We shall learn in Chapter 18 that the assignment of Katakana in EBCDIC created complications." (p173)
• I don't understand the Katakana EBCDIC problem, but at one point character order broke collation order, a la "abcdefghijkluvwxyzmnopqrst"
• Exotic EBCDIC control characters: "Digit Select", "Start of Significance", "Reverse Line Feed", "Graphic Escape", "Eight Ones".
• And now for act 2: ASCII tidbits!
• AT&T demanded that ASCII start with Null and end with Delete. That's why Delete is on the other side of ASCII from other control chars (212)
• "A" is ASCII 0x41 instead of 0x40 for compatibility with other standards. Why do those standards use 0x41? "The author does not know." (p238)
• ASCII 0x5E and 0x5F (^ and _) were originally ↑ and ←, two characters for use in flowcharting. (p242, but Fischer says they were for ALGOL)
• The UK wanted to have "10" and "11" after "9" (for pennies in a shilling), so unimportant symbols : and ; were put in those spots. (p241)
• Similarly, @ goes right before A (0x40) so it can be replaced with À in France and Italy. (p243)
• The equal sign was placed between < and > for mnemonic reasons: less than, equal to, greater than (p243)
• ASCII 0x05 was originally WRU, "Who Are You?" "Causes the receiving equipment to transmit back information identifying itself" (Fischer)
• The 1963 draft of ASCII had unassigned space where lowercase letters would later go. (p246) Lots of resistance to lowercases. (Fischer)
• Now it gets bonkers. ASCII was designed without regard to how it would look on punch cards! It's a disaster! Takes up 80 pages in Mackenzie.
• Here's the big problem. Punch cards have ≥10 rows. Most data is numeric. Hollerith coding punches row 0 for "0", row 1 for "1", etc.
• But ASCII encoding of digits punches row 5 and 6 for *every* column, "lacing" it. The punch cards will fall apart!
• Problem #2: Hollerith coding encodes space as 0x00. That is, no punches on the card. But ASCII space is 0x32, which punches row 6! So ugly!
• One proposed solution: just replace the "Space" bar on keypunches with a "Null" bar. REJECTED. (p255-256)
• Problem #3: Hollerith lets you "overpunch" a card to represent -5 in a single column. ASCII makes you put "-" and "5" in separate columns.
• (Disclaimer: I'm simplifying that last one a little. But people were really attached to their overpunches.)
• Solution: A standardized Hollerith-type encoding that works for ASCII or EBCDIC. Space is an empty column, no overpunching for negative #s.
• And that's Mackenzie: the history, politics, and design of some of the most influential standards. Not riveting, but full of great stuff.
• Bonus: headings that read like pulp SF novel titles: "The Duals of BCDIC", "Bendix Prime", "The Plomondon Proposal". 0x04

Some addenda acquired from readers while I performed that blob of text on identica/Twitter:

• Evan Prodromou and Kragen Sitaker both recommended "An annotated history of some character codes".
• spacehobo asked what those uncooperative IBM customers used the lozenge for. According to Mackenzie they used it to indicate "final totals as contrasted to subtotals" (p67) and to fill up blank spaces on printed checks. (I don't have a page reference for that one.)
• spacehobo also provides some illumination on the Katakana problem: "kana are syllabic, so there's a received grid ordering for the characters that has lanthanide/actinide-style break-out areas"
• Zack Weinberg complained: "All these struggles over what punctuation to include, but nobody suggests cutting back on the number of control characters. Nearly all of which are now useless."

  Actually, lots of people wanted MORE control characters. Lowercase letters didn't get into the original ASCII because people wanted to reserve that space for control characters. I think they were imagining that as more types of technology were invented, control characters would have to be added to ASCII for each one. So by now ASCII would be full of modem commands and graphics primitives, but have no lowercase letters.

• Eric Fischer had a lot of helpful comments:
  • Why is "A" ASCII 0x41 instead of 0x40? "Ferranti had A as 1 instead of 0 so they could have NUL be 0 in a 5-bit code with mode shifts between alpha and digits."
  • Apropos @: "I really think @ is 0x40 because it is the most US-centric character so the easiest to do without in a 5-bit subset where it's NUL... But the major proposal that put @ there (Allen Whitman 4/2/1962) was arguing mostly for keyboard layout."
  • Why was ASCII not designed for use on punch cards? "They really expected computers to use a 6-bit subset subtracting 0x20, putting space at 0x00. 8-bit bytes came as a surprise."
  • Who says ↑ and ← were intended for ALGOL? "Oct 23 1961 draft refers to ↑ as 'Exponential' and ← as 'Is replaced by'." (The flowchart thing sounded fishy from the start--why 2 arrows instead of 4?)

    Kragen Sitaker says: "They're also used in Smalltalk: ↑ is "return" and ← is :=". Dan Hirsch adds: "This is the reason that old smalltalk sources use _ as an assignment operator." spacehobo says: "I think I read in Jennings that := was meant to be ascii-art for ← once _ won out"