News You Can Bruise

On Friday I decoded a BCDIC punch card that my dad used to sign up for classes at UCLA in 1968. It says, "C 6088312496U40" What drove me to this? Well:

• Friday's BCDIC adventure inspired by C.E. Mackenzie, "Coded Character Sets, History and Development" (1979), a book ½ boring and ½ AMAZING.
• If I'm reading "About The Author" right, Charles Mackenzie was IBM's man on the committee that created ASCII.
• In celebration of his great book, today I broadcast the best bits that can be summarized in 140 characters. Why limit it like this?
• "How many readers of this book have experienced the aggravation of trying to squeeze 81 characters into an 80-column card?" (p269)
• Because I laughed at that quote the first time I read it, then realized I had in fact experienced that aggravation.
• Supplementary information from my friend @enf's paper "The Evolution of Character Codes, 1874-1968"
• Let's begin with [E]BCDIC tidbits! ¤ is "the international 'Currency Symbol'", a placeholder for your country-specific symbol. (EBCDIC 0x5B)
• ¤ is not to be confused with ⌑, the "lozenge". An IBM control character not intended for customer use. Spoiler: customers used it. (p67)
• If you had a German IBM keypunch, the hole patterns for #@$ were represented by ÄöÜ. Similarly for other European countries. (p90)
• If you had a "scientific" (FORTRAN) installation, the "commercial" characters %⌑&#@ showed up as ()+='. You couldn't have both in BCDICv2
• This meant FORTRAN code X=(A+B)*(C-D) might print as X#%A&B⌑*%C-D⌑. "Such program listings, though bizarre, were unambiguous." (p92)
• But COBOL "created a requirement for both the 'commercial' and 'scientific' graphics within a single application"! (p381) Thanks, COBOL.
• IBM unified "commercial" and "scientific" in BCDIC v3, but seemingly forgot to add the plus sign! BCDIC v3 was quickly replaced. (p100)
• BCDIC control characters were given symbols designed to discourage use by customers: ␢ Δ γ ⵐ ᚒ ✓, and one that looks like ᚒ rotated 90°
• "Graphics for control characters should have no intrisic meaning... How well the graphics meet the criterion the reader can judge." (p102)
• (I guess people who can suddenly use both + and % might not feel the need for Δ for a while.)
• Death of the lozenge: "Attempts to persuade the standards committees to assign [⌑] in ASCII were unavailing." Thus, omitted from EBCDIC too.
• But wait! Customers demanded ⌑ on their System/360 printers! "It was provided... although it ostensibly did not exist in EBCDIC." (p152)
• "We shall learn in Chapter 18 that the assignment of Katakana in EBCDIC created complications." (p173)
• I don't understand the Katakana EBCDIC problem, but at one point character order broke collation order, a la "abcdefghijkluvwxyzmnopqrst"
• Exotic EBCDIC control characters: "Digit Select", "Start of Significance", "Reverse Line Feed", "Graphic Escape", "Eight Ones".
• And now for act 2: ASCII tidbits!
• AT&T demanded that ASCII start with Null and end with Delete. That's why Delete is on the other side of ASCII from other control chars (212)
• "A" is ASCII 0x41 instead of 0x40 for compatibility with other standards. Why do those standards use 0x41? "The author does not know." (p238)
• ASCII 0x5E and 0x5F (^ and _) were originally ↑ and ←, two characters for use in flowcharting. (p242, but Fischer says they were for ALGOL)
• The UK wanted to have "10" and "11" after "9" (for pennies in a shilling), so unimportant symbols : and ; were put in those spots. (p241)
• Similarly, @ goes right before A (0x40) so it can be replaced with À in France and Italy. (p243)
• The equal sign was placed between < and > for mnemonic reasons: less than, equal to, greater than (p243)
• ASCII 0x05 was originally WRU, "Who Are You?" "Causes the receiving equipment to transmit back information identifying itself" (Fischer)
• The 1963 draft of ASCII had unassigned space where lowercase letters would later go. (p246) Lots of resistance to lowercases. (Fischer)
• Now it gets bonkers. ASCII was designed without regard to how it would look on punch cards! It's a disaster! Takes up 80 pages in Mackenzie.
• Here's the big problem. Punch cards have ≥10 rows. Most data is numeric. Hollerith coding punches row 0 for "0", row 1 for "1", etc.
• But ASCII encoding of digits punches row 5 and 6 for *every* column, "lacing" it. The punch cards will fall apart!
• Problem #2: Hollerith coding encodes space as 0x00. That is, no punches on the card. But ASCII space is 0x32, which punches row 6! So ugly!
• One proposed solution: just replace the "Space" bar on keypunches with a "Null" bar. REJECTED. (p255-256)
• Problem #3: Hollerith lets you "overpunch" a card to represent -5 in a single column. ASCII makes you put "-" and "5" in separate columns.
• (Disclaimer: I'm simplifying that last one a little. But people were really attached to their overpunches.)
• Solution: A standardized Hollerith-type encoding that works for ASCII or EBCDIC. Space is an empty column, no overpunching for negative #s.
• And that's Mackenzie: the history, politics, and design of some of the most influential standards. Not riveting, but full of great stuff.
• Bonus: headings that read like pulp SF novel titles: "The Duals of BCDIC", "Bendix Prime", "The Plomondon Proposal". 0x04

Some addenda acquired from readers while I performed that blob of text on identica/Twitter:

• Evan Prodromou and Kragen Sitaker both recommended "An annotated history of some character codes".
• spacehobo asked what those uncooperative IBM customers used the lozenge for. According to Mackenzie they used it to indicate "final totals as contrasted to subtotals" (p67) and to fill up blank spaces on printed checks. (I don't have a page reference for that one.)
• spacehobo also provides some illumination on the Katakana problem: "kana are syllabic, so there's a received grid ordering for the characters that has lanthanide/actinide-style break-out areas"
• Zack Weinberg complained: "All these struggles over what punctuation to include, but nobody suggests cutting back on the number of control characters. Nearly all of which are now useless."

  Actually, lots of people wanted MORE control characters. Lowercase letters didn't get into the original ASCII because people wanted to reserve that space for control characters. I think they were imagining that as more types of technology were invented, control characters would have to be added to ASCII for each one. So by now ASCII would be full of modem commands and graphics primitives, but have no lowercase letters.

• Eric Fischer had a lot of helpful comments:
  • Why is "A" ASCII 0x41 instead of 0x40? "Ferranti had A as 1 instead of 0 so they could have NUL be 0 in a 5-bit code with mode shifts between alpha and digits."
  • Apropos @: "I really think @ is 0x40 because it is the most US-centric character so the easiest to do without in a 5-bit subset where it's NUL... But the major proposal that put @ there (Allen Whitman 4/2/1962) was arguing mostly for keyboard layout."
  • Why was ASCII not designed for use on punch cards? "They really expected computers to use a 6-bit subset subtracting 0x20, putting space at 0x00. 8-bit bytes came as a surprise."
  • Who says ↑ and ← were intended for ALGOL? "Oct 23 1961 draft refers to ↑ as 'Exponential' and ← as 'Is replaced by'." (The flowchart thing sounded fishy from the start--why 2 arrows instead of 4?)

    Kragen Sitaker says: "They're also used in Smalltalk: ↑ is "return" and ← is :=". Dan Hirsch adds: "This is the reason that old smalltalk sources use _ as an assignment operator." spacehobo says: "I think I read in Jennings that := was meant to be ascii-art for ← once _ won out"

Lots of stuff is overpackaged, but for some reason flour is underpackaged and it has always been. I've lived for a while and lots of things have changed but one thing I can remember all the way back to childhood is the flour shelf in the supermarket being dusted with spillage from flimsy paper bags of flour that have grown leaks or rips. Flour's pretty cheap so I guess the supermarket and the flour mill can afford to lose merchandise. But really how much would it hurt to use the next thicker grade of paper? Also, apparently I'm Pat from Achewood.

Pieces of ship. How many cargo ships sink every year? In Dangerous Waters I read some huge number like 200. Can that be right? Do they get recovered like that ship did or do they just grow barnacles and wait for future archaeologists or treasure hunters?

For the combination of Nick Moffitt and me that inhabits us all.

• This train is bound for Gorey, this train.
• What's your function?

Trying to figure out the best way to present the licensing interface for NewsBruiser, I decided to see which Creative Commons licenses were the most popular. I ran a Google link: search for all 11 Creative Commons licenses, plus additional CC-provided licenses like the public domain dedication 'license' and the CC-GNU GPL and LGPL. I used the canonical license link: the URL I would link to if I were licensing something. It turns out that only four or five of the 15+ licenses are used by any substantial number of people.

The top five licenses are all Attribution-type licenses, with the most popular being Attribution-NonCommercial with nearly 25,000 hits. The least popular Creative Commons license, with only 13 takers, is the CC-GNU LGPL license.

License Design Implications

It looks like Creative Commons has enough licenses. The core licenses have proven very effective at meeting people's needs, and there's no need to keep minting new ones because as far as I can tell the new ones don't get used. This might indicate that the new licenses are created in response to "You should have such-and-such a license" feedback rather than in response to people who don't like any of the existing Creative Commons licenses.

For instance, consider the Creative Commons "Music Sharing License", which is effectively the same as Attribution-NonCommercial-NoDerivs. It's a lot more popular than its forlorn twin, but here "popularity" means 177 hits on Google instead of 16. It's dwarfed by its own commentary: Googling for "music sharing license" gets you 530 results talking about the latest license from Creative Commons. Compare to a string search for, eg. "Attribution-Noncommercial-Sharealike". You get people actually using the license, not talking about it. Creating a new license gets Creative Commons some buzz, but not much more; it looks like everyone who wants a license already has one they like.

The GNU licenses were probably created on behalf of GPL fanatics who kept bugging the Creative Commons people "why don't you have a GPL license?". Now that they're here, few people use them. My guess is, most people who want to use the GPL are using the actual GPL.

I didn't include them (or Music Sharing) in the survey because they're for music, but the URLs to the Sampling and Sampling Plus licenses are also mentioned almost nowhere on the web. This might be because no one uses them, because they're newer, because they don't get used correctly, or because they get used in ways that Google can't pick up. My methodology isn't as good for licenses not used by web pages (which probably also affects the GPL/LGPL numbers), so I will reserve judgement on these--but again, I searched for links to the URL I'd link to if I were licensing something under that Creative Commons license. The fact that the most popular licenses get tens of thousands of hits indicate that there's not some systemic disconnect where people don't link to the license they're using.

On the flip side, it does look like the CC public domain dedication is getting some good use--it's the sixth most popular license. I think this and the Founders' Copyright license (about which below) feed into a core competency of Creative Commons--being a trusted repository for copyright assignments.

UI Implications

If you want to let someone choose a Creative Commons license (in addition to whatever non-CC licenses), you don't need a bunch of orthogonal sets of radio buttons like the Creative Commons license picker has. You can provide four individual radio buttons and an 'other' field. This will handle 84% of the cases without greatly inconveniencing the other 16% of the people. (You want an 'other' field anyway, since people come up with all kinds of crazy non-CC licenses for their stuff.)

I couldn't measure the success of the Founder's Copyright because I couldn't find one canonical URL for the license. However, it's a good idea and I'm going to make it an option in NewsBruiser. This brings up my other UI design factor: in addition to the four most popular Creative Commons licenses, you can add whatever ones you like and think people should use. They'll be more likely to be used if they're options and not an 'other'.

Data below, if you're interested:

Relative popularity of Creative Commons text licenses
License	Google hits	% of total	First result
Attribution-NonCommercial-ShareAlike	24900	31.63	Photographers' and Illustrators' Artist Corners | Creative Commons
Attribution-NoDerivs-NonCommercial	21000	26.68	molly.com
Attribution	10300	13.08	Photographers' and Illustrators' Artist Corners | Creative Commons
Attribution-ShareAlike	10100	12.83	Tools
Attribution-NonCommercial	5060	6.43	MSNBC - GlennReynolds.com
Public-Domain	3080	3.91	Mike Linksvayer
Attribution-NoDerivs	1460	1.85	Among Other Things
ShareAlike	1060	1.35	ACM SIGWEB - Conferences
NonCommercial-ShareAlike	769	0.98	Trademarks, Free Speech, and ChillingEffects.org
NoDerivs-NonCommercial	479	0.61	blog@garogreen.com
NonCommercial	323	0.41	Search Engines Directory
NoDerivs	126	0.16	Browse Top Level > Moving Images > Brick Films > LEGO
CC-GNU-GPL	31	0.04	blogkomm: download
Attribution-NonCommercial-NoDerivs	16	0.02	Photographers' and Illustrators' Artist Corners | Creative Commons
CC-GNU-LGPL	13	0.02	Carlsbad Cubes - Wolf Paulus' Web Journal

Sumana's story of her visit to a call center in Bangalore is up on Salon. Forget The New Pentagon Papers, this is the real stuff! Sumana doesn't like the Magic Marker-ish drawing that adorns the story, but I do. I should write my sci-fi story about outsourcing, along with all the other sci-fi stories I'll never actually write.

It's a Christmas miracle! In this episode, programmers and government put aside their differences and enjoy a cool comma-delimited file containing ZIP code information. What are those mysterious last two numbers after the latitude and longitude?

Le Calendrier Républicain.

"This was an attempt to de-Christianize the calendar, but it was an unpopular move, because now there were 9 work days between each day of rest, whereas the Gregorian Calendar had only 6 work days between each Sunday."

From seth-trips, I encourage you to petition for late(r)-night BART service.

Update: Nick says the problem is mainly technological, so clearly what we need is to petition the laws of physics themselves.

Let's say you had a thing produced by someone, and you needed to classify it according to some arbitrary system. Well, you could use the North American Industry Classification System, which reads like the largest RPG treasure table ever. Except you couldn't, because you'd get distracted and start looking at totally unrelated stuff, never find what you were looking for, and the Japanese would take over with their more concise Pacific Rim Industry Classification System, which divides industries into "Stuff" and "More stuff". A random sampling from the NAICS list on the US Census web site:

Canned vegetables, except hominy and mushrooms
Receipts for contract and commission work on women's, misses', juniors', and girls' raincoats and other waterproof outergarments
Hollowware (including toiletware, ecclesiastical ware, novelties, trophies, baby goods, and other platedware)
Urea
Other burial caskets and coffins and metal vaults [plastics, fiberglass, foam, masonite, cardboard, fiberboard, etc., and all children's]
Pattern publishing, including clothing patterns [exclude industrial patterns]

(Foam coffins? For when your beer goes flat? Are they real coffins or not?)

There must be a cool toy in here somewhere, but the only thing I can think of is a little simulation of an economy where people start companies dealing in producing and combining the products of various industries, and there'd be no room for you in that simulation. Those simulated people don't need your reductionist thinking about whether it's a good idea to turn letterpress printing inks and fiber optic cable into stamped and spun utensils, cooking and kitchen, aluminum.

Tell-all PDF slideshow Havenco: What Really Happened. Includes line "Informal and inconsistent restrictions were placed on Havenco's operations, but largely ignored, in the truest form of banana republic.", and plug for author's less-Cryptonomiconesque "you can't catch us, we're the gingerbread man" server hosting scheme.

Update: Riana writes:

Note amusing typo on p. 5 of the PDF that causes the phrase "the underling problems." I'm sure Prince Michael thought/thinks his underling ex-CTO is a problem :)

Presenting the Microsimulation of road traffic. Lets you simulate the causes and easing of traffic jams. It would be great to modify it to let you experiment with antitraffic.

Anirvan points out that the ISBN space is being extended to 13 digits. It's the IPv6 of publishing.

Jason Kottke presents a geographically accurate map of the London Underground. That should make Rachel happy.

Now whenever I see a map of the Underground I have a partial mapping of the stations onto memories, which is a nicely weird experience. If I were to go on another vacation to Washington DC and take the subway there I could observe the formation of this mapping in near-real time. As it is, I have only a couple vague memories from my trip when I was 6: being between stations underneath the Potomac, imagining humidity and dripping water; being told by my father that Foggy Bottom was called that because it was actually underwater, at the bottom of the Potomac; standing on the Triangle platform watching a train arrive.

Of course, I have memories of various kinds associated with over half of the BART stops. There must be a point at which you've used a subway system so much that the stops stop being useful as mnemonic devices for remembering things you did.

Bonus game: without looking at the source code, can you find the link to Crummy on kottke.org?

A while ago I wondered if I could think of any way of encrypting a physical object. I thought of a couple ways of doing it, one of which requires no future technology but which is very time-consuming and only works for certain types of objects. I'll leave that alone for the time being. The other way I thought of requires nanotech, and it's a special case of a cool-sounding general technology. I want to know whether anyone else has used this idea, and how. (And how!)

The general idea is that you send a swarm of nano-bots at an object and compress it into a block of very, very dense nanobot matter. The block has the same mass as the original object, but it comes in a standard shape--maybe the size of a brick, with a protrusion on top for attaching a crane hook or similar lifting device. Now, if you want to encrypt the object, then as part of the compression you simply scramble the smart matter so that the nanobots won't know what goes where without your private key.

Problems: this would probably take a lot of energy. If you tried it on something living, it would probably die unless the compression was incredibly fast. If you could do this you could also use nanobots to make a copy of the object, encrypt the data generated by the nanobots, destroy the object, and recreate it from the encrypted data as many times as you wanted, which is more useful. (Rebuttal: this technology ensures continuity of matter, and avoids the embarrassing situation where you need to reconstruct something but don't have enough spare mass on hand, and the Johnsons are coming over! Also, storing a molecular copy of an object may require about the same amount of mass, in which case the two technologies would be pretty much the same.)

Has anyone thought of such a technology, or encountered it anywhere in fiction? I can think of a couple things I'm working on (very, very slowly) where I could use this.