« August 2007 | Main | October 2007 »

September 2007 Archives

September 5, 2007

Social software design, circa 1998

You are three quick steps away from joining rdos-general.

No I'm not. I'm one-click away from leaving...

September 6, 2007

History Lesson

"Some consider UNIX to be the second most important invention to come out of AT&T Bell Labs after the transistor." [1]
"The AT&T lawyers, concerned with consent-decree compliance, had believed it was safe to allow universities to have Unix." [2]
"Did the consent decree of 1956, then, kick off open source?" [3]
"There was a period in the early 1980s when Tops-20 commanded as fervent a culture of partisans as Unix or ITS - but DEC's decision to scrap all the internal rivals to the VAX architecture and its VMS OS killed the DEC-20 and put an end to Tops-20's brief period of popularity. DEC attempted to convince TOPS-20 users to convert to VMS, but instead, by the late 1980s, most of the TOPS-20 hackers had migrated to Unix." [4]
"Instead they hired David Cutler, who had played an important role in the development of VMS at DEC. VMS was a successful and innovative industrial OS in its days, and Digital had been working on it since the 1970's. Cutler took some 20 former Digital employees with him, and he and his team began the development of NT." [5]
"There are two things Cutler hates: Unix and GUIs. You don't say the word 'Unix' in his presence - you literally do not say it. You shouldn't say 'C++' either, as Bill himself gradually learned." [6]
"Windows-NT is VMS reimplemented." [7]
"At DEC Cutler is widely credited for terminating the 1979-80 Desktop RSTS project and scrapping the manufacturing prototype. Compared to the subsequently announced IBM-PC, RSTS had 40,000 running applications, ANSI languages, and a DBMS. RSTS had a reputation as a robust, stable and reliable multi-user, multi-tasking operating system." [8]
"Although e-mail had been available for use within a computer since the early 1960s, e-mail between computers got its start on DEC-20s... This involved not just new software and protocols, but also new notation: the now-ubiquitous user@host address format." [9]
"The DEC-20 was like a clipper ship, the highest expression of a technology which many believed obsolete -- the large central timesharing computer... Meanwhile, UNIX was taking over the computing world... For these reasons, we decided that it was time to start converting from TOPS-20 to UNIX.

VMS was not chosen for several reasons. First, we were feeling somewhat betrayed by DEC's abandonment of TOPS-20, and did not want to leave ourselves open to the same treatment in the future... Furthermore, UNIX has networking and communications for all our major requirements: Ethernet, TCP/IP, DECnet, BITNET, RS-232 and LAT terminals, Kermit.

The "user-friendly" shell provided by the TOPS-20 Exec, which gives help to those who need it but does not penalize expert users, is probably the feature that was missed the most." [10]

"From a technical perspective, the biggest mistake we made in VMS was not writing it in a high level language. At that time, we had a group of very accomplished assembly language programmers, some stringent size constraints, and no compiler with the appropriate quality for operating system development." [11]
"Developers designed both VMS and NT to overcome the weaknesses of UNIX, which Cutler once described as a "junk OS designed by a committee of Ph.D.s."

Coming from the mainframe world, Cutler's never been a fan of the PC. He considered the Intel x86 line of microprocessors and the OSs that relied on that chip family, to be something of a joke. So in the early days of NT development, circa 1988 to 1990, Cutler focused on a RISC-based chip from MIPS and demanded that his programmers write code that would work on any processor, rather than Intel x86-specific code, which might have been faster but would have been less portable. Cutler's disdain for the x86 continued through the 1990s, as Microsoft ported its NT code-base to other platforms, including DEC's Alpha and IBM's PowerPC.

Of course, NT's cross-platform capabilities didn't seem to matter much by the late 1990s as Microsoft canceled almost every non-x86 port. Cutler's beloved MIPS chip fell first, followed by the PowerPC, and, finally, the Alpha, which had been sold as part of DEC to Compaq." [?]

"When the PDP-11 computer arrived at Bell Labs, Dennis Ritchie built on B to create a new language called C which inherited Thompson's taste for concise syntax, and had a powerful mix of high-level functionality and the detailed features required to program an operating system. Most of the components of Unix were eventually rewritten in C, culminating with the kernel itself in 1973. Because of its convenience and power, C went on to become the most popular programming language in the world over the next quarter century.

This development of Unix in C had two important consequences: It made it much easier to port Unix to newly developed computers. It also made Unix easy to customize and improve by any programmer that could learn the high-level C programming language. Many did learn C, and went on to experiment with modifications to the operating system, producing many useful new extensions and enhancements." [12]

"In the early 1990's we visited Microsoft to try to ensure that their new OS Windows NT would be available on IA32. We met with Dave Cutler, and he was adamant that IA32 was doomed and would we please get lost so he could target Alpha and then whatever 64-bit architecture was certain to replace IA32 by Intel. It was not a polite disagreement; that guy (Cutler) HATED IA32 and wasn't reluctant to transfer his displeasure to IA32's representatives (us). What an ugly business meeting. Smart guy, though." [13]


quotes have been edited for clarity and conciseness


d|i|g|i|t|a|l :)

September 10, 2007

Keith Squared

Heh. Those crazy engineers.

September 25, 2007

My compiler vs. the monkeys

In high school I took part in a computer science fair in my hometown of Pittsburgh. It was city-wide, with folks from schools all over the area participating. The judging was held at Monroeville Mall. (The same mall where 'Night of the Living Dead' was shot, btw :)

My entry was a compiler for a custom language I had invented, implemented in Apple Pascal. I forget the details of the language, but the thing worked, and produced 6502 assembler as output. The assembler could be transferred to a DOS 3.3 disk and turned into a runnable program with the BIG MAC assembler. The compiler was mostly recursive descent, except for the expression parsing, where I used precedence parsing. Some local dad who had a job as a mainframe programmer had shown me the railroad switching algorithm to parse expressions, and I thought it was incredibly cool, so it got used too even though it wasn't really necessary.

(I saw a chapter on precedence parsing made it into Beautiful Code...about time that technique got some pr.)

Anyway, on the designated Saturday I showed up and set up my Apple II on a table. Only when the judges came by -- who were respresentatives from a local Radio Shack -- did I realize that my demo was a little lacking. Watching a compiler run is not very exiting, and the cumbersome process to move the intermediate output code over to DOS where it could be assembled and run didn't help. The judges were baffled by what I had done -- it wasn't clear they knew what a compiler was -- and my mumbled explanations didn't help. They moved on.

Later I learned that a program that animated some monkeys on a screen had won the competition.

My buddy didn't fare much better. He had implemented some kind of font-rendering system for his Epson MX-80 dot matrix printer. Had hand-designed all sorts of new fonts for it too, kerning, crazy stuff, it was some kind of proto apple II postscripty-thing. Frankly I was mystified by his project but it seemed like an awful amount of work and rather impressive. I felt better that he hadn't won either.

I took away an important life lesson from this experience.

Yeah yeah, the value of the demo... presentation, practice, performance. That thin shiny stuff sometimes blows away deep heavy stuff. No... I learned that staged competitions with subjective judging by small panels suck.

September 27, 2007

Kosmix releases Google GFS workalike 'KFS' as open source

Search startup Kosmix has released a C++ implementation of the Google File System as open source. This parallels the existing Hadoop/HDFS project which is written in Java. The Kosmix team has deep engineering talent, including a strong track record, and having recently built a web-scale crawler and search engine from scratch. Google has a set of tools that the rest of the industry needs in order to compete...it's cool that folks are stepping up to the task and leveraging the open source model to try to provide some balance.

KFS arrives with an impressive set of features for an alpha release:

  • Incremental scalability - New chunkserver nodes can be added as storage needs increase; the system automatically adapts to the new nodes.

  • Availability - Replication is used to provide availability due to chunk server failures.

  • Re-balancing - Periodically, the meta-server may rebalance the chunks amongst chunkservers. This is done to help with balancing disk space utilization amongst nodes.

  • Data integrity - To handle disk corruptions to data blocks, data blocks are checksummed. Checksum verification is done on each read; whenever there is a checksum mismatch, re-replication is used to recover the corrupted chunk.

  • Client side fail-over - During reads, if the client library determines that the chunkserver it is communicating with is unreachable, the client library will fail-over to another chunkserver and continue the read. This fail-over is transparent to the application.

  • Language support - KFS client library can be accessed from C++, Java, and Python.

  • FUSE support on Linux - By mounting KFS via FUSE, this support allows existing linux utilities (such as, ls) to interface with KFS.

  • Leases - KFS client library uses caching to improve performance. Leases are used to support cache consistency.

Every startup that scales beyond a single machine needs platform technology to build their application and run their cluster. If enough folks adopt the code and contribute, the hope is that it could become something like the gcc/linux/perl of the cluster storage layer.

About September 2007

This page contains all entries posted to Skrentablog in September 2007. They are listed from oldest to newest.

August 2007 is the previous archive.

October 2007 is the next archive.

Many more can be found on the main index page or by looking through the archives.

Powered by
Movable Type 3.33