2010-02-12

Big Data

When I was a kid -- 7 years old -- I got my first computer. At that time, using a computer meant programming a computer. My Commodore VIC20 didn't have a storage device when I got it (I got a tape drive later on). So, if you turned it off, the in memory program was gone. You had to type it in again the next time you started up the computer.

I remember learning about programming. One of my first programs was just:

10 PRINT "TROY"

I was delighted to see my name show up on the screen. The computer knew me now. I was being reflected back from the screen in glowing pixels. I'd taken a little bit of me, and put it into the computer, and the computer was letting me look at it, and learn about myself, and learn about the computer, at the same time. I was hooked.

10 FOR I = 1 TO 10
20 PRINT "TROY"
30 NEXT


This blew my mind. While there was something magical about seeing my name on the screen the first time. Seeing it ten times was just really really exciting. Something about the quantity was just really cool... and how FAST it happened. It put my name up there ten times as fast as it put it up there one time!! Thrilling!!

10 PRINT "TROY"
20 GOTO 10

The screen was filled with my name.... and it didn't stop. I got up and I ran to tell my mom. I drug her downstairs into the basement and babbled on about the computer and how it was just going and going, and how I did it. GOTO mom... GOTO! My excitement was more than I could contain. I was officially a rocket scientist now. My excitement, like the program was INFINITE!!! I created something that was endless, infinitely long. There was no way to count how many times it put my name on the screen. My computer, at that moment, knew only one thing -- my name, my program, and it would run it forever until I told it to stop. Happily. This was a kind of love and dedication that was far beyond what any person could ever give. There was something really deep here, between me and the computer.

Fast forward now, 24 years later. I'm a software engineer for a living now, and have been for a while. At my company, I recently got promoted to a fancier title "Director of Software Development", and all the responsibility for success lies on my shoulders. People answer my job advertisements with the salutation "Mr. Howard". That part freaks me out.

I'm still excited by big data. That infinite loop of TROYs on my screen was just the start. Now I design systems that process terabytes of data at a time on hundreds of servers. One of the most fascinating parts of my job these days is still the same as when I was a kid. I love hitting "Run" on a unit test, and seeing what happens. I feel good when it's successful once. My next step, almost without fail, is to see what happens when it runs ten times in a row.. Then 100... Then 1000... 10000... 100000... I just keep adding zeros until the thing breaks down, or until I get bored with it.

Big Data is still exciting, still fascinating. I've now given my computer programs more interesting sample material to work from, and so their world view has expanded. Now instead of only knowing me and my name, my programs know all the details of the personal and business lives of thousands of people whose email is processed by the programs. I think my computer still loves me more than any of them though. Secretly, somewhere in there, I know there's an infinite loop on a background thread that's just cycling over the string "TROY"... forever.