migration

moving there from here.

My new favourite tool

During the course of my work, I use a hex editor a lot.

Specifically, I use a hex editor most for reverse engineering binary file formats that have no documentation or for fixing corrupted files, and the like. One thing that I've always wanted, was some way to view the binary contents as structured data.. Like "Starting at this byte offset, consider the next four bytes to be an integer, and show me that integer, then, using that integer, take that many bytes immediately following it, consider them to be string data, and decode as UTF8 or EBCDIC... etc.

All that is fairly complex, and generally well beyond the facilities of anything short of a fairly low level and full-featured programming language.

Well, looks like the folks over at SweetScape realized this is a workflow that at least SOME people need to have... and so they built the perfect tool for doing that. It's called the 010 Editor. It has this great feature called Binary Templates as well as scripts. It's the bomb, and it's accelerated my reverse engineering work by at least an order of magnitude.

I'm buying a license, and anyone know know me will realize that's a pretty big deal. I'm a big fan of FOSS and generally try to use as much of it as possible, avoiding commercial apps... but this is a big exception. SweetScape is a small company run by a father and son team. Hardly "The Man". Lowell and Graeme Sweet -- you rock the block and treat the bytes right. ;)

Big Data

When I was a kid -- 7 years old -- I got my first computer. At that time, using a computer meant programming a computer. My Commodore VIC20 didn't have a storage device when I got it (I got a tape drive later on). So, if you turned it off, the in memory program was gone. You had to type it in again the next time you started up the computer.

I remember learning about programming. One of my first programs was just:

10 PRINT "TROY"

I was delighted to see my name show up on the screen. The computer knew me now. I was being reflected back from the screen in glowing pixels. I'd taken a little bit of me, and put it into the computer, and the computer was letting me look at it, and learn about myself, and learn about the computer, at the same time. I was hooked.

10 FOR I = 1 TO 10
20 PRINT "TROY"
30 NEXT

This blew my mind. While there was something magical about seeing my name on the screen the first time. Seeing it ten times was just really really exciting. Something about the quantity was just really cool... and how FAST it happened. It put my name up there ten times as fast as it put it up there one time!! Thrilling!!

10 PRINT "TROY"

20 GOTO 10

The screen was filled with my name.... and it didn't stop. I got up and I ran to tell my mom. I drug her downstairs into the basement and babbled on about the computer and how it was just going and going, and how I did it. GOTO mom... GOTO! My excitement was more than I could contain. I was officially a rocket scientist now. My excitement, like the program was INFINITE!!! I created something that was endless, infinitely long. There was no way to count how many times it put my name on the screen. My computer, at that moment, knew only one thing -- my name, my program, and it would run it forever until I told it to stop. Happily. This was a kind of love and dedication that was far beyond what any person could ever give. There was something really deep here, between me and the computer.

Fast forward now, 24 years later. I'm a software engineer for a living now, and have been for a while. At my company, I recently got promoted to a fancier title "Director of Software Development", and all the responsibility for success lies on my shoulders. People answer my job advertisements with the salutation "Mr. Howard". That part freaks me out.

I'm still excited by big data. That infinite loop of TROYs on my screen was just the start. Now I design systems that process terabytes of data at a time on hundreds of servers. One of the most fascinating parts of my job these days is still the same as when I was a kid. I love hitting "Run" on a unit test, and seeing what happens. I feel good when it's successful once. My next step, almost without fail, is to see what happens when it runs ten times in a row.. Then 100... Then 1000... 10000... 100000... I just keep adding zeros until the thing breaks down, or until I get bored with it.

Big Data is still exciting, still fascinating. I've now given my computer programs more interesting sample material to work from, and so their world view has expanded. Now instead of only knowing me and my name, my programs know all the details of the personal and business lives of thousands of people whose email is processed by the programs. I think my computer still loves me more than any of them though. Secretly, somewhere in there, I know there's an infinite loop on a background thread that's just cycling over the string "TROY"... forever.