Monday, January 08, 2007

Programmer ego boosters

OK, so I got this insane project in the works that does all sorts of magical transformations on the ludicrous crap compilers spit out.

Its generic in nature - throw any OBJ's at it, 16 bit, 32 bit, from any vendor. Assembler, C/C++, Pascal, Fortran, COBOL, BASIC, whatever, I don't care. I'll make sense of it and rewrite the retard compiler's output into something better.

Pretty heady intricate stuff, altering the output from compilers. The vast majority of programmers never even look at what their compilers are puking out let alone think about trying modify it, but that's the kind of project I like. Bit banging, low level and VERY NASTY where there's zero wiggle room, and hand waving don't cut it. What you do is either 100% right on or its flat out broken.

The past few days I've been working on a fixup optimizer. Its non-trivial since fixups and their plethora of variations and permutations are genuinely nasty and difficult. There's a ton of bits in record fields that have all sorts of different meanings depending on the settings of other bits, sometimes an index is present, sometimes not, sometimes there is a displacement, sometimes not, blah, blah, blah.

Working with fixups is like the "twisty little mazes" in the old Adventure game. Its easy to get in, not so easy to get out, and your head is on a swivel reading spec pages trying to find the exit.

So I whipped out about 1,000 lines of code to do some of these fixup optimizations and wonder of wonders it worked 100% on the first shot. Its not often that you crank out 1,000 lines of crazy intricate code in a few days that works on the first go and continued to work against the outputs from about a dozen different compilers I've thrown at it so far.

Its a good feeling.


MikeT said...

Heh. And to think I am just now going to be starting to take assembly language at the local community college (Northern Virginia CC, one of the nation's best). The book is on x86 assembly, which I hear is revolting compared to MIPS or PowerPC, but I'm still looking forward to having more than a basic idea of what you're talking about :)

Purple Avenger said...

I have a love/hate relationship with the x86 architecture.

Its quite non-orthogonal, and is a bit light on registers, which to some makes it "bad", but for someone who knows how to exploit its quirks, it can often get things done in a faster and more compact manner than some of the "good" architectures.

Compiler writers hate the x86 because of its non-orthogonality and dearth of registers. Its hard a hard architecture to optimize for unless you spend a lot of time working on exploiting those quirks. "portable" optimizers (like GCC's) can only do so much with higher level optimization and often just punt and resort to generating pretty retarded code.

This one of goals of my project - to analyze the code itself and improve on it with a battery of peephole optimizations and recognition of side effects.

Most compilers aren't very good at recognizing side effects in the flags and predictable convenient garbage values left in registers.

Back in 84' I wrote a static flow analyzer that would mark out all the basic blocks in binary code. That just needs to be updated some for new instructions and such, but the fundamental logic of it will remain the same

E and J said...

So I whipped out about 1,000 lines of code to do some of these fixup optimizations and wonder of wonders it worked 100% on the first shot

So I take it you don't work for Microsoft then...

Purple Avenger said...

So I take it you don't work for Microsoft then...

I used to work at Borland and IBM.

Microsoft has some very good people and it would be a mistake to dismiss them. Anders, Peter, and Brad(while he was still there) are all top notch. They were lured away from Borland though ;->

E and J said...

I was just going for a cheap laugh. Almost any software company nowadays have to have at least some excellent programmers because otherwise they would go out of business.

The management at Microsoft however I do have issues with - things like the whole 'patch tuesday' thing which gives the hackers a whole month to abuse any flaws, for example.

Purple Avenger said...

The fundamental problem here is the lead time it takes to do a full regression test on a product as complex as modern versions of Windows.

Getting a simple patch out when it has to be provided for multiple supported releases is a non-trivial task. I've been there with IBM, I know.

Its Hobbson's choice - you can drop support for everything in the past and vastly simplify the process (and piss off corporate customers a LOT, because they spent months certifying a particular release in their environment) or get on the multiple release support treadmill which is where MS is today.

E and J said...

I won't even start on how bad testing was in my old company - the test environment was known to have incorrect DLLs, so you test in the test environment, then had to test in production to fix any DLL problems!

Everytime we released a new program you'd have to test it in production on a few machines to see if that worked.

And lets not even go to how the seperate machines had different DLLs on them.

Or how DLLs have incorrect version numbers and you put an older vesion over a new one (just look at the crystal report stand alone and the 'freebie' version chucked in with VB!)

I'm glad I left (several years ago now). Last I heard they are still trying to roll out XP because of regression testing, and how we had a few dos-based programs that killed the CPU!

Purple Avenger said...

Many organizations struggle with a lack of discipline when it comes to SCM process.

Its really a lack of leadership issue. Sometimes certain people need to be shot and management is hesitant to do that because they think they need them so badly.