A Comfortable Couch

Tuesday, January 04, 2005

Formula Engine Rewrite

This is the story of when I re-wrote the Lotus Notes Formula Engine. It's a pretty cool story, it turns out Vader is Luke's father and I get frozen in carbonite. No wait, that's something else. Enjoy!

Quitting Iris

I've made my living writing software now since 1995. But it wasn't until 1999 that I became a hardcore engineer. That was the first time I quit my job at Iris Associates. Iris was the organization that created and developed Lotus Notes and I worked on the template team. Notes itself isn't really an application, it's more like an OS, and templates are the applications that Notes runs. Describing how Notes templates work is a difficult thing, there is really nothing else quite like it the software world, but I guess I'd describe Notes templates as sort of like web applications. As a template developer, you must deal with back-end storage issues, you are writing in scripting languages for both the back-end and front-end functionality and rendering stuff, it's sort of like working with ASP, HTML and JavaScript...sort of.

While I was on the template team I developed the original Notes Welcome page, I did work on e-mail and a lot of work on some obscure administrative applications, among other things. At the time I quit I was working on new features for the R5 point releases. But I hated it. I hated pushing around pixels all day, negotiating with the Quickplace and Sametime teams who wanted tie-ins and functionality on the Welcome page. I frequently worked 80 hours a week, was stressed and was seriously burning out. Why did I work so hard at something I hated? I don't know, I think because I wanted desperately to get away from it. So I thought if I just worked harder, someone would be impressed enough to give me a shot working on something I wanted to work on. Something more geeky and computer sciencey, instead of the touchy-feely UI crap where marketing types argue with me all day about how they think it should work and look. But I think I just ended up trapping myself. Some months later I was told by another project leader that I was barred from moving to his team by my managers because I was too vital to the template team.

UI stuff is hard, but not many people actually appreciate how difficult it is to do correctly. Everyone thinks they're an expert, but very few have actually spent any time thinking seriously about it, reading any article or books on the subject, watching users behind one way mirrors during usability tests and seeing how easily users get confused and distracted. So one day I'd had enough, I couldn't stand it anymore, I was tired of battling all day with people who thought their "little" tweak would have no repercussions whatsoever. Being a UI guy isn't just producing UIs, it also requires negotiating with lots of people, at least at Iris it did, which was the part I liked the least about the work. That's probably why Notes historically has such a maligned UI, because the culture there made it difficult to produce anything decent. So after a particularly frustrating run-in that I won't go into, I quit. I put in my two weeks and that was going to be that.

Completely Unqualified

Then a very friendly guy named Wai Ki Yip called me before my last day. He was a manager and engineer and he offered me a job on his team, the Language Services team. The Language Services team was responsible for the programming languages found in Notes and Domino: Formula Language, LotusScript, Java, and JavaScript. He wanted me to take over Formula Language, which was the original and most tightly integrated language in the product. Actually, unlike the other languages, it only existed in Notes and Domino and nowhere else. If you've used spreadsheets formulas, it kind of resembles that. There was no one who "owned" the language, no one who understood how its engine and compiler worked, and Wai Ki's group had only recently taken it over. It was pretty much an orphaned area of the product, but a very important area nonetheless. It was used for building view indexes, in Notes and Domino forms for field validation, dynamic rendering, application logic, in macros, in agents, and dozens of other places in the product. As a template developer I used Formula Language every day. For certain things it is highly productive, but for others it's completely unsuitable. Until Notes 4 it was the ONLY programming language in the product, and even though it was a simple language lacking even the most basic constructs that everyone takes for granted (like looping) it was possible to build very powerful applications using just that.

Oddly, writing applications in Lotus Notes and Formula Language was the very first paid thing I did as a software developer. I started with Lotus Notes V3 in 1995 during my first college internship at a software engineering group at IBM in Charlotte. They used Notes internally and I was put to work on the internal Notes applications, maintaining existing ones and writing new ones as needed. Then I worked as a consultant, writing custom Notes applications for mostly Fortune 500 companies. I became so good at it (and sick of consulting) that I went to work at Iris to develop some of the templates that ship with the product. So it seemed very strange that I would get offered the job of taking over a language that only a few years before I was using just like millions of other customers, I'm still not sure how I got from one to the other.

No one on the language services team, or even at Iris, understood the runtime engine internals completely, also known as "Compute" (the compile/decompiler was called "Compile", great names eh?). There was no documentation, the code was barely commented, many variables had one and two character names, wild pointer arithmetic was everywhere and not even the stored byte code format the compiler produced was documented. Bugs were piling up and there was no one to fix them. On top of that the engine was becoming a bottleneck for server performance, particularly for building view indexes. The engine needed to be rewritten from the ground up, it needed to be made maintainable, it needed to be faster, and it needed to be completely backwards compatible..

So here was I was, offered this position that I clearly wasn't qualified for. I had no experience with language runtimes or compilers, I knew very little about C and didn't know anything about C++, I had never dealt with platform byte ordering and packing and all the other issues associated with writing something for eight different operating systems, I had never even used proper version control. But none of that mattered to me. It seemed to me like an amazing opportunity and I would be doing exactly the kind of stuff I enjoy most. I wouldn't be bothered all day by marketing folks asking could the forms be mauve, I wouldn't have to worry about the wording of some field label on some feature that no one even knew how to get to, I wouldn't have to deal with icons that were too Anglo-Saxon, too masculine or too feminine, I wouldn't have to worry that some important VP wanted a new button that would confuse 90% of users. I would just write code. So I said yes.

To this day I don't know what possessed Wai Ki to offer me the job. It doesn't make sense really. I can't help but wonder if I was thrown into the fire just because they thought I couldn't do it, that I'd try it, fail miserably and go back to the template team. Maybe, or maybe he thought I would really be able to do it, maybe he saw something in me. All I know is that I was terrified. I knew the odds were far against me, and if I failed it would be because I wasn't good enough. I couldn't blame anybody but myself. But still I said yes, I was going to give it my best shot.

I remember talking with friend that overheard a very senior engineer saying I didn't have any real chance of pulling it off, that the old engine was very optimized and mature and was nearly fast as could be reasonably expected. Another engineer who had done a little work with the engine told me I had a very difficult task ahead, and if I pulled it off it would be a miracle. This was the kind of stuff that really motivated me, I love it when people think I'm going to fail. It adds one more reason to do it, to prove wrong the people who don't believe in me.

Getting to Work

So I moved my Pee Wee Herman doll and tropical fish lights from the client development area to my new office in the server development area and I immediately got to work on the backlog bug queue. See, even though I was supposed to eventually rewrite the engine for the R6, the engine in R5 was buggy as shit. There were crashes and weird behaviors all the time, but since nobody "owned" the engine, no one fixed the bugs. The bugs often weren't reproducible and the code was tricky. It's not like there weren't people qualified to fix the bugs, but there simply weren't enough GOOD people to fix the bugs. And for most of the bugs, you had to spend a lot of time learning the engine before you could make heads or tails of the code, which as I said before was notoriously terse and hard to understand. Iris was a lot like most engineering organizations, most of the good work was done by handful of talented people, the others often produced barely functional crap that would need to be cleaned up someday. So since the all the good engineers were working on other bugs and features already, the engine got neglected. Plus no one wanted to do it.

So I started on the bugs and I picked the easiest ones first. It was slow going as I didn't know C very well and I didn't know the Visual C++ debugger at all. Really it was pretty damn painful. But I spent my first couple of months just on the bugs, and I learned C and I learned the engine and I learned the joys of rampant pointer arithmetic and two letter variable names and no comments. I still remember those first weeks, it seemed like I looked at the code all day and dreamt about it all night. My brain seemed like it was rewiring itself so that it could understand what was going on. I learned slowly at first, then faster. Eventually I could actually look at the code and read what it was doing instead of slowly trying to figure it out it line by line.

So I spent the first four months just learning and fixing. Sometimes I'd cause regressions, especially early on. At one point I made so many mistakes that the build team talked to Wai Ki about me, I was slowing them down and if I kept it up they would take away my code submission privileges. I'd have to get my work reviewed by someone else for each code submission. So I learned to get much more careful and thorough, and eventually I cleared out a rather large backlog of bugs without unleashing total chaos. I had accomplished step one.

After I cleared all the bugs out, it was time to start rewriting the engine. The first thing I did was make clear documentation of the compiled format for the formula language byte code, down to the individual byte. Then I decided on the design for the new engine in my head, presented it to Wai Ki on a white board in my office explaining high level architecture and what it would accomplish. It was nothing like the old engine. Where the old engine was naive and slow, the new engine was going to be slick and efficient. And the new architecture would allow for all sorts of constructs that simply weren't possible in the old engine. For example an embarrassingly simple thing like looping wasn't possible with the old engine, it just wasn't. It went against its whole architecture. My new architecture was profoundly different and quite ambitious.

Now you might think that I produced a bunch of design documents and specifications and presented them to the various senior engineers and architects, but I didn't. I remember being surprised by this myself. Even Wai Ki didn't have much to say about my design or how it should be implemented. The philosophy was that if I did those things, everyone would meddle with the design and nothing would get done. It's truly easier to ask forgiveness than to ask permission, not to mention things get done a lot faster if you just do them. It sounds weird, but that was big part of the culture at Iris. Just get it done and do it right, do whatever it takes. So that's the way I worked, I was just going to go headfirst into it, damn the torpedoes.

Since most new areas of the product were being written in C++, I decide also to write the new engine in C++. I started reading Bjarne Stroustrup's The C++ Programming Language. I only got about one third through the book before I decided that I understood as much as I could without actually coding, and so I started coding. At first I didn't really know what I was doing, it was such slow going I probably only wrote about 1,000 lines of code the first month. I asked other engineers really dumb C++ questions, the kind of questions someone asks when they don't even know what's important. I would often get puzzled looks, not because the questions where so dumb, but because some guy who's rewriting the core language of a mature product like Lotus Notes should know the answers to those questions. But I slogged on. At one point I realized that I had written a large part of the code all wrong, that I was trying to use C++ in the wrong way (a very easy thing to do), and I had to throw away most of the code I had written.

Near the beginning of the project, Wai Ki and I set milestones for my work. Actually, he just asked me the order I was going to do things and then how long each thing would take. I just made uneducated guesses at the tasks and their completion times and he wrote them down. For some reason I really thought I needed to meet that self-made schedule, I had said that's what I was going to do and dammit I was going to do it.

Slow Going

At one point early on I became panicked, I thought not only there was no way my engine would be complete by the R6 feature deadline and the product would ship without it, I was scared to death that the new engine would never really work, that there was going to be some fatal flaw in my design that made it fundamentally incompatible with the old engine, or the new engine was going to be slower and hence useless. But through all this Wai Ki seemed to have unflagging confidence in me, he calmly told me that I just need to take my time, do it right and not get so stressed. At one point I was in his office talking about the all the work and how behind I was and how everything looked bleak, and the longer I talked the higher and louder the pitch of my voice got. He told me I was "freaking out," to which I replied, "Yes! YES! ....I'M FREAKING OUT!!!" But it was his calmness and confidence in me that gave me the courage to throw away large chunks of work when I saw it was bad. See, if I was panicked about it, then I'd have kept the shitty code around in a wrong-headed attempt to just get something built and worry about all the problems later, which is a very bad thing to do.

So I kept at it. It took me nearly six months to produce about 8,000 lines of code. The new engine had a lot of architecture to be built, there was engine startup and shutdown, reading and validating the byte code, formula optimization analysis and building the computation structures, memory and resource management, interfacing with various contexts in the product, the error handling model, variable scope resolution, and making all the internals object oriented so the individual @function and operators could be written easily and clearly (and no pointer arithmetic, I abstracted it all away). During this time I made some very unconventional decisions, the kind that taken in isolation sound crazy, but when combined into a complete architecture make good sense. It took six months to write enough code so the engine could actually run a formula. I was a couple of months behind my arbitrary schedule but I finally had something. I remember the point at which there was nothing else to write to actually run the engine, it was a surreal moment, I felt like there was no way I actually could have reached that point, but somehow I did.

Once the new engine was to a point where it could actually run, I needed to find something to test it out. I needed to see if it could actually run real formulas and if it was as fast as the old engine and could produce the same results. So I picked the most commonly run formulas I could find; the view formulas for the All Documents view in Notes mail. These were the formulas that got run for every mail message that gets delivered in Notes mail. I implemented all the operators and @Functions used in the formulas so it could run in the new engine.

I still remember that day I turned it on and turned it loose on a Notes mail file, the day my new engine truly came to life and did something useful. I ran the new engine and the old against the same mail file, and compared the results. The computation results were byte for byte identical, and much to my amazement, the new engine was just barely slower than the old engine. This was very good news because the old engine had been tuned and tweaked years ago for optimal performance, even I had recently added to the old engine's performance during my bug fixing phase when I found some inefficiencies that had crept in. So here was my brand new untuned engine, hot off the presses and keeping up with the old, and I was a proud papa. I was so excited I could barely sleep that first night I got it running, because I knew the next day I was going to start profiling the new code and find its slow spots. I expected to get the new engine to be maybe 50% faster than the old, and I would have succeeded. I would have built something faster, I would have done something that extremely smart and well-respected engineers said couldn't be done.

I profiled the engine, found the hotspots and destroyed them. They say you'll always be surprised by what the profiler shows, and I remember that I was. But one by one I picked off the hotspots, the easiest, most obvious ones first and then the trickier ones. And something amazing happened, the new engine wasn't just faster, it was a shitload faster (shitload is an engineering term meaning great amount). The measurable performance gain was 300% faster (I would later discover it was even faster when running in other contexts). I was very excited, I could barely believe what I had done. I checked and rechecked to make sure I was right. Everything pointed to the new engine being blazingly faster. It felt like I worked a miracle, I was giddy as school girl.

More Code, Lots More

It took me six months from starting coding to getting the engine to this point, to getting it to 8,000 lines of code that just had minimal functionality, the job wasn't even close to being complete. I had to rewrite the over 200 remaining @functions and operators before I could check the new engine into the daily build. I remember I had a big paper list of @Functions I worked from, and as I rewrote each one to work with the new engine I crossed it off my list. This part went so fast I barely remember anything from this two month period. I woke and coded and coded all day everyday. It was like I was on autopilot, my brain felt like a machine. Each @function I completed was like a hit of a drug and I was a junkie looking for the next fix. It's all a blur now, the code all runs together in my memory. It took me just over two months to complete, but when it was done I had written over 15,000 more lines of code. During the six months writing the first 8000 lines, I produced an average of around 60 lines a day. During that two month period my production had jumped to around 350 lines a day. Of course it was a lot easier to write that code, I didn't have to worry about architectural issues much, just individual @functions.

Once I had the new engine completely rewritten and working on a side branch I ran it for about two weeks with it as my primary mail client. Then we ran it on several production Iris mail servers. This was a great test of code. At Iris we always ran experimental code on the production mail servers. You find so many bugs that way because it's a real world production environment. I was on call constantly, when the production servers went down I was usually the first on the scene debugging the problem. If it was my bug I'd fix it and rebuild and reload it on the server, if it was someone else's they'd fix it and put it into the daily build and I'd pick up the changes the following day. When everything was fairly stable, I checked-in the new engine code in late August of 2000. I was actually ahead of schedule. Lots of problems were found and I fixed them as soon as they came in. Some were crash bugs, some where incompatibilities with the old engine, some were just plain weirdness. But soon the problems slowed to a trickle and found myself with very little to do.

That's when I started adding all the new stuff that I wanted to add to the engine. I remember I posted a question to a forum dedicated for professionals who worked with Notes and Domino (The Business Partner Technical Forum), and I asked what sorts of features they would want to see in the new engine. That question got over a thousand replies. Some of the requests where reasonable, some where stupid, some I just didn't understand. But I read each and every one and tried my best to consider them. In all when I was done adding all the new enhancements, the total code base had swelled to over 30,000 lines.

And through all this the new formula engine continued to get tested. The new engine was pretty reliable from the get-go, it very rarely crashed and when it did I relentlessly hunted down the cause until it was found and fixed. After about the first six months there were pretty much no crash bugs at all, almost all the bugs that came in after that point were slight incompatibilities with the old engine. More often than not, the old engine had it wrong, it had some indefensible behavior that was clearly not what was intended and often the formulas that relied on these behaviors did so unintentionally. It's like two wrongs making a right, but regardless if the old engine was wrong or right, the new had to have the same broken behavior. In software we call that being "bugward compatible". It sucks, and I absolutely hated that I had to go in and purposely mess up my carefully crafted, elegant code but that's what I had to do. That's necessary for a truly stable and backwards compatible product.

I remember when the Microsoft Windows source code got leaked on the net and somebody had searched all the code for expletives. There were tons of angry comments like this:
// the fucking alpha cpp compiler seems to fuck up the goddam type "LPITEMIDLIST", so to work
// around the fucking peice of shit compiler we pass the last param as an void *instead of a LPITEMIDLIST
and:
* for idiots like MS-Access 2.0 who SetWindowPos( SWP_BOZO
* and blow away themselves on the shell, then lets
* just ignore their plea to be removed from the tray

I completely understand why the coders put in comments like that, because you feel like it's your job to ruin your own beautiful creation because someone else wasn't smart enough to use an API correctly, so you vent. I think I left at least one nasty comment like that on a bad day, but I felt like that much more than once.

All Done

Notes and Domino 6 was eventually released more than a year later than originally anticipated. Had I known, my schedule probably wouldn't have been so aggressive, but it was a good thing for the new engine because it got so much more real world beta testing. I had already left IBM before the final release, but I kept tabs on what was going on and the new engine was a big success. While a couple of serious but rare backward compatibility bugs were found and fixed after release, the new engine was very solid, as far as I know it never crashes.

The thing I really liked was getting feedback from people who coded Notes applications exclusively using Formula Language. There is a large population of developers who never bothered to learn more structured languages. They would tell me how all the new features made their job so much easier, and they could now build applications they couldn't before. Of course lots of people liked that I just made the language more powerful so they didn't have to use those other languages. I loved that my work got many of these people excited to upgrade to Notes 6 and that it played a part in making Notes 6 a big success. Many people make their living building Notes applications, and they bet everyday on us. I wanted to make those people right, I wanted them to be successful because they believed in us and the product, and I felt like I helped make that happen.

Rewriting the Formula Engine was only one of the things I did during R6 development, I actually spent less than half my time on it. I also took over unread marks and developed the now patented unread mark replication system. Unread marks are a simple thing in concept, but in reality they are very difficult thing to do correctly, particularly in a distributed system. Before my work on them they were very much broken, with both flawed implementation and flawed design.

I also spent many months during crunch time working on the Database Team fixing bugs and performance problems. These were some of the hardest bugs I had ever faced, frequently we had no reproducible scenario, just a memory and stack dump, sometimes not even that. But just because a bug wasn't reproducible didn't mean we could ignore it. I spent countless hours pouring over hundreds of thousands of lines of code, trying to find the code flaw that would later crash the system. For some reason I really enjoyed it, I learned a great deal about how to write and debug high performance server and database code. Just to see if I could do it, I set a goal for myself that every week I would try to fix more database bugs than any other member of the team. This was a private goal, not something shared with the team, but it was something to keep it really challenging for me. More often than not I met that goal, and the faster I got, the tougher the bugs the database team gave me. Eventually I got one bug that had been crashing servers for six months, but remained unfound. It had stymied at least three other engineers before me, but I don't give up easily. I probably spent 80 hours on that bug alone, but I found and fixed it. It was really a lot of fun, but I'm weird like that.

My time at Iris during R6 was the best I've ever had in my professional career, almost every day was a new challenge and game to play. But ultimately IBM killed Iris. In late 2001 IBM folded Iris Associates and converted all the employees to IBM. All the energy the Iris building had was seeping out. Really it was more pushed out by the BS that IBM management kept trying to sell us, like that nothing important would change and you'd still be able to work unfettered. Yeah right, just ask anyone who still works there how much time they spend battling bureaucracy vs. writing code. But such is big corporate management, they don't even realize how much they're damaging things, they are essentially politicians not engineers. It's sad, but I'm grateful for the time I had at Iris anyway, if it was still there I'd go back in a second.

Update: My follow-up can be found here.

51 Comments:

Anonymous said...

That is a great story, Damien... thanks for giving us a peek into what went into the Formula rewrite. You did a great job, and it's even more astounding knowing how little experience you had going in.

Thomas "Duffbert" Duff
http://www.twduff.com

10:06 PMlink  
Ed Brill said...

Nice write-up, man. I think you'll get a lot of respect for opening up ... hope it felt good to purge all that energy out of your system.

My thoughts are at http://www.edbrill.com/ebrill/edbrill.nsf/dx/damien-katz-formula-engine-rewrite

10:12 PMlink  
Anonymous said...

Damien,

Thanks for sharing this fascinating story. I was going to reply in the form of a "Formula" but am too lazy. What fascinated me was the fact that you had to not only clean up uncommented code but do it while learning several new languages. Jeez........ I assume you haven't hit 40 yet :-)

Thanks again for sharing.

Bruce Elgort
http://www.openntf.org
http://www.bruceelgort.com

10:32 PMlink  
Anonymous said...

@If(@Author = "Damien Katz"; @ReplaceSubstring("Formulas suck"; "suck"; "rule"); @Failure("Too little documentation. Rewrite required.")

-- Nathan T. Freeman

3:22 AMlink  
Anonymous said...

Thanks for telling that story Damien - it was a great read - and I bow down to you - thats some job you did.

You know I always wondered how the hell the formula engine got rewritten - when you see everything else going on around notes technologies I couldnt imagine for a minute it was on their "todo" radar.

I'm not sure you should get a career change - unless you are totally burned out from developing (I really know what thats about - its happened to me a couple of times - some people just put so much energy into things and dont know when to stop!). You must have so much ability that surely some product development dept/company would pay big bucks for you?

Steve Castledine
http://www.dominoblog.com

5:06 AMlink  
Anonymous said...

Fantastic story, thank you. Did you ever get the chance to talk with Wai Ki Yip why he had choosen you for this task? Assigning a rewrite of core functionality to a C++ novice is not the usual way of getting things done.
Henning Heinz, Munich Germany

5:12 AMlink  
Ben Poole said...

An amazing story: I thought you did a brilliant job on the engine, but to find out you did it essentially as a "newbie" blows my mind.

6:21 AMlink  
Anonymous said...

So this is how the West was won?!
And now the world is your oyster!

Much needed and not a minute too soon the new engine. Fantastic!

Slawek

7:24 AMlink  
Anonymous said...

A good story and a great opportunity to give a big thanks to Damien and all the other hard working developers on the Notes/Domino team.

I had some exposure to the Iris guys back when I was working in Lotus Development UK, we used to kick the developers' asses in Halflife Deathmatch running across the WAN. Great fun and we actually got some work done too. I remember one of the guys, I think it was Bruce Kahn, agreeing to fix an R5 bug for me if I got more frags than him!

Cheers Damien...I too would go back to the Lotus I remember and love...even it they yellow has a blue tint these days.


Ben Rose
http://blog.jaffacake.net

8:33 AMlink  
Pete said...

Great post Damien; very accurate too. As to why Wai-ki picked you, I would say it was because you already had a reputation as a real go getter.
As for people doubting you could pull it off, add me to the list. I recall a conversation I had at happy hour one Friday where I said something very similar to the engineer you mention. Obviously, I was wrong and humbly apologies.

8:57 AMlink  
Anonymous said...

Great story Damien, thanks for posting it! Your rewrite of the formula language is something I've been curious of for some time. Years ago I read a notes.net article you posted describing the architectural changes (which, by the way, piqued my interest in languages and compilers which I crave to learn more about) and after reading this I can't believe you were able to accomplish all that you did. Truly one of the most amazing developer stories I've heard of. You did an amazing job,

Jordan Matthiesen

11:26 AMlink  
Andrew said...

I remember somebody telling me (maybe Ned or Bob) that the reason that you were going to succeed is that you were too young to know that you were going to fail.

I guess they were right!

11:52 AMlink  
Anonymous said...

Thanks for sharing, Damien. I have heard bits and pieces of this before, but to read it all at once is awesome.

I posted some more thoughts about it on my blog as well - and to make sure others see it (it's "required reading" ;) ).

http://www.lotusgeek.com/SapphireOak/LotusGeekBlog.nsf/d6plinks/ROLR-68CPBQ

Thanks again!

Rock
http://www.LotusGeek.com

1:25 PMlink  
Anonymous said...

Damien,

Excellent, excellent piece and believe me your efforts have not been lost on the Domino community.

You should have sold this for publication to the highest bidder:-)

Christopher Byrne
http://www.controlscaddy.com/

2:20 PMlink  
Dan Schwarz said...

Damien-

As a former Iris employee myself I know exactly what you were up against when tackling the formula engine. In my four years at Iris (1993-1997) I worked on some @Command enhancements (probably generating some of the hacks you had to work around :-/) The old formula engine was exactly as you describe. Unowned, undocumented, and damn scary with a vicious learning curve. Bravo for putting the code out of its misery and building a better replacement!

3:10 PMlink  
Ray Ozzie said...

Great post, Damien. I'm proud to have provided you the opportunity for such an experience by creating such a corpus of write-only code. One thing you didn't mention, though: IMHO, COMPILE was even less maintainable than COMPUTE, no?

A bit more history, FYI: The first "compile/compute" engine I wrote for NSF had an english language like syntax that was focused on testing the values of attribute/value pairs, e.g.:
color = "red" AND class = "fruit"

That was in early '85. The syntax ran out of steam pretty quickly, and I needed something better. I was very comfortable writing NSF and the Editor and whatnot, but had no compiler background so I transliterated the source code of the @Function compiler/runtime for Jon Sachs' "1-2-3 C" prototype implementation. (Jon wrote 1-2-3 in well-commented elegant 8086 assembler, and as a reference model for other implementations provided another core implementation cleanly written in C, but my recollection is that it was sparsely commented. Widely derived from at Lotus.)

It was a basic stack-oriented virtual machine architecture that suited the spreadsheet well, and did the job reasonably for what I needed. I started adding text-based primitives and the concepts of "lists", not really grokking the surrounding framework but happy nonetheless that I could keep moving the product forward. (Iris was only Tim and Len and me at the time. Or maybe also Steve and Al; I forget exactly.)

By my [perhaps mis-] recollection, the biggest nightmare came when we desperately needed to shrink the code, as we did in those days. (We needed to make the Notes client run efficiently in a 250KB working set on Windows 2.) So, I decided to do MAJOR surgery to COMPUTE (and the entire EDITOR) one weekend by enabling the entire subsystem to be compiled for either NEAR or FAR addressing models. We used the 16-bit NEAR addressing model on Windows, and the FAR addressing model on OS/2 where we had virtual memory. (And the NEAR addressing limit introduced my favorite error message: "Too Many Concurrent Formula Evaluations".) It worked like a charm, but the source began to get messy. Then, because because the near heap turned out not to be enough even on Windows, I used the new "pools" memory manager for NEAR addressing of many pools of memory, but the macros ultimately made hash out of the source code. The way the macros implemented base+relative segment addressing got really confusing. sigh!

Anyway, hopefully all of that history is gone from the code base by now. The Notes world owes a debt of gratitude to Damien for cleaning this up!

5:49 PMlink  
vowe said...

Great Story, Damien.

Ray, when do you restart your own blog? It sure was interesting.

7:26 PMlink  
Anonymous said...

Kotaus to Damien!
Next Mission (?): Write an engine to run @Formulas in a J2EE container?
:-) stw
http://www.wissel.net/

8:43 PMlink  
Anonymous said...

That is a great story-- and quite an accomplishment-- but good god man: IT'S LOTUS NOTES. We use it every day by corporate mandate, and every soul-crushing minute of using this application is unbearable. And understand that I love computers. I spend almost every waking moment in front of one. Lotus Notes is the first application I've ever used that sucks every iota of joy from using the machine.

I used to make fun of Lotus Notes, for little things like breaking every UI and keyboard convention I've ever learned in 10 years of using windows. I've since decided that Lotus Notes is like a mentally handicapped child. There's no point in making fun of it; it can't help the way it was born.

You seem like a smart developer, and your story is inspiring and interesting.. but I am truly sorry that your greatest moment is associated with such an aberrant product.

11:10 PMlink  
Anonymous said...

That's a great post, Damien. Thanks.

11:51 PMlink  
Anonymous said...

This is very interesting, but it also shows some interesting flaws in how you approach problems.

1) You make it sound like you hated dealing with marketing people who suggested UI changes because you felt they weren't always following usability standards. Perhaps, but having been on both sides, I can tell you that many engineers selectively choose which usability standards to follow: the choices tend to be correlated with ease of implementation. Yes, I cannot argue that marketing people will propose bad choices... but I've seen many more cases where engineers, even after multiple usability training courses, still made basic errors which hurt the users, but are easy to code. I don't know if that's the case here... but I wonder if some of the marketing choices were good ideas which were difficult to implement, or would require rewrites which, in your earlier stage of development, would have required code tossing that you didn't want to do at that time.

2) You talk so positively of the "get it done" and "code it" mentality of Iris, and of the "ask for apologies rather than permission" approach to coding. Its interesting to read that in comparison to an article about the development of Oracle 10g, http://www.oracle.com/technology/oramag/oracle/03-sep/o53making10g.html. In this article, they point out that
"The emphasis on planning significantly shortened the Oracle Database 10g development cycle and helped to condense the amount of time developers spent writing code. "If you look at the entire release cycle, the actual time developers spent writing code is about 25 percent of the overall project," says Kumar. "That may sound surprising, but it makes sense, since the rest of the time you're either planning for the release, writing specs, or testing.""
They think that writing enterprise software involves spending more time planning it out than coding it. You imply (and sort of state) that all that planning would have interfered with getting it done. Perhaps... or perhaps others knew ways which might have made your design even better. That's part of working in teams vs. the cowboy approach.

And maybe IBM has lots of bureaucracy, or maybe some of that bureaucracy is really dealing with product features and management issues. Again, as you pointed out, you learned lots and grew up during this process. You talk of all you learned about your ability to code better, and solve tough problems. After I finished this story, I wondered "where is the 'learning how to work well with teams? share learning? coordinate on specs?" That seems to be missing from this story of personal triumph in coding.

I love hearing about stories at Iris; it seems like it was a special place. Having met Ned B, Bob C, and Andrew W, it's clear Iris was full full of smart folks like yourself. And I'm glad you got a chance to challenge yourself and succeed. But I wonder: were some of the things you mention disliking due to your lack of training and maturity at the time? Do you wonder if perhaps, now that you've had a trial by fire of fixing messy code, you would approach these conflicts differently if you could coach your younger self?

Michael
wexler at yahoo dot com.

12:00 AMlink  
Anonymous said...

wow, what an awesome & interesting story indeed. you're the man Damien...thanks to you and more power.

ben perales

12:28 AMlink  
Richard Shergold said...

Damien

Thanks - that's a great story. I'm using your R6 formula language every day and can only say your hard work was well worth it. Best of luck with Couch, I hope it comes off fo you.

8:47 AMlink  
Anonymous said...

Lotus should give up the Workplace joke and give this one honest man a job to redesign all the tables and RichText stuff.

1:46 PMlink  
paco said...

Thanks for the great story Damien. As someone who used and tested the new formula engine (and new functions) early on, I can say it was an enjoyable experience.

Although the internals of the old engine were gooey, I enjoyed using it from the outside, and preferred it even after the advent of LotusScript. I did not even mind the lack of looping. It meant you had to think harder and design better, capitalizing on Notes' strengths rather than just coding the first thing that came to mind. I loved lists, although I could never figure out sets.

I understand bugward compatibility, but it seems like there should be a better solution than breaking beautiful working code. A few releases left the old code alone and introduced new opcodes (@V2If, etc.), but I suppose a lot of those would make a real mess.

Having moved on from IBM (mostly due to bureaucracy), I still have fond memories of formulas (old and new). And I hope the new engine makes it easier to write the formula debugger that I am sure folks still want.

4:18 PMlink  
Anonymous said...

This story is a great example of Iris and why it ultimately became a no fun bastard child of IBM. Take a 4th or 5th generation employee of Iris and baby him until he produces some good work,then watch as he becomes a megalomaniac pompous blowhard who thinks he is a genious named Damien. The reason Iris now sucks is because Ray Ozzie sold it out to IBM, left the company, and willed the company to a bunch of no talent technocrats. Luckily, the word "I" is easy to spell. Must have really helped you write this drivel.

10:59 PMlink  
Anonymous said...

Man, what's with all the negative comments and criticism towards Damien in the replies? WTF is the matter with these people?

-- Nathan T. Freeman
Co-founder, OpenNTF.org

3:46 AMlink  
Ben Poole said...

> WTF is the matter with these people?

They're morons. There are a lot of them about. And they all post anonymously to boot. How brave.

8:15 AMlink  
Damien said...

Cool, I've got a troll. I gotta go get my care and feeding manual. Every megalomaniac pompous blowhard needs a good troll.

1:12 PMlink  
Bob Brodsky said...

Damien,

You did a great job and so did your predecessors! I have used every release of Notes since 1.0 and it has improved with every release. I know the formula engine as a user and I think that the formula engine is one of Notes' major strengths: I can produce a working prototype of an application and demonstrate it before some other technologies can get all of their software and servers installed! Users/customers are MUCH better at telling you that they like or dislike what they see than telling you what they want/need.

Anyway, Great Work! Some day we will get @ThisFieldHelp into the product too.

Bob

2:13 PMlink  
Anonymous said...

Thanx for a very exiting story. I'm very impressed you actually made it wihtout a great experience in C++.

I hope you are proud of what you did, since it must have been a very emotional period of you life. I also hope you will get more opportunities/challanges like that in the future.

The reason why I truly loved this story is because I have a similar experience as yours. Or at least I can identify myself with your story. My experience, however was not related to a succes like Lotus Notes/Domino, but my very first job was to be responsible for the core systems of a reporting and consolidation application that consisted of 200,000 lines of C-code. This was back in 1989 and Windows was not an option.

My first assignment was to rewrite the form-engine that was used in the various parts of the applications. I do not remember every specifics of it, but your story made my brain work for thoose memories.

Again thank you for the most interesting story I have read in a blogg so far.

Henrik Grönvall
henrik.gronvall@synergica.com

6:59 PMlink  
Anonymous said...

Very pathetic. What a bunch of losers living in the past of Lotus Notes. Hiding is more like it.
Try getting into the present and getting some WORK done.
As opposed to resting on your fat ass laurels like you are some sort of megalomaniac pompous blowhard.
I remember Damien at Iris , he was a pompous blowhard.
Ray Ozzie is the biggest phony of all time. Mr. Sellout.
He sold out Iris to IBM, and he has already sold out Groove to Microsoft.

10:53 PMlink  
Dave Harris said...

Great story, Damien

Although I don't get to do any development anymore, I remember getting involved with Notes around 4.11 - wheen looping and the rest of your additions came in, I started really missing working with the simplicity of the Notes client, rather than having to code everything for the web.

Thanks for the hard work, and thanks for sharing the story.

3:02 AMlink  
Anonymous said...

Thanks Damien! This is the most inspiring and interesting notes-related blog I have ever read.

11:42 AMlink  
Anonymous said...

This is for "Anonymous" who seems obsessed with the phrase "pompous blowhard":

Wow, you sure are sure of yourself, talking smack and attacking Damien and others. And man, how brave you are attacking Ray Ozzie himself! Woohoo! Let me congratulate you Mr.... Mr...

Wait, that's right - you didn't leave a name. Not too proud of those words, huh? Don't have the testicles to own them?

It is amazing how brave people become sitting in front of a keyboard, especially when posting anonymously. Why don't you grow yourself a pair and actually post your name, and tell us what you have contributed to society that prevents you from being the waste of human flesh you presently appear to be.

Oh, and as for the comment about "getting some work done" - while I am sure that you're out there spending months (and someone else's money) writing bloated code that does some basic task, I am actually producing apps that will run circles around most "standards based" apps, and I am writing them in a fraction of the time - all because some "pompous blowhards" like Damien, Ray, and a host of others (minus you) put in some great work on a platform that has been around longer than you've had pubic hair.

There, that should keep the troll full for awhile.

Rock
http://www.LotusGeek.com

10:12 PMlink  
Anonymous said...

Dear Moron Troll;

While you don't tell us who you are, a few things are apparent.

You are stuck in a corporate role staring at a computer with no control over your own destiny and no ability to make real decisions. Thus, a low level I.T. dweeb not moving upward any time soon. You likely think you know how things work, but in fact couldn't project manage your way out of a paper sack (well, at least its clear you can't get out of an 8x8 cube). As to programming? That ACE certification class doesn't count.

I'll take any large scale secure web site requirement spec you can name, and develop sooner, cheaper, and it will be more secure than anything you can come close to with Domino or any other product -- by at least 200%.

Andrew Pollack
Developer, Admin, Network Admin, project manager, firefighter, consultant -- who does not live in someone elses 8x8 cube, but puts food on the table based solely on his own skills.

oh, and byte me.

10:51 PMlink  
Anonymous said...

No one should bother addressing the moron troll. People like this are numerous and just don't get it. We have a mix of really cool notes apps and others which are somewhat clunky. The cool one got their coolness from being "high-use" and evolved with time. The clunky stuff is most often low-use or near abandonment. Our culture has become accustomed to getting fast solutions which this product offers. We like the fact that entire projects can be farmed out to a single guy instead of separate people for data, ui, web, etc. Notes is a perfect fit for the "lean business" movement in that it offers quick solutions to complex problems. Abandoned applications are no biggie because investments in them are often small. Applications which "take off" are spruced up as needed and evolve into complex (cool) apps. Notes provides its customers a "darwin" of software - an animal most able to adapt quickly to changing business needs.
Back to the formula language - no one mentioned the that since the hide-when feature uses formula, it was dramatically improved - seems I get a lot of these working the first time now which is wonderful.

Mike dotFulbright atAlcoa dotcom

11:36 AMlink  
~C4Chaos said...

i've had experience using the "old" Lotus Notes Formula language. it was useful but it sucked. i remember coding it in notepad just to preserve the formatting.

your story is uber-inspiring. your winning attitude rocks. godspeed.

6:52 PMlink  
Anonymous said...

What a great story! Quite an impressive adventure.
Brings back memories!
I'll never forget calling the Lotus Support staff in '90-'91 (only 2 individuals, I believe one was Martin Cox) while working with beta of Notes Version 2. I recall how difficult it was trying to get that Formula language to actually do something that reminded me of real programming! At that time they became quite familiar with me. And then teaching myself C to use the Notes API. Thanks for the story!

- Jerry Gassie

12:37 PMlink  
Anonymous said...

Interesting story! It reinforces my belief that everyone should write a programming language at some point in their career.

It can be somewhat surprising how easy it is to write a Turing-complete language which can calculate anything that any other programming language can calculate. It's a remarkable feeling when you finally add some feature and realize that the language is suddenly Turing-complete. It's an incredible feeling of power, and the process of developing a programming language of your own makes you a better programmer, and helps you to understand other languages much more easily.

Great story. I had much of the same experience--being tasked, as a young programmer, to develop a new programming language, and I found it to be one of the most interesting and rewarding things I've ever done. Enough so that I've built my own programming language, Frink, just for fun. I wish that more people got the same opportunity!

--Alan Eliasen

6:29 PMlink  
Anonymous said...

I'll take any megalomaniac pompous blowhard who's willing to work on Database bugs! It's a good thing I didn't know about your goal to fix more bugs than anyone else on the team because I surely would have obliged you by assigning more! Seriously, thanks for the help in N/D 6. I was just treading water on that release and your work was much apprciated.

-Kimilee

12:05 PMlink  
Anonymous said...

Thanks for the article. Hey can you give us an insite on what ever happened to the formula debugger ?

2:34 PMlink  
Damien said...

I took out the old formula debugger, it was very crashy and was completely incompatible with the new engine. I wanted to write solid debugger but I worked on unread marks and database problems instead.

10:16 AMlink  
eknori said...

Hi and thanks for the article again.

Today I worked on one of my applications using @formulas.

Did you ever think about making "functions" possible in formula language.

This is what I'm thinking of.

I use the following code in my app

_SourceField:=MESSAGES;
_seperator:="=";
_ErrMsgHeader:="Language settings";
_ErrMsgText:="ERROR: " ;
_ErrNotFound:= " not found";

key := "msgTicketError";
_ConstErr:=
@If(@Trim(@Middle(_SourceField; key + _seperator ;";"))="";
@Return(@Prompt([Ok];_ErrMsgHeader;_ErrMsgText + key + _ErrNotFound ));
@Trim(@Middle(_SourceField; key + _seperator ;";")));
key := "msgTicketNotAvailable";
_ConstNotFoundInConfig:=
@If(@Trim(@Middle(_SourceField; key + _seperator ;";"))="";
@Return(@Prompt([Ok];_ErrMsgHeader;_ErrMsgText + key + _ErrNotFound ));
@Trim(@Middle(_SourceField; key + _seperator ;";")));
key := "msgTicketWarning";
_ConstWarning:=
@If(@Trim(@Middle(_SourceField; key + _seperator ;";"))="";
@Return(@Prompt([Ok];_ErrMsgHeader;_ErrMsgText + key + _ErrNotFound ));
@Trim(@Middle(_SourceField; key + _seperator ;";")));
key := "msgTicketUserNull";

...

As you can see, one block of code is repeating.

Wouldn't it be nice to have something like

_SourceField:=MESSAGES;
_seperator:="=";
_ErrMsgHeader:="Language settings";
_ErrMsgText:="ERROR: " ;
_ErrNotFound:= " not found";

@define(Macro$):=

[ @If(@Trim(@Middle(_SourceField; key + _seperator ;";"))="";
@Return(@Prompt([Ok];_ErrMsgHeader;_ErrMsgText + key + _ErrNotFound ));
@Trim(@Middle(_SourceField; key + _seperator ;";"))) ];


key := "msgTicketError";
_ConstErr:= Macro$;
key := "msgTicketNotAvailable";
_ConstNotFoundInConfig:=Macro$;
key := "msgTicketWarning";
_ConstWarning:=Macro$;
key := "msgTicketUserNull";
_ConstUser:=Macro$;
key := "msgTicketProbNull";
_ConstProblem:=Macro$;
key := "msgTicketProdNull";
_ConstProdukt:=Macro$;
key := "msgTicketParam";
_ConstParameter:=Macro$;

1:46 PMlink  
eknori said...

or even better:

_ConstErr:=Macro$(key);

;-)

1:56 PMlink  
Lionel Conforto said...

Hi Damien,

Thank you for the story. It was a two-times positive experience: first, your story was really interesting and exciting; second, I read all the comments from those people that I know and I understood that major members of the Notes family came to congratulate you. That's really a nice tribute to you.

Thank you again for your work and for the story !

BR, Lionel

12:53 PMlink  
Ken Pespisa said...

Great story Damien. It gives great insight into life at Iris. I found the story "impossible to stop reading" (what's the Web-oriented saying for the old cliche "the book was so good I couldn't put it down")

I remember meeting you at Lotusphere 2001 in the "Meet the Developers" area. You stood out in my mind because you were very enthusiastic about the formula language rewrite. I also remember you being excited for the potential of @Transform, something I've found pretty useful now that we've moved to R6. Anyway, it's cool to understand why you were so excited about the new features. You deserved to be. I had no idea how far the language nor your career had come.

Great work and good luck with the new venture!

3:54 PMlink  
anonymousname said...

I'm still a bigot after 11.5 years in tech sales at Lotus. Right now I'm exporting data in CSV format from a system, so that it can be loaded

10:41 AMlink  
anonymousname said...

Good one. After over 10 years at Lotus in support, tech sales and consulting, it was time to move on. I left in 2001. IBM killed the spirit. Now I'm using Domino to read CSV files exported from another system, because no one in my new company understands collaboration software, they use email for everything and their intranet web server software sucks. That's universal around the company. The relational system I'm exporting from does not have full text search, and I can't influence the team that runs it. [sigh] And with Domino evolving I wonder where the market share will be in a few years. Is it really a dead technology as these other respondents are suggesting?

10:47 AMlink  
Anonymous said...

Haa

It's bad girl!

Ma shlomekh?

Grand Cayman Real Estate
Used Car Dealer Dallas
Camera Dvr Security System
Attorney Civil Right Wyoming
American Banker Insurance Company Of Florida
Back Head In Pain Right Side
Concrete Balcony
California Bad Credit Mortgage
Agent California Estate Fortuna In Real
Virginia Bankruptcy Law
Use Car Prices
Pain In Lower Left Side Of Back

Ngeyavalilisa

6:24 AMlink  
Anonymous said...

Erratic Speech Lexapro

Lexapro And Sleeping Pills

Lexapro And Itching

Effects Good Lexapro Side

Difficult Form Thoughts Lexapro

Help For Lexapro Withdrawal

How Long Do Lexapro Side Effects Last

11:03 AMlink  

Post a Comment

<< Home