Betting on the wrong horse

I’m a software engineer. After decades of hanging around — and being one of these — computer geeks, I’ve come to realize a common trait among us: we’re relentless optimizers. Not just optimizers of code (to make it run faster or more efficiently), but of all manner of decisions in our lives.

We optimize the route we take to walk to (and through) the supermarket; in fact, I bet most of us can articulate an explicit algorithm for generating the route. We optimize our purchasing and lifestyle decisions. For example, which wallet we buy and how we organize its contents. We’ll do odd-seeming things like investing copious amounts of time identifying an optimal brand and model of footwear or clothing, and then buy multiple instances of it so that we can entirely remove the subject from our future consideration.

And of course, we optimize our technology choices: that is, the languages we write in, the frameworks we use, and the tools that we put into our software toolkit. "Optimizing" here means that we strive to make an "optimal" decision. That is, one which, after taking trade-offs and externalities into consideration, delivers the greatest benefit at the lowest cost.

Sometimes, even though we do our best to make the "right" decision (or at least, one which another rational person would endorse given the same information as a basis), there exists a time scale over which that once-optimal decision can start to look wrong. Maybe it made sense at the time, but new information comes to light or circumstances change, and we decide to revise our choices. (Incidentally, we optimize that process too, trying to calculate the point at which the cost of switching is sufficiently low compared to the benefit that we expect to obtain in doing so.)

When this happens, we somehow feel that we "bet on the wrong horse". Like the mentats in Frank Herbert’s Dune series, our decisions can only be as good as the quality of our input information permits, no matter how rational we may try to be. As such, there’s always an element of chance, of the unknown, that makes this most calculated of processes feel somewhat like wagering. Perhaps that’s why we agonize over the decisions for so long: because we know that they are bets, and we’re loathe to lose them.

A case study: Secret management

By the late 1990s I had started using random-ish, site-specific passwords for all of my internet accounts. This was before the days of Facebook Connect and the like, so every single website, forum, issue tracker and so on that I frequented had its own authentication scheme, requiring a username and password. I needed somewhere secure to store these, and a plain-text document wasn’t going to cut it. At the time, I stuck them in a FileMaker Pro database. This meant that I could easily sort and search them, and I could protect the database as a whole with a password of its own.

The end of the FileMaker era

While this worked, there was something about it that didn’t feel quite right. These were the halcyon days of the early internet, when MySQL, Apache and [the LAMP stack](https://en.wikipedia.org/wiki/LAMP_(software_bundle) were on the rise. Locking up my data in a non-free, non-open database felt like a recipe for the dreaded vendor lock-in. All of this in the context of the looming transition to OS X, which made me feel less than certain about the safety of relying on FileMaker as I switched from OS 9 to a drastically different operating system (even more drastic, arguably than the subsequent migration from PowerPC-powered OS X to OS X running on Intel processors). In the end, FileMaker ran just fine on OS X, but the damage was done. I was spooked.

The Keychain: into Apple’s loving embrace

These misgivings lead me in the early 2000s to switch to storing my secrets in the built-in Apple Keychain. I went with the Keychain because it provided me with encrypted storage without any fragile dependencies on third-party software. I knew it would work from day one on any subsequent OS upgrades. I knew it would always be free (or at least a free component within the operating system, although I needn’t have even worried about that because Apple eventually made that free too). Working with Apple APIs as an OS X software developer, and lurking on Apple security mailing lists and interacting with and observing their developers, gave me confidence in the quality of the security of the Keychain design and implementation.

As such, I was explicitly choosing vendor lock-in, but one that I felt relatively comfortable with because the vendor in question was Apple and I knew that I was going to be using an Apple OS for the foreseeable future (which, in the world of tech, means "For at least the next five years").

The Keychain had been around since the days of Mac OS 8.6. By the time I started using it in this way in OS X, it had a fairly nice management interface called "Keychain Access", which could be used to store, search, and view not only passwords but also arbitrary textual content as so-called "Secure Notes".

In hindsight, going with the Keychain was a good bet. It fulfilled security expectations over the years that I used it. The vendor lock-in effect was there as I’d known it would be, but it didn’t hurt me too much because I remained firmly and willingly entrenched in Apple’s ecosystem walled garden by choice.

Nevertheless, with each major operating system upgrade I would receive a reminder that any eventual move away from the Keychain would be a somewhat painful one. You see, I’d take these upgrades as opportunities to do a bit of spring cleaning, explicitly migrating across the Keychain items that I cared about, and leaving the others behind. This, because the Keychain is simultaneously user-managed (containing the items you explicitly add to it) and application-managed (containing items added programmatically via API calls made by running programs); it was this latter category of items that made the Keychain particularly prone to accumulating cruft: hundreds of application-generated items with unintelligible descriptions, among which it was hard to discern the essential items and which rapidly devolved into an unmanageable mess.

While the Keychain Access app made it trivial to open multiple keychains, moving items between them was a tedious process requiring you to re-enter your password and click an "Allow" button for each item (annoying, even if you temporarily set a blank password to expedite the process). This was nigh on unbearable. It made me glad that OS upgrades tended to come along only once every year or so.

What we’re talking about here is a "soft" form of vendor lock-in where you can get your data out if you really want to, but the software is light on "export" functionality (or devoid of it), so it’s going to be pretty darn unpleasant to do so. Still, it’s possible. Apple doesn’t provide a handy export-to-CSV menu item in Keychain Access, and I doubt they ever will, but it is possible to get your data out with a combination of tricks: Apple provides a security command line tool that can be used to get a raw dump of password items in the keychain, and I hacked together this secure-notes-exporter tool that can be used to get the secure notes out.

Famous LastPass words

What eventually pushed me over the edge was the desire to have seamless access to my credentials across my devices. For a long while I thought it was nuts to want to store your credentials in the cloud, even if they were encrypted (defense in depth and all that, and even if the crypto is secure, the implementation all too often is flawed). Nevertheless, as time passed, I found myself relying on my mobile device more and more, and the desire for universal access to credentials became increasingly pressing. At some point I decided that I wanted to trade a little bit of the security of the Keychain "local-only" model for a big chunk of sanity and convenience.

Apple did come out with iCloud Keychain in late 2013 for the purpose of seamless cross-device sharing, but I was wary of any cloud-based offering from Apple, given their atrocious track record in this department. I did take it for a spin, because I didn’t want to increase my exposure to third-party vendors if I could help it, but I found it to contain a narrow, insufficient subset of the range of features I could get if I switched to LastPass.

I spent a long time weighing up that particular move. If I was going to pick a third-party vendor I wanted this to be a once-off migration. The other main competitor I considered was 1Password from AgileBits. At that time, AgileBits didn’t make the cut because I saw it as:

• Made by a small company, and a relatively new one at that, so I wasn’t sure how long I could count on it to survive.
• Having a very weak cross-platform story at the time (and to this day, still much stronger support for Apple OSs than the alternatives).
• Having relatively expensive licensing, with new version upgrades tied to payments.

On the other hand, LastPass had a much larger user base (despite coming out in 2008, a couple of years after the first release of 1Password, in 2006), had a more compelling cross-platform proposition, and was based on a simple but reasonable subscription pricing model. Part of me wondered if the UI was hideously ugly because they’d dedicated the bulk of their engineering resources to security. Still, it was a close call. I naturally searched the web for reviews and opinions, and I asked for input from colleagues that I trusted: opinions were divided. In the end, I had to go with a gut choice, and my gut told me that LastPass was probably the right call. Later on the security wizards at work went through a selection process of their own and ended up choosing LastPass for the entire company, so I felt somewhat vindicated.

With the recent acquisition of LastPass by the much-maligned LogMeIn, I wondered if I’d made a mistake. I didn’t think anything drastic was likely to happen overnight (although I have had bad experiences in the past when a company whose products or services I used was acquired by another, like when Nettica was acquired by Dyn who promptly increased the prices by 3600%), but I knew that the odds of things remaining favorable had somewhat declined. The future suddenly looked less certain. Had I bet on the wrong horse?

1Password

Funnily enough, just as this happened, AgileBits announced a new "Teams" variant of their 1Password product, and the new functionality addresses some of my concerns that I’d had with it, such as the reliance on yet another third party (Dropbox) for synchronization and limited flexibility for sharing with others (specifically the within-family sharing use case). Additionally, in the intervening two years, AgileBits had chalked up a few more runs on the board, shown an ongoing track record of good customer service and polished UI, had no security scandals, and had bolstered its cross-platform support.

Another nice thing they’d done in the intervening period was bake in support for TOTP passwords, which would allow me to eliminate my dependency on Authy or Google Authenticator. Note that storing your TOTP second factor in the same place as you store your passwords is less secure than storing them on an entirely separate device, but there is still some benefit to using TOTP even in this weakened form, and I am already running around with both LastPass and Authy on my laptop and my phone, albeit with different passphrases.

I decided to take 1Password for Teams for a spin, and by and large have been pretty impressed by it for both the individual-user use case, and the family-sharing one. The UI is very polished indeed, making it a pleasure to use. I find that it fills out forms and detects sites and password changes at least as reliably as LastPass did. The search experience is much snappier than LastPass. The software seems fairly approachable to non-technical users (based on my observation of my wife using it, who had previously done manual secret management with the Apple Keychain and a set of not-quite-random-enough credentials).

The jury is out on the security trade-offs in the Teams design: the implementation is closed and proprietary just like LastPass’s was, although the details of the design are disclosed with some detail — at a level that I would describe as "computer-savvy but layperson with respect to cryptography" — in an in-progress white paper. The level of openness about the security considerations here is about on par with that of LastPass and Apple itself.

I’m decidedly not a cryptography layperson, but nor am I an expert. My sense is that the design is likely to be cryptographically sound, but the quality of the implementation is difficult to judge. I can only hope that the security aspects of the implementation are as carefully and thoroughly handled as the details of the UI and UX are.

There are some interesting trade-offs in there. They’ve very clearly tried to tread the balance between security and flexibility, and you can see these in terms of some levels of access being protected via strong cryptographic guarantees and others by server or even client enforcement of policy. For what it’s worth, I think most of these trade-offs are sensible enough, and none of them are a deal-breaker for my use case. One noteworthy security choice they’ve made is to eschew conventional two-factor authentication (eg. via TOTP) and instead use a separate "Account Key" in combination with the password in order to set up new devices. I think this is pretty novel — and I say that in a non-judgemental way as neither good nor bad, but rather something that I can’t see myself having come up with — and it’s a decision that I think I can live with. One nice thing about not using TOTP: it means that if I make 1Password itself the place where I keep all my TOTP secrets, then I don’t need to maintain a second, separate TOTP source such as Authy or Google Authenticator in order to set up 1Password (seeing as I wouldn’t be able to use 1Password to set up 1Password).

In terms of threat model, I’m not too worried about the consequences of adversaries getting into AgileBits’ servers and getting their hands on encrypted vault data. I think that sharing my credentials (and full administrative power over them) with my wife is actually a far larger increase of attack surface area than is letting AgileBits keep an encrypted copy of my stuff.

To LastPass’s credit, they provide very good export functionality. I was able to get everything out of LastPass (many hundreds of items) and into 1Password for the trial without any hiccups. Likewise, I was able to move data from the standard version of 1Password to the Teams version with ease. It is possible to export from 1Password, although I was pretty underwhelmed with the quality of the CSV-formatted data. I hope this improves, lest I find that 1Password is another "wrong horse" kind of scenario. At the moment, AgileBits is being tight-lipped about pricing of the Teams product once it gets out of beta (it is currently free), so I am hoping there won’t be any deal-breakers there; at the same time, I think it’s unlikely they would price it above the threshold I am prepared to pay for something as important as providing a secure home for my family’s digital assets.

The moral of the story

In this blog post I’ve told a story of how I made a series of technology choices over a lengthy time frame (almost 20-years, which if you convert to "dog years" gives you well over a century, which is about how long this feels on a technological time scale). At each point I endeavoured to make an "optimal" decision, one that would give me the most benefit at an appropriate price. I was wary of lock-in effects, and only regretfully allowed them into play as a calculated risk in the name of angling for the right cost-benefit trade-off.

Still, no matter how hard you try, over this kind of time scale, and sometimes even over much shorter ones, you’ll find yourself in the situation of having to revise, undo, or remake your decisions. This is the stuff of life in the internet age, I guess. Just keep on doing your best: things generally work out all right in the end.

Addendum: Good, bad, and ugly bets

In preparing this blog post I couldn’t help but look back over the years and reflect on some of the bets I’ve made. I’ve definitely had my fair share of doozies, some amusing prediction fails, and also a handful of really solid prognostications. Here follows a selection of illustrative examples:

The Ugly: Stuff that was really totally just wrong

• Wikitext: In the mid-2000s I bet on wikitext (used by thousands of editors on millions of Wikipedia pages) instead of Markdown, dismissing the latter as a hacked-together Perl script that wasn’t as powerful and was just one among a field of all-too-similar competitors (like Textile). I even wrote a super-fast wikitext translator (fun project, by the way) for use on my website. Now, I have thousands of wikitext documents, and the world has a flourishing Markdown ecosystem. I’ll eventually convert my old content, which — thankfully — can be done in an entirely automated fashion.

The Bad: Mistakes that probably didn’t hurt me too much

• Ruby: I like Ruby, and have since I first started using it in the mid-2000s, but I want to have a rational relationship with my programming language, not an affective one. I knew it was slow, I knew it wasn’t great for concurrency, but what I didn’t know was that the fast-and-loose, hipster group-thinking ecosystem would eventually drive me up the wall.
• Rails: I was able to build a lot of stuff fast with Rails, and if I were in a real hurry to prove a prototype concept at a startup I could (still) do worse than hiring a few Rails devs and bashing something out, but I stayed on the upgrade treadmill too long and ate an unpalatable amount of Rails omakase. When the JS ecosystem itself matured enough to catch up with and then eclipse the Ruby/Rails universe, I happily jumped ship and left my duck-typing, metaprogramming days behind me.
• Slate: I moved from the simple convenience of Divvy plus Stay for the infinite programmability of Slate. It promptly became abandonware, although it mostly continued to serve me well for several years after that. I eventually switched to Hammerspoon.
• GitHub: I initially rejected GitHub as "just gitweb (which is free, open source software) with pretty CSS", and figured most of the adoption was coming from besotted Rails fan-boys swooning at the sleek sans serif elegance of GitHub’s design. I considered the forking model chaotic and crazy. I totally underestimated the importance of the imminent network effects which eventually made GitHub the now de facto home for open source.
• Twitter: Another one that I offhandedly dismissed as inane and silly, once again underestimating the power of network effects and emergent behavior within a constrained system.
• Facebook: I considered it as superfluous (that’s what email is for, right?) and begrudgingly joined around 2007 to exchange content with people who didn’t share my point of view about email being the channel for network-mediated human communication. Now I work at Facebook, and use it intensively to connect with friends and family across the globe.

The Good: Bets that turned out well

I don’t always bet on the wrong horse. Some bets that turned out nicely:

• Mac: I bought my first Mac in 1996 (probably before it was actually a good idea) and have been buying Macs exclusively ever since.
• Git: Started using Git in 2007. A rare thing in the world of technology: a decision that looks as appropriate today as it did 8 years ago. (Still, on the way there I had stops at RCS, CVS, Subversion and SVK, most of which I wish I could have just skipped right over.)
• Vim: In 2009 I switched to Vim. It’s possible that within a year or two NeoVim will have progressed enough (and Vim will have stood almost-still long enough) that a switch will be in order, but I am not too worried about that because it should be a relatively low-cost move; the real win here was adopting the modal editing.

Discuss: Facebook - Hacker News - Reddit - Twitter