In this special edition of the Web Setup, I spoke with Olivier Lacan who works on the team at Code School, teaching people around the world how to become web programmers through video lessons, challenges and more. I call it a special edition because Olivier has given has an unprecedented amount of detail that will give you insight into how modern developer-driven organizations are built & run. From desks & computers to each web service they use, Olivier provides color and background on why they've made the choices they have. I highly recommend taking the time to read this interview and hope you'll enjoy it as much as I did.
Hi Olivier, tell me about your team. What kind of office setup do you have?
Envy Labs as a whole is made up of about 40 people. Over half of those work full-time on Code School. Two years ago you only needed a few fingers to count the number of people who could work full-time on Code School, it's a little crazy how much we've grown.
Envy Labs has been promiscuous for a while when it comes to office space. We started inside of a co-working space called CoLab in downtown Orlando. We moved offices 4 times in 3 years since then. Since the beginning we've been in mostly open environments. Sometimes we've had developers in one large space, designers in a smaller one, with project managers and marketing folks scattered in smaller offices. But we like to shake things up, so for a while we had a mix of developers and designers grouped together by projects and now by Code School teams.
Our most recent move in August 2013 was the first time we signed a lease of more than a year and we actually built out a huge space in a new building in downtown Orlando. We crafted this space around how we sit and work: 7 rows of 3 people facing 3 people, next to floor to ceiling windows.
We finally ditched the Ikea desks and got wide (but shallow) desks with articulated monitor mounts for most of us. Ergotron mounts for people with 24" displays (various Dell LED models), Brettford MobilePro Desk Mount for the few fancy people with 27" Apple Cinema Displays, and really basic mounts with a hydraulic system to raise the screen (similar to office chairs) for unused desks.
Every desk has casters. We can reconfigure desks around groups of people working on the same project if needed. To be honest, we haven't done that much yet because power cables tie us down and not every desk have their own power strips. More recently, we bought a handful of GeekDesks which we setup in a pairing/standup area at the back of the office and some people liked it so much they got GeekDesks for their main desk.
What hardware do you use?
Almost everybody uses a Mac. There are a few nomads who prefer the Air. Those who want more pixels or performance go with the Retina MacBook Pro. A few people still hold on to their ancient MacBook Pros. I think we had one or two Windows users at some point because apparently Excel's Mac version isn't great, but I believe even they switched. There's also Thomas with his Alienware Linux laptop and double 24" vertical monitors. He spends a lot of time on Linux terminals doing SysOps work.
Most of us use the Apple wireless keyboard and Magic Mouse combo with Griffin raisers to hold our laptops. We have two Kinesis (a Cherry MX Brown and a Cherry MX Red) keyboard master typists and two hipsters with obnoxiously loud Das Keyboard clackity-clack factories. I recently ended up trying the Logitech Easy-Switch. I like it more than any of the mechanical keyboards I've tried so far, I might have an incurable disease.
And what software can your team not live without?
I could probably speak for most of us when saying that GitHub changed our lives. I don't think I've ever experienced such a low amount of friction with a tool like this. I'm guessing the fact that the GitHub team uses GitHub more than anybody else has a lot to do with it. That and their knack at reducing features to their essence make a serious difference. The amount of flexibility GitHub provides us to collaborate on software development, and software production in general is staggering. Our marketing and support teams use GitHub too.
After years of happy Campfire usage, we switched to HipChat about 8 months ago. HipChat won us over with useful features like integrated private messaging and email notifications for @mentions and private message while you're away. That means we don't have to swap back and forth between Flint (the best Campfire client ever, which I will sorely miss) and Messages. While HipChat has native Mac, Linux, iPhone, and Android support, the clients are far from perfect. The truly upsetting parts are their terrible search feature and atrocious emoji support. Over time though, we noticed something trouble. While Campfire was suffering from product design abandon, it seemed to be on bare life support, HipChat felt like it was being driven in weird directions. While there are obvious show-stopping bugs and important core features and details missing, all the progress we saw over almost year didn't address any of that. Instead they kept building brand new features of questionable usefulness like embedded audio & video chat. This might harsh but it was fertile ground for discontent; now a growing contingent of our team is looking at Slack with envy (pun intended).
The amount of thoughtfulness and attention to detail I have seen in Slack over just a few weeks of trial usage has been inspiring. I don't think I've seen a better built tool for teams in years. There's a very good chance we'll switch soon, despite the steep price increase compared to HipChat.
For communication we also use Basecamp, but since we've been so developer-centric in the past, we've had a hard time not duplicating things from GitHub. It's nice for broad discussions about the company or product direction, but we often struggle — at least with Code School projects — to find Basecamp's proper place in our workflow. Generally I'd say our Marketing and Education teams along with project managers are the heaviest Basecamp users. I'm a huge fan of their calendar however, which (with Fantastical for OS X & iOS) is the only proper modern implementation of a calendar: it's not a dumb copy of a physical calendar, it takes advantage of the dynamic nature of software by displaying the current week at the top of the page, not the beginning of the current month. It's crucial because any software calendar that begins with the first day of the month dooms itself to be at least 50% irrelevant half of each month — which is such a waste.
After switching from the nearly abandoned Airbrake (at the time) to the really good (and open source!) Sentry, we've finally settled on Bugsnag to monitor exceptions for Code School's main internal apps. The app is really great at surfacing important exceptional events, their impact on customers, and the relevant bits of information in order to quickly get the debugging process started. They also went through a really great redesign recently which now allows us to see which bugs were introduced or seen in the last deploy, or filter by bugs that were first seen recently or a long time ago. Again, for me it boils down to thoughtfulness. Despite equivalent features, the Bugsnag team nails the crucial little details which puts them above the fray.
After swinging between several solutions to track business events, we've settled down on KISSMetrics. It's one of the best event-centric business analytics tools we've used so far. I've spent a lot of time recently with my codeschool.com teammate Katie Delfin working on tracking discrete customer events all throughout codeschool.com and it's really nice how KISSmetrics sort of guides you into a few best practices with the way it defines default metrics and funnels for which it expects a few specific events. It not only helps our sales and marketing team do their work more effectively, it also helps us understand what customers really do and how our business is doing.
While we originally built out an internal library to abstract all metric and event gathering inside codeschool.com we recently threw in the towel and decided to pay the team at Segment.io to deal with that so we don't have to. We noticed that there are quite a few services that we integrated with and that their APIs kept evolving which required upkeep and time our (relatively small) team just didn't have. And occasionally our marketing team needed us to send data to a new service. Now we just have to flip a switch on Segment, and we can start pumping events into Librato as well. I really like how Segment allows you to send events to a specific service they integrate with, for instance if you want to send SysOp metrics to Librato and not pollute KISSmetrics with them if you're not gonna need them.
On the code side of things, my team relies heavily on Code Climate and Gemnasium. At a high level, Code Climate is a great tool to see the vitals of a large codebase and ensure that — generally speaking — code quality is going in the right direction. As of this post, Code School — a three year-old app — has a GPA of 3.29 on Code Climate with about 50,000 lines of code (20,000 logic, 30,000 tests). We already knew what our god objects were before using it, but it gives us multiple types of insight:
- The Trends allow us to identify classes with heavy churn (modifications) which allows us to ensure that we break things up into smaller classes to reduce the risk of introducing bugs in critical pieces of functionality. The correlation of churn & quality says a lot about how dangerous certain classes have become. When we need to refactor, these are the classes we usually target.
- Having Code Climate do a cursory check on common security flaws in the codebase is really useful because these issues are often very hard to notice when you're knee deep in code.
- The more recently introduced GitHub Pull Request integration makes it even easier to do code reviews to quickly see if a PR dramatically decreases code quality or security. Those are posted to our chatroom so there's also a subtle shame factor whenever you merge something in that dramatically decreases the perceived code quality of a few classes. It can encourage that sometimes beneficial last minute refactor.
On the Gemnasium side, they do the very hard work of tracking security releases for all of our dependencies, and notifying us when any one of them has become a danger in any of our apps. At our scale, it's not realistic anymore for us to manually track how up-to-date all of our production dependencies are and whether they have become liabilities. Gemnasium also makes it easier to upgrade dependencies by parsing the CHANGELOG files for many of the open source Ruby libraries we use. It gives us a diff of the relevant changes between the version of the library we use in production and the latest stable release. I haven't upgraded gems on a Rails app recently without using Gemnasium. That said, I truly wish the open source community would get its act together when it comes to changelogs. There almost as many formats and name conventions as there are projects on GitHub: History.md, changes.txt, CHANGELOG.md, etc. Worse, some maintainers just casually dump commit logs in their changelogs which provides absolutely no value to their users, especially when you try to evaluate whether it's safe to upgrade. I really want to setup a similar site to GitHub's choosealicense.com for CHANGELOGs, if anybody reading this is interested, please get in touch.
I can't remember when we first started using Travis CI Pro but I want to erase the memories of when we didn't. We have about 85% coverage on the app that powers codeschool.com but our most important features are covered by our specs and we have reams of unit tests to ensure everything at leasts works as intended individually. Our Git workflow is simple: master is always deployable, and unless you're doing a hotfix for a serious production bug, you're working on a branch from master. We regularly merge master back into long-running branch to keep them up-to-date and readily mergeable since Travis can only run builds on mergeable branches.
If a branch or pull request doesn't have any outstanding issues or to-dos associated with it, this means we can simply check Travis to see how the branch is doing with regards to our test suite. A great example of this was a long-running Rails 3.2 branch I've been working on. It was decoupled from our daily development efforts but I slowly kept it running and made failing specs pass one after the other. It's been green for a while now and I'm excited to deploy it soon.
I could go on and on so here's a quick list of apps we use pretty much every single day: Pingdom for basic uptime monitoring and to power status.codeschool.com, Gemfury to host all our internal gems, DNSimple for everything domain-related, Logentries to which we pump all our production and staging logs for analysis and debugging, MailChimp & Mandrill which we use for all mailers and transactional emails, Shopify which powers shop.codeschool.com, UserVoice which is very handy to let the truly most requested features bubble to the top, etc.
Last but not least, and I'm not sucking up because it's your blog, we like Meldium a LOT. When I joined the company all the passwords for apps we shared were stored inside of a Backpack file. Yeah.
I had to hold a gasp when someone first gave me access to them. I originally migrated all of us to Passpack. While they provided the security and granularity I was looking for, I was forced to create a screencast to explain to a team of professional software developers how to start using it. The user experience was just not there.
When it comes to security, user experience is the most important thing to nail down because if users find your security tedious — no matter how important that security is — they'll find a way to circumvent it. And they did. Our marketing team was using LastPass (which has very similar issues in my opinion) and after tinkering with the foolish idea of building my own solution (maintaining security-sensitive software is a full-time job) I briefly considered using GitHub's stillborn Swordfish app but the team features were non-existent.
That's when Meldium appeared. You solved many of the core issues I needed addressed. The design decision of not letting people you share credentials with actually see the passwords at all is brilliant. Beyond the simple security advantage, it enforces centralization. No one on the team can copy a password somewhere else, unless we specifically let them. That means no one will ever have an outdated password when (not if) we need to rotate passwords. It also means that I don't have to worry about someone leaving the team with a stash of passwords I then have to update on all the services we use. So many hours of worry and tedium we get back.
The actual implementation continually blows my mind. You're basically using a man-in-the-middle attack for good, if I don't betray my poor understanding of the concept. A less malignant analogy would be to liken it to the way telephone companies operated switchboards before individual telephone numbers connected people directly. You had to call the central and ask the operator (i.e. Meldium) to talk to your friend Frank McFakeperson in Ipsum, South Dakota (i.e. MailChimp) and the operator would patch you through to them seamlessly. Feel free to use that to make a kick-ass retro ad one day.
How does your team choose & adopt new applications?
Being software developers ourselves, and picky ones at that, we obviously pay close attention to design. By that I don't just mean aesthetics. Although a team that knows how to present its product inspires much more confidence than one that doesn't. That's especially true for business-criticial software that we're likely to invest a lot of money in.
By design, I also mean:
- How thoughtful is the tool?
- How careful are the considerations?
- How well does the tool solve the problem at hand?
- Am I jealous that I didn't build this myself?
Pricing is of course important. We ditched CoTweet when they switched to a ridiculous pricing model after a redesign where they asked us for thousands of dollars right off the bat. There's nothing more offensive to developers than a company that tries to force them to talk to a sales person on the phone. It's easy to understand, really. If you're a shoemaker who builds dress shoes and you go out to buy pricey boots, you expect to skip the pleasantries and the kind of sales pitch they use on regular customers.
And yet somehow, all around you see these business-to-business or "enterprise" services who expect you to get in touch with a sales person so they can hold your hand, give you a "personal demo" of the product, as if the product can't stand on its own merits. It's not just condescending, it also indicates lack of confidence in the product. I usually drive right past these services to find someone who tells me what problem their product solves, how they solve it in a way that is relevant to us, and how much money they want for it.
If they offer a free trial for me to play around with the product, it can be a great idea. That is, if they have spent the time to create a good "blank slate" experience, which useronboard.com provides tons of amazing examples of. If your tool requires a lot of data to become useful, then show me example data that allows me to see it working the way it eventually will. Many good services out there don't do that, which means they waste the advantage of their free trials, and I find my inbox full of cool products I've never really had a chance to evaluate because I don't have time to either hook them up to our apps or create test apps.
Mobile applications that allow you to see what the full-fledged web service offers while "on the go" are nice when tightly-bound to our core business (analytics, error reporting, business metrics). I wouldn't say it's an absolute deal-breaker otherwise: GitHub for instance is quite crippled on mobile and that's not too much of a problem.
With all these tools, how do you onboard new people and help them get productive on day 1?
We don't hire often enough to justify building bootstrapping scripts for our development environment, especially since it's not that hard to set up. Tools like Dropbox, Meldium, and GitHub make it very easy to get people ready for work on day 1. The only hurdles we have left to face are more specific to our production stack, which goes to show how much easier things have become.
On the information side of things, I built a tool called Orientation. It allows us to disseminate internal information by writing short articles on the simplest topics. It allows new people to do fuzzy searches in a similar way to Alfred and Spotlight. I plan to open-source it soon because I think centralized knowledge is one of the biggest bottlenecks of modern companies. It's far too common for people to explain the same thing over and over again to different people. We don't tolerate it for customer support (we end up building FAQs), I don't understand why we do for co-worker support. Orientation also allows anybody in the team to look up anybody else by face, by what they work on (not their title) or their name and email. It's nice when people are too shy to ask someone what they do or if they forget someone's name which starts to happen when you grow beyond 20 people.
Do people in your team work remotely? Any tricks/tools to help manage this?
Remote work isn't something we're doing right yet, but we're working on it. Just a year ago we only had one person on the team who was remote full-time, now I'm the second one (and I'm six hours ahead in Paris, France) and we have several other part-time remote people who don't live in Orlando and drive to the office only a few times a week. Many of us are remote part-time, on Tuesdays and Thursdays, so we can focus better by avoiding context-switching and commutes, taking care of our young children and/or pets, etc.
The problem is that since most people aren't completely disconnected from the office, we still struggle to be considerate to people who can't meet at the office to catch up on recent progress. I recently set up what I like to call our "Envy Deck," which is simply an iMac with a much better webcam than the default iSight and a highly sensitive omnidirectional microphone. Using a cool service called Appear.in it allows us to keep an open audio/video link between every remoter and the office. Appear.in uses the experimental WebRTC API which basically means everybody is connected peer-to-peer. The big draw over something like Dropcam is that it's a much better multi-way system. You can always see people's faces and they can speak back, so in way it allows them to be "present" at the office. We had it always on at first, but it's a little distracting and a bit creepy when you forget about, so now we just activate that spacetime portal whenever we feel like it.
Persistent public chatrooms with either Campfire, HipChat, Slack or IRC are a big deal too. It's very important for people who are remote to feel like they're in the loop, which means that if a conversation can happen on a discussion room instead of a physical meeting room, it should. We try to have as many chat rooms as we have projects in order to increase the signal-to-noise ratio.
Another great tool is ScreenHero. It has allowed me to remote pair program with Tyler, an Envy developer in Tennessee, on open source projects like rubygems.org; to have one-on-one conversations with John who needed to understand our product better when he joined our sales team; to have debugging sessions with new team members or front-end developers instead of wasting time chatting back and forth with very little context. ScreenHero makes it incredibly easy, even if someone uses tmux with Vim and the other Sublime Text or Atom. It's even great to use ScreenHero to avoid making someone in the office switch contexts when you need a sanity check, since it's always better to look at code than try to describe a weird problem you're having.
What would be your dream setup? What would you wish were easier or better?
I hope browser vendors dramatically improve the performance of WebRTC so it's not so CPU-intensive as it is today. It has gotten much better in recent builds of Chrome though. I would love for us to have multiple "decks" in the office where you can hop on to talk to someone who's remote or even pair with them that way.
I wish it were easier to ensure everybody on our team uses proper security. Meldium is a good first step but too few business service providers offer proper encryption for communications or two-factor authentication.
I've recently started using Librato to create the same kind of "information radiators" (large LED screens with actionable business metrics displayed on them) that I've heard the Square team uses throughout their office. It's already helped us identify abnormal activity for Code School and investigate it. I think aggregating and surfacing data about our business in a highly digestible way to encourage problem solving.
I truly wish secure communications for individuals and teams were easier. Keybase.io and GPGTools are interesting improvements that inject much-needed design thinking into the security and cryptography worlds. This frustrates me so much I recently gave a talk about it called Rational Security which does feature a bunch more tools like this.