In this segment of To Be Continuous, Edith Harbaugh and Paul Biggar talk about the fear of shipping and if code is an asset.
Edith is CEO and cofounder of LaunchDarkly, a continuous delivery tool of “feature flags as a service.” LaunchDarkly powers software teams to quickly launch, measure and control their features.
Paul is the CEO and founder of CircleCI, a continuous integration and deployment platform that automates development workflows and IT operations, allowing developers to ship quality software significantly faster.
[soundcloud url=”https://api.soundcloud.com/tracks/229127881?secret_token=s-fslKp” params=”auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=false” width=”100%” height=”166″ iframe=”true” /]
Paul: Okay, so one of the things we wanted to talk about this week was the idea of shipping velocity and how it’s affected by team structure, sort of miscellaneous factors that aren’t part of continuous delivery.
Paul: So at the end of the last episode, Edith, you said, I think this was a quote from Yammer VP, “The organization you design is the software you built.”
Edith: Yeah, it was actually from the CTO Adam Pisoni and it really struck home from me because I see this at so many companies, they have a lot of engineers and they wonder why they have a lot of code but not a lot of product. And it ties back to what you just said of specializing the product management role.
Paul: Right. I think that this a name. I think it’s Conway’s Law and the way that I saw that expressed is that if you have, if you’re building a compiler and you have four different teams that are building a compiler, you’ll end up with a four-pass compiler.
Edith: There’s no one vision.
Paul: Right. When looking at lots of different teams and team structures, the interesting one that I found was the Heroku one.
And they have a language team and they have an add-ons team and they have sort of sharp delineations in their software, or in their stack, that allows them to really focus on one particular area because they’re such sharp demarcations between the different areas of the product.
Edith: I think that’s good if you’re a fairly mature product. I think in the early days of Heroku that would not have worked at all.
Paul: I wouldn’t say it’s quite from the early days, but relative to now it was quite early, I think they had that sin Cedar, which was around 2010.
Edith: I just more meant when you’re an early stage start-up, sometimes you change your entire product.
Paul: Well okay, yes, yes. I mean, absolutely. I think once you get into, once you get past the first stage of the product, and if you’re able to draw very good interfaces between how your customers understand what your product is.
Edith: I don’t know, I mean I’ve seen this go bad in so many organizations, where you have entrenched engineering organizations that care more about staying on their current project, then actually about where the market is going.
You know, like we’ve always worked on this, so we need to stay here because we don’t know anything else versus being able to evolve to where the market is going.
Paul: Right. This reminds me a little bit of something that I’m working on at the moment. We brought in some UX experts to look at our app and to help us sort of transform it into something that was a little more usable.
And they did a fantastic job and I spent this afternoon reading some of the reports. But what was difficult was understanding where the product needed to be. So for us in particular, there’s not enough focus on the deployment.
There’s a lot of focus on the build and there isn’t really sort of a broader look at what do engineers actually do when they’re trying to do continuous delivery. And so we ended up with what was in the product was redone in a really fantastic way, but there wasn’t much affordance made for here’s the thing that actually needs to be in the product.
And when you talk to customers, it’s hard for customers to tell you oh here’s the thing that you actually need to be or they look within the box you’ve drawn for them.
Edith: Yeah. I say this ’cause I had a similar evolution to you, I actually started off in engineering. And when I was in engineering it was very obvious what we should built next, extremely obvious. And so I always thought that our product manager was an idiot for not seeing as clearly as me. When I became a product manager, I realized how myopic I had been as an engineer.
Paul: Can you give an example of what was the next thing to build that in retrospect was wrong?
Edith: I would see all the little bug fixes that we should be doing instead of the next big features. Or not even the next big features, but the next big product.
Paul: I think big or small, or big picture versus small picture, is a good way to distinguish these two. I think that it’s very easy when you’re talking about product management to get the idea that a product manager knows everything and the engineer is just an implementer.
And I think this is where a lot of the resistance to product managers comes from with an engineering organization’s the idea that they’re going to be relegated to mirror kind of peasants in the…
Edith: Code monkeys.
Paul: Code monkeys, there we go.
Edith: You know, nobody wants to be a code monkey. That doesn’t sound very fun.
Paul: Right, right. I would disagree with that, but I think that’s–
Edith: Wait a second, we never disagree.
Paul: It is very frustrating trying to understand everything. And on the other hand, it’s very satisfying to ship things and to get your stuff in front to customers. So very often, the ability to just be a code monkey for a certain period of time is this sort of soothing feeling of just shipping software that fixes a lot of small problems.
I remember reading one of the famous GitHub guys, I don’t remember which one it was, but let’s assume it was Kyle Neath, that spends a lot of time on big projects. And in between the big projects, he needs to find what is the next big project to work on.
And it’s often very frustrating, or very you go down certain rabbit holes and whatever, and you end up kind of not shipping things, or you end up getting frustrated or whatever. And what he likes to do then, is just reach to the back log and just take a bunch of small fixes. And he spent like two weeks of just like implementing very small things.
You don’t need to think about it, and it’s cathartic and it lets you ship and so a couple of weeks as a code monkey I think is a very useful thing to sort of refresh the head and that sort of thing.
Edith: I agree, but I think nobody wants to do that full-time and I’ll also challenge something else you said, which is everybody wants to ship.
I think there are a lot of people who find shipping terrifying. and they’d rather keep holding stuff on until it’s perfect.. Like, I’ve certainly been with in situations like this where it’s like–
Paul: Right, where we can’t ship because it’s not perfect yet or it’s not complete.
Edith: Yeah, and as an engineer you have this real battle of well what if people want this, or what if they want this or this might be not quite right.
Paul: Yeah. The personal strategy that I use to manage that is to try to write the blog post that you’re going to launch this with and to, cause very often you’ll be like, “oh I can’t launch this cause it hasn’t got this feature, it hasn’t got this feature.”
And in the blog post, assuming that you’re going to tell people how to use it, or you’re writing the doc maybe, if not the blog post. You get that sort of feedback as you’re trying to explain to your user how to do this you’re going to say, all you need to do is this, and you’ll realize that this is seven steps long instead of one step long.
Edith: Yeah, the Amazon model. So at Amazon, they actually start with writing the press release first.
Paul: Okay, right.
Edith: And everything and that’s a really good guide back.
Because too often, people do the other end of they’ve built this gargantuna thing and they’re trying to write a press release or blog post and they’re like, woah, we’ve built all this stuff but there’s nothing actually to talk about.
Paul: Right. And part of that, and something which I think engineers have a difficult time thinking about is how to get this widely in use. So you can build the feature, but its no use having built it if no one uses it. So you need to build the bread crumbs, you need to figure out the ways that are subtly hinted that this is the feature that you want when it’s the feature that you want and to draw people’s attention to it.
And sometimes that’s putting it in docs and sometimes that’s doing a big announcement, but more often it’s trying to get the product in a place where the UX naturally implies the right path or the right direction for users to discover parts of the product.
Edith: Yeah, I mean the whole idea of responsive design, and I think even more, and this goes back to why I started LaunchDarkly, is you might have built it, but nobody might want to use it.
Like, you could have put all this effort into building it and done all these bread crumbs that nobody follows. So that’s the idea of you actually start doing the bread crumbs first and see if people start following that path.
Paul: So with LaunchDarkly, I’m guessing that the way that you see whether someone is using it is whether it’s enabled for them. Is that right?
Edith: Well no, so what we do is we allow people to turn on features for certain users.
Just turning on a feature for a certain user doesn’t necessarily mean that they start using it.
Paul: Right, exactly. So do you tie this to mixed panel usage? Or some sort of analytic stuff?
Edith: We could tie it to different back ends, like we tie it to New Relic, we tie it to actually optimize so you can see if people are even, and we have our own internal analytics.
Paul: Gotcha, okay. So this is the thing for me, that I started a project recently, and the first thing I did is built the dashboards for adoption. And we’re still at the stage in the project where there’s no adoption, or there’s tiny amount of adoption amongst early users.
A trickle. A trickle that you can’t even see on the graphs.
Edith: So it’s more like a fine mist.
Paul: A fine mist. But what you need to get is you need to get to the place where everyone is using this. ‘Cause if you just build it, they’re not going to come.
They need to be told about it, they need to understand how to use it, and getting those first customers to using it and where it’s deployed amongst them is it gives you incredible feedback about how one actually ships that software to the larger customer base.
Edith: Totally agree. I mean, this is classic Lean principles of just making sure some people can use it well before rolling it out further.
Paul: We discovered a part of the product that exactly three customers were using.
Edith: How did that make you feel?
Paul: Well, we didn’t actually know it was a feature. This was the idea that you could do deployments in parallel. So at CircleCI we paralyze your build and so the idea is that basically we take your test suite and split it across 20 or 50 or whatever machines.
But it turns out that that applied to deployment as well. And there were exactly three customers using that. And one of them had a valid use case for it. Out of thousands of customers, exactly one valid use case was there.
Edith: So what did you do with the feature?
Paul: We killed it.
Edith: Did you tell him?
Paul: I hope so. Yeah, I know. I think we reached out to that guy. There was another way for them to do it.
Edith:
The painful part of project management is also when you have a feature that you like to kill but that a subset of power users loves.
So like at TripIt, we’re a mobile travel itinerary but we let people do a printout. And one time we’re like, oh nobody prints anymore, let’s just kill it. Turns out that people print and they really really like printouts.
Paul: Right, right. I understand that, yeah.
Edith: Like particularly if you’re traveling to a foreign country.
Paul: Yeah, you’re not going to have internet or your phone’s going to be dead.
Edith: Or you want to show something to a passport guard.
Paul: Yeah, or a local. Without handing over your phone, like here’s what I’m doing.
Edith: Yeah, so they were furious with us.
Paul: Right. So, had you killed it at that stage?
Edith: Oh we killed it. Like we were just like oh, we didn’t have good analytics on people printing, so we just said oh nobody’s printing, let’s kill it.
Paul: Oh, wow wow.
Edith: So our analytics later was that people complained. Quite loudly.
Paul: And so had you properly killed it at that point, or had you merely disabled it to see if it went away?
Edith: Let’s see. We disabled it. I think we could get it back but people were really really upset.
Paul:
I like the thing of shipping something turned off, rather than actually deleting the code. Even though it’s incredibly cathartic to delete the code and to hide and it and remove it. But the turning something off with a feature flag is just a lot better way to sunset something.
Edith: Why do you think it’s cathartic?
Paul: Oh deleting code is amazing. It’s like my favorite thing to do.
Edith: It’s funny. My cofounder John, he was from Ex Atlassian, and he said the winner of their hack competition was always the person who deleted the most code.
Paul: Right, right. That makes perfect sense.
Edith: Cause that’s what they wanted a reward is tidiness.
Paul: Yep, yep. It’s very much related to the idea of product management and validating things and making sure that you don’t build too much of the product. Code is not an asset.
Code is an asset in the financial sense of it in that you think you want it but you actually don’t. You actually want the best performance with the least amount of code slash asset available.
Edith: Yeah it’s like, so a friend asked me once, should I pay my developer’s more if they write more lines of code? I was like, no. It was like no, that’s a really easy metric to gain.
Paul: Right. So we were talking about deleting features by feature flagging them. I think that this is an awesome way to delete a feature because it’s very, very easy to get back. It’s much easier to get back than a rollback.
Rollbacks are this thing that nobody wants to do because they’re very, very painful. Especially if a ton of code has come in between. So if you ship something, or if you delete something by literally putting a feature flag in around it, and then you ship the code and then it’s still on, and you turned it off for a certain amount of people, see if anyone complains, see that it still works and that you know, show it to ten people, and then you delete it for everyone just by flicking the flag.
And then if you get someone saying, you know, we really, really, really need print it, You can turn it back on for them while you have a think about how you really want to solve this problem.
Edith: Yeah, I think feature flags is a really misleading term.
So feature flags implies that it’s always on or off, when really it’s more of a feature control.
It’s a way to encapsulate a portion of functionality such that you have total control over it, from the sunrise of it, you know, from launching it to certain people, seeing their reaction, getting analytics, and then all the way to the end, as you said, every feature eventually you want to kill.
For more on Continuous Delivery and to find out about new episodes you can follow the show on Twitter at @continuouscast.
