19301709934-16429307

During Denver Startup Week, we hung out with some expert product folks. If you want to be a product manager or are wondering what it takes to be an amazing product manager — check out this panel for helpful insight.



Want more startup and product content? Subscribe to our newsletter.

Video Transcription

Jared: Hi everybody. We’re aibout to start. I just wanted to give everybody a few more minutes to funnel in, but let’s get this thing started. My name is Jared and I work at Galvanize. Jared, like the Galleria of Jewelry if you ever heard of that place. I just want to get a feel for who’s in the audience today. Who here is a product manager? Raise your hand. Who here wants to be a product manager?

Natty: By the way, this is not a hand raise. This is a hand raise.

Jared: Good one. If anybody would like to come up closer to the front, we’ve got a lot more seats left, by the way. So got a lot of seats up here, everybody who’s funneling into the back. Who here is a software developer? Raise your hand. Cool.

Who here has heard of Galvanize before? Raise your hand. Wow, that’s everybody. Are you aware that we’re in Galvanize? Raise your hand. Without any further ado, thank you for being here everybody. We’re going to kick this off. We’ll start out by introducing our panelists. Let’s start with Eric.

Eric: Microphones, yay. Oh, I can hear myself. That’s awful. I’m Eric Carlson. I’m Director of Product at Kapost, which is a small startup up in Boulder. Actually, not so small anymore. I guess probably around 90 people now. I have about 10 years of product management experience, a whole weird, wide range of things. Ultimately, my background’s in human factors, machine learning, and software development, back before I got into the PM game. That’s me.

Natty: Hello everyone. My name is Natty Zola. I’m the Managing Director for Techstars in Boulder. I’m not really in a product role right now, but I’ve been in product for eight years before this role, everything from two person companies to a hundred person companies, so I have a range of experience. My background is in finance, design, software, and startup-ey stuff. So that’s me. Thanks for coming.

Dan: Thanks. So my name is Dan Podsedly. I guess I’m the General Manager for Pivotal Tracker. Who here has heard of Tracker, by the way? That’s awesome. Okay, so I work for Pivotal. We started as a little consultancy in a little tiny room above the Gap in San Francisco, en route to what’s now, I guess, about a 1400 person company that does a bunch of things all related to modernizing how software is built. I used to be an engineer and then gradually became a PM, doing everything the hard way. And now I do a little bit more than just a PM, but that’s my most interesting part of my past.

Jared: My name’s Jared Polivka and I have a diverse background. And I’m really excited about the panelists that are here today because they have shaped my background, all three of you have, so thank you for being here. I moved from North Carolina in 2011, to Boulder, because I read “Do More Faster” and because I read about Everlater, which Natty founded. I was like, “Wow, that’s awesome. I want to get into startups and I want to build things.”

And then I ended up working with Eric at Kapost as a Product Manager. And we worked together very closely. I learned a lot from Eric and I learned a lot from Mike Lewis. And I have Dan to thank for the tool Pivotal Tracker, which is what we use. And I have many dreams about using Pivotal Tracker and doing acceptance criteria and testing. Thank you, Dan.

What does being a product manager mean to you?

Jared: Before Galvanize, I served as a PM with Eric, and before that I was a Marketing Manager at Uber. And I’m really glad that everybody’s here today. Thank you for attending this panel. I’m just going to kick it off with the first question. What does being a Product Manager mean to you?

Natty: I think this is actually maybe the hardest question that you’re asking today, because I think it’s a really hard thing to define. To me, the Product Manager is that person that’s in the middle of everything that’s going on. And by everything, I don’t mean the stuff that’s happening internally in the company, but also the things that are happening externally with the product and the customer base. So the product manager is really responsible for taking a lot of inputs and processing that, communicating what’s happening in the products to the rest of the stakeholders that are involved, both internal and external.

That doesn’t mean that they’re doing marketing copy, but it means that they’re informing marketing on what the user behavior is within in the product. So I think about it as that piece in the middle that’s making a lot of connections between all of the other functions and is really responsible for what is the product, what are we deciding to build, and really a deep understanding of what do the customers need. What are their needs and what do they want the product to be? So I think that’s probably my best definition of what a Product Manager is but I think it’s a difficult function to define.

Eric: Yeah, not to copy too much of that, but to me it means getting to do something I love every day. And that means also not doing the same thing every day. It’s the most multi-disciplinary role in a company, right? So you have to be comfortable on the technical side, on the user side, on the business side, and put all those together. Really, the intersection of those three is a brain teaser. Figuring out that solution that notches into those three just right, fits that bill, is ultimately very, very satisfying. So I love it and I evolved to it naturally, but being a Product Manager now means I made it and I get to do what I love.

Dan: Everything that they said. Just to reiterate, it’s such a broad range of skills and the things that you have to do every day just depends on what you encounter that day, right? And I think the important thing is being able to adapt and be nimble and go from this big picture, we’re going to go after that hill and here’s why, and then actually get into the details and into the weeds and find an incremental path and work so closely with your engineering team, represent the business, represent the users. You’re a communicator, but you’re also…I like to use military metaphors even though sometimes I get really strange reactions to that, but you’re like the lieutenant of the squad or whatever that is going after that hill.

And you’re just equipped, and you are driving it. You’re not necessarily doing all of the hands-on stuff, but you’re close to it and you can steer and you can encounter obstacles and find a way around it. But communication, I think, is the biggest piece of it. You are a communicator. You are channeling communication. You’re a decider or you are representing decisions, and you are communicating them.

Product Manager team composition

Jared: Speaking of your squad, what’s your team composition like?

Dan: Well, there’s the ideal, the actual, where we are, where we’re going, but I think the ideal team size…it also depends on what we’re really talking about a Product Manager and the Product Manager’s team. In maybe a medium to a large situation where you have a 50 person group that is building a product, you actually have multiple…you want to think of yourself as one team. You want to celebrate your wins as one team. You want to be on a mission together towards something, but you’re going to have multiple smaller groups of people.

So whether those are the teams or those are your pods or squads, whatever you call them, that’s a level of the hands-on Product Manager. So let’s say…well it might be called the Product Owner these days, right? There’s one PM who’s the vision, the very external but very collaborative with the internal group. The PM is working with potentially two or three or four Product Owners that represent and own specific areas. So putting myself in that position of the Product Owner, what is the ideal team? It’s probably four to six developers.

At Pivotal we actually think of them as pairs, because we’re all about pairing. So for us, it’s two to three pairs. So not necessarily a dedicated UX or product designer, but somebody available so there’s a full-time UX role there as part of these teams or these pods. And then the Product Owner. That’s the squad. And then surrounded by roles or there are other roles that are part of the mix ideally. You have an exploratory tester thinking across the whole system.

Maybe they’re better with the individual team, or some part, or they’re at the team level and they represent the wider view. But you want to have those roles. And you don’t want to be much bigger than that as the tactical, on the ground, going after the hill squad.

Eric: So at Kapost, we have been growing pretty quickly, and it recently brought up some interesting questions about team structure. Going from, I want to say, half a dozen developers up to 20 in the span of 6, 9 months was an interesting test of our product structure, how we’d set it up, and how we wanted to operate. The decision we faced was, do we bring on additional PMs and define the scope of each PM narrowly to a slice of the product?

So each Product Manager would own a narrow slice and having that narrow slice would let us both do the strategic things, but also work your way down to the tactical and really, really function as both Product Manager and Product Owner. Or do we separate those Product Manager and Product Owner roles? Have a Product Manager that handles more the strategic side and embed a Product Owner with the developers? I know there’s a lot of debate about that.

The danger of splitting the Product Manager role and the Product Owner role is that the Product Manager is the external facing, so you under the market, you’re spending a lot of time with users. You’re analyzing usage. You’re understanding the business cases. You really want that knowledge to be embedded in the product, so you want that person to be intimately connected to development. But on the practical side, what we found would happen is any time engineering needs came up that demanded the Product Manager’s time, they ultimately outweighed the strategic needs.

Strategic is so squishy that it was very easy to say, “Well, I’ll devote an extra couple of hours to engineering this week and then I’ll catch up on the strategic side next week.” And we would never catch up. So we ultimately opted to bring in more Product Owners, have Product Owners own a narrow slice, but then the ultimate management of the product and ownership of the product concept, at a higher level, spans too.

So we have, right now, two Product Managers split across four development teams. Each development team has a Product Owner that stays much closer to them. And then we try to overlap in the design space for a feature. That’s where we try to get some mind share established between the Product Manager and those external facing goals, and the Product Owner who needs to know them for the nuts and bolts decisions of development. So it works fairly well so far. I think that overlap is essential. If you have a clean break between the two it will break down. But I think it’s the right thing to do.

Dan: Just one more thing. Just to reiterate, that problem of how you slice your pods or teams within your product is the hardest thing ever. And what might make sense today, might not make sense a month from now depending on what you’re working on. If you go really vertical, that’s nice, because the POs and those teams can really just execute. They’re pretty isolated. They can go top to bottom.

Ideally, those are attached to certain isolatable or naturally separateable parts of the code base and the application. But then you end up with silos. And then when you do something that spans across them, it’s really hard. So you can also slice horizontally: front end, middle end, back end. But then every single feature that you do ends up spanning these things and then you’re having nothing but meetings and assumptions about dependencies. Now you have to have, whatever, GRR or something that lets you express [inaudible 00:12:11]. It’s crazy. So the ideal is somewhere between the two, and it also depends on the day.

Natty: How many people here work for a company that has fewer than 15 people in it? How many people work for a company that has over 100 people in it? Dang. Pretty even. How many people are in the middle, who didn’t raise their hand for the others? Okay. I wanted to make sure that we were tailoring the answer to who’s in the audience here and unfortunately, I don’t think there’s enough of a trend to have one answer. But I think everything that these gentlemen said was spot on.

The only thing I would add is that I think there’s been a shift and a valuable resource that, on the ideal product team is someone dedicated to analytics and quantitative assessment of what’s happening in the product, because that’s just so fundamental to understanding how it’s going. And, if you can afford it, someone who’s doing the qualitative side, so a customer researcher. We at MapQuest, where I ran consumer products previously, we didn’t have the budget to have those folks on every single product team, but we had them in the product team and they rotated around through the different squads.

Any squad was the Product Manager, 6 to 10 engineers, and a designer usually. We had this Qualitative Lead and a Quantitative Lead and those people were responsible for understanding the user experience and the user behaviors across all of the products, and then bringing that insight into each one of those teams. If you can afford that, that was the best money we ever spent because our products were so much better by having dedicated resources who were bringing that insight into the strategic conversation. So that’s the only thing I would add to the recommendations that they had on the ideal side.

On the realistic side, you’ve got what you’ve got and a Product Manager always has fewer resources than they need and so you have to balance that. And that’s where the comments earlier around having to wear many hats and be responsible for a lot of different functions. Usually the PM has to backfill when you’re missing one of those key skill sets. So you’re in there looking at Google Analytics or Omniture or Mixpanel. You’re out there talking with customers at the coffee shop and doing quick beta testing. You’re running wire frames by potential users. You’re the Product Manager.

If you don’t have role, that’s what you have to do. And so even if you’re not trained in that or skilled on it, just reading up on it, practicing it, going out there and not being afraid to test those things and dive in is a phenomenal skill to being a great Product Manager. So I think that probably, if I had to describe a trait of the best Product Managers I’ve seen, it’s a fearlessness, because you have to do all these things. You never have the resources you need, so just go do it. And if you’re wrong, you’re wrong.

Just make sure every cycle is a minimal as possible, and you’re getting maximum learning. So I think the reality is there’s this ideal team that very few companies can afford to have happen. In the real case, hire as many of those positions as you can and where you can’t, you, as the Product Manager, have to fill that role.

What’s the ideal iteration for agile development?

Jared: Natty, I know you’re a big proponent of agile. I guess I just wanted to ask you guys what is your flavor of agile and how do you structure your iterations in your flow? What’s your ideal iteration?

Natty: Yeah, that’s a great question. And I am passionate about it. I’m passionate about anything agile and lean. And there’s some folks in here who actually just went through Techstars and if you want to ask them about it, we preach lean theories all day every day, probably obnoxiously. But I actually did a talk about this at the last Denver Startup Week around how at MapQuest we structured our team. And so we had this playbook. And we had how we get work done, how we organize ourselves, and what we prioritize are three different things.

And I’m happy to go into any of those, but on a how we get work done, we did a modified scrum at MapQuest where it was basically scrum but modified for how our teams were set up and for how our executive structure wanted products delivered. But I highly encourage any lean methodology. What you’re trying to optimize for is the fastest iterations in the lowest amount of time to learning possible. You’re not optimizing for how fast you’re going to ship products, although usually that is an outcome of that. What you’re optimizing for in any lean process is fastest time to learning.

And that may mean you have to ship a product to learn, but it also may mean that you go out and talk to customers instead of shipping a product. So the thing that I always encourage Product Managers to optimize for is how fast can you make the learnings so that you make fewer mistakes and you start making the right product sooner? And that’s what mean and agile and scrum is all designed to do. So from a high level, any method or flavor of agile I recommend. If you’re not doing that, let’s talk afterwards and I can give you some quick tips on how to make the transition.

Eric: Yeah, we also work as agile to use. I’d be real surprised if anybody sat up here and said, “We’re waterfall.” But we work in agile on top of the normal process, so really most important thing I’ve learned doing agile is that there’s no one process that works for any team or even any feature. So there’s no canonical implementation of agile, or if there is it doesn’t exist in the real world anywhere. We have experimented a lot with this structure of how we lay out our iterations. We actually just moved from two week to one week iterations. We’ve also tried three weeks. That’s a bad idea. But it is very much worth experimenting with.

Even every few months try to throw a curveball in and see what works for your team. We found that one week iterations for the team we have right now and for the features we’re working on really work best. The other thing that I’ll mention, we adapted…we had been pointing in the same meeting where we were determining iteration goals. And that’s how I was used to doing things. We recently broke that apart. So prior to an iteration, we now meet up about halfway through the prior week to go through the feature and point.

And what that’s allowed us to do is bring the developers into the process, and if they point out things that, “Hey, have you thought of this? Are you doing that? Why are we doing it this way?” Spreading out, giving yourself some time between pointing and determining iteration goals allows you time to go back and to chew on it and to take that feedback into the process. One of the best things about agile is that it’s a team sport.

Everyone there has mind share for what you’re doing. The developers are involved in the concept. They understand why they’re able to chip in. So spreading those two things apart just allows them to give feedback better.

Jared: Can you raise your hand if you’re familiar with t-shirt sizing and pointing? I just want to make sure that we’re all using the same terminology. Eric, do you want to just give a brief…

Eric: Sure, absolutely. So in agile you have stories and then stories you are assigning points to. Now, points at the end of the day, what you’re interested in may be how long is going to take to develop these features? At Kapost anyway, points though are a proxy for complexity and I think that’s baked into the definition of agile, in fact. The idea is that it’s very difficult to look at a feature and say, “How long is this going to take?”

Anyone with a development background knows that’s hard. I was terrible at it when I was a developer. I was off by a factor of three to four usually. Timing is really hard to estimate. Complexity and unknowns are easier to estimate. And so you can look at each story and say, “How many points of complexity is that?” We use a three point scale. I’ve been trying to push my team to Fibonacci for a long time now, mostly for nerdy reasons, but I like a more granular pointing scale. The lead developer on one of the teams doesn’t, so we’re at a three point scale.

But ultimately, we’re assigning points to complexity and over time, as you develop the velocity for your team, you began to get an average for how that complexity translates into time. And that lets you estimate how long a feature is going to take to implement.

Dan: You said so many amazing things I have nothing to say. But just to maybe elaborate on that last point a little bit, it’s absolutely about complexity, but for us it’s even more about the conversation. So you’re not really trying to measure actual complexity or eventually map it to time. That’ll just all work itself out. You want to be consistent. That rhythm of estimating, and we do it weekly, it’s always a weekly thing.

Iterations are these…they’re just guideposts and they give you structure for some process like getting together in a room and talking about what you’re about to do, working through that, estimating as a team. And by the way, team sport really resonates. Lately, I’ve been weaving into every single conversation that I have with people around anything to do with this, team sport, team sport, team sport. Software is a team sport. So that’s huge.

But yeah, that weekly rhythm, the iteration, it’s more about just having a rhythm, getting into a room. These are the things…we’re on this mission, remember? So we don’t forget, this is our roadmap, just to reiterate. Every week, right? Then get into these are the stories we want to make our way through. Let’s estimate them as a team. And we’re doing that because it drives the conversation. Looking at something and thinking about complexity makes you start wondering about assumptions. Does it include, I don’t know, IE support? No, no, no. Okay, good, because I was going to say that’s an eight point story otherwise.

So you’re uncovering these things. You’re asking questions. You’re learning just through that conversation. And the PM, this is just this huge opportunity to refine, to reprioritize, to pull things out that might have been assumed to be part of [inaudible 00:22:04] story. And you do it as a team. You go around the room. You put your hand up, like one, two, three…one. And when you have everybody with their hand up saying one except one person in the corner saying five, that is a signal for some sort of an assumption that has been made differently.

Great, let’s uncover it. Let’s clarify the assumption. You learn something. Maybe you create a separate for the harder part that only one person thought about and you didn’t. Throw it on ice. Maybe it comes back into the backlog later. Maybe it’s less important. So that is such a key part of it and the rhythm is really just getting a room, having a retrospective at the end of the iteration. And the retrospective, in my mind, is one of the most important parts of being agile.

Agile means so many things to different people. It’s been repurposed to mean some bad things. People are like, “Well, we’re post-agile now.” And I’ve never understood what that meant. Post-agile? If you define agile to be you’re evolving, you’re nimble, you’re course correcting. What is the post of that? You’re no longer doing that? Regardless, the retrospective is where you identify what’s working, what’s not working, let some steam out, have a beer, and identify ways to course correct with whatever is not working and double down on things that are working. And I think that’s what really being agile is.

Eric: I just want to second the retro part. Retros are super important. I’ve worked at companies before where it was too big of a time investment, so when they were looking to get leaner that’s the part they tried to cut or did cut. And that really, it saps away the team identity, it takes away appreciation for the engineers’ work, and it allows less opportunity to do that discussion and have everyone gain that common understanding. Really try to make time for the retro. It’s well worth it. It’s a hard thing to quantify, just [inaudible 00:23:52] say, but it’s a super important part of the agile process.

How do you organize research for product planning?

Jared: So how do you organize your research for product planning? How do you go about that?

Eric: Yeah, so I’ve become a fan of something called dual track agile or dual track scrum recently. And really all that means is you make a roadmap for research as well. So dual track agile separates things into a development track, so these are things that are baked and ready to develop. It’s the execution track where you have things that can go in the icebox, developers can pick them up and work through them.

And then you have your research track, projects that are coming up. Maybe it aligns with the roadmap. Maybe you know you have to do these things or maybe they’re squishier, you just want to find out about something. You can lay out that research roadmap. Doing that allows you to explicitly plan for it, make time for it, and then apply the agile process to yourself, too. It’s not just a development tool. It’s something that you can take and apply to yourself to make sure you have time, to make sure you understand your own goals and deadlines. So I’m a big fan of that.

And then once you’re on that track I would also just emphasize sharing that with your team. I spend a lot of time talking to my teams. Sharing it with your team is important. Sharing it around the company is important. One of the things that gets lost between product…and even development teams, but also product and other departments in the company, tends to be the why.

You’ll tell people what you’re developing, here’s our roadmap, you can slap up on a monitor somewhere and point to it, but if you don’t describe why the roadmap is what it is, then they’re not getting the full value they need to have faith in your decisions and then to go forth and market and sell and do that sort of thing. So it’s really important to distill that research in a way that other people can get their hands on too.

Natty: I think research is the most powerful tool that any PM has in their pocket, because what I’ve seen is if you don’t do your research and you don’t have your feelers into what’s happening in the product and what your customers want, the other stakeholders are driving the roadmap and that’s the worst place to end up, because all of a sudden you are just executing what someone else envisions. And it’s unlikely that what someone else envisions is what your customers actually want.

So research is so important in your tool kit and so I think both on the qualitative and quantitative side, the more time you can spend researching your users, what are their patterns, what do they need, what do they want, then you can start turning that into the why you choose to build one feature over another feature. And then when you have pushback from stakeholders, which inevitably happens, you have unbiased, objective information around why you made the decision to go that way.

Or if you have to set the roadmap, you have to present that, you’re not going in there saying, “Well, I think users want this,” or, “I heard from one user who wants this.” You need that research to make the case to control your destiny and control your roadmap. At the end of the day, PMs are working for the users. You are the advocate for the users and what they need. So if you don’t know what they want and you’re not doing that research, and you’re going off of your hunch, the likelihood of you being wrong is just incredibly high.

And things change, so you have to always be out there getting the pulse and seeing what’s happening in your product. So to me, the research is that underlying foundation. It’s the most powerful tool you have in the tool kit, and it’s the biggest protector or armor that you have as a PM to control your destiny, especially if you’re in an environment…like if you raised your hand and you’re in a company of over 100 people, I’m sure you’re feeling this pain, where your roadmap is just getting whipped aside left and right from executive A who prefers this design or executive B who their son-in-law thinks this about the product and so they bring it in.

And all of sudden, you’re like, “What am I doing here?” That’s the worst place you can be. So do the research. And it’s super hard, but that’ why I said earlier if you can hire people to do qualitative and quantitative research for you, best hires. I would trade an engineer for a qualitative or a quantitative researcher in a heartbeat. So, my two cents.

Dan: Yeah, you don’t want to be the PM who basically just justifies why we’re doing something because you think it’s a good idea, but it’s something. A lot of PMs, they tend to be pretty smart, or they think they’re smart. I don’t know about myself, but I sometimes trust my intuition and judgment a little too much and I always know what the solution is and I know what we need to do, right? And I’ve learned over time that I don’t always know.

There’s a little bit of intuition and there’s a vision, but it should totally be backed by very concrete things. Even just to represent that in front of your team, you can’t represent it if it’s really just because I think it’s a good idea. Eventually, you just lose trust. People will not want to work on that team. It has to be deep. The other thing I want to say I think parallels what you’ve been talking about or what you were talking about with the dual track. We try to put it more under a design roadmap.

There is a roadmap, which is the, what problems are we trying to solve or what are the goals? We want to increase the appeal of our product to Product Managers as an example. So well obviously the next question there is why is it not appealing currently? Is it because they’re not aware of it? Is it because it doesn’t have certain features, it doesn’t solve certain problems? But we try to frame our design. So we have big picture design. Then we have more inlying design that goes along with we’re actually working on something, we have stories, and sometimes even our developers are paired with designers to flesh things out.

But the bigger picture design is more product design. And it starts with research. So that’s where I think there’s an opportunity for the Product Owner or PM to collaborate, really you spend a lot of time with the product designer. And again, there’s a whole other conversation or panel about what design is and the spectrum of that, but assuming design means product design, you are starting with framing the problem. You are getting into who are we really considering here? What type of people? Finding those people to talk to them and then when you have an existing product, then we have this luxury…Tracker is used by a lot of people, so we can pretty much go in any direction and run into somebody using it and cool.

So we’re considering PMs…Hell, this room right here would be really awesome for some product research so, Matt, if you can just close the door. We’re going to [inaudible 00:30:30]. No, but seriously. You find some way to access the people that you are trying to solve for, look at how they work, look at what they’re doing, look at what they’re running into. What is their problem? Perhaps just solving it today in some way, like PMs reporting progress to their stakeholders. Well, how are they doing that right now?

Let’s actually ask them for the specific things that they share with their stakeholders. Can we have that? Maybe you can, whatever, [inaudible 00:30:54] some stuff in there, but could we please have that because that really gets at the needs. And then we can start designing solutions for those needs. To me, research is somewhat tied and coupled with product design. It that’s the kind of design that you’re doing, I think that’s the right kind of design, that’s the right way to approach it. And then also, just having tools in place like Mixpanel.

You can actually find people that are doing things with your product. Just based on what Mixpanel tells you, you can narrow, you can scope it down to, “I want to find people in the city I’m in today that frequently use this particular feature or don’t, or use it in this particular way,” and narrow it down and reach out to them and say, “Hey…” You don’t want to be like, “Well, we’ve been watching.” But there is a…”We notice that you’ve been using our product heavily and,” most of the time you know what people are just based on LinkedIn, so it’s not that creepy to assume that you know why they are a Product Manager.

But use all these different tools to get to those people. Surveys are also useful. So maybe that’s the next question, is the execution on how you do the research, but I think it’s a mix. Talking to people should be number one and that’s part of the research, surveys and other things. And on that topic…

Jared: How did you know?

Dan: Right, so how do we understand our users? This is, again, as a Product Manager this is maybe your number one mission, to really understand them because you have to represent them and you can’t solve their problems and their needs without understanding them. Maybe this was the previous question. Without the research, there are people that can get away with just knowing. Like I know…it’s just that vision thing, we shouldn’t discount it.

Maybe you arrived at that vision basically because you were in that world or maybe you’re just amazingly smart and you can imagine all these things and you read a lot of things at night, but Elon Musk, for example, or Steve Jobs, they were the visionaries. They just knew. They just know. They’re brilliant. They’re not human. And we all have a little bit of that in us but not enough of it to just do it. So understanding users, again, you have to know what…break them down into personas. That’s an old thing but that really works.

So we want to make our product better for Product Managers. There’s an obvious persona there. Maybe that breaks down. There’s different types of Product Managers. There’s big, managing a 100 person group that’s building something massive, down to the CEO of a small startup that’s basically playing that role because there’s nobody else and they’re doing everything including product management. And that lets you maybe even prioritize, like, “Which one of these are we going after? Are we talking about the big PM or the small PM?”

Maybe give them names, put pictures on them. That helps to really just imagine and put yourself in their shoes. And then again, what are they trying to do? What are their needs? Which of those needs are we trying to solve for? And that segues into how can we solve these things? And then you can start testing some of these. And you know how you test, of course, depends on the situation, but that puts you in a good position. And again, going out there and talking to people just uncovers so much and shatters so many assumptions because you really just have no idea until you get in there.

Eric: So I think sometimes companies put too much emphasis on looking at usage data when they talk about research. You can A/B test. You can look at how people use your product, but ultimately that’s people drawing inside the lines that you’re giving them. And so it can only tell you so much. So personas are an excellent example of trying to break out of that and really understand your users and who they are more than just how they interact with your product.

The other thing I’ll add to that, and this is a bit of human factor as an early lead to my background getting out and looking at users’ contexts for using your products is very important. So are they using your products as one of 50 tabs while they’re doing 20 other things? Are they using your products while they’re walking from meeting to meeting? Are they using them on the road as they’re driving to work?

Looking at just the overall context and how people are considering, how they’re interacting not just with your product but what else is in their mind while they’re doing that can reveal some important things and lead you in research directions that you wouldn’t have expected otherwise. Personas tend to focus on who is this person, but expanding on that to what do they do, how do they work? Making it pretty broad can be very useful. Now you end up with this book of a fictional person, which no one ever reads, so that’s not useful so don’t go overboard with it. But definitely looking beyond stats and A/B tests is very valuable.

Natty: Yeah, the way I…I have another dimension of that. And the way that we use personas, and I highly recommend them, but a different way to think about personas that I’ve found more effective than the demographic personas, which are most common and you read 90% of the literature on personas they are demographic personas, which is female, age 35 to 48, median income. That’s actually really not that helpful. What’s better are situational personas.

And I highly recommend you change that…or not change that, because I think both are valuable, but spend some time thinking about what are the…Instead of defining personas by demographic, define it by situation. So I’ll give you an example. When we were at MapQuest, everyone…well, I shouldn’t say everyone, a lot of older people use MapQuest. Younger people are starting to adopt it again. I highly recommend it. It’s awesome.

But what we found was when we did personas, it was 75% of the U.S. qualified in our three personas. That’s not really that actionable. So we took new personas. How about the persona which is the person who lands at the airport for the first time in a new city on a business trip? I can actually develop products for that person much easier than I can develop products for the 50 to 60 year old male from the South. I know when that person gets to the airport, what do they want to see in a mapping application? What are their concerns? What are their anxieties? Where do they most want to go?

So should I put up a, “Hey, want to rent a car?” button that shows up when I see you’re at an airport? Do I want to put a, “Here’s some quick eats before you get on your flight,” button when I see that they just went to a rental car place and then went into the terminal? So those, if you change your personas to what are the situational personas that you want to appeal to, it can usually give you a little bit more actionable strategies in your product development. You still probably want to tie those to actual demographic personas, but I think it’s a little…it’s a nuance. It’s a little bit more powerful.

And then the second thing that I highly recommend are customer journey maps. And there’s a ton of literature on those, but I think it’s the best way to understand the experience with your product, because the reality is the experience with your product is not just the usage of your product. It’s the whole experience around it. When did they realize they had a problem? What’s the research they did to find a solution? What are the criteria that they had in their head to make a decision around which tool they wanted to use to solve that problem? Their experience with your product, do they refer your product to other people? Do they cancel your product?

You’ve got to understand that whole ecosystem and the best framework I know to do that is customer journey mapping, so I would Google that, look it up. There’s a ton of great free literature on it. If you find anything from the Stanford D school, that’s the best stuff that I found.

Qualitative Research Resources and Recommendations

Jared: So we’re focusing a lot of qualitative research with this, and especially, these are all qualitative techniques. Do you guys have any resources or recommendations for the audience? Just the simple act of being the detective and asking questions, there’s a lot on just how to formulate your questions and how to ask good questions. Do you have any resources that you’d like to share with the audience, just for some of the people who want to be PMs or want to refine their process? Do you have any good place to start?

Eric: So if you look in the formal field of human factors, there’s cognitive work analysis and cognitive task analysis. If you look those up, those provide really good places to start. These are formal breakdowns of how people behave in context, so while they’re doing a task. The differences are minor between the two. Once tries to posit internal states and actually figure out what the person’s thinking, and the other says it doesn’t matter.

It only matters how they behave. But those fields are good things and Wikipedia articles are great to start on both. And then, to his point, demographics is so easy everyone knows how to do it, but look up ethnographic surveying. You want your personas to be a lot more ethnographic than they are demographic. Googleable terms.

Natty: I don’t think there’s a…I don’t know of a silver bullet for how to ask good questions or a great resource on that. I think you get better at asking questions the more you do it, and the more you immerse yourself with your users and get to know them you’ll figure out what are the important questions to ask.

Dan: Just one thing that we found useful recently when talking to people is you obviously want to get at the needs and take biases out of it, but sometimes just asking, “Well how would you solve this?” “How would you like to solve,” and it reframes the thinking a little bit. And you don’t want to jump right into that. You really want to get at the needs beyond the, “I just want this button.” But there is a progression toward that and getting people to start thinking constructively about your product. It puts them in a different position and they start thinking about it differently.

Eric: So one more thing on that topic. My cheat sheet when I’m sitting in front of a user asking questions and trying to figure out how they work, they’re usually by the areas that I try to think on. And I actually write them down in my notebook, so as we’re talking I remember to try to think from those perspectives. So I’ll look at their objectives, what are they trying to accomplish? I’ll look at their tasks, how are they breaking down progress toward those objectives into discreet chunks of work? What are their methods for accomplishing those tasks? What artifacts in the environment come out of that? So do they have Post-it notes plastered all over the place? Do they write on their hand? Do they just memorize everything?

So what artifacts in the environment. And then the last is the environment, so document the context inside of which they’re working. Just having those five things sitting in front of you while you’re interviewing users or maybe, if you’re lucky enough, you could shadow your users for a day and just watch how they work. Those are the things to keep in mind and really focus on.

Hypothesis Creation and Testing for Product Managers

Jared: How do you approach hypothesis creation and testing? So you always have assumptions. Sometimes you test it with research and your research will shatter your assumptions. Then sometimes it seems like the only way to get feedback is to build something. So how do you go about creating hypothesis and testing it as a PM?

Eric: General methods? Yeah, that’s hard. The one thing that jumps to mind there is definitely use hypotheses. I remember learning about the scientific method in third grade or something and thinking, “Why do you need to take a guess before you learn anything? You just look and then you say what you found.” But there’s no such thing as looking in an unbiased way. There’s no such thing as a neutral observation or if you do that, you’re not going to get anywhere you want to be.

Hypotheses are really there so that you can structure your observations, so that you know who to talk to and what to look at. Beyond that, I’m afraid I don’t know of any general way besides doing it a whole bunch and trying, trying, trying until you find a method that works for you.

Natty: Yeah, I think you hit on a really powerful thing, which is it’s so easy to just go out and do a test and see the learning. Yep, that’s what I thought was going to happen. But the way our brains work, you’re actually unable to recollect what you actually thought was going to happen beforehand if you didn’t write it down and put it somewhere else. And if you’re not comparing what you thought was going to happen with what happened, even if it’s accurate, you’re not learning. You’re just not learning.

And one personal motto I have is no theory, no learning. So if you don’t have a theory about something, you’re unable to learn about it. So it’s not only right. It’s about learning. So just do it. Just write it down. Have a hypothesis. I know it’s spot on.

Dan: Yeah, this is a big topic and it’s just such a range here. And it also depends on what you are focused on. You’re building a brand new product. It’s probably a lot easier…and I should say I think it’s…no matter what, it’s good to frame the problems with some sort of hypothesis. We think this will solve this and here’s how we’re going to know that it solved it. And so ideally there’s some metrics around everything. It’s not always conclusive.

You say, “Well, we want at least a thousand person using this every day or this many times,” maybe do the frequency or engagement or whatever. But it’s never black and white as to what is successful enough. It’s like, “Well, it’s 23% of our users. Not that high.” But it’s also which users are they? Oh, these are the really important ones, so actually it’s more important. So anyway, framing it in a way that gets us close to being able to measure is ideal. But then in terms of how you test, it depends on what it is.

In our case, and we’ve been very guilty of doing really big things for a long time before getting enough elevation it’s actually going to work, because you get into this…and some of it’s real. You can’t just give…put something out there that’s completely simplistic. Obviously, there’s a lot of tests you can do just manually like mock something up. Mock up a couple things and walk through the people and say, “This is going to really address your needs.” And you can gauge based on their reactions or you can get into the concrete things, and you start to feel better about the testing.

But ultimately, you might have to build something. So for a new app, probably even several simple things or even landing pages that are like, “Oh, I want to do that.” Now you’re measuring clicks toward that. You’re testing people’s interest in it or some sort of action around it, but you haven’t built it. When you’ve got maybe a mature product, it’s a little hard to do that because the solutions might need to be a little bit more involved and if you put something out there that’s really minimal, it’s probably not enough. It might give you false signals. It might give you false negatives because you haven’t connected the pieces.

So you’ve got to connect the pieces enough, but then there’s this danger of, “We’re going to work for six months and build this amazing feature that’s going to have falcon wing doors and they’re going to love it. It’s going to be great.” And then you launch it and you get 17% or whatever in Mixpanel. And then it’s like, “Well, is it because the pieces are connected but they’re not connected enough yet? There are not enough pieces there?” It’s not discoverable. People just don’t like it. It’s not solving their needs. Then you’ve got to really dig into it and start asking why. Why is it only 17%? Okay, well why that?

And then maybe you get into some of the multiple why experiments that ultimately you might have to go talk to people. Why is this not working for you? So then there is going to be other things you have to consider.

How to Build Data Collection into Your Product Process

Jared: So we covered qualitative research and hypothesis testing. In an ideal world, if we were at Airbnb let’s say, every team has a data scientist on the team. That’s amazing. So how do you build data collection into your product process and into your product and how do you decide what to collect? There’s so much data that you could collect on users and their behavior. How do you navigate that process?

Dan: Well, if you’re the NSA, it’s very simple. You just collect it all. But it is actually…it’s not an unreasonable starting point, especially if you’re building something new. If you’ve got an existing app that has a million touchpoints, maybe it’s a little harder to retrofit. But these days there’s such good tools for this. We’re putting [inaudible 00:47:10] Mixpanel because it’s just really easy to hook up.

Regardless of what kind of an app it is, web or mobile, it’s easy to hook up and then you can start instrumenting pretty much everything. There is a downside. You start building up so much data and then your Mixpanel monthly bill gets huge, but kudos to them. They built a great tool and we’re super happy to give them a ton of money every month because we have such good access and such good visibility to what people are doing. And you can slice it arbitrary ways. You can build funnels in it. You can segment. You can look at what people are doing live.

It’s a little creepy because you literally see people’s faces. Oh, this person just clicked this button, and then they did this. And this is where they are right now. Okay, maybe I shouldn’t be saying that. I’m going to…You’re like, “I’m not using Tracker anymore because it’s spying on me.” But these days that’s just what you have to do. Obviously, it’s done with the intent of having this awareness that allows you to make your product better.

So I think everybody is pretty conscious of using this data in a very sensitive way and we certainly do, but the cool thing is that you can instrument pretty much anything and they have great tools that collect this data. And they actually make it really easy to understand and to analyze and go deep, and just to get a pulse of what’s going on with the application. Without that visibility, you’re just blind. You’re just super blind.

Eric: So we use Mixpanel as well, and it’s led to an interesting problem, which I don’t know of a good solution, so if anyone knows one out there and wants to tell me after that’d be great. But when we develop a feature we come up with a laundry list of things we want to track about that. You build in, Mixpanel’s great because you can do it really quickly with little effort. When we start getting that data, we can look at it, and Mixpanel has turned into a tool where, for PMs, you go in to check one thing and then four hours later it’s like going to TV Tropes on the internet, just disappear into…So the data there is great.

There’s a ton of data. The piece that I’m missing, this is a problem of having too good of data, is I can set up funnels to look for patterns that I think of. I can look for things that confirm what I already know to look for. What I’m missing is a way to find a fingerprint in data that means something, so some common pattern that’s meaningful, and have something tell me that it’s something to look out for. So I’m really, right now, on the hunt for a good patter analysis tool that I can plug in to Mixpanel data so that I can start looking at those common patterns and know, and then go out in the field, talk to users, and figure out what that means. If anyone knows one, please tell me.

Natty: I think all the tools have their pros and cons, so instead of focusing on tools, my recommendation is start with the hypothesis and make sure you’re testing and you can get results back from that hypothesis. And that’s how you can start figuring out what you’re going to track and collect. And then I sort of think about it the same way that engineers in here probably think about testing, which is if you goal aim for 100% test coverage, all you’re doing is writing tests.

So if you aim for 100% data coverage, tracking everything beyond the basic install, you’re going to spend your whole time analyzing data and not actually finding insights or taking action on them. So I think about it as test all your hypotheses, build tracker on all your hypotheses, get the high level metrics and data that comes from free from almost any application that you install, and then just like in testing, you write tests for the high risk areas of your product, and write tests when a bug happens so that it doesn’t happen again.

So I would take the same approach with when you tag something to track it is, is it high value? Is there high value? Tag it. If something breaks because of something or there’s a risk point, tag that as well. So that’s how I would start and hopefully that’s a practical way to not spend your whole time tagging and tracking in whatever tool you choose.

Dan: Just to tie back to something we were discussing earlier. Having that broad coverage of instrumentation, say, with Mixpanel. Again, you have to know what you’re looking for. You have to have some hypotheses when you go in there. Or if you do a data science project on your usage data, that’s not just like, “Well data science, tell me some things I don’t know. Just tell me what I should do or what should we build.” That would be pretty awesome, right? It just looks for patterns and makes recommendations. That would be amazing.

Maybe one day we’ll get there. So you have to really [inaudible 00:51:46]. You have to know your world and your users enough to make these hypotheses or guesses and then you go validate them and verify them with the data. But the other thing you get here is you never know what you might want to dive into. Six months later, we’re considering improving something around how stories get deleted. So great, now we have this data. We can dig in and see how often this certain problem happens and who is it happening to? Maybe there is something interesting there.

And then it also lets you actually reach out to those people and say, “You know, we’ve noticed that you delete stories a lot. Why? Are you doing it on purpose?” Again, you don’t want to be creepy about it, like we just show up like, “Here we are.” But you can. Mixpanel actually lets you do that. You can literally just target people who delete stories more than 10 times a day and something and something else and do it in the morning, not in the afternoon. And push a button and a little notification or a little thing pops up with your face on it. “Hi, I’m Dan. I’d like to know why you’re deleting so many stories.” You could do that so easily. Obviously, you have to be a little bit conscious of how you exercise this power, but it’s very powerful.

How To Prioritize Your Product Roadmap

Jared: There’s a lot of literature on this topic, but how do you prioritize your roadmap and how do you decide what to build?

Dan: And I’m the lucky one with the microphone who gets to go first on this one. This is hard. This is also just like a…this is who you are as a product manager. You have to be thinking about this all the time. You have to be talking to people. You have to be getting inputs from various stakeholders, maybe a little bit of…you just have to pick a direction and go, but tie it to concrete things. Like having a mix of strategic things on the roadmap and tactical, so you can keep shipping some things while you work towards something bigger, representing it in some form.

But I think keeping it lightweight is important, because you can get really into like, “We’re going to build this elaborate roadmap,” and then what is that? That’s like a Gantt chart and Microsoft Project with all these dependency assumptions and it gets all complex. And then three weeks later, your world has just changed. You’ve learned some things, whatever stuff happened. Oh my God. How are we going to modify this roadmap? So what we do is we keep the roadmap in basically just a Google spreadsheet and it’s simple.

And we call these things big epics or portfolio epics, whatever. These represent the…again, not necessarily features. Sometimes they are bigger features but they’re more like problems to solve. And this is your roadmap and it’s just about prioritizing it and making it visible and clear. And this is your…when you get into the weeds and you’re like, “Oh, man. All this stuff happened and this bug’s here and somebody just said something as we need to respond to it so we better do this. You can just lose sight of what is your path, and so the roadmap represents your path.

And it is an evolving thing and you have to be able to evolve it as you go, but it’s more just to capture the mission, make it visible. That is the hill that we’re going toward and these are the big steps along the way. But all the things that go into prioritizing it, there are many, and this is…I don’t think there’s something very specific like you do this and you do this and then you have a roadmap. And I struggle with this, because this is hard. There’s so many things you could do. There’s things you have to test.

There’s just some things you have to do because you’re now with this big company and you have to move your, whatever, recurring billing from this thing to that thing to that thing. So I don t want to do it. It’s got to be in the roadmap. We want to do these other things for our users. And it’s balancing all these things. You’re the balancer and good luck.

Eric: So roadmap planning is really hard because oftentimes you’re comparing apples and oranges. You want one feature for this reason and this other feature for this other reason. The way I approach the problem is you don’t prioritize features, you prioritize objectives. So what are you trying to accomplish right now? Whether those are high-level business or maybe one step below.

So let’s say your objectives at the company level are retention, so user retention, or sales velocity. You can ascribe a priority to those, or maybe just a percentage priority where you want to spend 70% of your resources improving retention and 30% against velocity. Once you either prioritize or break up between those, then you can ascribe features to those objectives. So how well will this feature that you have in mind, how well will it server retention or how well will it serve velocity? It gives you a common denominator or a common unit along which to compare features, and then you can fairly easily compare apples and apples.

So you come up with some hypothesis of this feature will serve to improve retention by X, Y, and Z. This other feature will serve by A, B, C. I have some computational modeling in my background, so I tend to get quantitative with it then. Jared, I subjected to my quantitative model spreadsheet where you actually, as a team, we came up with a model of what we thought was driving this certain business patterns that we were seeing.

And then we could input features into that model and try and quantify the benefit of features against the objective, prioritized objectives. And it really makes coming up with one cohesive roadmap a lot easier, because now you have a common unit on which to compare objectives.

Jared: I love that model.

Natty: I totally agree with what they said, so I’ll take a different track and give some separate feedback, which is I think this is your goal also, which is it should not be assumed that you will get to prioritize your roadmap as a PM. And it goes back to my earlier comments around research being a really critical tool for you. Again, especially those companies that are larger than a startup, you know there are a lot of people having an influence and prioritizing your roadmap for you.

And your goal should be to establish your credibility, get the data, deliver results so that you can prioritize and you’re given the trust to prioritize that roadmap. So I think first and foremost, this is your goal is to have this problem. And to these guys’ point, it is incredibly difficult to prioritize this stuff. The one other thing I would add is that I love a quote that Hemingway said, which he said…he sent someone a letter and at the end he said, “I would have sent you a shorter letter if I had enough time.”

And I think that’s an interesting way to think about product strategy also, which is it’s not about more. It’s about quality. And don’t go down the rabbit hole of trying to add feature after feature after feature because some users asked for it. Find out what really matters and make that part great, and spend a ton of time making that part great, even if people are on the periphery asking for things around it. It’s rare that those things will actually make a difference. And so I tend to focus on quality over quantity of features. And I love the perspective of going for it by objective as opposed to feature. So maybe a different way to answer that question.

How do Product Managers decide when to remove a feature?

Jared: So how do you decide when to remove a feature?

Natty: I think it’s something we should all do way more often. It’s hard to dedicate the time to taking something out, especially when you have other stakeholders. They want to see progress, moving forward, new stuff, additions, more, more, more, more. But actually, again, going back to my other comment, I started to think less is more and so we should be doing this a lot more often. I don’t know that there’s a black and white answer on this. You can look at the data. You can look at your objective as a company.

You can look at who your users are. As those things are changing and evolving and you’re refining them, there’s probably parts of your product that are less relevant and fewer people are using them. And it’s great to trim the tree if you can. I think it’s hard to get the approval and buy-in to spend the time doing it, but it’s really worthwhile, because all that stuff hangs around. It has to be cleaned up eventually, and the further you go, the bigger the debt.

Jared: I should rephrase this question as how do you decide when to remove it or re-implement or redesign that same feature?

Eric: Would you like to reconsider your answer?

Natty: No.

Eric: This is a little bit…well preview would be overstating but it’s a side benefit of looking at objectives when you’re planning features. You’re implementing a feature to accomplish a goal, a goal for your business. You know why you’re putting this thing out there in the world. If you thought ahead about that, it gives you a framework under which to analyze is that doing what we wanted it to do?

Maybe people are using it for something completely unrelated that you realize is useful, too. But often than not, it’s not going to be. Unless you’re Twitter, no one’s finding a use for features that you just randomly throw out in the world. So for the most part, you can look at that feature and say, “Why did we build this? Is it accomplishing that thing?” And then the decision whether to retool it or to just destroy it is a qualitative one that’s very much feature by feature. A lot of that comes down to cost benefit analysis and looking at what you think it would take to retool it versus what the realistic likelihood is of getting that benefit out of there.

That’s also where user testing comes in. If the decision between retooling and destroying something, A/B testing is very popular right now. We’ll implement two ways. We’ll do twice the work and then find out which way one works or which way it works. Google Ventures has a great graph on their website that I’m totally going to mangle trying to recollect it, but the four step process of design, implement, test, analyze, that’s the typical cycle for a product. And that’s A/B testing what you’re trying to do with the two different roads you want to go down. They have a graph where they’ve crossed out develop and test or develop and analyze.

See, I told you I’d mangle it. But the point is you can jump straight from design to analysis. You can do paper prototype walkthroughs. You can plug it in, if you want to, into something like Envision that lets you put clickable models. But just doing back of the napkin with users and putting it in front of them and saying, “Here’s your goal. What would you do? Where would you start?” Okay. You did that. Then you scratch out another one. This is what you see then. How would that work? You can measure. It is possible to measure how well something will work before you retool it, so before you take something entirely behind the woodshed, it might be worth walking through with users on paper whether you can retool it realistically or not.

And that goes the same for new features too. In an ideal world, I would love to sit up here and say that I’ve never implemented something that I didn’t have user-driven proof that it would work. That would be a complete and total lie, but that is the standard that I’d love to get to someday, and there are ways to do it.

Dan: Yeah, we know this world. Our app has been around for a long time and some of the earlier features were built because we thought they were a good idea, just because a good engineer pair working on it. These days, it’s a very different world. So there’s a lot of features that we look at and it’s like, “Ah, it’s not being used that much. It’d be nice to get rid of it.” But then you get into the, “Well, getting rid of it has a cost.”

Even if it’s just, “Well, there’s still some number of people using it,” and if you just rip it out you have to worry about how are we going to communicate that? Do we pre-communicate? Do we say, “It’s going to go away?” And then they’re going to react and say, “No, please, don’t make it go away.” Okay, well we won’t or what? Is there a threshold that only 10 people complain and that’s okay, we’re just going to rip it out? Ripping it out then, obviously, has some complexity. There’s a button here. Okay, now the button’s gone. Now there’s just empty space.

What are we going to put there? We’ll move some things around. It pulls you in and you have to do more. And then there’s the after the fact. “I was using this feature and I hate you because it’s not there anymore and I’m going to pull my entire, whatever, team away from your product.” So looking at it, how much of a cost is it to keep it? And there’s some concrete things there. There is the code for it is really hard to maintain. Every time we touch something else we have to go dig into this mess. Or there is feature complexity.

This feature is making other features more complicated because when it’s in this mode that this feature represents, then other features have to work a different way. And it’s hard to add new features because of this existing feature. Okay, great. Now we have a real reason to consider removing it. But let’s look at how it’s getting used. Obviously, there’s an assumption that you already know that it’s not getting heavy use. Otherwise you wouldn’t be thinking about removing it.

So you go into Mixpanel, whatever, and you find that there’s 237 people a month that use this feature out of whatever, some big number, so it’s a low percentage. Okay, is that low enough to remove it? Well, that’s not enough. Who are they, why are they using it? And it might turn out that the people that are using this feature are actually all CTOs or C whatever Os of your customers and if you take that feature away, they’ll be pissed. You don’t want to piss them off.

Okay, well what are they using this for? Go after their needs and that goes back to what you guys were saying. What was the original intent of this? Can it be solved in a different, better way? And the other thing is, maybe it’s not getting used much because people don’t know about it. And so you can’t conclude…you can’t jump to conclusions. Maybe only 237 people are using it because they happened to stumble upon it. We didn’t a good enough job of promoting the feature or making it discoverable or making it intuitively understandable as to what it’s for.

So this is a big one. There’s a lot that goes into it. And yeah, it doesn’t happen enough in today’s products. People are like, “Oh, that’s my baby. I’m not going to rip that out,” or, “It’s too hard,” or, “Well, I know this person is using it and I don’t want to piss them off.” So we’ll just leave it there and move on to the next question.”

Eric: SaaS definitely makes this harder, because they’re paying you and they like to know what they’re getting by paying you, so when you change that it’s a hard conversation. If you’re lucky enough to work on a product that you sell and forget, then I’m a little jealous.

How to Deal with Unruly Stakeholders

Jared: Just want to double-check with Ana in the back. What time do we have the room until? It’s 1:15? We’re good on time. Just double-checking. So we might skip one of the questions. Who here in the room has ever had their roadmap blown up by a stakeholder? Or somebody who’s like, “I need you to build this. Screw everything else you’re doing.” Yeah, that’s not cool. How do you guys deal with that?

Eric: So for me, I mentioned earlier the why is very important when you communicate with other groups and departments. The why really helps give you authority and also establish that there is a direction. There is a reason the roadmap is the roadmap. And it can lead very naturally to a conversation about the cost. So when something comes in, it gives you a basis to say, “Here’s why we’re going to do what we do. Now let’s talk about why you want this and compare the two.”

It’s a much more natural conversation. Just I like this feature. No, I like this feature. Ultimately, when you do have things cruise in that you just have to take on, I think that’s an inevitable part of being a PM. The other part there, though, is a discussion of cost when things blow up. So it’s not fair to expect sales to understand why their request is causing so many headaches. They live in sales world. They don’t know development. They don’t know how long you’ve been working on something and how jarring it is to change mid-stream, for example.

Have a conversation around cost and try to take…just like when you’re talking to your users and you’re trying to understand their perspective, think of other groups in your company the same way. Treat them with kid gloves and make sure they understand exactly the cost of those requests. And over time, hopefully you’ll find that those requests go down anyway. They won’t disappear entirely though.

Dan: Yeah, this is, again, this is what you just do as a PM and you have to set your expectations. In fact, embrace this. This is our roadmap based on what we know today. But we know that tomorrow something’s going to come up. And maybe it’s not these giant things like some big, new customer wants this and they know your CEO. That’s a big one. It’s a hard one. Go have some conversations, whatever. But smaller things will always pop up. You’re always going to learn, and the things popping up kind of represent the learning.

Maybe you shipped something and that uncovered some new area and so now you’re getting requests for that. Or bugs or whatever or realize there’s something that doesn’t quite fit. And so your roadmap better be nimble and you should have a process around that roadmap. And your whole team should be nimble enough to handle it, but you don’t want to be constantly just reacting, reacting, reacting because it’s like, “What are we doing? We’re just reacting.” It’s stressful. It’s hard.

And so you’ve got to be that person who just maintain the balance of we’ll react to some things, some things we’ll just…Lead us to updating our roadmap because we’ve learned some things, but let’s not forget the roadmap. So it’s this balance of going that direction but sometimes you might have to go around some things that pop up and/or realize that actually we should be over there. And that’s perfectly fine. That’s just part of being agile.

How to give your company visibility into the product roadmap

Jared: I’m going to combine two questions together and bridge them. The first one is how do you give the rest of the company and the other stakeholders visibility into the product and your process? It’s really important. I think we touched on it. I was a group leader there. And then also, how do you foster culture on your team and keep your team’s morale high, and also product vision?

Eric: Well, I’ve already prattled on about communicating why, so that’s my trick for communicating roadmap, not just the roadmap but the foundation behind it. On the culture side, I really like to do something that I think is typically considered back in agile, which is my stand-ups last a long time in the mornings. I’ve had half a dozen developers on the team that I do stand-up with every day, and it’ll go 30 minutes some days, 40 minutes sometimes. Stand-up is the one part of every day where everyone’s in the same room and putting their heads together.

So if they want to dive into technical issues and have a little mini retro and get some mind share in there, that’s great. If you want to spend time joking around and showing your vacation pictures, that’s not a bad thing either. I don’t like to lead the stand-up as a stay on top. What did you do yesterday? What are you doing today? Are you blocked? Next. I think that is…and some people do that. Some scrum masters buy into that and that’s fine. I understand their point, but as the one touchpoint I have, guaranteed, with my team every day, I will take that time to get to know them better and I think we’re better off for it.

Natty: What was the first part of the question again?

Jared: Visibility across the ranks.

Natty: Oh right. I think that’s just a key role of the PM is you spend a vast majority of your time communicating. And that’s…you should be good at it, and if you’re not good at it, spend a bunch of time working on it in practice. But that’s…I think that we’ve mentioned so many things that are critical in the PMs tool kit, but that’s a big one. And then on the culture side, I think culture is potentially the only defensible thing that you have in your company and so defining how you…culture really is how you make decisions and how you work, and so everything you do as the PM sets the tone for culture, because you are that person in the middle.

So the decisions you make, the process you come to reach them, the way you empower your team, the way you interact with your team, the way you communicate, everything you do is watched by everyone else. And so you are really a key piece of setting the tone and the culture. So there’s no right culture. There’s a right culture for everyone and for every company, so you have to figure out what that is. And then you, as the PM, need to know that you’re really important in terms of driving it and setting it. And so when you violate your culture, it has a huge impact because of your role.

Dan: For us, just to use some concrete examples. Pivotal Lab started as a little tiny company and our culture started as very much focused on common sense, get things done, make them real, concrete, write them down, work in small teams, everybody same page, same visibility, here’s why we’re doing it, all that. But not Pivotal Labs is I don’t even know how many people, make hundreds, 500, 600, 800, over I don’t know how many cities now, 12, 15, 20. And so this has been a big thing. How do we keep that culture as we grow.

So some things that stand out are when you work as teams, when you have team culture, that’s a pretty easy one versus you have a bunch of individuals working in tandem. What is culture in that sort of environment? People are just individuals doing work individually. When you have teams, teams have a culture. You have a team culture. Moving people between teams on some reasonable frequency helps as well, so you don’t have siloes. You get a company of 10 teams, don’t put them in different rooms and they never talk to each other.

Obviously stand-ups, big stand-ups help, but actually moving people around, but maintaining some consistency so that you have continuity of just knowledge and then what were you working on. For us though, really the big thing is pairing. So because people pair…developers do sometimes, designers do, sometimes PMs do, but we really encourage pairing. You are working with another person, because I’m forced to either verbalize everything instead of I’m just going to do this, but then it also the benefit, you have this continuous learning of not just technical things or whatever, product awareness, but culture.

You are sharing everything as you pair, and then as you move people around, they pair with new people. That really spreads and it keeps spreading and it’s the same culture. It evolves but it doesn’t deteriorate. It just improves. And this is how we’ve managed to scale up across all these different cities. You pretty much go to any Pivotal Labs office and it’s there’s something very thick and similar between all of them and it’s good. I think it’s the teams, pairing, movement. And then stand-ups and have some team events. Go out. We try to do, at least once a year, a picnic.

We’re all of us…at least the Tracker team, we’re going to go out and have a picnic and spend the whole day not working, drinking low alcohol beer because it’s all that’s allowed there, playing games, stupid games, do stupid things. But then you feel like, “Ah, yeah. We just reconnected.” And that keeps that culture going. And that’s just one of the many ways, but those are some for us.

Advice for People Who Want to be Product Managers

Jared: This is the last question before we open it up for Q&A. What’s your advice to people who want to get into product management? All three of you have gotten into the field in different ways. You each have your own unique story. And how would you suggest people in the audience make the shift, make the transition to becoming a Product Manager?

Dan: I think there are different directions into it and they all have merit. We talked about all these things that are important and I think we…what’s come across, it isn’t just this one thing. You have this one scale. You’re a great PM or you will do well in that role. It’s definitely a spectrum that ranges from soft skills, communication, maybe empathy, reacting to things that happen with people in teams. On the other end of it is some technical comfort level, because these days apps are pretty deep and if you’re building mobile apps you should be aware of the constraints and be a little bit on top of what’s going on in the industry.

You have to be a little bit of a marketer. You’re building this for a market. You’re trying to move it towards that as a business aspect of it. So there’s a spectrum and there’s many paths to it. And we see people come from different walks of life, and there is no this works, this doesn’t work. I think you have to have a certain…like this is the kind of thing you want to do. You want to organize and herd and move in a certain direction. You’re comfortable sitting down with developers and really get in to some technical things.

But just enough so you have enough awareness you can make prioritization decisions. So we see a lot of engineers go from engineering to product management. And it’s great because they have that technical background. They can really connect with what’s going on in the team. But too much of it is also not ideal, because then you’re just thinking like an engineer and everything’s an engineering problem. So you have to learn a lot of those softer skills and communication, have some understanding of marketing, and just business basics.

Coming from the business side, the marketing side, I think the investment is in learn about how software is built these days and how it’s done incrementally and make yourself comfortable working in the fine grain. Maybe build something. These days, there’s so many resources for building software. Obviously, Galvanize is one of them. There’s so many. You can do things online where…just walk through building a little rails app or a little app or whatever.

I think that’s a necessary investment if you’re coming from a non-technical background. So it just depends on the direction. There’s always an investment, and you want to be somewhere in the middle.

Eric: Yeah, and like that, you just have to start doing it. So if you’re a developer moving into PM, moving toward, or you want to become a PM, you know the technical side really well but maybe you don’t know a lot about why the app is like it is or how people are using things out in the world. So start digging into usage data. Start getting curious about that stuff. Start being the knowledgeable person on your team about other facets of the project. And over time you build up the second and third and fourth, fifth, sixth legs of that…not a tripod anymore, but hexapod that is really the basis of a PM’s skill. You just have to start somewhere and start digging into it.

Natty: Yeah, they captured it great. The only other thing I’d add is passion. I think it’s a hard job. I think it’s usually not…you’re not able to actually create any of the things that happen with your product. Everyone else is creating them. You’re coordinating it and that can be challenging. So you need to have passion for the product. So don’t be a PM is you’re not passionate about the product. Go be a PM somewhere where you are passionate about the product.

Eric: One more thing I’ll throw in from when I started making that, heading that direction myself. I actually didn’t know Product Manager was a thing when I started doing it, so I had a panic attack that I used to be a developer and my resume was a list of languages and things I’d built. Look, I can whiteboard code for you. I’m a good coder. PM is a bunch of soft skills and it’s terrifying at first that you now soft skilled yourself out of any one job. It’s like jack of all trades. But there is a really valuable combination of skills. You just have to set your eyes on it and go for it.

Dan: One thing that stands out for me as I see people going from more traditional or business or even product management or marketing roles into what we see as the ideal Product Manager-Product Owner, is it’s being able to operate at a high level, thinking big, vision. Drop down, and then think in little detail and thing incrementally. It’s think big and shoot for big things but then you have to map that and translate that to incremental progress. And that goes hand in hand with writing good stories in Tracker or whatever.

Yeah, we want to solve some specific problem around engagement or make it so that PMs can report progress to their stakeholders, but what does that actually mean? What are we going to start with? We start here, then we’ll go here, and then we’ll go here. Oh, we’re going to discover something here, so at that point we’re going to be going this direction when you get there. So taking a big picture, thinking, “I can do research and capture these big things in a document,” but great, now let’s talk about how to actually get there. Where do we start?

And maybe using things, like imagining a life in the day of whatever the persona is, that there’s a progression. And to me, that’s the biggest thing is going from we want to be there. We want it to be awesome. Yeah, but how are we going to get there? Without going too far, you’re not the engineer. You shouldn’t be specifically like, “We’re going to do this exact thing on the left side of this.” Leave that up for design, product design and engineering to solve the hows, the specific hows. But you have to actually think through the product incrementally and how it’s going to get from where it is here or now to whatever the ideal is, what the goal is.

Jared: I just want to thank all three of you again for being here today. Let’s stand up and take a bow, Product Managers. So I’m going to run around with the mic and give it to whoever has questions. Thank you guys, again, for being here today. And let’s open it up.

Natty: The only thing I wanted to add is I think if you want the best one page overview of what product management is, look up Ben Horowitz’s “Good Product Manager, Bad Product Manager”. It’s maybe 20 years old right now, but I think it still resonates. And I’ll tweet it afterwards if you have trouble finding it.

Jared: We only have a few minutes so if we don’t get to your question, feel free to send an email at Jared@galvanize.com. It’s J-A-R-E-D@galvanize.com and I’ll ping one of these three and we’ll tweet it out, we’ll tweet out the answer.

Man in Audience: So most of what you’ve been talking about is very software focused. Do you have anything that you would change in your answers [inaudible 01:23:00] for products that are hardware?

Eric: Don’t know. Software is what I know, so it’s what I understand. Hardware always seemed like a big scary pain because of the development time involved and the bake-in that you have. I think the one thing that goes across any product, whether it’s software or hardware, whether you’re making bridges, [inaudible 01:23:29], whatever, is the ability to test things before you build them.

So hardware more than anything, you need to have a strong idea of the strengths and weaknesses of what you’re building ahead of time and really invest in that level of planning so that you don’t get halfway through a production cycle where you actually have things that you can’t change and you find out after the fact.

Dan: And some of the stuff still applies. Framing problems, understanding needs, who’s this for, trying to keep the cycle as short as physically possible given manufacturing cycles and all that. But communicating, so keeping a vision, reacting to things as they happen. A lot of that stuff applies pretty much any project in my mind.

Woman in Audience: Probably reject.

Jared: [inaudible 01:24:24]

Woman in Audience: Okay. Hi. So I was wondering how you guys all and how you have in the past measure and prove progress with a product team, whether it’s individuals, individual products themselves but TPI [inaudible 01:24:43]?

Eric: Just shortly, the important thing there is, and again, broken record about objectives. Measure things in terms of what that feature or change or product is supposed to do or what you wanted it to do for your business. It’s really tempting to measure things in clicks and numbers of users, but ultimately you need to look at that bottom line of what goal were you trying to accomplish by getting those clicks, by getting those number of users.

It’s just a really easy pitfall. There’s a term…let’s see if I can remember this. There are measures of effectiveness and there are measures of…oh, I forget the other one. Measures of use, I guess it is, something along those lines. You need to be looking at those measures of effectiveness. They’ll achieve the objective rather than is it use [inaudible 01:25:34].

Natty: Yeah, I think the other way to describe that is you want metrics that prove value and that prove growth. So you want to have…I don’t think you can really optimize more than three to five metrics unless you’re doing it for just one hypothesis, but from a macro level have three to five metrics. Half that are testing value and half that are testing growth. If that’s helpful.

Dan: And I’m not sure if you meant metrics specifically around the product, like how well is the product doing?

Woman in Audience: The team itself.

Dan: Right, team itself. Yeah, that’s interesting and that’s actually something that we’re really trying to focus on. How do you measure that your team is going well or your project or whatever, but let’s say it’s a team because that’s really what it comes down to. So what are the things that you want to improve and are they improving? Is there a…[inaudible 01:26:32] New Relic. New Relic is an application performance monitoring tool that everybody uses. It’s really great. But what’s the New Relic for your team?

How do you know that things are improving, that you’re working better? And there are some semi-standard agile metrics that you can look at, like how long does it take you to work through stories or features? Is that getting worse or better? How often are you shipping? Are you shipping continuously? Is that actually changing? Are you starting to slow down because you have more of a heavyweight process? How much churn is there? How many rejections on stories?

These are not all necessarily clear as to what’s better or worse. It’s more how are these changing and why? And try to uncover some of these trends that might be indicating some things that are hiding. Maybe you need to move some people around. Maybe you have too many people on one team, there’s too many lines of communication. So that is a very interesting area and I think there hasn’t been that much focus on it yet.

Woman in Audience: Do you have time?

Jared: We can do one more.

Woman in Audience: Okay. So as someone who…sorry.

Jared: That’s fine.

Woman in Audience: Someone who wants to get into product management, but kind of, sort of have those soft skills that you guys mentioned. What are practical ways to market yourself to get into a position? It’s hard to apply when your resume is just this conglomeration of random things. So I guess what would you guys look for in hiring?

Dan: Well, since I’m holding this and we’re trying to hire Product Owners, definitely it starts with you show some enthusiasm and interest and passion in what we do. If somebody just sends a resume and it’s like, “I did all these things,” especially when it’s a broad spectrum, it’s just immediately like, “We’re not sure what this person is really looking for. Are they just throwing their resume everywhere?” So that’s more general. It goes for pretty much any…I’m applying for a job. I have to…you have to make it relevant to the person who’s on the receiving end.

And even if you don’t have a lot of experience in that particular area, if you show that you have a lot of enthusiasm, here’s why you’re interested in working at so and so company, because I have seen your product in use here and I saw the results that it had and I really want to be part of that. I want to make it better. I don’t have a ton of experience, but I do have some of these relevant pieces and I want to learn and this is the area that I think, or this is the place where I think I can get that learning.

That comes across so strongly, when the person takes the time to really capture that. And it’s not easy to put that into words sometimes, especially when you don’t have necessarily a lot of that concrete experience to base it on, but it really gets, at least, my attention. I know this is what stands out for a lot of the others who are hiring managers. It’s like, “This person took the time to tell me why they’re interested in working here. At the very least, I want to have a conversation with this person.” And that’s already a door that’s open.

Eric: I know when we’re hiring junior PMs, first time PMs, what can help is using PM-like language on your resume. As superficial as that sounds, if you’re a developer and you’re listing off your accomplishments as a developer, so developer it’s, “I built X, Y, and Z.” But if you instead list it as, “I built a feature that contributed this to the bottom line of the company, that accomplished this,” now you’re speaking our language. A PM resume looks very much like that. I raised sales 6%. I led the team to accomplish these things.

So speaking the right language, it just communicates that you’re coming at this from the right mindset. We’ve hired business analysts. We’ve hired developers that come out of [inaudible 01:30:31]. The other thing, developers can get involved with open source projects with GitHub, that sort of…GitHub, just look up a project on GitHub, you can see a developer’s experience. So it’s harder for PMs. If you have the technical skills to implement something yourself though, and then you can do that and cast yourself into the role of look, I came up with this.

I analyzed the business opportunity. I came up with the core features, defined an MBP. It gives you at least a context in which to discuss those things. It’s not easy. It takes a ton of time, but that’s one approach.

Natty: I’ll add another. So we’ve got the approach, the resume, and then, I think, let’s say you get the interview. I highly recommend do a ton of research on the company and come in with a perspective on it and some ideas. The best people I’ve hired, we ended up spending the whole time talking about the interview about how they’ve improved the product or how they understood the users. And at the end of the interview, I realized, “Oh my God. I don’t know anything about you, but you know a ton about us and I want to hire you.”

So I think if you get the approach right, get the resume right, and get that interview, not all interviews will go that way but be prepared for that and come with a perspective, come with ideas, come with enthusiasm for that product, and that will make you stand out.

Jared: I’ve never hired PMs, but being on the opposite side of the table, coming from doing marketing at Uber and getting hired by Eric and Mike at Kapost, ask for a creative assignment. So if you get the interview…or even make one of your own even if they don’t ask you for it. So by that, I mean I would improve this part of the product in this way. Or given these circumstances I would do this. And lay out some research, do some research, know the product really well like Natty said.

But I interviewed with a couple companies before working at Kapost. One of them was Upworthy. And at Upworthy they gave me this intense graded exercise where I had to prioritize all the features in the roadmap and design a feature from scratch, and do a wire frame, and lay out my spec and just do all this stuff. Kapost asked me to do the same thing, so I’d already had some practice doing that and I nailed that exercise, so I’d say be ready to do that. And if they don’t have that in the interview process, ask for it, because it’s really good practice and it’ll let you flex your creative problem solving.

Eric: Also mention that hiring managers, it’s really hard on the PM position because it’s a hard job to list the concrete skills for. Hiring managers don’t know what the hell they’re looking for on a resume. So it’s a position where networking probably has a greater effect than most others. Jared does that really well. We learned about him because he knew Mike and I think made it a point to get to know Mike for that reason, and it worked out great. So more than an engineer, more than any other position where it’s easy to say, “Here are the languages I know,” you have to introduce yourself, because if it goes through HR, you might just die in a stack.

Jared: Thank you so much everybody.