Building effective security OKRs

9 min readNov 9, 2020

This is an abridged version of my LocoMocoSec 2020 conference talk

How do you set the strategy for a security team? If your goals are vague, people on your team will work on whatever they’re interested in because there’s no clear mission. If you don’t measure impact, your organization will hesitate to give you resources because you won’t be able to show what they get in return.

OKRs let your security team do work that everyone in the organization values. John Doerr’s book Measure What Matters calls out four organizational OKR “superpowers” that also apply to security teams.

Focus and Commit to Priorities
Align and Connect for Teamwork
Track for Accountability
Stretch for Amazing

Maybe you’ve seen first-hand how OKRs lead to amazing results. Or, maybe they’re mandated by your organization. Either way, how should security teams get the most value out of the OKR process?

Security objectives

Define the mission with high-level security objectives

First, make sure you have a set of security objectives that make sense outside the security team. They should describe how security supports the organizational mission. What risks are your team focused on preventing? If your organization has cascading OKRs, think about how security can be framed as a company-level objective.

Michal Zalewski describes the importance of objectives for your security team:

Rather than focusing on tactical objectives and policy documents, try to write down a concise mission statement explaining why you are a team in the first place, what specific business outcomes you are aiming for, how do you prioritize it, and how you want it all to change in a year or two.

A good example of high-level objectives is in Gitlab’s information security team’s handbook. They talk about three tenets:

Secure the Product
Protect the Company
Assure the Customer

Note that the first two are about reducing actual risk, whereas the third is about perceived risk. Reducing actual and perceived risks can both align with organizational objectives.

Create balanced security objectives

I work at LaunchDarkly and our customers have high security standards. To assure them, we undergo ISO 27001 and SOC 2 certifications. Most of the effort spent obtaining these certifications isn’t about reducing actual risk. Instead, it’s meant to meet reduce perceived risk and assure the customer so we can enable revenue goals. Following these standards also reduce actual risk. We can have initiatives that fulfill many objectives.

In fact, having many objectives that balance each other out is a good thing. You don’t want to achieve goals while ignoring harmful side-effects. One high-level objective I’ve introduced at LaunchDarkly is to enable rapid delivery of value. It is designed to be in tension with other objectives. We can reduce risk towards 0. We can build the strongest customer assurance in the industry. If we slow the product delivery team to a grinding halt, we’ve failed.

Think of your security team as a product

If you want to build a successful security team, make sure people want what you’re providing. Think like a product manager.

Instead of a company-wide rollout of a cumbersome new security tool, can you ship a minimum viable product, learn, and iterate?
Security tools can balloon CI build times. Can you have an objective around developer experience?
Can you send out NPS surveys after your threat modeling exercises? Would other teams recommend your security team to other people?

These kinds of objectives provide a good tension with your risk reduction objectives. They ensure you have a sustainable relationship with the rest of your organization.

Long-term objectives drive short-term objectives

After you’ve defined high-level objectives, you can create quarterly objectives that have higher resolution.

Objective: Reduce risk of a data breach by preventing vulnerabilities in our production web application

Assure the customer by conducting a quarterly penetration test

Remember that these short-term objectives may map to many long-term objectives. I’ll talk about the process for brainstorming and prioritizing objectives later.

Security key results

Key results should be decidable

Vague key results are too open for interpretation. The “process is documented” or “false positive rate is improved”. If two people might disagree on whether you hit the key result, change it.

Key results should be actively tracked

Another problem I’ve seen is that the key results sound good but the metric isn’t defined or tracked.

Key result: Reduce vulnerabilities by 50%

If vulnerability data isn’t tracked and queryable, this key result isn’t useful. This type of key results can be a symptom of teams creating OKRs at the last minute. It’s important to do pre-work for OKRs. Make sure that you have systems in place to measure what you intend to improve before you set the key results target.

Challenge and inspire to the team

Define the direction you want to go with an objective and the criteria for what it looks like when you’ll get there with a key result. You can set aggressive targets for key results and give people ownership and autonomy to meet them. It’s a great way to get creative solutions to problems and it gets people engaged in their work.

Drive team learning with accountability

Another big failure mode with key results is to “set and forget”. If you don’t track whether you hit your target, you won’t improve. Discipline around planning should extend into grading and analysis.

Example security OKR: code vulnerabilities

Based on these guidelines for effective objectives and key results, I’ll walk through an example OKR. Let’s start with an objective:

Objective: Reduce risk of a data breach by preventing vulnerabilities in our production web application

How do you measure your progress with a key result? You can only measure vulnerability data if you have consistent metadata. At LaunchDarkly, we use Jira as a single source of truth for tracking vulnerability information. We created custom issues types with the required metadata fields.

Type (CWE)
Severity (CVSS)
Service
Team
Source
Time introduced
Time identified
Time mitigated

It’s important to weed out “potential vulnerabilities”. We have a triaged bug-bounty program. We have regular manual penetration tests. These very rarely cause false-positives. We also have vulnerability scanners across our layers — OS, Cloud, web, and code — where we tend to see a lot of noise. We have a process for manual triage of these results to make sure our vulnerability data ties to real rather than potential risk.

Once we have good vulnerability metadata, we can start to track vulnerability metrics. Some examples metrics:

Time to detection of vulnerabilities
Time to remediation of vulnerabilities
Average vulnerabilities over time

All of these can be viewed by type, source, team, and so on.

Key result: Reduce average time to detection of vulnerabilities in production web application by 50%

We’re going to reduce the average time to detect vulnerabilities by 50%. This should reduce the probability of a data breach. We’re able to detect the vulnerability before an attacker exploits it.

Measured by Average(Sum(Time identified — Time introduced))

We can now state our key result as a hard metric. If you have a single source of truth for vulnerability data, dump it to an analytics tool. Create a dashboard that people can check on progress at any time.

Having this well-defined key result makes the impact of initiatives much clearer. Does the team think implementing a static analysis tool will reduce the time to find vulnerabilities? The OKR links the initiative with the top-level company goals and justifies the project.

Sometimes you work backwards. If you think that a security project is important, describe what impact it will have with a key result. How does that tie with objectives?

Consider measuring risk

Another style of OKR you could consider is to use risk itself as a metric.

I found the book How to Measure Anything in Cybersecurity Risk to be a foundational shift in how to think about cybersecurity risk. Some key concepts:

Risk assessment is a critical foundation of security
We are always doing risk assessment — formally or informally
Expert estimation is effective even with incomplete data
Instead of risk matrices, measure risk as dollars

I’d recommend this book, but it gets dense. There’s some templates on the website if you’re looking to get started.

Ryan McGeehan’s simple risk analysis is a more concise take. He describes how to forecast risk with expert estimation. He has a blog post about using this process to create OKRs. He suggests starting by writing an objective with a “probabilistic risk scenario”. An example scenario: “an adversary accessing production from a developer laptop in Q3”. You do a forecasting exercise and make the result your baseline. Now you do your work, making progress as usual. At the end of the quarter, you do another forecasting exercise and compare with the baseline.

You can measure risk as a metric or use the language of risk to clarify the impact. Either way, spend time documenting how your security work ties to risk.

Process for creating Security OKRs

I hope you have a better idea of what good security OKRs look like. Here’s our team’s process for building OKRs.

Assess prior OKRs

First we meet as a team for a post-mortem on the previous quarter OKRs. We discuss what worked and what didn’t. If we didn’t meet a key result, we may want to try again, change it, or take a new approach.

Do a live group risk assessment

There are a couple ways I’ve asked the team to prepare for an OKR brainstorm. The first is an informal risk exercise. We come up with the most likely and impactful risk scenarios. What keeps people up at night? Where do we know we don’t have adequate protections? What stories have we heard about trends that make us concerned? This frames our ideas with our risk reduction objective.

Find inspiration through an article or video

Another preparation strategy is to ask everyone on the team to read an article, book, or paper or watch a video together. I’ve used the video of Clint Gibler’s “An Opinionated Guide to Scaling Your Company’s Security” talk from AppSecCali. It’s a great compilation of talks from leading application security teams. This can help open you up to more ideas, leading to a more creative brainstorm.

Do a bottoms-up brainstorming exercise

During the bottom-up brainstorming exercise I gather ideas from the team about what they want to work on next quarter. I like to encourage people to think about it beforehand. I remind them to be good advocates for an idea, they should come prepared with data. They should show the impact of their proposed projects.

Use post-it notes and voting

I get people to write down their ideas on post-it notes. Each person puts them on the wall, clustering related ideas. After narrowing down our ideas to a set of common themes, we run a voting exercise. Each team member ranks their top five ideas. We can then calculate the priority of each project, according to the team.

Set up time to review with other teams

After our team brainstorm, I reach out to other teams to get a sense of their quarterly OKRs. The DevOps or IT team are often leverage points for security efforts. Where possible, I like to establish shared objectives. I use the “two sides of a bridge” analogy. We agree to build out separate parts of a complementary project that meet in the middle:

You deploy MDM, we configure the security policies in it
You build the observability dashboard, we add the authentication and access control layer

Stay tuned in to company wide goals

It’s also a good idea if there are company-wide goals to ensure that you’re aware of them and supporting them. Are there any new initiatives that need security focus? It may reduce our bandwidth.

Prioritize objectives based on impact

At this point, I have the materials for a good first draft of OKRs. Since I’m responsible for prioritization, I need to decide what to cut. I rank the projects based on their impact and effort. There are two other things to consider. One is confidence — how likely is this to result in the impact? Am I comfortable taking on high-risk, high-impact projects? The second is cost of delay. A risk that exposes the organization today is higher priority than something that prevents a future problem.

Document objectives, key results, and initiatives

I’ve seen OKRs that are just a few sentences. I think it’s important to invest time in writing OKRs with more detail. How does this objective tie in to higher-level objectives? Why did we focus on this objective over other objectives? How are we measuring the key result? How did we choose the target for the key result?

Publish OKRs to get feedback and alignment

Next, I send out our OKRs to the organization for feedback and alignment. Once the OKRs are set, I ask people in my 1:1s what they want to work on. I assign ownership for meeting the KR based on their interest and capability. It’s a great way to build autonomy and career growth for individual contributors.

Execute, rinse, and repeat

At this point, the team can start documenting project plans. As the quarter progresses we have a monthly check-in on projects and key results. If it’s clear that we’re not going to hit a key result, we can consider changing it. The goal here is to create more and more predictable outcomes, which let us place bigger and bigger bets.

Questions? Feedback? Let me know on Twitter: @alsmola