Transcript

00:07
All right, hey, and thanks for having me. Let's get the sharing going on here.  And let me know if you can see that.

00:20
I hope you are.  It's  showing up and I'm seeing thumbs up. So all good.  Um, so welcome to my talk on supercharging your security pipeline or managing vulnerabilities in a multi-tool jungle.  I am Mikko Nilsson.  I am the product security lead for customer intelligence R&D here at  SAS.  Uh, I've been at SAS for 15 years,

00:48
So I'm older or have been here longer than most of the furniture. I've seen a lot of evolution going on at SAS with our products and solutions. In those 15 years,  I've mostly worked with our customer intelligence solutions in various different roles,  ranging from presales to technical architect to global enablement consultant.

01:18
Uh, but I've also done a short stint as the information security manager for SAS in the Nordics. So, quite a lot of different perspectives of SAS and SAS solutions in those 15 years.  And before we dive into the details, I also want to state that I'm very much a generalist. I'm not a developer. I'm not a test engineer or a secure code guru.

01:48
So what I do at SAS is high-level things, and I try to make sure that we point our security program in the right way and provide the correct metrics up the chain.

02:05
And before we go into the jungle,  I'll share a few words about SAS and the solution that I work with.

02:15
So at SAS, our vision is to be the most trustworthy data and AI partner that powers the world's decisions. And our mission is to provide knowledge in the moments that matter.

02:33
And we do that by delivering the fastest, most productive, and trustworthy solutions.

02:44
One of those solutions that I work with is called Customer Intelligence 360. And it is a marketing solution that's designed to reimagine your marketing with trusted analytics and powerful decisioning. CI 360 is a hybrid multi-tenant software as a service offering. And what we mean by hybrid is that some of the software is provided as software as a service.

03:14
Through the cloud, and parts of the software are installed on-premises, wherever the customer's data may reside. And the reason we do this is to minimize the amount of data that customers need to actually move into the cloud. And that, of course, minimizes risk to the customer, and there's less data that they need to be worried about in a third-party cloud.

03:43
We follow DevSecOps methodologies and CI-CD with CI 360 to deliver features for our customers at high velocity.

03:57
Customer intelligence 360 is really built from three different parts. We collect, identify, and curate audiences, regardless of where the customer data may reside. That's part of the on-premises part.

04:15
And then based on those audiences and based on that data, we build journeys to analyze, enhance, and orchestrate  that customer journey. And then finally, we have destinations where we connect, deliver, and record messages to our customers.

04:39
In my talk today, I'm first going to describe our journey in software development, as well as the evolution of our application security program with that software development journey. Then I'll talk about DefectDojo. And not because I'm trying to sell DefectDojo to you, because I'm not affiliated with DefectDojo, but because we at SAS have decided to use DefectDojo for centralized management.

05:09
of vulnerabilities. And regardless of which solution you might choose to use, DefectDojo being one of them, the premise remains, regardless of which software you choose for your organization to do application security posture management.

05:27
And finally, I'll share some lessons learned, some outcomes, and tips I found valuable during our journey towards centralized vulnerability management.

05:41
So to do that, I first want to give some context by talking a bit of our journey to where we are today.

05:52
SAS has been developing software since 1976, so quite a long time. And we've gone through many phases of how software is being developed. We've started with waterfall development. And then  in CI360 specifically, we moved into monthly iterations of software development. And now we're at a stage of continuous deployment.

06:21
Along with that, our security program has had to evolve. So with waterfall development,  we did phased testing where security testing was part of the development cycle per se. And you had a dedicated time where you did your security testing, made sure everything is okay, and then delivered your software out the door. Now, obviously that doesn't work.

06:50
With monthly iterations anymore, you're going to have less and less time to do that. So to be able to meet that need, we developed security scan automation, which was integrated with the pipeline. And when we created monthly iterations, we ran scan automation at a specific point in that monthly phase. It's kind of phase testing, but it's kind of step two of that,  uh, which, which enables automation. And with continuous deployment.

07:20
We don't really have that option anymore. We can't pick a specific phase where we do scanning. So we've had to do pipeline integrated scanning and we've had to start looking  at our security posture from a centralized perspective and to build a better holistic view  of our security posture. And that's where centralized management of vulnerability steps in.

07:49
So why does all of this matter? Why do we do this? Why do we evolve our software steps  to be faster and more rapid? And one of the drivers here I want to talk about is of course security.

08:05
This research is from Mandiant,  and there's a source right below if you want to go look at that. This one's from 2023,  but there's probably a more recent one coming out now or already came out.  But the point of this slide is that the time to exploit of vulnerabilities  is decreasing constantly. I mean, if we look back at 2018,

08:34
The average time to exploit a vulnerability was around 63 days, according to this research. And they've done recurring research, um, kind of on a biannual basis, I guess. And every time they've done research, the time to exploit has gone down significantly. First from 63 to 44 days, then to 32 days. And the latest one, only five days of average.

09:04
or an exploit to a vulnerability. So if we think about that in the context of how you develop software and how you deliver patches or fixes to your software, it's pretty clear that if there's a critical vulnerability, you need to fix that really fast. And to understand  where you need to fix and what you need to do centralized vulnerability management, I think is part of that answer. Now let's look at some more stats here.

09:33
The  split between zero day and end day vulnerabilities where zero day vulnerabilities are vulnerabilities which are unknown to the vendor and just disclosed without a patch being available. And end day vulnerabilities are known to the vendor and you may have a patch available. So from patch release,  that's a end day  vulnerability.

10:02
ratio of zero days  towards  end days  is  also  increasing. So we're having more and more zero days instead of end day exploits. And with end day exploits, of course, the benefit is you have a patch that you can apply. And now it's just the amount of time that it takes for you to apply that patch. But with zero day, the risk is that you don't have a patch available once that vulnerability is disclosed.

10:33
Now, if we look at the end days here and see how long it takes for attackers to exploit vulnerabilities that have been disclosed or patch may be available.  And why this is significant is  if we talk about patch management and where we do vulnerability management, how we remediate the bugs, then  it is important to understand how fast we need to apply that patch, right? So 12 %

11:03
of the end days were exploited within one day of that vulnerability being disclosed. 29 % within one week and 56 % within one month. And 44 % of end days took longer to exploit than one month of the release of the patch or disclosure of the vulnerability.

11:29
So that's not the only thing that's driving  the efforts  to  remediate faster  and  to do centralized vulnerability management. And this is where we head into the jungle. So welcome to the jungle. We've got fun and games.  I heard a song like that  at one point. You  may have heard that before as well. What do I mean by jungle? Well,

11:56
When we're developing software, we have various sources  of vulnerabilities.  So we may be doing static application security testing, scanning our code  and seeing if our code introduces weaknesses. In addition to that, we're probably doing SCA scanning or software composition analysis where we're checking if the third party open source libraries that we're including in our software.

12:25
also contain vulnerabilities. Now maybe you're developing software for the cloud.  And if you're deploying on the cloud, you should probably be looking at your cloud security posture as well. So maybe you're running some sort of cloud security posture management, which is checking your cloud configuration for any issues as you deploy your software.

12:48
And maybe you're doing DAST scanning as well. And DAST scanning is dynamic application security testing.  If you're developing a web application, dynamic application security testing tests the web application itself  for vulnerabilities. So that's further right in the development pipeline because you actually need a web application to test. But already here, we're introducing one more problem.

13:19
Dynamic application security testing may reveal issues with your code. So you may have a weakness with your code, or it may reveal an issue with your third-party libraries, which is identified by software composition and analysis. So now you're already dealing with duplicate findings coming from different tools.  And those duplications may occur over

13:47
different realms and different tools. So you don't necessarily know where that bug was introduced from, and you have two issues to deal with, which may be the same thing.

14:00
And we can add more tools into this  mix or the jungle. So you may be doing secret scanning. You're scanning your source code for embedded secrets or maybe your GitHub repositories and making sure no one's committing  keys or, or, passwords into, into your source code management. Maybe you're developing things  from an image container perspective. Maybe you're using container based things and you're probably scanning those as well.

14:29
There's overlap with all of these. Maybe you run a bug bounty program or you do penetration testing of your software. Those findings may touch various different areas of this.

14:45
And in addition to that, so you have this massive tool set of security things where you're producing a lot of findings and you somehow need to figure out how you correlate those, how you prioritize those. But addition to that,  if you're an agile company  and you want to evolve your security program, perhaps you want to replace some of those tools, or maybe you want to test some of those tools against another vendor.

15:13
And this is where it gets even more complex because now you have two different tools doing the same thing, potentially where you're trying to test which tool is better and which tool you want to move to, or perhaps you've already made that decision, but you need to transition over to the other tool  and to make sure that things are okay. You want to verify that you're producing  similar findings from both different tools.

15:43
And I mean, there are niche tools as well, which work very well for a specific purpose. Like let's say secret scanning in this picture here, you might be using a specific tool to do secret scanning, even though most of that is covered through your static application security testing. So that's the other consideration here.

16:06
All right, so how do we handle all of this? Well, that's where kind of DefectDojo steps into play.

16:14
And DefectDojo is there to do your centralized management. So what is it? You're probably familiar with DefectDojo because you're listening to this webcast, but let's go through what  I believe DefectDojo is and what it does. I may be wrong here, so call me out if I am. DefectDojo is an open source application security posture management tool.

16:43
an ASPM tool. And DefectDojo enables you to correlate findings across different security tools. It also builds an inventory of your applications and your endpoints. So it's almost like an asset inventory in that sense, but from a software application perspective. And it has capabilities, powerful capabilities, to de-duplicate findings

17:13
And then from a triage and vulnerability management perspective, you have powerful bulk tasks and filters, which your triage can then use to  filter out the noise before it's sent out to developers for remediation. All of that is powered by powerful APIs and reporting capabilities.

17:41
And you can get DefectDojo, either the open source version by going to GitHub, or you can license the pro version by going to defectdojo.com. At SAS, we're currently using the open-source version of DefectDojo.

18:02
Okay. So what else is in there? Well, DefectDojo has integrations for various different things. There are almost 200 tools in various different categories, including static application, security testing, software composition, software bill of materials, type tools, infrastructure scanning, threat modeling, and bug bounty imports. And in addition to that,

18:31
If you can't find a specific tool integration that fits your tool, there are more generic import formats as well. So DefectDojo can ingest AWS security findings format files, Cyclone DX  files,  are software bill and material standard,  and Oasis static analysis results interchange format files, which some static analysis tools produce.

19:00
And if that's not enough, you can certainly try to format the findings that you get out of your tool into a generic finding format. So in DefectDojo, you can import things in CSV or JSON format as long as it conforms to the standard that DefectDojo expects. So you can build a parser, or maybe you have in-house developed tools that do some things.

19:28
and you want to pass those into DefectDojo. Now, all of that is powered by APIs as well. So it's not just UI-based. And a lot of the things that we've built at SAS utilize the underlying API  structure of DefectDojo. And that's what we've been really thrilled about.

19:50
Okay, so let's bring that into a bit more context and into what we do in customer intelligence at SAS.  And I'm going to talk about our security pipeline and kind of how that looks from a DefectDojo perspective.  And before I get into that detail, though, I want to mention that we kind of do two different things right now. Before DefectDojo was in place, we already implemented

20:20
security automation within our pipeline. And we have a gated automatic pipeline that runs tests as soon as developers merge changes into their main branch. And that pipeline is gated. If a developer tries to introduce a new critical finding or high-priority finding, they're stopped, and the merge doesn't go in.

20:46
So it needs to be remedied before it's introduced. And if we find security issues that are already in production, so a brand new vulnerability for a third-party library, for example, then we set a timer within the pipeline according to our SLAs.  And once that pipeline expires or the timeline expires, the SLA expires, it blocks the pipeline, and it prevents the developer from

21:16
introduce anything else into the into the main branch before they fix the vulnerability. And we have automatic ticketing in place as well. But as we started building DefectDojo, we're now in the transition phase where we're actually dismantling that existing automation and creating an integration with DefectDojo where our pipeline.

21:44
Really, actually talks to DefectDojo, and DefectDojo is the single place of truth, which controls what happens within the pipeline itself. But we're not quite there yet. Okay. So let's talk about the pipeline, then a bit more. Um, we have various types of scanners running within our pipeline. You do sassed and SCA scanning. We scan for secrets. We do dust scanning.

22:13
And then we scan infrastructure parts, like image scanning.  We have IAST and cloud security posture management producing findings. And all those findings are produced.  And before we had DefectDojo, we had a pretty direct,  uh, mapping from security tool to the developer, and it opened a JIRA ticket.  And we didn't have really any manual triage or or visibility in between.

22:43
And in addition to that, we have interactive engagements that produce findings. So we have a private bug bounty program that we run for  CI.  And we also do regular penetration testing of our software. And those also produce findings. And previously, managing these findings in a centralized fashion has been pretty much impossible. So that's why we introduced DefectDojo. And DefectDojo is the central place.

23:13
where all those findings land. And it enables us to do triage, deduplication, and reporting on all the different security findings that we have, regardless of their source. And once we've done that triage function, we also have the capability then to push those findings into JIRA, which then becomes a ticket and is assigned to a specific developer for remediation.

23:41
Or maybe it's an infrastructure engineer,  depending on the type of the issue. And we can also do alerting from DefectDojo. So we can alert channels and teams and maybe even alert on-call teams if needed. And I really should change the title here to our future security pipeline because we're not quite here yet. And I'll talk a bit about our challenges that we've met along the way, but

24:11
That is the vision. DefectDojo will be the center place for all vulnerabilities and centralized vulnerability management.

24:23
So let's talk a bit about our challenges.

24:28
Uh, first of all, we've gone through migrations of security tools before implementing DefectDojo.  In hindsight, we should have implemented DefectDojo first and then done a security tooling migration. And the prime reason for that is there's a larger developer impact when we change tools on developers.

24:56
and don't have something in between. It means that developers need to be more familiar with the actual tools producing the findings. But when we have DefectDojo in between, it's easier for developers to react to findings because we can normalize the output from a tool so that it becomes easier for a developer to see it.

25:25
Uh, another reason that we've had some delays here is that we have an ongoing transition between our collaboration and version control platforms.  And that's led to some complexity, uh, with our DefectDojo implementation.  And, um, we've had to manage slightly different security SLAs, and maybe the remediation paths are slightly different depending on which collaboration tool you're on.

25:53
Which version control platform you're on?

25:58
Another thing that has caused some delay is that with software development from 1976  and a long-time user of Jira, we have heavily customized our Jira, and we were not able to use the native DefectDojo Jira support that's in there right now. And we had to actually write custom scripts to integrate with our Jira instance.

26:31
Next, I'll talk a bit about the kind of end-user perspective of DefectDojo. So I'll describe different profiles for this. I'll talk about DefectDojo from a developer perspective, a triage perspective, a service owner perspective, and a manager perspective.

26:55
So let's get started with the developer experience of DefectDojo and how it looks like here at SAS. So first of all, our JIRA tickets are automatically created, and they're also automatically resolved. And that's a big change to the past, where we were able to produce tickets, but it was on to the developers to actually close the tickets when they

27:24
Had remediated issues, and we didn't have that to wait kind of automation. But now we,  have that automation in place and the ticket is created automatically and with some tools, depending on how confident we are on the findings being real. We have completely automated that process. So when a security tool reports a finding and we have a high degree of confidence that it's a valid finding, we automatically push that into JIRA.

27:54
So there's no manual triage in between.

27:58
And once the issue is resolved, so once a developer merges a change into main, we have automation which closes the ticket once the security tool is no longer reporting that finding.

28:13
The tickets include links to  DefectDojo and the reporting security tool as well in case the developer needs to go deeper into that finding to find out how to remediate the issue. We try to include remediation instructions if possible. If the tool reports them, we put them in there in the ticket. And then we also drive goal dates based on our internal SLAs.

28:43
Look at that ticket in a bit more detail. On the top there, we have a normalized title.  So we report what type of tool is reporting this issue.  It's an SCA tool here in this case,  reporting a high severity issue for denial of service. And then I've just scrubbed out the service name because that's irrelevant. And then we populate that ticket with some labels that are relevant to us, and maybe

29:12
relevant to the teams. So in this case, we can, for example, see that this ticket is a security issue. It's from Dojo and it applies to a third-party component called Jetty server. And then we also populate the CVE information. So this finding is related to CVE 2024 8184. And then we have some further details down the line there.

29:43
In the ticket, we also have a direct link to DefectDojo. So if the developer wants to look at it in DefectDojo and see, maybe they want to see if there's other issues, is this a deduplicated issue, or something like that, then that information would be through the DefectDojo link. And then we produce information for the developer on how this should be remediated if the tool provides that.

30:13
Our tool here, Sneak, is reporting one instance of denial of service, and there's a finding-related dependency below there. And there's also a direct link to the tool itself. The Sneak Java link there goes to Sneak. And this issue is related to JetEU server 9.4.53 within that service.

30:49
All right, yeah. Next, let's talk about the triage experience. Oh, sorry,  skipped a slide there.  This is the view in DefectDojo. I just wanted to show that real quick as well before moving on to triage experience. So in DefectDojo, once you click through,  and this is a different issue that I'm looking at, the finding will look like this.

31:16
And it will have further details. So you'll see the severity, what the SLA is, and the status of this finding here is inactive, but the developers also have access to this type of information straight through their JIRA tickets.

31:36
So if we have tools which don't have a high degree of confidence, then we need to do some triage on our findings.  And when we're doing triage,  there's a few things there I'd like to mention.  If we have automatic triage, then there's nothing to do. There's automatic assignments, and they're pushed to JIRA. But we can track which JIRA ticket they're

32:05
assigned and what the Jira ticket numbers are.  If we need to do manual triage of findings,  the one thing that I really enjoy about DefectDojo is the fact that it has really powerful filters.  And you also have custom details or custom properties that you can apply to your functions, which you can then use.  In my example here, for example,

32:31
We have a JIRA assignee so that we know who tickets should be assigned to for a service and which project they should go to. And we also have information on what the source is. So the source of this project is GitHub.

32:48
Within DefectDojo, as I mentioned, have bulk capabilities. So if you have a lot of findings that you need to deal with, then you can use bulk edit to,  for example, mark false positives or mark things as out of scope if you determine things are out of scope.

33:10
You also have capabilities to merge findings. So,  um,  in my example here, we have a SAS finding on stored XSS in a single file.  And in my triage, I've determined that these probably could be fixed as one, uh, similar change or similar fix. So, from a developer perspective, I'd really like this to be one action on their side. So within DefectDojo, I have the.

33:40
capability then to merge these findings as one and pass it on to the developer as a single action rather than multiple.

33:51
So, from a service owner experience,  there's also value in DefectDojo.  And within customer intelligence, we have a lot of different service owners that are responsible for various different services within the application itself.

34:07
When they open DefectDojo, they can find their service within DefectDojo and see an overall picture of what's going on with their service. They can see how many findings are open currently, and they can see details about, for example, what their SLA is that they're expected to meet.

34:32
They also have visibility for all the findings that relate to this service. So they can go under findings and see how they're doing over there. And then they can see how they're tracking towards their SLAs, maybe, or what are the highest severity issues within their service. And then they have capabilities within DefectDojo to configure alerting based on their service. So.

35:02
For example, maybe they want alerts when an SLA is breached or when risk acceptance expires.

35:16
Finally, I'll talk a bit about the manager experience as well.  We can move up a level from the services level, and within DefectDojo, immediately when you log in, you get an overview of your total security posture state. You can see how many active engagements you have, how many findings are active,  and your historical finding severity.

35:47
And as a manager, I can also define different  SLAs per service, and we can then assign those to the services that we deem need a different  SLA. Perhaps you need a more accelerated SLA for internet-facing services.

36:08
And yeah, finally, I also want to mention ASVS benchmarks, which are a thing that we use at SAS to validate the security of our applications.  The service level provides benchmarks for ASVS, where you can do a simple checkbox exercise on where you stand with your application security posture from an ASVS stamp.

36:38
Okay, finally, I want to share some outcomes from our evolution, partly due to DefectDojo, but also due to our evolution of our security program. And then finally, I'll share some tips that you can take with you if you want to go on a similar journey as we've done, or start implementing centralized management of vulnerabilities.

37:05
So first of all,  with our evolution, we've achieved faster meantime to resolution. And that kind of meets the need of fixing things faster and fixing things sooner.  We've been very big advocates of a shift left mindset. So we've implemented security tool IDE integration so that developers actually can resolve issues even before.

37:35
They commit their code, and before they try to merge their code. So before our automated pipeline scans scan,  developers already have the capability to remediate issues.  We've introduced dependabot auto mergers,  which update things as updates become available.  And that solves a lot of issues with our SCA scans.

38:04
And with continuous development, obviously, we're driving towards smaller and faster changes combined with better tests. And that, again, accelerates our mean time to resolution from a security perspective.

38:22
With DefectDojo, we've managed to get less resource utilization from a developer perspective,  and partly also due to the automation. And we also have less findings to deal with because we're doing duplications,  deduplication within DefectDojo.  From a developer perspective, developers now only work on verified and triaged findings rather than

38:50
on all of the things that a security tool produces. And because of the automation, developers spend less time in JIRA checking boxes and closing tickets, and more time on developing features, and also less time on working with security issues, which should be the goal.

39:12
And from a manager and the security lead perspective, we've gained a lot better situational awareness of our total security posture. Uh,  and the the kind of capability to do vulnerability impact assessments, uh, based on metadata and based on data on the services that may have a vulnerability and.

39:39
Sometimes that's even automated. We don't even have to do anything.

39:47
Okay. A few tips before, um, before I end my spilling, get to the questions. If you're not doing centrally the central management of your vulnerabilities yet, you should probably start right now because it will help you as you, uh, mature and as you introduce more tools into your security tool stack.  And one of the nice things about DefectDojo is that there's a demo available.

40:16
where you can go in and start kind of introducing yourself to what this is about, if you're not familiar yet. And it's always online.  Details are here  : demo.defectdojo.org. And in the documentation, you can find the login credentials to the demo as well. I highly recommend you go there and invest again, DefectDojo, if you haven't done that yet.

40:41
One important tip before you start implementing DefectDojo is to give thought on how you structure your organization and DefectDojo. There's no right answer to this. It depends on how you're thinking about and how you're structured within your organization.  DefectDojo organizes things in a hierarchical fashion. So you have product types.

41:09
You have products, have engagements, you have tests, you have findings, and you have endpoints. And for us at SAS,  if we start from product types, we have categories of products that have slightly different criteria that they meet.  And those are what we regard product types. And products for us are the individual services that make up.

41:39
a product type.  Services have their own owners, and we've decided that a product in DefectDojo speak means service from our perspective. Engagements are the various different test types.  Software composition analysis, static application security testing, those are engagements or interactive engagements. It's a pen test.

42:09
The tests are the individual tests within that engagement type. And those tests then produce findings. So those are the actual security issues that are produced. And endpoints, depending on your test type, the findings may be associated to an endpoint.

42:32
And the final thought I'll say, if you ever think that I could use JIRA to do this,  and it's not something related to project management, then I highly recommend you don't do that because there's probably an actual tool that you can use to do that thing, like DefectDojo. So, using JIRA, for example, for application security posture management, probably not the best idea.

43:02
You should probably get a tool like DefectDojo to do that. And yeah, GRF, for example, probably not the best tool to drive a Formula One car, but I heard they're doing that as well. So on that bombshell, I'm done and we can get to the questions, and I look forward to meeting you and the jungle as well.