Allie Howe Hey, my name is Allie Howe. I'm the host of the Insecure Agents podcast. We're here today at RSA. I'm here with Kyle Bhiro and Josh here today with Pensar to talk about offensive security agents and what we're all up to at RSA. So, Kyle, would you like to introduce yourself?

Kyle Bhiro 100% thank you so much Allie for having us. I'm so excited to be here with you. I'm Kyle. I'm one of the cofounders of Pensar. We build state-of-the-art offensive security systems. Our team is here with us from New York. I'm, I'm excited to, you know, pass it off to Josh and we're gonna, we're gonna unpack some cool stuff.

Allie Howe I'm excited to talk. To you and Kyle's also the Co founder of Insecure Agents.

Kyle Bhiro Yes, you know, we started Insecure Agents just before recording, but we're relishing on a year of Insecure Agents and how exciting that has been. I'm so I'm so happy to be here with you, I think. This is gonna be fun.

Allie Howe Yeah, super fun. Josh,

Josh Kotrous yes, thank you, Kyle, and thank you so much, Allie, for having us. This is incredible, But yeah, I'm Josh Kotrous. I'm the CTO of Pensar.

Allie Howe Awesome. What has stood out to you so far at RSA? I know you guys are from New York. You flew in here. Why did you come all this way?

Kyle Bhiro So we do the circuit, we go to all the major security conferences. This is the start. Some people have been here for B sides and now flowing into this entire week and then you know, you've got your rest of like summer camp and Black Hat and Defcon and so forth. I have a little bit of maybe like a contrarian take, which I may. It's not too hot of a take. It just I'm going to say it really loud and people at RSA might yell at me. It's security on security here. We know that it is buyer on buyer and I think it's a great showing of what everybody's been working on for the last couple of months.

For us, we're here, you know, excited off of a pretty hot launch. It's a great time to celebrate that. And it's also just really exciting to see what other companies are working on in addition to seeing the people that I want to see some customer facing experience. That's that's that's why we're here.

Allie Howe Totally. And obviously, like I've been, I've known Pensar for a while because I've known you for a while. And this launch is a huge launch for you guys because it seems like, you know, a huge step away from or a continued step away from like SAST scanning. You were scanning LLMs and finding LLM related vulnerabilities that nobody else was like 6-8 months ago. We were doing like OS hackathons together. So you were definitely like ahead of the curve then. And now you were just released Apex's offensive security agent. That's better than models off the shelf. Are you still doing like the SAST scanning or like how do you see this like space changing?

Kyle Bhiro Yeah, and so the the point of maybe like our journey together, which is like well over a year, like touching into like 2 years now. I think when we like we first started working together, the narrative was AI powered SAST. And I think that's something that still plays its role and is important in the space. But to say it lightly, Pensar has evolved. I do want to spend a couple minutes like talking about Apex and I have like a mastermind behind Apex. Josh, you can give away better like intro and like why we decided to build Apex.

Josh Kotrous So yeah, as you, as you mentioned, we were largely focused on like AI powered SAST that's kind of on the 1st iteration of the company. And what we had found is, you know, SAST can still only really go so far and agents are only good as like the context that you're providing them. They're not mind readers. So something that we've really focused on with kind of like the offensive security agent harness and with Apex is kind of like this validation layer, having the agent actually forced or first validation in runtime by generating a POC, validating the output of that POC, and then iterating on it if it needs to and changing direction if it needs to.

That validation layer we found to be very, very important in reducing false positives to almost near 0. So yeah.

Allie Howe Amazing. Yeah, that's huge. And you said you've been trying it with enterprise customers. How's that going?

Josh Kotrous Yeah. It's been going really, really great. We've tried it on a few different enterprises, some like large financial institutions and we've been able to find like 0 day vulnerabilities in these customer systems. So those are vulnerabilities that they are brand new to them. They're critical level vulnerabilities and something that needs to be patched relatively quickly. And then that's actually like one of the metrics that they're measuring like our success on is if we're able to bring like a critical level vulnerability that is something that they haven't seen before and get it remediated quickly. We've had a lot of success in that space.

Allie Howe Yeah, wow. I went to a talk recently that Alex Stamos gave at a snooze neck read a security event a couple weeks ago and he gave a talk called AI is Eating Security. And he was talking about how models have gone so good now that they can find vulnerabilities that humans never found, like bugs in Linux from like 2003, for example. And basically the main take away was we're finding, you know, the old bugs, but also like new zero day bugs as well. And companies like especially like only a handful of companies had to like worry about a true like 0 day. But now every company does because everyone like the scale AI has enabled attackers to target everyone essentially with the breadth and depth available to them with AI.

So how does Apex help with this new emerging threat?

Kyle Bhiro Yeah, first I want to give a shout out to Snusek and Flee. If Flee actually watches this podcast, I think he does follow the podcast. This new tech is very cool. I think the question of adversarial and attackers taking advantage of agents and LLMs and maybe even just like the tools that we've, you know, gone on to like publicly released, it's a horizontally scaling threat. And so now to simplify, the attackers have access to the tools and a volume that maybe the good guys are unable to like keep up their defenses for. My Co founder actually put out a piece of writing around this and the I just have a lot of excitement more so from like our point of view and releasing Apex, that the tooling that we're putting out is going to be tooling that's in the hands of the good guys.

And I'd actually say that we've been lagging behind. Good guys overwhelmingly have been behind the curve because it is not a new piece of news that threat actors are using LLMs to bang against our, our, our interests. And how like Apex does that? If you want to like, make a comment on on that.

Josh Kotrous I'd say like one of the largest value props of this is with some of the organizations that we've worked with, they just have attack services that are just so large. Like if you were to go to like a manual pen testing firm, they can only scope it to be. So scope the engagement to be so large, whether it's time constraints, cost constraints, whatever. And what AI enables is we can really achieve full coverage because we can spawn, you know, a swarm of pen testing agents that run continuously 24/7 and every commit. And then that ensures that, you know every single change is tested consistently and that you really have full coverage in your security program.

Allie Howe Yeah, that's amazing. I know a lot of like practitioners listen to this podcast as well, like developers and security practitioners. And this space is moving so quickly and everyone is. Also, I'm trying to target software factories where my agents write all of my code, they review all of my code, and the humans are just out of the loop as much as possible. How does Apex fit into that world? Was it integrated into the agent harness? Are you testing it? CICD time? What is? Where does it insert itself?

Josh Kotrous Yeah. So really allow optionality so developers can fit it in whatever makes sense to their workflow. So we actually have like a locally hosted open source version of the product that can live in your terminal. And if you wanted to run a pen test on like your local dev environment of the application, you're totally able to do that. The way that we actually dog food it internally is it lives in our CICD pipeline. And in that way, every single time we deploy to our staging environment, we run a pen test against those applicable changes and then Pensar opens up issues within our within our repo for us to fix those with pull requests for the patches as well.

Kyle Bhiro To add to that, I think the mention of like practitioners in the audience that is like listening to insecure agents. The the reason that we've like gone deep on this like open source release and the interest and actually for the journey of the company, we started the company working on an open source CLI and that being almost two years ago, we've come back to this. What feels like the starting point. The reason why is a lot of companies in the space are taking a very opaque stance. They're not interested in and sharing and having their practitioners be able to play and like touch and understand these tools.

We fundamentally disagree with that. I think it's like really important one for maybe distribution purposes. If you wanted to actually like use these tools and get into the the hands of the practitioner, that's important. Two also just from an understanding point of view and who we're appealing to from a technical decision maker or developer, or if it's a security engineer, they have to feel it and touch it and understand how these tools work. Point on, you know, the people that are here at RSA, I think that's really resonated with a lot of people. We've even done a little bit of walking up to some of the perceived other players in the space who just not interested in taking a stance like this.

Allie Howe Interesting. Yeah. And also for the, I know people, decision makers listen to our podcast also that have budgets and are probably looking at this space going, OK, can I consolidate my budget now that SAST scanning and DAST and continuous pen testing now it's kind of all rolled into the same thing. What are you seeing on your end?

Kyle Bhiro I just had lunch with a security leader who is probably, I, I would say the status quo shrinking security budgets despite maybe like growing budgets in other areas, like growing engineering budgets. There are instances from other companies that we've worked with, we've actually seen growing security budgets. I, you know, applaud and feel those. I hope those companies are grateful to, to maybe have like an expansive resources and be able to procure more tools. I'll make an example for both, right? Let's say for what you mentioned, you have like your SAST solution, DAST solution and a couple other tools that are taking up pretty much all of your spend.

There's a point of view that from a decision maker perspective, it might be more economical to like RIP out two and replace with 1. And I understand that I actually saw this week like Wiz released like their red agent, which is essentially also in like the threat detection space and could be a solution that, you know, plops into like the Wiz environment and now is satisfying more coverage than just like cloud security, for example. I think that's going to like appeal to a lot of people. Now, on the other side, there are companies that have internal policies or maybe pressure from like their compliance and their Slas and their regulators that say, no, you actually have to have such a degree of like attack coverage.

You know, internal policy. I'm referencing like this one specific organization that we worked with that requires them to pretty much buy and have coverage for like every, every tool and every space, right? So they have a SAST, they have a DAST, they have like an automated pen testing system. They even have internal pen testers and an external firm, third party firm. The, I'm sure it's costly to say that, to say that lightly, but you know, for different reasons. Let's say it's just for like maximum coverage. The example that we produced earlier, which is like if we could find even one critical that everything else missed, it's worth it if that's something that you care about in your security strategy.

If for maximum coverage purposes, there's something that like checks as many boxes as like Pensar does, which does eat up a lot of like issues that we'd find maybe from like SAST and DAST solutions. That's great. I also do want to say that SAST plays its, its place, has its place, right? And these automating testing solutions, I think also have their place. Some are, you know, more critical to your security strategy than others. Overwhelmingly though, I'll wrap up with the conversation that we've had here at RSA. I think one of the themes that RSA this year has been this concept of like continuous security testing.

Allie Howe I agree. I think especially in the wake of the SOC 2 compliance drama, we'll name the company that like went down point in time audits have kind of lost their, I don't know, just like luster. I guess. I might definitely see this trend where we're moving away from like point in time audits because there's like agents change so quickly, LLMs changes, users interact with them, not just code deploys. I'll put it in on it really doesn't make sense too much, especially like a once a year thing. I mean, these systems need to be tested probably daily. I don't know, like what's your take on that?

Like every hour, daily, Like it's just all the time.

Kyle Bhiro Yeah. So on the the first item that you mentioned, which is a pretty dramatic example of, you know, maybe like the the SOC 2 company that will go unnamed in that specific example as well. Like how crazy is that? A company could have like an exposed like you could like exfiltrate data from a security and compliance company. It's like very dramatic. But to go. On to go on. And you know, maybe unpack like the frequency at which like what does continuous mean? There's there's one example that I want to bring up and maybe you have some thoughts on this too. So we met with a head of pen testing in a large like global pen testing organization here at RSA, and he made a really good example.

I'll leave his name out. And he mentioned that their manual pen testing business and the traditional pen testing model is hourly rates for a human, right? And so let's say you block off like 8 hours a day for a human and you're paying them for that rate, and that's probably a couple thousand, a couple, $1000 a day per human. Continuous model doesn't mean 24 hours in a day. You just take that amount and multiply it by three. It's not how it works. If it did work like that, pen testing would largely be not economical for a lot of customers. Now the way that we do it, if that's testing on a PR basis, do you want to jump into like?

Josh Kotrous Yeah. I guess like ideally in like the most perfect world you would be able to test your application anytime the code changes in a way that's relevant to security. So changes in in the business logic, changes to to the core actual application logic. That's possible nowadays, obviously with like continuous autonomous pen testing, but that's where we found like our platform and approach to be really successful is because like even if you get a manual pen test done, especially with the velocity that teams are working at nowadays, I mean, there's some companies that we've worked with that are pushing like a hundred 200 PRs a day between their team to the repository.

Like the week after that pen test is already out of date. Things have changed. You have either introduced new endpoints, removed endpoints that are no longer relevant, and then that testing is no longer relevant. So yeah, that's why we really, really believe in bringing and testing directly into your CICD. And then in an ideal world, which is again possible today, is being able to test every single security relevant. Code change. Before it ever gets pushed to production.

Allie Howe Yeah, that makes sense. What's one thing that you're looking to get out of RSA this year?

Kyle Bhiro We've been walking up and down the street and it is just a very reaffirming realization to see people who know us bumping into you through the conference, bumping into the other characters that like we were currently seeing work with through the year. RSA is one of those conferences where I get to see everybody in person. It's that's important. And then the second part of course is in beautiful spaces like this and Pensar also participates in that. We have our own like customer facing experiences that we put together. We're well known for our ground game in New York. We throw some of the largest like security and engineering events.

And actually we've been very intentional with what we're doing here at RSA. Little bit more low key. I've actually not announced a lot of what we're doing, just inviting like closed group, closed conversations, security decision makers, a lot of like, you know, if a company of the same size is looking to like get an inch on like what I think is worth doing, making sure chairs are filled with buyer personas. That is how to maximize value, especially when it's a large carve out of resources and spend, but also just time like being here. That's that's what we want to get out of RSA.

Allie Howe 100% like that makes sense. And for anyone that's in New York, you do do the best, like events, like security events there in New York, for sure. I always enjoy like participating in those and going to those. But yeah, I want to start back to your launch because I saw it on Twitter, like it went viral. Did you have like 2 million views or something?

Kyle Bhiro Yeah, yeah, yeah. So the launch was last week.

Allie Howe That's incredible.

Kyle Bhiro And it was just, I would say months and like hard work from the team actually throwing together the launch in like the days, like leading up to it. There were a handful of like user reviews and testing that we had done. Kudos to Josh, kudos to the engineering team. My Co founders spent like the two or three days leading up to the launch just making sure everything was like also bulletproof. But like the presentation of the launch goes really well. A lot of things launched last week. A lot of companies have made large announcements if it was capital or products that they're rolling out. The timing is very obvious.

It's ahead of RSA and the reception is very +1 of the things that we are leaning into and we want to do is obviously with like the open source push. We're working on building a community around that. And so we're inviting people to like come into the channels and talk to us. If that's through our Discord, if that's making contributions to the project. Very, very encouraged the launch in my book, very happy, very happy with it. And now and then also for us, we have a lot more that we're working on. There's more that we have yet to like talk about and you know, put out.

I hope that the cadence of shipping and features and exciting announcements from Pensar continues to be what you know, we call like a heartbeat. Exciting launch after exciting launch after exciting launch. Just keep an eye out for more exciting projects to come from US.

Josh Kotrous One thing that we're really passionate about is giving tools to empower these people to be as successful as possible and be as productive as possible and seeing like them, work with it, enjoy it, and like findings that come out of it has been really, really incredible to see and great feedback to receipt. Want to add something to that?

Kyle Bhiro There was a example. There's a lot of people like coming to us and celebrating and they're very proud of maybe they like gone and submitted a bug bounty or like Apex help them like rank in the top three on like ACTF.

Allie Howe One question I have is how are you staying ahead of like the model providers or what makes you so much better so?

Josh Kotrous It's not necessarily I'd say that we're trying to be better than the model providers. I think a lot of it really comes down to like a harness that we built that then wraps like the underlying model itself. Apex is compatible with like, so we host our own inference. It's Anthropic models under the hood, but it's through like private inference so that your data is not actually going to an external provider. But it's really about like the harness itself. We want to be able to get as much performance like out of the models. We think the model providers are going to carry models forward. They're going to make improvements there.

And it's really about like the harness that we wrap itself with to extract as much performance as possible. And just like how we actually do that is if you were to compare like Apex to if you're just used Claude code, for example, for the pen testing use case, there's a lot of like guardrails that they have built directly natively in the Claude code as well as behind the scenes. Unlike in Anthropic's inference that it will just refuse to do a lot of the things that with kind of the harness that we built, it is a lot less restrictive and kind of teeing off of that as well using our inference.

One thing that we've ran into with our own developers even is their accounts have like gotten banned or suspended from Anthropic because they're afraid of them doing things that are against the terms of service. And whereas like using our inference, it's not going to be a problem. So that's kind of how we're solving for that.

Allie Howe What's it like reacting to model changes? So if you're saying like, well, our harness gets around the restrictedness by a clever prompt, I don't know what it is you all actually do. That's proprietary for sure. I bet like when you have model change like how do you like know, like how you monitor that and like how quick can you change it?

Josh Kotrous Yeah, great question. So we actually have like all of our benchmarks that we've measured Apex against all open source as well. So let's mean that the teams put together, they're called the Argus benchmarks that acts as effectively our eval that we can run the agent against when the new model comes out and see how it's actually performed. It's performed right? Like we've compared it against like PentestGPT wrapper, some of the other like open source security harnesses that exist out there. I mean, we've had significantly better performance than those like on our benchmark. And our benchmark covers everything from relatively simple vulnerabilities to more complex networks, setups and configurations to be able to prove that, you know, this can actually perform well in the real world.

And then as you know, models change, new models come out, we can benchmark its performance and see see how it does. So right now we have those benchmarks released, I believe we cover like Haiku and Opus. And as these models get better and as scaling laws apply, it's it's only only going to improve, improve in performance.

Kyle Bhiro Something that I would love for you to maybe like touch on is like why we decided to go and build Argus and, you know, giving a little bit of context there.

Allie Howe What's Argus?

Kyle Bhiro Argus is our benchmark. Yeah. We've created internally at Pensar and the reason why we did that, maybe just give a little bit of context. There's a public validation benchmark from the Expo team that has been the standard in the space for a long time, despite Expo coming out and saying that this has been a project that has been collecting dust. So why did we decide to go and do that?

Josh Kotrous Yeah. So we made our own benchmark simply because the the ones that exist currently in the industry that a lot of things are being measured against just don't really represent applications, how they behave in the real world. So they're like in very small demo applications that don't have like any networking around them, have very loose controls in the 1st place. I mean, then also like a lot of them just like weren't working applications to begin with. So actually like getting the benchmarks up and running wasn't super cohesive. But then like the actual coverage of examples that they're testing didn't necessarily replicate real world systems. So that's where we started exploring, you know, how can we introduce like WAF rules into, into our benchmarks as well.

And some of these like more complex things that agent used to work around. Also things like multi chain attack and tests as well to make sure that OK the agent isn't just finding its first vulnerability and stopping there, but actually finding the vulnerability and then reasoning and thinking around how it can actually exploit more things based off of that vulnerability.

Allie Howe Amazing. Yeah, that's super cool. Do you have any like benchmarks from the community as well? Is it open source like the benchmarks?

Josh Kotrous Yep, the benchmarks are open source. Those are all within our GitHub, I believe they're linked in the Apex repo. But yeah, we've also tested that like against other tools that are open sourcing system on those benchmarks have performed even better than those. So that's been really, really cool to see. But those are fully open source and as long as as well as the results with like the agent traces as well and so fully transparent in terms of like the path of the agent took and and the results and how it. Got there.

Kyle Bhiro And we openly encourage anybody else to like test against benchmark as well. If for some reason somebody here at RSA were to come and like you know challenge Pensar for their number one spot on the benchmark. Very, very interested in like seeing that. I hope that the interest for like Us one open sourcing Apex and also Argus is to just push the entire space forward if anything but that.

Allie Howe That's amazing. No, that's super great. I hope you should do an event like that. Maybe like come and test the benchmarks. Amazing. Well, super cool. If someone wants to try Apex, where can they do that?

Kyle Bhiro Yeah, just go to our website www. pensarai. com. There is a little header at the top with a link that will take you directly to the GitHub. The GitHub link is Pensar is github. com/pensar AI slash apex. If you have questions about about it, We are an open book, an open door. Anybody at the team would be happy to talk to you, so feel free to reach out to myself. I'm all over on platforms. Josh as well, very generous with his time. So we would be excited to talk to anybody who has curiosities around the release.

Allie Howe Amazing. Yeah. I hope people engage with you in the conversation and then follow you on Luma 2 in New York.

Kyle Bhiro Yes. Follow us on Luma. We run one of the largest security and engineering communities in New York. Please join us.

Allie Howe Yeah, no, highly recommend. Well, thank you so much for coming out and I hope you guys have a great rest of your RSA.

Kyle Bhiro Yeah, Allie, thank you for having us. It's always a pleasure to see you. I hope RSA is good to you this year as well. And I'm sure we're going to see each other in the summer for the rest, for the rest of the circuit.

Allie Howe Absolutely.

Josh Kotrous Thanks for. Having us.