00:00.00
Jonathan Johnson
Everybody thank you for joining us on another episode of the podcast bright and early today for some of us some in the afternoon. Um I I'm one of your hosts Johnny we have Luke and Jared with us. We also have an amazing guest of a Nas. Um Nos please introduce yourself.

00:16.40
Nas
Ah, hello. Well I don't have a lot of interaction to do I am not not bench on Twitter I've been in infosec for more than five years now and I do some hot takes on Twitter that angry a lot of people on osds and stuff. Believe half of it is intentional and half of it is for fun. So that's it for me. Yeah, and I do a little bit of research on the side. Yeah.

00:37.86
Jared Atkinson
The the key. The key is to make sure that nobody is able to discern whether what part is fun and which part is intentional. That's like.

00:46.10
Jonathan Johnson
Yeah, the the trolling piece on Twitter is real. You just can't let people know that you're actually just trolling.

00:51.35
Jared Atkinson
Yeah, but that's ah Johnny likes to do hot takes on osds as well. So we could we could plan that for a little later in the in the.

00:58.48
Jonathan Johnson
Yeah, just kind of just kind of assume it like the 45 minute Mark I assume that like our audience is starting to like probably fall asleep because jar has probably gone on with like a 2 fifteen twenty minute rant to that point and so um I like to like spice it up with something pretty hot at that 45 minute Mark so prepare I don't know what it is yet today.

01:08.71
Jared Atkinson
Wow.

01:18.31
Jonathan Johnson
Haven't thought that far ahead. So.

01:20.41
Jared Atkinson
We It's whatever we're like divinely inspired to talk about the ah Okay, so so we had ah just to kind of kick things off we we had told you that we we like to kind of just go wherever the wind blows us I guess but to start off there was a conversation that we had with memmet.

01:20.97
Nas
We'll think of something.

01:23.55
Jonathan Johnson
Yeah.

01:39.36
Jared Atkinson
Um, cyber monk on Twitter and ah in the conversation. Ah you you started talking about the relationship between narrow and broad detections and that's something that I think a lot of organizations struggle with it's like what's the appropriate level of specificity. Maybe we'll say um. When you're building detections because there's there's obvious tradeoffs and so I I was curious kind of what your thought process or experience was with trying to manage narrow versus broad detections and whether one's better. You know all kind of that that type of conversation.

02:09.30
Nas
sure sure yeah, yeah well I have a a little story about this obvious issue. Well ah in the in in our previous company. The previous company I worked with. We had an edr solution. And and that edr solution didn't offer an alerting mechanism so we couldn't create any alerts and the the synchronization or the communication between the cm solution and the the edr wasn't that great because they didn't support each other. So basically when you enable all the rules. A bunch of false positive as it is obvious but when you try to tune it. You don't have this option to tune them so the obvious route we took is to open tickets with support and ask them about this specific issue on the on the side. Did a little bit of reverse engineering on on the edr platform and discovered that the the rule are kind of broad meaning they they don't specify enough conditions or correlation for it to be specific to your organization. For example, so when we ask. Support about that. They talked a they mentioned a couple of points. 1 of them is that ah, they are targeting a a larger market than Algeria so they are targeting Europe United States and and Africa and Middle East so they need to to make it work for. Almost everyone and we had kind of a special environment so that that got me thinking on the idea of broad versus narrow you know because ah broading a detection make it very useful to everyone but will generate a lot of force positive and require a lot of tuning but narrow it down. Required some ah level of expertise that not a lot of organizations that just want to buy an edr simply due to to to comply to some compliance stuff what will know how to handle. So yeah, this is the just of the idea that I was thinking about.

04:06.68
Jared Atkinson
Sure sure.

04:12.49
Jared Atkinson
Yeah, yeah, and I think there's a interesting aspect of like yeah Ed is this category and we kind of treat them all as if they're the same to some degree. Obviously we do competitions to see which one's best for us. But like we I think we think they're more similar than they are different I suppose. But the 1 thing whenever you're comparing things. It's always important not to consider ah necessarily how similar they are but like how different they are on the margins because the details are where things matter and I think the I think every edr vendor has a target demographic and so you had mentioned kind of in the. And the precall that it's like well it would be allowing for the tuning of detections or allowing for custom detections is really beneficial for for maybe folks like us that have a grasp of the different data that's available to us. Maybe a better idea of what the the different attack techniques look like and how to how to. Get an idea of how they work that type of thing. Um, but most most organizations probably don't have that right? and so when you're an edr vendor. You almost have to say who is our target demographic and like what are their requirements and and when you do that you may be foregoing what's. Best overall for your most skilled customer sets in favor of what's most usable for the broadest bit of market I suppose.

05:38.63
Nas
Exactly Yeah, how.

05:38.77
Jonathan Johnson
Yeah I think when it comes to like edrs and the alerting mechanisms that kind of are built in within them I think of that is almost like ah like a second benefit that they provide um obviously like the telemetry they provide is the first so what is like the raw telemetry um with. When I think of like alerting from edrs I don't think of like precise detections specific to my environment. Um, simply because again like they are a tool that's in multiple organizations. So like how if I if I'm doing one organization and have a specific edr vendor. And I wanted to make it specific to my organization. Probably what I would do I would see if they have like ah an api I'd feed that a api in my sim assuming that the sim could handle that and um, I'd probably like take their third party alerts and start to tune them like manually in in that analytic probably maybe. Um, but I think the question I guess is in my head is when it comes to an edr vendor would you rather them give you the configuration potential to be able to tune those alerts or do you think the value of the broad alerts. Um. Are are better just to get insight into something that's there because if if it automatically was tuning to your environment. What if something was bypassing that quote unquote tuning I guess.

06:59.70
Nas
Yeah, well well I'm of the mindset. Ah because you talked about ingesting it to the the cm and doing an api but I believe that the first step that anyone should do is to understand the alertsa are being generated and try to understand them because. You know there that there is no convention between edrs or av vendor on the type of alert. So for example, if I say alap brud force or but careba roing or whatever this alert this specific alert its definition differs from idr vendors to id vendors and without knowing the original idea behind that alert.

07:30.68
Jonathan Johnson
A.

07:36.50
Nas
Create some abstraction that is sometimes not necessary and it is bad for the long run because if I suppose that for example, let's say that ah the alap brute force and it is a basic alap brute force so somebody is testing against an specific account via aldap and the threshold is 10 or 5 but you internally.

07:52.71
Jonathan Johnson
Yeah.

07:55.64
Nas
You Define a specific threshold that is different from that. So at least you need to understand that part a little bit because ah in my experience at least I find it very interesting how the naming convention and the logic behind the alert Differ greatly. So There are some alerts. But when you look at them. For example, it's Mimicast Gen one but actuality is just a signature for the original Mimmicast and you have the the alert starting with Ml and machine learning and whatever and those are the most interesting to get ahead of the curve.

08:20.89
Jonathan Johnson
Um, yeah.

08:32.10
Nas
So Because for example, let's say that an alert gets Detected. An issue gets detected and you want to understand if this issue is truly a a real a real alert or it is a false positive without offering an inside into the alert Itself. You are blinded some way so you can't even ingest it and base your logic on it if you know what? I mean? Yeah, okay.

08:58.29
Jonathan Johnson
Yeah, yeah, yeah.

08:59.65
Jared Atkinson
Yeah,, there's ah, there's an issue with the opaque nature of built-in detections. Um, which is like you said it's like if we detect we like this detection says it detects curber roasting Buts The question is it's like how does it detect Curb roasting because they're like it's. Practically impossible for a single detection to be comprehensive enough to detect all iterations of curberosin that are possible and so and like and so there there needs to be some analysis like what you're saying into the actual mechanism that's being used or what the threshold is For instance. Ah, to determine that an alert should be fired so that you actually know what assumptions are being made So There's this in like kind of risk management. There's this principle called the better wrong than vague principle which is in any complex system which ah you could say a detection rule. Um, is a complex system right? because you're considering multiple variables and there's different situations and all that kind of stuff. Um you you must necessarily make assumptions because you don't understand the phenomenon well enough to to not make assumptions and and it's better for us to make those assumptions transparently so that. They could be criticized right? Um, it's better to do that than it is to not be knowledgeable about those assumptions and the problem is is when when alerts from Edr vendors are presented assuming that they don't present them transparently which most don't right? most consider their rules to be. Ah, Proprietary. Let's say or like intellectual property of some sort and so they and they don't want to share them and maybe ah, an alternative way to look at it is um, maybe ah I'm assuming their there'm like we know for a fact that they don't share their information I'm kind of assuming why.

10:31.60
Jonathan Johnson
Um, yeah.

10:50.54
Jared Atkinson
Right now like I don't know for sure but 1 might be that it's proprietary. The other is that potentially they think that if their rule their detection rules are known that gives the attackers an upper hand against their product potentially but the problem the problem is is that that leaves me as the person the customer. Um. Kind of like having to make assumptions based off of the name like you said it's like oh I detect kboost or ldap brute force. Well, it's like do you really? And what do you mean by curber curber roasting because what you mean by curber roasting might not be the same as me. It reminds me of carbon black had not to pick on carbon black but they they had a black hat. Had a big. Um yeah, as before Johnny was in the industry probably but um, they had a big display that said we we catch Mimi Cats or we stop mimi cats I think it's bad I think it might have been. We stop either way it doesn't matter we something Mimi cats.

11:30.30
Jonathan Johnson
Um, oh this is like ten years ago man dude sheesh

11:41.22
Jonathan Johnson
I'm pretty sure as we detect mimiette or something like that.

11:47.57
Jared Atkinson
And ah, let's just say it's we stop and and Benjamin Delpi of course you know posted on Twitter a picture of that and then he posted a picture of him running mimicats on a carbon black endpoint like a car endpoint that had carbon black installed and he's like well obviously you don't and the problem the problem is is that the concept of. We stop Mimi Cats is an abstraction layer because what do you mean by stop and what do you mean by Mimi Cats because like we we tend to act like mimicats is this thing that we all understand but we we don't necessarily understand it right because you could mean the exact version of Mimi Cats that's on the release page on github. And so you're stopping it based off of a hash and technically you are you are saying a truthful statement but that like obviously that's not what we like what we should expect. That's not a reasonable expectation and so you have to have a broader definition of mime cats and it's like okay well. Maybe we mean anything that is derived from the memecats source code and it's like okay well that's better than just this in unique version right? But ah like when I think of mime cats I think of the capabilities that mime cats uses right? Not necessarily derived from that source code but mimekats introduced this idea that you could dump. In plain text for memory among many other capabilities that it has and so it's like I don't I don't only want to detect mime cats I want to detect anything that implements that attack technique and so there's there's all kinds of different definitions and then like what do you mean by stop and so there's this There's this whole. Problem with this abstraction of if we talk about things without understanding how they work under the hood then we're going to make some bad assumptions or we we tend to make bad assumptions. Let's say and that that's a dangerous dangerous game to play.

13:32.53
Jonathan Johnson
Yeah I think I think when it comes to edr alerting and telemetry in general There's like this false sense of security that comes with it and I think what what I mean but when I say that is like well I also think there's an unrealistic expectation to the vendor as well. And maybe that's due to them the way they market or maybe it's the way that people view edr vendors like I think the majority time when people talk about alerting from edr vendors or at least whenever I've seen bakeoffs happen. They're like does it detect x and like cool can I like write that off as like this is coveted. Covered in our environment and doesn't correlate to a miter attack technique. The the issue with that is like again we don't know at what level of abstraction is that detection being applied. Also it's unrealistic I by the way I figured out what my hot take is going to be at the 45 minute Mark um, anyways, so ah.

14:22.60
Jared Atkinson
No boy all right? Keep it to yourself for now.

14:26.22
Jonathan Johnson
Um, ah I think also it's like it's unrealistic for us to ask the vendor. What is this detection doing because it is proprietary that'd be like me going to specter ops and saying like hey I want to see all your detections. You know what? I mean like it's just like not realistic I don't think I mean those some things are for her hey hold on hold on hold on hold you talk for 15 minutes let me talk for 5 we. Don't give me some nothing.

14:43.20
Jared Atkinson
Yeah, but when we give when what when if we make it I'm just saying I'm just saying that it's reasonable when you pay somebody for something that they you know are relatively transparent for it.

14:53.40
Jonathan Johnson
Yeah, but you're also pay. Yeah I'm not saying like the transparency piece shouldn't be there what I'm saying is like it's not realistic for like what stops someone from saying like taking a carbon black detection that they have built in and then next year in the contract ends they go to crowdstrike. Tell Crashchek how it's done or someone switches jobs and's like hey this is how Crashchek's doing it and the detection built out now. It's like I think we get so hung up and this is where I think the unreless expectation of edr vendors is applied. We get so hung up on the alerting that the edr is doing and not enough on the telemetry. That is there so we can customize alerting for our organization and environment so we can get to the abstract level of what we may want like I think if we did some type of I don't want to say the word reversing because I don't really think that's the correct correct terminology to use but some testing on the edr vendor to see like okay like. Kerberosine happens when I launch rubus and I happen to just run like Kerberos right? and it's like cool great like I have a very precise detector here. That's awesome. But like I think a lot of times when people look at that they sign it off like hey we have Kberos attacking great. But the issue is like it's not the abstract level that you may want and so it's at least a start and it covers some type of gap whether that gap is relatively small. Um, but we cannot expect edr vendors to cover a huge gap in our environment in terms of alerting I think that's where the telemetry is for and that's why you have to have another team. Creating Detections customized to your environment.

16:27.95
Nas
But you but you know I had I had an idea about this speaking of the detection because I agree with you? The detection logic is proprietary but but let's say that there is a way to express what she detect without revealing the logic. So Let's say that. Ah. I Want to detect that someone is dumping memory and I believe Specter Robes has an article on your idea of Capability Abstraction. So You see the idea of Capability abstraction similar to that. Let's say that the description of the rule shows something like that.

16:55.28
Jared Atkinson
Sure yep.

17:05.20
Nas
So you know at which level of abstraction. This rule is targeting. Let's let's say that I want to detect mi mika. So Mimikas has several layer of layer of abstract abstraction. So the rule only needs to say that. For example I detect at api level or I detect it at 2 level or command line level.

17:07.75
Jonathan Johnson
E.

17:24.81
Nas
And with this with this simple knowledge I can initiate discussion with the vendor. For example I can tell him that by the way I have other telemetry for example or I want you to enhance it to so that it can detect Api level abstraction for example, just to initiate that this.

17:39.42
Jonathan Johnson
Um, yeah.

17:44.11
Nas
Discussion. But when you say that? Ah, for example, any any vendor or any antiir or any anyone you go to he will give you a title for the alert and a small description about the concept. Not even the idea of the attack itself. So when you say Minimicads mimikats has a ton of functionality. So.

17:57.24
Jonathan Johnson
Um, yeah.

18:03.77
Nas
Which functionality of mimmicature detecting. So tell me that I'm detecting the yeah this is this is the idea that I want vendors to to to to start the conversation with and another part is that ah I believe this is my hot take for today. Infosecc Twitter is that ah I believe if yeah in physics.

18:04.82
Jonathan Johnson
Yeah, no I agree with that? yeah.

18:23.66
Nas
You know when you study physics for example and you yeah when when you study yeah when you study physics you start with Calculus Calculus hard and whatever. But when you when you grow up, you'll learn about string theory and the multi-dimension idea and and.

18:24.11
Jared Atkinson
Um I like where this is going.

18:26.71
Jonathan Johnson
You say physics and Jared's Jared's woken up over there.

18:42.41
Nas
You have ah a whole debate on that idea. But in reality only a small subset of all physics engineers or or physics are discussing string theory and that is what is happening similar in infosect data. For example, a bunch of researcher and academic people and people interested in the. Ideas how they work and not necessarily. They want the results are discussing a a a high level of something that most people want to acquire. So for example, ah, when you when you meet when you meet a lot of people and you say I have iriar here in Algeria for example I give you an insight here in Algeria.

19:07.70
Jonathan Johnson
Um, ah depth.

19:20.95
Nas
Where where infosec isn't that advanced from the blue team sign so red teaming is is advanced technically but from the blue teaming side. It still people are still learning about detection engineering and stuff so when you when you go by an edr you will never have the discussion of customizing the alert of of. Creating your own detection or for example, what is the telemetry level only a subset of people require this knowledge. So just imagine that and let's enhance this to the whole world. So let's say if if you are an Edf vendor and only ten percent of the people are requiring and. Chatting about multiple telemetry points and offering alerting and offering customization. But the rest of the world require only the default alerts and require only the edr and they are happy technically because they don't know what they are missing on. Happy with with the the edr provide and this idea of ah, very technical requirements and very low requirements. This contrast between the 2 is what interests me between in infosecctuator and the real world. So this is that.

20:30.85
Jared Atkinson
Um, yeah I think there's.

20:30.99
Jonathan Johnson
yeah yeah I think Jared and I had a conversation about this a couple weeks ago. Essentially where like and in Twitter in general, there's ah, there's a big push in at least the detection space. Um, where there's so much like conceptual talk that sometimes. That sometimes ah there isn't enough implementation talk on Twitter in terms of the blue on the blue side. Um I think that's why like in the terms of red team people generally view red team as being quote unquote more technical than blue because like the majority of blogs that go out are just like straight to the point cutthroat.

20:54.85
Jared Atkinson
Um, yeah.

21:08.50
Jonathan Johnson
Dry all technical stuff and then just move On. There's like the impact of that like great impact pone somebody. You don't really have to explain what the impact of that is into ah a deeper depth. So I agree with what you're saying there I Also agree with what you're saying about like the edr vendors I think there is a push that needs to be happened. Like expectancy of them to apply more advanced features. But then also give a better description like you like you mentioned like at least say at what level of abstraction is this alert doing like I don't necessarily need to know what your logic is what I need to know is like where am I covered and can I have confidence. And that level of abstraction in terms of your edr detection I Let Jerk go now.

21:48.67
Jared Atkinson
I I've got like 3 different threads to kind of wrap up so like I ah like on the like transparency of of alert logic like so I personally view my value proposition. So this may not be like a spectro ops view may it might be I don't know I don't. Haven't had this conversation explicitly. But I view my value proposition as what I will come up with tomorrow. Not what I came up with yesterday if that makes sense right? So like um, it's like yeah let's say Luke leaves and goes and or let's say Johnny leaves and goes and works for red canary. He.

22:22.91
Jonathan Johnson
Which happened. Yeah.

22:25.83
Jared Atkinson
He could take things that he developed at Spector ops and implement them at at red canary which you know I don't know that we're competitors necessarily but potentially somebody that has an mdr may be less likely to hire us to consult them on how to build out their detection program I don't know um, but but like yeah, that's great. That's fine. They. They may have caught up in some sense. Maybe they're ahead in another sense right? but but like I'm going to come up with new I I'm confident in my ability to come up with new ideas tomorrow that I don't have today right? and so that's that's my value proposition and so that's why I like I don't. I don't like the argument of like the intellectual property like oh well somebody can steal this and go work for like take it to a different edr vendor because it's like you shouldn't be. You shouldn't be sitting there. You know fat and happy and you're in like in your current state. You should always be striving to move forward right um. There there is potentially an argument that if all the detections are public then that gives attackers the ability to navigate the detections like easier. Let's say um, maybe that just means that you need you need more comprehensive detections or maybe you're. Your detections are too superficial if that's easily achievable by an attacker. But um I mean that's that's a much deeper conversation. Let's say we could which we could get into in ah in a second but like ah I think I think an interesting aspect that you brought up Nas was the idea that there's.

23:46.23
Jonathan Johnson
That's my hot take by the way.

23:56.80
Jared Atkinson
There's a spectrum of resolution with which you could look at any problem right? And so like I like to say that it like it's from the subatomic level which is like the most specific level that we know of and there may be further but like in physics we don't have We don't have the ability to go further than Subatomic let's say right?? um. And then there's the cosmic level and um, you may be looking at the same phenomenon literally but depending on the level of resolution with which you look at the Phenomenon you're going to have a different experience or maybe see different things right? So um, there's this book that I read called The. Ah. What is it called an ecological approach to visual perception and I think visual perception is really interesting to me because I view Edr Telemetry or just like telemetry collection in general as equivalent to like the human perceptual system. So like we have human beings have sensors like our eyes and our ears and our. And our tongue and our nose which collect telemetry from from our environment similarly to you know if you you could translate that into like infosec where we have edr sensors which might be our eyes and you have network security monitoring sensors which might be our ears right? So you have you have these sensors similar to humans right? and. Those sensors are targeting a specific level of of resolution right? So like our eyes can See. We can't see subatomic particles with our eyes but we know that they're there because we have certain tools that allow us to do that similarly like we can't We can't see you know galaxies or I'm not like a big astronomy. Person but like we can't see things out in the cosmos without these gigantic telescopes right? But they enhance our ability to see different things. The question is is what is the appropriate level of analysis for the job that we're doing so like like you said sometimes we get in this argument where we're talking about the subatomic level. But the prat like the level that we should be practically working at is maybe the level that we see naturally using the edr right?? um or using our eyes. Yeah.

25:55.87
Nas
Exactly and if you'll excuse me there is there is the idea that is often discussed. It's about the command line detection. You know I had I had a project called Malcia Malicious Command line on github and the idea behind this project I got it.

26:04.40
Jared Atkinson
Oh man. Thanks.

26:10.20
Jared Atkinson
Um, okay.

26:15.35
Nas
Because I was reading a lot of reporting from fire eye from rat canary and the bunch of people out there and most of the time I don't want to say 100% just not to be wrong, but most of the time the the the attackers are using.

26:27.44
Jared Atkinson
Um, yeah.

26:34.50
Nas
Command line arguments without. For example, we we often have the discussion that we can obfuscate or we can remove parent child relationship using Api we can change the command line by looking at the structure of the of the p web structure for example, but.

26:43.16
Jared Atkinson
Um, yep.

26:53.46
Nas
Oftentimes this isn't used at all. So for example, you you use tools like nl test or I don't know Reg ah any any anything in old bus project. So most of them I use with command line. So.

26:54.35
Jared Atkinson
Um, sure.

27:04.70
Jared Atkinson
Um, sure sure.

27:10.76
Nas
Between this idea of Subatomic and cosmic. Ah, oftentimes we lose the the sight that if we implemented for example, 50% of our detection work Command Line. We detected 90% of the attack Because. Ah, even if the attackers are ah really Using. For example, advanced to advanced route kits and advanced bypasses and whatever 10 percent of their arsenal is using simple tools that we can detect using Command line and. Having this discussion is very very useful if you are in this level in this subatomic level. You're talking very specific very narrow research. But if you're a a normal guy command line will sometimes suffice in quotes to detect most at act.

27:48.30
Jared Atkinson
Um, yep.

27:57.73
Jared Atkinson
Sure.

28:03.61
Nas
But any level of the yeah.

28:03.90
Jared Atkinson
Now following like that better better better wrong than vague principle. Um I think it's and I think what you say is generally true, right? So like you you will probably especially if you're not like an organization that has a super tight risk profile right? So um. Kind of depending on depending on how you view your risk. But I I think you probably using command line will detect most attacks. But I think it's important I think one of the reasons why like I tend to get into the the let's say the subatomic level um is because it's important for me to know what assumptions I'm building on right. And so if I'm if I'm building my detections around the command line. It's important for me to know those things that you talked about like being able to spoof the parent process parent process to evade some sort of like relationship type thing or the ability to do like all the offfuscation that Daniel Bohanan has talked about. For instance, um, because it's I guess it's it's important to know what assumptions are being baked in so that you can assume the risk properly if that makes sense right? So like because it is possible. It is possible for an attacker to you know ah. Get access and still information without using command lines in any meaningful way and so like it's okay to assume that you will catch most things using the command line but you should know that you're making an assumption if that makes sense.

29:34.16
Nas
Yeah.

29:36.32
Jared Atkinson
And so like you don't want to you don't want to be doing it ignorantly. You want to be doing it with full knowledge of hey like we're going. We're going to do the best we can with what we're given. Ah, but we acknowledge that there is like a subset of possibilities that that we're going to miss.

29:49.90
Jonathan Johnson
Yeah, and ideally there's like some type of detection layered approach that's happening there right? So if like if something is quote unquote bypassing 1 abstraction like layer of the detection then you have like a lower one. So look. Let's let's step away from like the conceptual idea here and.

30:07.62
Jared Atkinson
Um, now boy.

30:08.13
Jonathan Johnson
Look into the practicals like say we have um I don't know take anything some Powershell command light right? and we are looking for I don't know some something I don't really know. Let's just make something up here. Um, you could you could do like a signature. I guess I guess not for Powershell, let's say if that was mimmicats right? You do ah, let's talk mimmicats I guess since that was like the theme earlier you could do a signature on mimmicats itself on the hash. Great. That's that's easily bypass you take a layer lower. You could do like a command line right of let's say like dc sync great cool. Okay, that's probably going to be easy bypass someone as offs usecation.

30:30.11
Jared Atkinson
Are.

30:43.95
Jonathan Johnson
Maybe there's some type of amsy that can be applied there. Um some type of amz logs which I love amz logs I think that's like ah a telemetry source that's like commonly overlooked. Um, then we can go a layer deeper cool. Okay, so like let's look at like the network protocol slash rpc that's being called or even the api level. Um, layer because at that point like if we can get to limitry into that piece that is a broader scope. So like what is the number 1 Api that is used or what is the Rpc call that is used for like Dc sync. It's yeah nc changes. That's what everybody cares about and there's like these other little things that we can do in terms of like. Um, the object in the dns class that could like limit the scope to be like um and also like the extended rights. But let's just look at like if we're looking specifically the ah the rpc layer so get and see changes. Um, the issue is like get and see changes. It's called anytime you do like um, a group policy update as well. And it has to call it to the domain controller. So if we only like layer if we only call and look for that then like that's going to be a pretty broad detection and you're going to see if that's going to happen a lot from workstations in general. So then? Okay, now the question is we has to the broader we go I think. Generally the more tuning that might need to be applied for the false positive or signal tonoise ratio not the false positive false negative route but more of like the signal tonoise ratio that might be applied there. Um, where the higher the abstraction that you have a detection.

32:11.40
Jared Atkinson
Um, yes, yeah.

32:18.32
Jonathan Johnson
The less tuning you probably have to do because like the tuning is already implicit inside of that logic due to the fact that you're looking for something as specific as a command line. So I think like when it comes to these different layers ideally and this is a concept that I don't think is very well like.

32:25.33
Jared Atkinson
Um, I think that's right, yeah.

32:37.20
Jonathan Johnson
Applied and worse organizations and I think this is the reason being is due to resources I don't really know I'm not in every organization but I could foresee that happening I've seen this happen at clients when I worked at spectorops right? is like you want a detection for X. Great rewout a detection and then someone might be like hey we need to do like. Ah, lower abstraction level to kind of like do some detection layering here now you have limitations you have telemetry limit limitations. Um, you might have tooling limitations the sense of like sims because like the majority of sims in my head or like not great like I hate Splunk For example, um. Also hate like elasticsearch in the terms of like utilizing um like lucene because of correlation Issues. So I don't want to say I hate the products I Want to say like they're lacking terms of like analytic correlation and so like that's why I like things like cousto.

33:17.58
Jared Atkinson
Um, because of because of correlation issues right too late.

33:32.84
Jonathan Johnson
And I like things like Jupiter um, right? like those are my ideal things because I can bring in whatever telemetry source I want I can start to manipulate the data anyway I want in order to make it fit and now I can do broader correlations that need to happen if you're going to do a broader detection strategy. Um. But again with those like lower abstractions. Um, there are gonna the the noise the signal to noise ratio could be potentially higher so you have to like keep that in mind but the layering piece isn't happening a lot of times and that's because the deeper you go in that abstraction the more like ah. I don't know how we're going to be able to get this to work in a resource non resource intensive way.

34:12.98
Jared Atkinson
I think there's like ah there's a layering issue. So like I think too often we view so like Nas are you familiar with the funnel of fidelity thing that I wrote up. Okay, so um, the the idea there of the funnel of fidelity is that we have multiple phases in the detection process right? um.

34:21.51
Nas
Um, they are.

34:32.36
Jared Atkinson
And the problem is is that we have a relatively unlimited or unquanttifiable problem set that we're dealing with which is like people are trying to attack our organization. Let's say and we have a finite number of resources that we have to figure out how to apply properly right? Or as best as we can. Um and I think. And detection is one of the phases right? So there's a phase collection which is how do we collect telemetry to know what's happening with our within our environment and there is there like there's more telemetry available out there than we collect right? So there is a filter there right? We're filtering out telemetry which we deem to be non-valuable and ah. And like if we had more compute resources or more storage resources. We would potentially collect more telemetry right? So there's there's a finite set of resources that limit what we collect then there's detection which is I have all these raw events right? And maybe this is. Maybe this is implemented within your edr or maybe it's implemented within your sim. So different organizations are going to implement this differently. But it's like I have all these raw events how do I identify the events that I should start spending greater amounts of resources on which are typically becomes like human human analysis that type of thing. Um, and I think one of the problems that we do is we view detection as being the phase in which we discern malicious from benign right? Which first of all, what do those words even mean but like let's just act like we all understand what it means as opposed to what I think we should be doing which is identifying. Interesting or unexplained events. Um, and like I don't like at detection I don't necessarily care about whether or not it's malicious I care about whether or not I want to invest more of my finite resources into investigating it if that makes sense. And that and with that perspective it allows me to ah have a broader approach because I know that I'm going to whittle it down later on through different different phases and the the key of the multiple phase approach is that you want to be applying as. As little resources as possible to gather additional context right? and so things like soar app like soar soar platforms that allow you to automate the collection of additional resources are really valuable so that you can kind of push the human intervention. Let's say or human analytics. Ah. Further down the pipeline so that you're dealing with less like humans have to deal with less and less or maybe more and more of what they what they should be doing that's predictable is done for them because humans are probably your biggest bottleneck as far as resources I would imagine in most most cases.

37:11.76
Nas
I yeah on that idea I will I will tell you this from my experience. Let's say. For example, you had a team and that team isn't an expert on detection and the idea that I developed is we shouldn't impose a level if you know the pyramid of pain. So. Level of detection. Yeah, so so let's say that you had a team and that team cannot invest in knowledge or gaining new information because there is there are a lot of tickets opened ah opened and there are a lot of issues. So I believe that we shouldn't impose. A a certain level just because there is a level above it or below it. So I should say let ah let let me let me explain it more more thoroughly. So we spoke about api level detection and command line level detection and the the the distinctionion between the 2 and you spoke about.

37:56.44
Jared Atkinson
Sure sure.

38:04.16
Jared Atkinson
Um, yeah.

38:09.29
Nas
The idea that you should know what you're doing and I can argue that even if you don't know what you're doing. You should always go for what you know, even if it is bad. Yeah, because because just simply by enforcing the idea that your detection will be bypassed.

38:10.64
Jared Atkinson
Um, yeah.

38:18.34
Jared Atkinson
Yeah, well true. Yep.

38:28.84
Nas
Is a better assumption to make because in my opinion and and I will tell you why because the moment I say for example that my command line level detection is bypassable. Of course it is bypassable but maybe in my whole career in that company. Nobody would have bypassed it simply because it's by ask for there. Go ahead.

38:49.30
Jared Atkinson
Yeah I guess I guess the problem is is that if if your perception is based at the command line level. You actually don't have the toolset to discern whether or not it had been bypassed.

39:02.35
Nas
Yeah, sure sure sure sure yeah you up up I Agree agree agree, but but my ah my my my idea is let's say that I'm unaware I should do what I know for now and later.

39:03.44
Jared Atkinson
So like I don't think that I don't think that you can actually you can't You can't explain the negative. Yeah, okay.

39:15.49
Jared Atkinson
Um, yeah I agree with that as well. Yeah.

39:21.90
Nas
When when when I grow and when when I learn more stuff I should implement other stuff. So the idea is that because the the discussion is always limited for for example, ah someone share a a a command line or for example, Sigma rules a lot of sigma rules are based on on Command line detections.

39:39.60
Jared Atkinson
Um, yep.

39:40.50
Nas
And the people argue that there that I'm not effective or whatever but I say that go for it and on the other hand build another layer that will help you on the long run. Yeah, that's that's the idea that I was going for.

39:46.86
Jared Atkinson
Um, yeah, yes, yeah I think I agree like ah something is better than nothing and yeah.

39:56.32
Jonathan Johnson
Yeah, the eighty twenty kind of thing like I think I think in that piece too. It's like if you if you implement something you don't know nor don't understand and it happens to be wrong and someone like questions you on that like you you don't have any integrity behind that you said oh some.

39:56.95
Nas
Yeah.

40:15.38
Jonathan Johnson
Some person I follow on Twitter that has 36000 followers did it so I just assume that they were right I assume nobody's right typically like I just like until I test it and I can see it with my eyes or they unless I have like a really good blog post like I'm just like yeah hey yeah like that looks cool, but how can I how is that impactful how is that like.

40:21.10
Jared Atkinson
Um, yeah.

40:33.81
Jonathan Johnson
Do I really truly understand it because if you go and implement something that you truly don't understand and say you leave one day then that thing is stuck there and it could be doing your organization more harm than good because it gives you a thought of security where it's like hey where I'm covered in x abstraction of y technique. And I were like hey guys we're good. But then you have like a red team's like Yail. We've been. We've been bypassing this for the past like 5 years like what are you doing? you know what I mean and so like I think like and not being able to explain the logic behind that is also an issue. Um, and so that's why I believe like it's like if you're new to a subject. It's it's good to play around with it but don't implement it. In prod until you feel like you you truly understand what's happening.

41:12.92
Jared Atkinson
Um, okay, 2 2 3 2 things to kind of ah so I agree that something's better than nothing my criticism not of you but ah in general of like I've observed that people get stuck at the command line. Ah. Level of analysis. Let's say let's say that if we're talking about the subatomic to the cosmic. Let's say command line is kind of like the most. It's the most obvious level so it's the level that we're like pre preordained. Let's say to to already pay attention to and so what my observation is that people are happy. At the command line level without ever considering what's more if that makes sense so like my my warning is the the command line level will solve I agree probably 80% of the problem and like it's it's a good good place to start, but you should always be. Concerned in general even if you can't explain why about where that fails if that makes sense because the the general like I tend to be more sensitive to false the threat of false negatives because by the by their nature. You don't know that they exist right? And so like you have to. You almost have to assume or act as if they do exist because there's nothing that's going to tell you that a false negative occurred or that somebody bypassed I mean maybe you could you could say that maybe nothing's not the right thing. So like if you get ransomwaed then there was a false negative obviously like if you didn't know about it right. But there is a subset of attacks that won't raise that that threshold to where it's obvious to you or um, what what did you just talk about Johnny because I wanted to comment on that as well.

42:54.96
Jonathan Johnson
Ah, like the unknowing like the not truly knowing the subject that you're implementing in your organization.

42:58.50
Jared Atkinson
Give me more than that. Oh okay I think there's ah okay, so this is this is maybe my criticism of like ah machine learning approaches in general. But I think it it applies more generally than that as well. So it's it's not unique to machine learning. It's just something that. Ah. Is very common when people implement machine learning. So there's this There's this idea that it's like I've run into customers that have implemented some next gen machine learning type software that's supposed to find detect detect stuff for them and I'll say hey like you know couldn't we do something. Something a little bit more simple in that to start with and they say oh well, we needed something and the problem. The problem is is that it's almost like the I think it's the second law of thermodynamics to where there's like a conservation of energy. Um, kind of concept to where um, there's a conservation of energy across the funnel and to some in some regards to where there's like. How much there's amount of energy that you put into building the detection analytic versus how much energy you put into triaging the alerts that are generated from that analytic right? and 1 of the problems that I see is that when we use machine learning machine learning is very opaque to the especially in the implementation that vendors. That vendors typically typically use and so what ends up happening is like and I'm I'm being hyperbolic to some degree but it'll produce an alert that says this is weird right? And so the cool thing for the for the consumer is that they've now ah like they've now delegated all of their detection responsibility to the machine learning platform. But the the burden of investigating those alerts is gigantic because it doesn't have the necessary context for an analyst to like it like when you have when you have a detection that says hey, ah this hash which we know is associated with Mimecats was found right.

44:37.30
Jonathan Johnson
In here.

44:52.16
Jared Atkinson
Ah, you have the context to know exactly what happened right? like it's It's not hard. The investigation is super easy because you did all the work on the detection side to find the hash and all that kind of stuff and so it's like okay well I know that it's memecast because nothing else would have that hash. Um, but when you have machine learning and like obviously this is. This is a layman explanation of it. But it's like hey this thing is strange. It's like well now that now it's your job to figure out why it's strange and whether you should care about that strangeness and that's not that's not an obvious process in my opinion and you're basically just kicking the can down the Road. Um.

45:28.17
Jonathan Johnson
Yeah I I have a very like weird take on machine learning in general because I almost feel like an I yeah let me eat preface. Yeah I like I think machine learning like could be great 100%

45:28.91
Jared Atkinson
In that case.

45:36.62
Jared Atkinson
I Think machine learning could be great by the way. So like I'm not I'm not trying to take a dump on machine learning I'm just saying that that's.

45:48.10
Jonathan Johnson
I think though the majority of organizations that are trying to implement machine learning aren't at that point yet, they're trying to overengineer a problem. Um because I think it sounds cool and I think it's going to be the the solution to all their problems and I think like the reality is. Majority organizations need to learn how to utilize their telemetry towards its full capacity and capability. Um, first in maybe other policies in place like preventative policies preventative mechanisms before implementing machine learning like for example, like if you have a malicious actor in an organization that's using ps exec. Just you know, straight out of the box. But then again, you're allowing every admin and and under the sun in your organization to pus exec well like machine learnings are kick pick kick like pick that up. But I'm assuming that's going to be people's kind of thought process is like oh like it's gonna be 1 of those weird nuances. So I think like. That's kind of my take on machine learning I think it could be very valuable I just haven't seen it implement it in a way practically that is super super valuable today.

46:52.45
Nas
And yep I agree I agree in my experience the machine learning that I encountered are beneficial from a research standpoint because sometimes that you have a binary and some av vendor or some ml algorithm flags it. And the first question is why and when once you understand why you will learn the logic behind the vendor and how they are training their algorithms to detect these types of these types of things. For example, one one day we received a case and that case was someone who was using an a. Generally developed application in visual basic and it's very old. It was developed in 2004 or 2003. So ah, the ml algorithm flagged it. So when we looked at it because it was using a known packer. He decided to flag it. But once we reported it to the vendor itself. It only took 24 hours and it wasn't detected anymore. But the issue is ah the solution was that they didn't remove the detection for the packer but they created a edge case for that specific. Specific binary which could be a double at sword to be honest, but it is very something. It's something very interesting from a research standpoint to see the idea of how can you detect malicious behavior based on training something over and over again. Yeah.

48:10.11
Jared Atkinson
Um, sure.

48:12.64
Jonathan Johnson
Um, yeah.

48:25.58
Nas
In the future. Maybe it would be the best thing ever made. But for now certainly.

48:27.34
Jonathan Johnson
Yeah I think I think to what you're saying I think you might agree with this but I talk about this sometimes is and it's like there's a difference between research telemetry and scalable telemetry and like I think like that commonly isn't understood across the board. Um.

48:37.31
Nas
Um, um, careful that sure.

48:47.33
Jonathan Johnson
Because people are like oh if like machine learning can do it then you could do it in your organization. It's like I personally today I don't I don't necessarily trust ml out of the box to really help be helpful.

48:56.92
Jared Atkinson
So yeah, so I view. Ah the research telemetry thing like and ml's actually gray at this. So if I'm trying to to discern for instance like what? what What makes a malicious service right? I have a number of just intuitively I have a number of different things. So like. If. It's set to auto run maybe that makes it more likely to be malicious, but obviously there's like tons of Auto Run services that are not malicious and so that's not enough. Obviously then it's like if it was created from a remote system then that that might be something that makes me suspicious about it if it was um, but that you know doesn't necessarily mean it's bad. Um, if it was created by a process other than services e xe that means that they didn't use the api to create it and that's that's suspicious to me. So there's there's a number of different things that but like um I don't know how much like what the weight of each of those things should be each of those different features will call them. Um. And I also don't know if there are potentially features that aren't intuitive that are actually more important and so one of the values of ml in my opinion and like obviously this is not the most like ah like I'm not super knowledgeable about ml in general so like take it for what it's worth But I think one of the one of those super benefits. Of it is that you can start to discern what features or like what contextual bits of information are most valuable or like most valuable to solving the problem right? So um to me like something being auto run one of the reasons why attackers use services for ah. For their attack is for persistence purposes and so like in order to do that you would want to set it to Autostart and so that seems intuitively obvious that that might be something that you would look for um but I could be wrong, right? because I only have so much experience with malicious services but ml in theory would be able to account for more.

50:53.25
Jonathan Johnson
I Don't know I almost feel like I disagree slightly and the reason why I say that is because I feel like if you have someone who's researching a topic and implementing like a detection to prod. They should be able to discern all these different variants and why an attacker at least I don't say all a good coverage or a good like.

50:53.62
Jared Atkinson
Um, okay.

51:13.41
Jonathan Johnson
Breadth of knowledge of the technique and why an attacker might do X versus Y as a variant or like some type of contextual piece there like to me Ml in this instance and again like you said I'm not an ml guy like I don't like look into it a lot. But. I Think the only reason why Ml might be applied in your scenario is for baselineing purposes in your organization like how many auto runs would I be triggering on versus like I don't see necessary What the difference between your scenario and someone researching a topic really really really well and implementing the detection would be.

51:41.10
Jared Atkinson
Know that the yeah the the the problem the problem is is that researching the attack only shows you half the problem right? because you have to also consider normal behavior. So like you can't you can't You can't say what like.

51:55.81
Jonathan Johnson
Um, yeah.

52:00.45
Jared Atkinson
What is common amongst all attacks right? That's easy. That's an easy problem. The hard problem is saying what's common amongst all attacks. That's not common amongst all benign versions and like and you only have so much so much cognitive capacity to process it like I like machine learning allows you to run that.

52:02.61
Jonathan Johnson
Um, yeah.

52:10.92
Jonathan Johnson
Um, yeah, still the.

52:20.50
Jared Atkinson
Problem millions of times so that's the that's the benefit like your your your ability to analyze. The problem is is here machine learning is theoretically you know here or whatever.

52:22.51
Jonathan Johnson
Um, yeah.

52:30.91
Nas
Yeah, regarding Ml I have yeah I have I have an article I've written on on an Ml engine for Cymantec called Criteria and the idea I could be wrong, but my my understanding of the semantic process.

52:32.61
Jonathan Johnson
I Guess yeah.

52:49.44
Nas
They have a couple of engines so they have signatures and after signatures they have some some behavioral stuff and whatever and the final piece of the puzzle is an ah ml engine that works on something called features. So as as you were saying so a lot of features and. Let's say that a a specific binary or a script bypasses all of this the final piece of the puzzle is criterion and this criterion because it was written in Java so the the features were were really interesting. Some of them were basic. For example, if the keyword was network or virus. The fw wordd. So it's typically a a virus and they I believe they have 36 or 40 features they sum all of those and pass them to a to a an um ml algorithm and this will spit out to a binary results.

53:34.99
Jared Atkinson
Um, yeah.

53:46.94
Nas
So it is malicious or not It was very interesting how easily by possible. It is from a from a technical standpoint but it is very interesting to see that according to them apparently those were the features that are that are most interesting to discern. Between a benign binary and a malicious one. So even the keywords are sometimes interesting and I believe I don't know if if it's align with your point Jared but the idea that ah ml training an algorithm a lot of times with some specific features. Reveal a nature that is hitting to us because we cannot analyze a lot of binaries at the same time and with a considerable amount of time so training them over and over again over time will reveal a hidden nature of benign behavior versus non benign behavior I believe.

54:25.65
Jared Atkinson
Um, yeah.

54:40.62
Jared Atkinson
Yeah I've been like really interested in the like relevance of myth or like religious myth or just like I guess all all myth is somewhat religious depending on your definition of religion. But um and how like we tend to since the enlightenment. In the west at least we tend to view ah truth or reality as propositional, right? This idea that you have to be able to rationalize or build propositions to describe what is true and kind of and that's like the fallacy that I think we fall into too much when we're doing detection because like you said there is it's um. It's easier to know so like I have a 2 year old right? and like you might say hey can you do this and they'll say why and it's like sometimes it's really hard to answer the why question um, and but it's like I know that you should do that because I've observed that that's the right thing to do numerous times but I can't tell you exactly what about that makes it the right thing to do. And like when we when we look at like stories or myth or like religious religious stories. Um, there's like an underlying ethic. That's like kind of almost evolved over time through verbal tradition and what people tend to keep and what people migrate towards and all that kind of stuff that says that like. We don't know exactly what the lesson is here and we don't know exactly why it's important but like it wouldn't be it wouldn't have it wouldn't still exist if it weren't important or there weren't something to it. Um, and so there's there's always there's always more to reality or more to truth this is at least my my understanding of like this mythological approach is There's always more that can be revealed from story that you can't necessarily rationalize or propositionalize and like you kind of just reminded me of that idea when you were talking about um that like the hidden idea but there's also from a technical perspective. There's this whole process of feature engineering which goes this is actually what caused me to think of this kind of this point in conversation is when Johnny said hey there's a difference between research telemetry and production telemetry um like production telemetry is inherently limited based on what's um. What edr vendors choose to give us right and there's actually this weird thing to where they actually like kind of converge on what they choose to give us bait. Yeah correct, there's a there is a delta there. There's a difference between what they can give us and what they choose to give us right and there there's also a difference between what they can give us now and what they can give us.

56:53.81
Jonathan Johnson
Choose and can give us. Yeah.

57:05.79
Jonathan Johnson
Yeah.

57:07.40
Jared Atkinson
In the future Potentially um, and the reason why they give us what they give us is because that's what we want them to give us right? but that doesn't mean that we're right like ah basically like their their features are ah very often driven by Consumer Consumer wants right.

57:16.14
Jonathan Johnson
Um, yeah.

57:25.60
Jared Atkinson
But like consumers aren't necessarily the source source of truth of what's what's best right? And so we have this weird thing to where our perception of like what is a service is colored by the telemetry that we collect from our Edr which may not be the best to like the best way to look at. Ah, service for instance and so that's where the the research idea and then like I think that could be mixed with some so something similar like I think that you could like obviously you're going to have way different results. So like I'm not trying to trivialize machine learning but like to Johnny's point I think you can basically do a low grade version of what machine learning is doing.

58:01.98
Jonathan Johnson
Yeah, yeah, ah I think I think yeah to that point Jared like again like let me preface like I think machine learning has a place but what I was trying to get at was like at a smaller scope at a smaller dataset I think a human could do that cognitive analysis there.

58:04.50
Jared Atkinson
Manually right through iterative iterative analysis.

58:10.55
Jared Atkinson
Um, yeah. Yeah.

58:21.73
Jonathan Johnson
Simply because like I'd rather not spend the money on Ml whenever I could be spending the money on either data collection or like upgrading my analysis platform. Um, and like I think there's levels right? and I think like Ml just happens to be a very high level that I think a lot of people aren't even like.

58:29.32
Jared Atkinson
Um, sure. Okay, you.

58:38.24
Jonathan Johnson
Touching the threshold of getting close to yet. Yeah, yeah, exactly I should ah should have word that better. But yeah I agree with you and go ahead I'm sorry.

58:38.67
Jared Atkinson
I don't I don't disagree with that. So yeah, okay so you're making a you're making a cost benefit kind of argument more so than tech. Okay, but yeah, so the yeah, but the the ideas is that there may be telemetry that's more valuable.

58:53.96
Jonathan Johnson
Um, yeah.

58:56.71
Jared Atkinson
Um, then what we currently collect and the way that you figure that out is through analysis at the research level and then ideally the conclusions that you would that you like and maybe you use Ml to figure that out at like from a research level and then and then ideally you propagate that learning right? that that new understanding to.

59:00.54
Jonathan Johnson
I agree. Yeah.

59:15.13
Jonathan Johnson
Um, yeah I.

59:15.86
Jared Atkinson
The edr vendors and then they provide it because like there's actually like ah just because you solve the problem. The best technically doesn't mean that you're going that your company's going to last because there's like there's ah, there's a problem to where um, some companies are kind of like ahead of the community. Let's say or ah ahead of the industry to where they have actually like.

59:30.80
Jonathan Johnson
Um, yeah.

59:35.26
Jared Atkinson
Provided a better a better ah product let's say from a technical perspective but the consumer market is not ready for that solution because we don't understand that that's better like yep.

59:42.85
Jonathan Johnson
Um, yeah, so they can't value that quite yet. Yeah I agree I think um and I think we talked about this in the last podcast Potentially when I talked about like edr vendors and like having like the base set and then they have more advanced telemetry. Um, that a lot of like different vendors are starting to apply I think it's important back to your original point of like going to the vendor and talking about telemetry I think this is why it's important to set up partnerships with whoever you're using, um, in general because from my perspective like as a researcher I've done this plenty of times where. I'll be looking at something and I'll look at the telemetry given and I'm like okay like let me kind of like take your like Edr telemetry and base it off windows security events and kind of like see you what's there but then also let me like if it's like ah. Kernel Object. Let me just open that up or let me look at like a structure and see what's potentially possible to grab but the piece is like now I'm going to have to write some type of code or at least walk the dog and and sense of the process of saying like. What would it take for me to get that attribute from that structure or from that object and like okay now that I know that is it realistic for me to go to the vendor and say like hey like this is like bomb Http://dotcom is there any way for us to actually like for your engineers to start engineering this into the platform if not like. I Think there's a lot of things that are like under the hood that would just be like phenomenal in the sense of detection but like the resource piece of like collecting that might be a lot.

01:01:19.82
Jared Atkinson
Um, yeah.

01:01:21.68
Nas
Yeah I just use this to to feel what it raise your hand so this this could fit very well into the idea of bypass and bypassing telemetry that exist and that doesn't exist and the prioritization problem that Idr vendor face.

01:01:25.80
Jonathan Johnson
Um, is.

01:01:39.34
Nas
So For example, let's say crowd strike or wherever they had the the best people or they still have the best people perhaps in windows internal stuff so they could collect. Ah I don't know red Canary Also do the same of collecting very very low level stuff. But. Edl vendors could collect but they have this priority issue where they should prioritize selling the the product with a a stability a certain stability then offering this pletra of features to anyone and only a certain population will use this. And this will feed into the idea of bypassing something. Did you bypass the telemetry or did you bypass the the detection and whatever and I had I had an idea about this bypass and because that is the the bypass and there is the other word where we often use.

01:02:34.18
Jared Atkinson
Evasion.

01:02:35.12
Nas
Yeah, evasion. Yeah correctly, so I look at it from a video game standpoint. So if you play a lot of about pgs. Ah sometimes when an enemy attacks you it. It's called an evade you evaded it. For example there there exists a direct attack into you and this attack you evaded it.

01:02:38.97
Jared Atkinson
Um, oh boy.

01:02:54.59
Nas
But a bypass you escape the enemy as a whole so you didn't fight it. You find you you you you find a way around it and this distinction between invasion and bypass I find really interesting. So ah sometimes when you look really deep into the telemetry generated by the by the Id vendor you'll find that it could.

01:02:57.19
Jared Atkinson
Um, ah, okay, yeah.

01:03:14.50
Nas
Technically detected but they didn't write the necessary detections and when you ask them? why you get stability you get performance reasons you get and you never get security reasons you only get R and D reasons and this is very interesting.

01:03:28.55
Jared Atkinson
Yeah there's ah I like to tell a story about how evasion is relative to the sensory capability of your target right? And so there's there's a story I think it might be apocryphal. Ah like there's there's conflicting evidence of whether it's. True or not but there's there's a there's a stealth jet that was really popular in like the desert storm war I guess from the us called the f 1 17 nighthawk and it was like the first real stealth stealth fighter jet or stealth maybe as a I don't know fighter. We'll say fighter. Um. And the and the idea was that http://lockheedlockheedmartin developed it and in the development they they found that shape was actually the the important feature of of stealth and so it's about like making sure that basically the radar signal did not bounce off and return in an expected pattern. Ah, to the to the station and and the interesting thing was it's like well the the jet wasn't ah wasn't invisible from my visual perception but like they weren't trying to hide from my eyes they were trying to hide from radar right? and so they they adjusted how they approached the problem ah based on. The basic perceptual capability of their Target. We'll say um and so there was this interesting story that came from it is when they first first brought it over to the Middle East I think they put they were in Saudi Arabia maybe and they put it in these openair hangers because it it also had this stealth paint on it and it had a ah. Ah, toxic fume so they couldn't put it inside of like a indoor facility and so they had it it under these open air hangers and they came back in the mornings and there was always these dead bats all around all around the jet and what they what they hypothesized was that because. Bats see using echolocation right? They don't have like actual visual perception. They ah they were not able to see the jet because it was evading them in a similar fashion to how it evaded radar which was always I don't know that this is actually true. So don't don't you know, crucify me if if I'm wrong, but um, this. But the story is kind of interesting and it's it's something to be a guide to us to how evasion is an interesting thing because you often see people on Twitter say oh, it's more stealthy to do it this way and it's like well what is stealth like what what makes something more stealthy and what they're saying is you know based on what I expect. The perceptual capability of the target to be It would be better if you did it this way because it's less less likely to be caught. Um, but like Johnny has a blog post that talks about the different types of evasions and you you alluded to them Nas was which is like am I am I evading your ability to capture telemetry in the first place.

01:06:19.40
Jared Atkinson
Or am I evading your detection or like even further am I just evading evading you because I'm able to complete what I need to do faster than your cycle to be able to remediate me and all those things. Ah those are all evasion will say maybe maybe 1 is a bypass I don't know based on how you just described it. Um, but the the interesting thing is it's like um, it doesn't matter. Well it does matter but like as long as I'm able to finish before you you remediate me then it doesn't really matter from my perspective as the attacker unless I have some sort of Intel goal to like not be detected. Let's say but um. As a defender. It's important to know where the problem manifested itself because the the solution is very different if the problem was well. They just finished faster than we were able to remediate them as opposed to well we we didn't even collect telemetry that would have told us that they were there in the first place. Those are completely different problems and they have completely different solutions and so it's important for us to discern like this is like a thing when when you're red teamed and the red team is successful. It's like where did the breakdown happen did the breakdown happen because we didn't collect telemetry because we had a bad detection. Did we alert on it. But then the you know tier 1 analyst marked it as a false positive did our did we not get to the remediation step fast enough. Did we do the remediation step and it didn't work as we expected it to as we expected it to um, all like that's an important important distinction to make and like I think. Too often. We don't make that distinction. We just view all evasion as being the same thing.

01:07:55.37
Jonathan Johnson
Um, yeah I think um.

01:08:01.65
Jared Atkinson
As a long pause I Thought you're just like yep, that's good.

01:08:04.49
Jonathan Johnson
Well I just disagree I agreed with you? Yeah so there't. Um, so I'll take ah I have a general question for both you too and Luke feel free to ask this or answer this. So um, we talk about like ost debate all the time. So.

01:08:22.64
Jared Atkinson
Here we go.

01:08:24.47
Jonathan Johnson
Ah, ah in a sense of like whether it's good or bad that red teams are or people are open source tooling their they're like rubus with will and micats things like that. Um I'll wait to give my opinion on this question but I want to hear you guys. Your guyss answer should like why hasn't there been like an open source tooling debate in terms of detection. So Do you think that it is right for people to be like open sourcing detections for multiple organizations or do you think people should be doing that because that's been an argument I've heard from my people on the red side is like. You know we release tools all the time but on the flip side. There's no,, There's no detection um like detections being written and putting out there as Well. So What are you guys as general thoughts on that.

01:09:16.60
Jared Atkinson
I Could go first or second I don't care which okay yes I think I think there's like ah there's a practical problem with detection in that we all have slightly different ah perceptual capability.

01:09:17.97
Nas
No, go go ahead. Jack go ahead.

01:09:24.93
Jonathan Johnson
Any.

01:09:31.47
Jonathan Johnson
8

01:09:33.47
Jared Atkinson
And there's an or like there's a machine behind detection and response that doesn't like that's much larger than the machine that's pushing like a red team operation right? So um, a lot of times open source tooling on the red side. Can function atomically right? So like anybody can just pick it up and and use it right? But the problem the problem is is that on the detection side. There's a there's an issue in the sense that we all have slightly different telemetry collection capabilities right? from a technical. Perspective. We all have slightly different naming conventions which makes it not obvious to be able to translate detections. Yeah there's a standardization problem and like things like Sigma try to solve that to some degree. Um, and then there's there's a whole like mechanism of process and procedures behind. Um.

01:10:16.62
Jonathan Johnson
Standardization problem. Yeah.

01:10:28.72
Jared Atkinson
Behind the detection and response program because like the detection the technical detection analytic like I just mentioned is only the beginning of the problem right? or the solution. Let's say because you you identify even if you knew for a fact that something was malicious. You still have to know how to how to deal with it.

01:10:35.28
Jonathan Johnson
You.

01:10:48.19
Jared Atkinson
Right? and dealing with it is not obvious and it's not It's not something that we practice enough I think that there should be a lot more practice and when we come up with remediation steps. We should like ah Jocko Willink is like ah he's a Navy so x navy seal commander and one of the things that he he pointed out was. A big difference between like let's say like your local police and the and special forces is that local police. They they operate so they're doing their day job 90% of the time with 10% training right? and so and so ah, yeah, and and the Navy And Navy seals there.

01:11:18.39
Jonathan Johnson
Um, oh I've heard this I've heard this podcast I believe Sorry didn't interrupt you.

01:11:26.43
Jared Atkinson
They're doing they're doing 90% training with only ten percent execution right? And so like I'm not saying that we should be doing 90% training in the infosec side but like how many times have we have we interacted with a customer where they they find something or maybe they have a legit incident and they have.

01:11:28.53
Jonathan Johnson
Um, yeah.

01:11:44.73
Jared Atkinson
A remediation playbook and they just throw it out the window because they've never practiced it. They don't actually know where to find it. You know all those problems and it's like um yeah I think that's like I think that kind of describes the problem for me is that it's not. It's not something you could just take and plug in because there's a whole bigger problem. There's less.

01:11:47.89
Jonathan Johnson
Um, yeah.

01:11:56.24
Jonathan Johnson
Um, yeah me.

01:12:04.61
Jared Atkinson
There's less certainty. Also.

01:12:05.52
Nas
Yeah I think I think the defensive side is very context dependent I have an anecdote regarding this because yeah Johnnie spoke about that before the idea of never trusting a detection. You see you always should try it.

01:12:06.53
Jonathan Johnson
Yep.

01:12:24.46
Nas
You always have people telling you that you should block any execution technically from temp or wherever where one day we had the greatest idea in the world and we we blocked anything related to Powershell Vbs wherever all those suspicious stuff.

01:12:29.64
Jared Atkinson
Um, yeah.

01:12:42.64
Nas
And we we tested it on a small subset of the company because we had 28000 computers so we tested it on a couple hundred and it works relatively well. But once we generalize that and I preface this by saying that we had the non-homogeneous environment. Not nothing to be proud of but it's the reality and the the thing that surprised us was the amount of software and the amount of legitimate software that that drops Powershell scripts to do updates that drops vbs scripts that do update literally uses the same thing as a vbs.

01:13:01.10
Jared Atkinson
Um, yeah.

01:13:20.43
Nas
Malicia Malware and a legitimate software will use this so this uncertainty and this context condition that is relatively high and it is directly linked with the defensive mechanisms.

01:13:20.58
Jared Atkinson
Um, yep.

01:13:36.77
Nas
So for example, the analogy that I'd like to do is let's say that Sony again a video game reference because of that's it I a video game nerd. Let's say that god of war is very optimized to PS four because it's it's written into 1 platform but you have something like.

01:13:42.91
Jared Atkinson
Um, yeah.

01:13:52.67
Jonathan Johnson
Um, such a good game by the way.

01:13:55.28
Nas
And screen For example is technically yeah, difficult difficult to optimize and you notice a lot of bugs if it is multiple platform rather than single platform and the idea of rating tools because we all use windows or linux.

01:14:08.85
Jared Atkinson
A.

01:14:15.23
Nas
So the the the malicious stuff is already optimized to that platform across all updates. But if you switch to the other side you have different versions of windows which are vulnerable to different vulnerabilities. You have different products who are monitoring the the computer itself.

01:14:18.67
Jonathan Johnson
E.

01:14:20.31
Jared Atkinson
Um.

01:14:34.96
Nas
And you have different people so a small company might have 1 person a bigger company. You might have 100 person so these conditions when you align them. It's very difficult to generalize a detection. So as you said sigma tries to solve this and I believe they are trying to solve this with with their Aurora product.

01:14:52.89
Jared Atkinson
Um, okay.

01:14:54.79
Nas
Making it a an endpoint agent that's using sigma and etw and a lot of stuff and it's pretty good. But I think the idea is that since the majority of people didn't consider security and are not considering security at a higher level For example, you have a company with 20000 employees but the security team is just 5 people and you cannot implement. You cannot change active directory because it's a 2008 server and you don't have a vulnerability management program so you have a lot of priorities even before you think of detecting something.

01:15:15.87
Jonathan Johnson
A.

01:15:29.90
Jared Atkinson
Um, yeah.

01:15:31.52
Nas
And the red team needs. Only 1 thing to download minimikats and you have a computer without antivirus and congratulations it got executed and he got a shell and he can do whatever he wants and this duality between red team and blue team is what what's so very interesting because I don't believe. Could exist until the vendors decides to be somewhat agnostic and they agree on some some level of uniformity. So let's say that a lot of vendors will agree that a naming convention of detections will be the same a minimum viable product.

01:15:59.17
Jared Atkinson
Um, yep.

01:16:08.79
Nas
Of Detection. So So let's say for example, every product you buy will detect ah this capability of mimicads out of the box and this definition of capability is across the board the same once we're having that then we can start to discuss. A general way to create detection for anyone I Believe yeah, this is my take.

01:16:29.66
Jonathan Johnson
Um, yeah.

01:16:30.46
Jared Atkinson
Yep, yeah, and 1 one of the problems is is that in order to standardize standardize like that you just describe 3 different abstraction models right? So there's the abstraction of how do we view the telemetry that we collect. There's the like the abstraction of what what is Mimi cats and then there's the abstraction of what is. What is detecting right? So there's ah like there's the the problem that we're facing is that we have to get consensus on the right direction across all of those things as a prerequisite to being able to have something that's widely shareable and I don't think that we have we don't have consensus on any of those things. Right now.

01:17:11.30
Jonathan Johnson
Yeah, so my take on it. Thanks for asking guys. Um is I I Ah I agree with both you guys? Yeah, so I think the thing for me is there's too many like there's too many standardization telemet tree issues.

01:17:14.95
Jared Atkinson
We didn't We didn't think that you needed needed us to lead you in here.

01:17:28.66
Jonathan Johnson
Um, and to me there's too many contextual pieces that might apply so the way I might write a detection is completely different than how like Jared or Nas or Luke might write a detection right? and so it's how I comprehend the problem or the technique and so the way I would rather go about it is and this is kind of like the theme with some of my blogs is. I'd rather talk about the research diving in abstracting the technique and explaining at what points am I writing the detection and then showing how I might come up with that analytic logic. So that way if someone wanted to do something similar. They could apply that same type of skill set. You know. Or they could feed give feedback to me and teach and teach me a better. You know a better format to do that way. So that's kind of how I think about it because I don't think 1 detection like I don't think you can really mirror detections across organizations as you know priorities change also telemetry telemetry. Telemetry changes, standardization changes, etc. So that's kind of how I how I view that that question. So.

01:18:29.30
Jared Atkinson
Yeah, and like like to let's say let's say we do have standardized data. We might not have a standardized back end of analysis to where like when we like we may not have a standardized expectation of what an alert is right.

01:18:37.39
Jonathan Johnson
Um, yeah.

01:18:44.88
Jonathan Johnson
Yeah, like it.

01:18:47.70
Jared Atkinson
So like ah like for instance I think I think an alert should be something that's relatively broad because I want to have follow on processes that narrow it in because like I operate on the assumption that like you can't get something back once you once you ignore it? Yeah, but there's other other organizations that don't like.

01:18:59.88
Jonathan Johnson
Yeah, you're more sensitive to false negatives.

01:19:06.99
Jared Atkinson
Maybe they chose not to or they just don't have the process to be able to do that follow on and so like they need something that's going to be more precise right? And so that's yeah, so there's there's tons of moving parts that I don't think we're standardized on and like that 1 you might not be able to get standardized on.

01:19:14.30
Jonathan Johnson
Um, yeah.

01:19:26.18
Jonathan Johnson
Yeah I've asked I've heard that question like some people have asked me that question before from like a red team perspective. They like why aren't there. Why isn't there a repository repository of like detections or things like that like there is like open source tooling I said well I think there's more going into it like I think there's more contextual pieces that you can't.

01:19:26.78
Jared Atkinson
Across the full industry.

01:19:45.24
Jonathan Johnson
Can't be accounted for there's too many variables when it comes to brand detection and I think also I think in general people are also more sensitive on releasing that because you're almost like hey like here's a detection that I have for X thing and someone could be like yo like this is dumb or.

01:19:47.46
Jared Atkinson
Um, well yeah.

01:20:00.60
Jared Atkinson
Well I think there's ah I think that's like so assuming that everybody was on the bandwagon of detection is just the first step in the process which means that you can be more false positive tolerant because you would have additional like additional phases.

01:20:02.14
Jonathan Johnson
People is typically like to show I think.

01:20:17.56
Jared Atkinson
Of analysis to narrow in your focus then you would have something similar to like the cryptography thing to where it's like that This is the better wrong than vague principle right? So it's it's better for me to make this thing public so that it could be criticized right? because ah but like once you get into the very specific.

01:20:18.21
Jonathan Johnson
Um, yep.

01:20:37.80
Jonathan Johnson
Um, yep.

01:20:37.62
Jared Atkinson
Criticism's obvious right? and so there's no there's no need to share the specifics but the broad detections. It makes sense to share because then it can be criticized and like I've learned ah numerous things about the flaws in my detection approach from like talking to Lee Christensen or will Schroeder for instance because they're like oh well. If you do that then I would just do this or Matt Graver is another example and it's like oh okay, well that's that's good to know. Um, now when you're like talking about the practical implementation. Ah, sometimes you like like Nos said you just have to choose. It's like okay we understand that there's a limitation there and we're just going to move forward anyway.

01:20:58.20
Jonathan Johnson
Um, yeah.

01:21:13.75
Jonathan Johnson
Yeah, don't let perfect be the what's the saying like don't let perfect be the enemy of good or something like that.

01:21:15.79
Jared Atkinson
Um, because we got to do something. We can't have the perfect solution. Yeah, but yeah, but there is value potentially in having like hey this is the abstraction map for how I view curberosing and based on that abstraction Map This is how I'm detecting that behavior and then like give people an opportunity to.

01:21:33.87
Jonathan Johnson
Um, yeah.

01:21:34.97
Jared Atkinson
Criticize it like. For instance I thought ah a lot of curberosing detections look for some sort of threshold of like ah somebody at like requesting mass kerbose service tickets right? and so the detection might say if somebody requests 10 so kboro service tickets then we're going to alert but like you don't have to do that.

01:21:52.74
Jonathan Johnson
Yeah.

01:21:54.74
Jared Atkinson
And I like originally I didn't know that until will was like oh I just look for like this single account and then I you know I discern using some other mechanism which account is the one that I'm going to Target and then I just request the ticket for that account specifically and so it's like okay well that the assumption that that detection made was that they must. Or that they will maybe that that not that they must but that they will request multiple service tickets and that's a bad assumption. Oh yeah.

01:22:16.53
Jonathan Johnson
Man and then you into the conversation of like of like pre attack detection. So like someone doing like a whole bunch of ldap queries in the organization. Chef kiss.

01:22:26.88
Jared Atkinson
Yeah, Well so like yeah with Ker roasting. There's ah, there's enumerating service accounts right? So There's an ldep query that would allow you to enumerate service accounts. But theoretically you can potentially learn what the service accounts are for most sent. Let's say so there's like there's potentially different ways to do that. Then there's the act of requesting the service ticket right? which is the like that is the kerberrosing. There's the cracking of the service ticket to get the password which that you know if you don't if you're not doing that Offline. You're you're an idiot. Basically so that that shouldn't be something that a defender should expect to catch and then there's the use of the Stolen password.

01:22:53.50
Jonathan Johnson
Can be offline.

01:23:05.81
Jared Atkinson
Those are kind of like the 4 opportunities of detection and so the question is is where are you detecting which one's better which one's more efficient.

01:23:11.75
Nas
You know there is there is There is one idea I played with which is sometimes when you go very granular you you will face something especially if you have a non-homogeneous environment. So let's say that everybody can do whatever they want and your detection when it goes to.

01:23:12.70
Jonathan Johnson
Yep.

01:23:31.60
Nas
A very specific level. It will detect everybody as malicious so you have to de abstracted a level and then you take the risk of if an attacker will simulate the same behavior as everybody you will not catch it and sometimes you you have to be okay with that.

01:23:37.66
Jared Atkinson
Yeah, yeah.

01:23:50.25
Nas
And go to the next phase of that attack of that specific chain and you focus on that because ah, it's very difficult, especially in old environment where security wasn't the very big building blocks of that environment so you have multiple machines multiple vulnerabilities.

01:23:51.35
Jared Atkinson
Um, yeah.

01:24:09.30
Nas
Everyone is an administrator even we had we had cases where people were installing their own antivirus. But if that's yeah, if that's a new one for you So vary a non-homogeneous system and you cannot make very direct assumptions.

01:24:15.37
Jared Atkinson
Oh good.

01:24:25.87
Nas
For example I used to do a very basic exercise where I go to the Edr platform and select process or parent process equals Powershell and I export all that Csv and that and that zsv I will analyze it and then do a statistical approach.

01:24:35.29
Jared Atkinson
Me.

01:24:44.14
Nas
Over the most command lines ah relationship between Powershell and the pan process etc to see what are the most common things and those common things I study them to then determine if they are normal behavior and how attackers often abuse that behavior. But it it is a very long process. Especially if you have very very different use cases and very different users across the board. So this is I think is very important to keep in mind.

01:25:07.10
Jared Atkinson
Yeah, yeah for sure for sure. Yeah, so yeah, that's ah so one thing is we've been talking about how to detect a single attack technique. But it's important I I don't know if this is exactly what you're going for, but this is what you made me think about is that. Ah.

01:25:10.54
Jonathan Johnson
Um, yeah.

01:25:25.41
Jared Atkinson
No attack technique exists in a vacuum right? So um, if you can detect you know 10 attack techniques at 80% the likelihood that you will attack or that you will detect an attack path is higher than 80% probably if that makes sense so there's some sort of.

01:25:26.10
Jonathan Johnson
Um, yeah.

01:25:44.21
Jared Atkinson
Aggregation capability or something going on there. Yeah, cool, um all right? So I think we're running out of time but Nas we appreciate you. This was great conversation. Thanks for joining us all the way from Algeria we we appreciate that and that's ah is good to meet you kind of.

01:25:57.30
Nas
Ha ha.

01:26:02.54
Jared Atkinson
I don't know if this is in real life but closer closer than on Twitter so this is the first time that we've had it like a real chat. So I appreciate it but been a been a fan of yours on Twitter for a while. Yeah.

01:26:08.41
Nas
Yeah, thanks, appreciate you all.

01:26:11.42
Jonathan Johnson
Um, yeah, thank you so much I appreciate it man.

01:26:13.76
Nas
Thank you, Thank you for the opportunity and sorry if I'm if I if I caused any service in the force.

01:26:20.24
Jared Atkinson
Um, oh no, no no, it's all great. It's all good. It's all good.

01:26:20.42
Jonathan Johnson
Um, that's that's the purpose of this podcast is just that's the purpose of the podcast man is like talk about the hard stuff that when it go over too. Well on Twitter and just kind of like.

01:26:31.48
Jared Atkinson
We we kind of started it I think with this idea that oh man. Awesome yeah, we started the the podcast with like we had run into so many situations to where somebody was given a hot take on Twitter and then we'd say you know well like I don't know that is that easy and then they're like well this isn't Twitter's not the right place to.

01:26:32.47
Nas
Yeah I'm a big fan. So it's a lot.

01:26:50.39
Jared Atkinson
Have this conversation. It's like okay well what if we make a podcast would you come on and be a guest and that that was kind of the start of it. So cool man yeah dude appreciate it. Yeah, definitely Luke. You got it stopped or you want to.

01:26:53.21
Jonathan Johnson
Um, yeah.

01:26:57.70
Nas
Amazing idea. Thanks a lot. Thank you.