Hashing It Out: Blockchian Security Part 1- Mehdi Zerouali

We really introduce ourselves, should we? Y'all couldn't, they know who I am, they know me. ♪♪ Welcome to the Hashing Out podcast, where we talk to the tech innovators behind blockchain infrastructure and decentralized networks. ♪♪ Your hosts are Dr. Corey Petty, currently doing research at Status and waiting for other people to keep up. In between, we're talking about marketing folks, we're talking about business as well. Jesse Santiago, a former electrical engineer, now working on decentralized storage at Status. We'll give you some shoulders, there you go. ♪♪ And with the Deep Voice and the Deep Questions, Deep Ferguson. It's gonna be crisp. And I'm the Hashing Out Showrunner, Christian Nogara. Oh, I wish it was. This episode focuses on security, with Medi. You know how to pronounce it. Zorwali, co-founder and director of Sigma Prime. Start off. Welcome to the show, welcome to Hashing Out. Medi, we met DevCon, I don't know, we met a while ago, and I just have opportune introduction from Evan Vendes, actually, at a bar. It's like, hey, here's my buddies, you're both a security, y'all should talk. Yes, you was a Berlin, probably 2017. That sounds right, yep. Yeah, I was it. And then we hung out and chatted and went to the different bar and drank, copiously. So why don't you do a favor and give our audience an introduction to who you are and what you do and what you're into? Sure, I can do. Thanks for having me. Always a pleasure chatting with you, Corey. Nice to meet the new faces on the podcast. My name's Medi, I'm a co-founder and director of Sigma Prime. We are an information security consultancy, focused on blockchain technology. This means that we provide security assessment services to a whole bunch of different blockchain projects. As you can imagine, a lot of smart contract security assessments, but we're also very comfortable by being deeper into the stack and helping a bunch of blockchain foundations secure their blockchain nodes or blockchain clients. So working with your foundation, for example, on a bunch of fuzzing exercises, allowing us to surface potential critical vulnerabilities on all the Ethereum clients. So yeah, that's keeping us really busy, as you can imagine these days. And that's sort of half of what we do. That's the half that I look after. And the other half of the team is dedicated to the development and maintenance of Lighthouse. Lighthouse is the rust implementation of the Ethereum consensus protocol. It is now responsible for about 35%, 40% of the users on the Ethereum network, which is great and scary at the same time. But yeah, Kudos to the team who's been doing an amazing job for the past four years, yeah, allowing us to, yeah, have a proof of stake Ethereum chain. I still can't believe it sometimes. Yeah, it helps smoothly with the manner. Yeah, that's me, Singapore Prime in a nutshell. Prior to starting Singapore Prime, I was a penetration test up, finding bugs in people's mobile apps, networks, web apps, et cetera. And yeah, and we've been around for about six and a half years. Thanks for that. I think it's proven to start off with the general concept of security. So like, what does it mean when you say security in the blockchain space? Yeah, that's interesting. When you ask the question, I automatically started thinking about some of my very, very first information security courses that I've actually went through in Sweden and there's this concept of the CIA triad, right? I'm sure that's probably familiar to you, Corey. So the CIA triad defines three properties that effectively made up the security of a system. You have C stands for confidentiality, I stands for integrity and A stands for availability, right? These are pretty self-explanatory, confidentiality, you want a certain piece of information to remain some private and confidential to a selected set of participants. Integrity, you want the assurance that the data hasn't been tampered with, hasn't been changed, and availability you want obviously to be able to access the data whenever you want, whenever the system is actually allowing you to. So you can probably define security using these three metrics. And when it comes to blockchains, we can probably dive into these a bit more specifically, but confidential it is obviously a challenge in most open networks and open blockchains. Things like zero knowledge proofs, obviously, will help tackle that. But integrity is kind of the whole reason we're doing this thing, right? Having the ability to trust that whatever information, whatever transaction will never be tampered with and a normal set of rules. And then availability is a very interesting one in the context of blockchains, which has to tie with the concept of liveness, right? For these networks, making sure that you actually have an operating platform that will be resilient to a lot of denial of service attacks, for example, or certain parts of the world going offline of the internet for certain reasons, network petitions, how do you recover from those, et cetera. So it probably helps approaching the problem in my view, if I try to map whatever blockchains you're trying to assess with these three criteria. Can confidentiality integrity available? There's a lot of different models, obviously, but this is a pretty simple one, I'd say. And those three metrics, like how do those three metrics pertain to like blockchain? Like what does security mean specifically in the blockchain space? Yeah, so it means a whole lot of things, right? So give me three things. Yes, there's three layers, there's sorry, two fundamental layers, perhaps in the blockchain world, I'd say, right? You've got the consensus layer and the execution layer. So execution essentially deals with transactions, user transactions, this is where we move funds around, this is where we create smart contracts, et cetera. And then you've got the consensus side of things. This is where your validators or your miners come to consensus and obviously agree on the canonical tip of the chain, which is updated every 12 seconds for Ethereum. So you can essentially mess with the security of a blockchain by messing with either of those layers, right? If you find the problem with the execution of a transaction, your network's probably gonna have a bad time. If you manage to break the consensus of that network, well, the only alternative you have left is probably a hard fault, right? So you probably don't want that because that might be conscientious. So the way we approach security assessments of blockchain software is by trying to find really that separation between the two layers, right? Like conceptually you have your execution, it's where your applications live, and then you have your core layer one, which is where the consensus lives. We obviously work on both layers, doing, as I said, a lot of smart contract security reviews, but also helping on the lower layers, blockchain networks, ship secure software. So what might be interesting is to perhaps dive a little bit into the consensus side of things, while the execution is, I guess, fairly well covered these days. You know, a lot of smart contract auditing shops, a lot of smart contract auditing freelancers that are these days hammering platforms like Code Arena, Immunify and whatnot. And I feel like we get a lot of information on what's going on on that side of the spectrum, but the consensus side is somewhat obscure to a lot of us. So if that's okay with you guys, I'll probably just dive into the consensus, and if we have time, we can tackle the application that you have with us now. Sounds great. Yeah, real quick. I would like to add on to that as, this is something that I tried to harp on when I focused on security, was when you hear the word security, as it pertains to blockchains or web three or whatever, everyone identifies that immediately with smart contract security. And up until I'd say recently, with efforts like yours and many others, it's become more aware in the broader ecosystem that there's a lot more going on or the hood outside of smart contracts. Like there's a lot that needs to happen correctly in order for even start thinking about the concept of smart contract security. Can you talk about like maybe first before you dig into some of the subtleties of execution layer or consensus layer security? Like why that is? Like why is it that people focus on smart contracts when they think about security and why now are we starting to think about other types of security and why they're important? Interesting. So I think that the general community is interested in building on the application layer. So in the context of EVM chains, this is developing your smart contracts in Slidity or Viper, compiling them into EVM bytecode and deploying that. So naturally, you're gonna have a lot more people involved with the application layer than the base layer. It's probably at a 150 people, 200 people around the world working on the base layer with the Ethereum foundation or with teams like ourselves. But you'll have probably a hundred or a thousand times more people actually working on the application layer, which makes sense. There's probably gonna be a lot more things to secure on the application layer as we ossify the base layer of Ethereum. Hopefully we shouldn't be seeing too many changes on that base layer in 510 years time. A lot of work currently being done to ossify that layer. We're not there yet, but hopefully that's the trajectory that we're taking. So I think it's natural that people have been focusing a lot more on Slidity EVM type stuff than they have been on the bare sort of lower levels. You'll find that the actual attack surface is somewhat perhaps limited on those blockchain networks. First of all, there aren't that many, right? There's like a limited set of them. It's probably, I don't know, the one's worth looking into maybe 30 of them, 20, not sure. Whereas obviously the number of smart contracts employed on Ethereum alone is that are actually worth investigating and potentially reporting bugs on, you know, the order of probably the thousands or the 10 of thousands. And it's also probably a bit harder. Like the security of the consensus layer is has to tie in with a lot of computer science concepts. Some of them quite evolved, whereas the EVM layer, well, I guess, you know, historically we've seen a lot of people coming in as smart contract security auditors who don't necessarily have conside backgrounds, right? They're just able to conceptualize the EVM, abstract the complexity away and just tackle it, you know, as they would tackle any others assessment, which is cool, which allowed us to, you know, beef up the actual number of people able to provide these security services on smart contracts. Whereas if you wanna do the same thing, like secure, help secure, get, or help secure, lighthouse or prison, that will certainly take a lot of expertise in both the actual programming language used, but also the fundamentals of programming and sound programming, if that makes sense. Yeah, hopefully that answers your question. Yeah. Okay, so am I good to dive into the consensus side of things? I've been diving away. All right, let's do it. So we've gone this for a bunch of networks, right? We've gone this for Ethereum network, Polkadot, scanning through the ones that are coming with NDAs. Yeah, anyway, a bunch of others, right? And you have these networks that operate on a lot of the most of the mainnet these days. Some of them may have different implementations of the same specification in different languages. We'll come back to this after. That's quite important and quite tricky to get right. And essentially, you can think of them as single pieces of software that connect to other similar software in a pay-to-pay fashion. They propagate messages. They exchange a whole bunch of messages and they all have to do, and this is Ethereum specific in this example, they all have to do something every five minutes or every six minutes, every validator around the world has to do something, right? It can be just vouching for what they think is the tip of the chain or it can be proposing a block themselves if it's their turn. So, yeah, it just helps set the scene. This is what we're dealing with on the consensus layout for Ethereum at least. So obviously, you want to think about this as an attacker, right, with a very serious mindset. What if I could connect to these networks and shut them down? That'd be pretty good, right? If I'm attempting to destroy Ethereum or I don't know, like I'm opening a short position on ETH and then I can destroy the Ethereum network for a certain period of time, I might actually make a decent quick buck. So, this is what we look at in terms and that ties in to the availability side of things, right? If I'm able to bring the network down, I can probably profit from this. So, the way we approach this particular statement or question is by, first of all, looking at the peer-to-peer networking stack. So, you may have heard of things like LibPDP, DEVPTP, et cetera. These are frameworks, modular frameworks that can be used by blockchain networks to power their peer-to-peer networking interfaces. So, if I'm able to send a single packet to a blockchain node and that packet will bring it down, well, I can probably send that same packet to all the blockchain nodes on that network and bring the whole network down or at least be able to bring a certain amount of network participants down, which may actually impact the finalization of the chain, for example. So, networking is always a hot area for us to look at. We wanna be able to demonstrate to our clients that through connecting to a single peer and sending a single TTP packet, we're actually able to take the node down. And this has some times to do with the way blockchain clients, powers and martial data that is fed to them on the network. So, they process packets, they get packets and they have to look at the content of the packet. And in that process, they, what we call deserialize the packet, right? The packet comes encoded, following a certain grammar, certain language, the blockchain node receives the package, decodes it and then processes it, sort of propagates it up the business logic stack. But the very first layer is where we can potentially sometimes get the blockchain node to completely crash. As it decodes a malicious package, it just does not know that malicious package does not adhere to that grammar that it's expecting. And in some cases, well, it's trying to decode it, following that grammar. And if you play it right, you can actually get them to just, what we call OOM, get an out of memory era on the actual host, which will lead to the whole host restarting up. So that's one avenue of bugs that we actively look for on the consensus layer. And then you have obviously the cryptography side of things. We participants of these networks usually, typically, use public key cryptography. So a public key, private key pair, define an account. And validators are responsible for signing some messages, consensus messages, using those keys, using the hot key, the private key. So we use BLS signatures. It's a relatively new scheme that was introduced maybe 20 years ago. The Ethereum network is one of the first projects really on earth, making use of BLS signatures. And BLS signatures are really clever and allow us to scale the number of participants in those networks really, really well. Really, really well. So when it comes to cryptography, there's a lot of things that could go wrong. First of all, never roll your own crypto. There's heaps of different packages out there. Please just make sure you use a battle tested, well reviewed cryptography library. Heaps of them, there's no excuses in 2023 to be rolling your own. Unless you're a cartographer and you've been working for years on peer reviewed research and come up with something very innovative and you wanna see what it would actually look like in production. Of course, but if you're building off existing components, well battle tested blocks, then just reuse what other brilliant people have done. And you'd be surprised in a lot of cases. So our clients are like, no, no, no, no, we'll do this better. We'll write it our own. Like the other, I don't wanna use an external package. I don't want to have to deal with a C code snippet that I then have to call into. I'll just write it myself. And it's extremely hard to get right. Photography development is very difficult. So in a lot of cases, we do find some of those bugs on cryptographic layout. So the bugs that we found before the launch of the Ethereum proof of stake chain were actually really interesting. We could essentially, as the identity of any user and we could provide a key or signature, sorry, that would always be being valid regardless of the data. This is due to the point at infinity when you deal with elliptic curves. And if you pass to your verification function, a point at infinity, then you'll always validate it. But at least that was the behavior of that particular library. So yeah, cryptography can be very, very tricky to get right. So again, strongly recommend not rolling your own. And this is again, another area where we've been successful in finding critical vulnerabilities on behalf of our clients. What's the success rate on the people that want to roll their own cryptography? Low. Oh yeah. Oh, very low. Yeah, funny. We do it. Yeah, cool. Yeah, yeah. Yeah, Mandy did his own implementation. Oh yeah, I've been meaning to actually check to look at his implementation. I think some of our guys have dived into his BLS implementation. Yeah. I think his challenge was to try to make it constant time if I'm not mistaken. He's done some fun things. I'm a little bit of a new. We'll talk after this. I'm a little bit of new, but I kind of understand, I mean, I don't even roll my own joints. But what does roll your own crypto mean? Okay. So it's basically like roll your own crypto. You can think of it. Try roll your own cryptography. You got a specific there for the audience members. Roll your own cryptography. Absolutely. It's like develop your own cryptography. You think that you can encrypt messages better than decades and decades of computer science research and cryptography research. Cubers. Yeah, I know, right. But some people think that they can't. And they're like, it's just, you know, I just need to glory these messages and pad them a little bit and shaw the whole thing. And look at this, you know, you can't recover this, can you? Okay. That's frustrating. That's really frustrating. It's like, nah, we've got cypress for a reason. So please use those curves that have been at least being saved by the community. Yeah, sorry. Then think. But yeah, it's either coming up with a new algorithm and you cipher that you think beats the rest. I call that developing or rolling your own crypto. Or you're implementing an existing algorithm, cipher that's been proven to work, but you're implementing it yourself. And you're not using the reference implementation or the bazillion numbers of well-battled tested libraries that have implemented this scheme before you. You think that you can do it better than that. Guess what? Probably not. Because there's a lot of effort that has gone into securing those libraries. They've gotten third party security assessments. They've gone through extensive rounds of buzzing. In some cases, they've also been formally verified. And you just rock up and think that you can, you can like just just now I'm better than I can go there. Yeah, exactly. Yeah, yeah, yeah, exactly. It was me, right? Yeah. Thank you for cleaning it up. Sure, easy. Yeah, I mean, I can talk about consensus stuff for agents, right? But we may want to cover something as I just before we move on, perhaps I quickly mentioned different implementations of the same specification. It is the case for the theorem network. It is the case for the Polkadot network where you have, say for Ethereum, a client written in Go, one in Rust, one in Java, one in NIM, one in JavaScript. So, you know, a whole heap of different flavors of the actual same spec. This is important, right? They're all building the same thing in different languages. What you find interesting in those cases when you have that landscape of different clients is trying to do something that we call, well, it's called biferential fuzzing, right? If you have five pieces of software doing the same thing concurrently, right, they connect it to each other. It's like a P2P mesh network where they all meant to be doing the same thing. If one of them diverges from the spec, then if there's a bug in one of them, well, what happens is that that particular client or the validator is running that client may actually fork off and build their own chain and have their own skewed vision of history, meaning that they won't actually be able to participate meaningfully in the consensus of Ethereum. So obviously, we wanna find these bugs before they hit production, right? Because what we're talking about here, if they are to be exploited, is a proper chain split. And it's probably one of the worst things that can happen in a blockchain, right? It's like a contentious fork. We don't even know what's the canonical chain anymore. We've got too many validators on both sides. Social recovery, i.e. our fork, is probably the only solution we have left. So yeah, we try to find those particular state transitions that may result in a discrepancy in dividends on any client. So on Ethereum, we've got four or five, maybe six different clients. We fuzz them all, and we do, as I said, differential fuzzing, where we compare the output on all of those clients for every single input we provide, and we make sure that the output is the same, because it should be the same. Again, doing the exact same thing in different languages. And if they don't actually, if there's a difference in output, then it's potentially a critical bug. And it's up to us to try to find where this came from and why we actually have this discrepancy. It's been great. We found about 40 bugs, some of them very critical bugs using these techniques on the Ethereum network. It's hard. My job is easy finding bugs, like shipping software without bugs is much harder. An interesting dovetail from that conversation is, why go through the effort of writing multiple implementations? If you have to do all this extra work to make sure that different implementations are quote unquote speaking the same language, like for every input, they all do have the exact same output. And you have to do all this work to make sure that's the case. Why build multiple implementations? That's a great question. And we didn't make it easy on ourselves. And by we, I'm in the Ethereum community by prioritizing client diversity. The Bitcoin network, for example, is at the polar opposite of that velocity. Or like, nah, nah, nah, nah, nah, wants to only have one reference implementation so that there aren't any cases where a potential network could fork. For Ethereum, in Ethereum, we have historically been a huge fan of client diversity. It actually saved us in the past, right? If you recall 2019, 19 or 18, 19, I believe. Bug on Go Ethereum, the largest Ethereum execution client. Bug on the execution side of things. And the only reason why we had no interruption to the operation of the network was thanks to the second client, Parity. It doesn't exist anymore. It's called Open Ethereum. And I'm not sure that exists anymore. Yeah. But we used to have a Rust execution client called Parity. So yeah, you know, guess was down, Parity was up. No, absolutely no impact on the end user, which is great. So client diversity allows us to have a way to fall back, right? If there's a bug affecting one implementation, well, you know, luckily, if we're lucky, that bug is not spread to the other implementations, which means that the rest of the network can then carry the load. So it's critical to the, going back to that A, in the CIA triad, availability, client diversity is critical to ensuring availability in the context of blockchain networks. That's our opinion. That's, I guess, the Ethereum family's opinion, but that can be debated. And, you know, some other networks are necessarily following the same philosophy. I think I identified the trade-off nicely of like, it's extra work to ensure that you have that all of the different implementations follow a specification. And I'm sure you understand fully that most people don't write specifications. So like, you need a lot of coordination effort in that process, which is hard when you have competing businesses writing software. But at the same time, you're giving a lot of guarantees on that thing that you need, which is availability. Absolutely. One thing that I don't think many people think about, outside of the Ethereum community, like I'm talking about the laypeople, are where nodes are actually hosted from. Is there any, I guess, security aspects associated with everybody trying to run a node at home versus, you know, using AWS or, you know, digital ocean or something like that? That's a great question. I think if I recall correctly, and we can try to dig up the numbers, but there's been a study published not too long ago that aims at tracking where those validators are, right? Like where they are geographically and what they're running on. And you'll find that the distribution is very interesting, actually, in the sense that a lot of operators do use cloud services. And then within cloud services, there's definitely a dominant part of AWS hosts. AWS is more expensive. It's probably the sort of prime cloud provider in that list. But they're very easy to use, I guess, right? Like people doing DevOps or have played with the cloud before have probably interacted with AWS. So that explains why they have such a big share of the network. And then you've got all the exchanges, the Kraken, the Binance, the Coinbase. These folks try to be fairly secretive about their DevOps operations. So we obviously have advanced working relationships with some of them so we know what they're running. A lot of them run stuff in the cloud. Some of them have their own data centers as well. And finally, I like that you mentioned the homestakers. Homestakers are great. There've always been a priority when it came to designing this thing. We designed it with the homestakers in mind. We want people to be able to run this stuff from home. And thanks to initiatives like Rocketpool that allows you to have liquid staking and say, you know, I'm interested in staking. I only have 16 ether, I don't have 32. Well, I can chuck 16 ether and collect 16 ether from the community. And with these 32 ether, I'm gonna now run a validator and I will be sharing the actual yield or rewards with the people that pulled those 16 together. This has actually resulted in a whole bunch of people running Rocketpool from home, which is great. Like if you try to optimize decentralization, you obviously want a node everywhere, right? You don't want all your nodes to be in America. You don't want all your nodes to be in Europe. You really want that network to hopefully have, sorry, have like a presence geographically everywhere. You want a presence in as many different data centers as possible, as many different cloud providers as possible. I think we're doing an okay job. Let me just say this. I think we're doing a decent job. Obviously, decentralizing more is definitely beneficial. A lot of concerns were floating around with Lido. Lido being a liquid staking protocol, but effectively only governed by a selected set of entities. So you've got about 30, 40 organizations that can act as validators for Lido, whereas Rocketpool is completely open, permissionless, anyone can come in and run a Rocketpool node. So we've seen Rocketpool eat up a lot of Lido's lunch over the past few weeks, and that's probably depends who you ask, but it's probably a good thing for decentralization, right? Because the more Rocketpool nodes are spun up, the more happiness, probably homestakers, most likely, whereas if you spin up more Lido, a validator as well, these are managed by professionals. I must say that Sigma Prime is a node operator on both on Lido and Rocket. I don't like to switch gears a little bit. We've been talking about security as it applies to a layer underneath smart contracts and what it means to think about security for consensus and what type of bugs are appropriate for breaking the nodes that run these things. But there's this concept in traditional security that I've always enjoyed, and that is the pyramid of pain. If you're not familiar with that, it's this pyramid of different types of indicators of compromise or IOCs, and how easy or difficult they are to mitigate, right? And so if you look at traditional security, at the base layer you have things called hash values, basically you can find them easily, but by removing a specific hash value as an indicator of compromise, the attacker doesn't have to do much to get past that, he just changes a byte in the software, you can't see it anymore, you can just keep going, right? And you go up the pain, as you go up the pyramid of pain, it changes, and at the very top you're basically, it's TTPs, so in order to mitigate these, if you're successful in mitigating these types of things, then you're forcing the attacker to change their behavior, and they're less likely to do that, therefore your overall security, security posture of software or organization is much better. And what I've been interested in, in the security industry, is the differences in traditional security and blockchain security, as it pertains to getting the user to change their behavior, and what are the different types of indicators of compromise that are different in this industry than traditional security? As an example, fishing is incredibly more detrimental, and the blockchain industry than it is in the traditional industry, while it's still detrimental, it's worse here, because if you lose your seed phrase, you lose everything. There's no way of getting it back. Like, can you talk about that as a concept? What's the difference between traditional security and web three security, as it pertains to like, the attack surface or what people go after, and how, yeah, that's general differentiation. Yeah, it's very interesting. In my view, corporate security or traditional legacy security, if you will, has the ability to layer a lot of defenses, right? And this is the approach really, when you talk to size those and tech giants, this is what they're doing. There's no such thing as a secure organization, it's just a very hard to hack organizations. And they understand this, and they're layering perimeter defenses, layered with like AVs, endpoint protection deployed on all workstations, plus lots of red teaming, lots of security testing, et cetera. They're making their attack surface as little as possible, so having some services that can potentially be considered sensitive, well, we'll just hide them behind a VPN, right? If our employees are the only people needing to connect to this sensitive thing, hide it behind VPN, then you're sweet, right? Then you first need to compromise an employee or compromise their workstation to be able to access that, you know, not saying that can't be done, obviously it can, but again, just layering those layers of defense, adding them on top of each other. We can do that relatively easily in the corporate world, right? Something is too critical, yeah, just put it inside DMZ, you know, out of the internet, something is potentially in terms of confidentiality, right? Like if going back to the seat of the CIA triad, if something is very critical and we don't want people to have access to it, just put it behind off, right? Just use authentication, user and authentication provider, or for example, tie it back to our active directory if we have one and, you know, within probably like a few hours, that piece of information is now protected, right? Against illegal access. We can't really do so much of that in our space, can we? So when you deploy a smart contract, by definition, the attack surface that is available to you as the builder or the deployer is the exact same that's available to anyone. I think North Koreans think, bad actors, they all play with the exact same rules. They all play with the exact same attack surface. So unfortunately, we won't have that luxury of adding perimeter defenses to help us sleep at night. We can't really do that. So there's a big difference there. Another thing is, I think monitoring. Monitoring is quite interesting. I think a lot of people have realized over the past few months that there's a strong need of being able to monitor my protocol closely, right? So people have thrown around a couple of interesting initiatives to be able to standardize how we're doing monitoring. So this is a very immature area of our industry, in my opinion, whereas the monitoring business in traditional security, holy cow. It's like these. Yeah, it's been around for over a decade, probably, definitely over a decade. And it's matured to a multi-billion dollar industry. So, yeah, a lot of work that we kind of need to do on the blue team side of things in the crypto world to match whatever we have already in the traditional space. These are just my views. We're the welcome to discriminate on the already- I mean, if you've listened to anything I've ever had to say, monitoring has always been the main thing I worry about. There's this concept that we said earlier. The majority of the people focused on smart contracts. They developed smart contracts. They tried to understand secure software development lifecycles. They'd spend a lot of money getting audits for a while exorbitant on amounts of money getting audits and waiting in line. And then they'd apply these things and then they walk away and pretend they're safe. And then they don't ever watch them. And the tools we have associated with doing both of the activities are very asymmetric, meaning that the people who, like all the tools we have for checking that a specific smart contract does what it's supposed to do is much larger. And the tools we have for checking that it is operating appropriately on the blockchain now, or like showing up when it does something that we don't want it to. Or money moves in a specific way that it's leading towards some risky scenario. And yeah, I can't, I'll sit on that soapbox all day long. Yeah, I think this is a very interesting area of research. I'm not sure if you would agree with me, that I haven't really seen explored too much lately is monitoring based on mempool activity. So I probably want to know about a potential exploitation before it happens, right? Well, I want to know about it after it happens, of course, but ideally I want to know as it happens, right? So if we are able to build a set of monitoring tools that effectively monitor mempool and not the on-chain transactions, this might allow us to potentially, hopefully avoid some of those hacks, right? And you can think of gas price wars as well, right? Like if there's a call being done to this governance contract that changes this particular parameter to a ridiculous value, act on it directly, bump the gas price, front run that transaction, and secure your protocol. So there's, I think a lot of things that can be done. But by the way, these, been talking about this with my co-founder, there's heaps of ways that you can use that. So you have to be extremely careful how you build such an infrastructure. But I think it's definitely worth exploring. And it could have actually saved a few billion dollars over the past year. To add on the question that I asked you in the first place, these differences, I'd like to mention that one of the call marks of blockchain networks is relinquishing control and giving it over to users. And that usually means that we also give them the responsibility of managing the risk associated with that control. And so, since we no longer have it, we have to educate or make appropriate defaults that help them navigate that situation or understand how to navigate themselves whenever it comes up. And that's, I think a very differentiated part of security in crypto is that traditional security is, don't worry about it, we'll take care of it. We've kept you from being able to shoot yourself in the foot, you can't do anything. And if you can, we'll fix it for you because we have that control and see if they trust them. Whereas, it's almost anathema to most blockchain principles is we literally can't do it. So if you do it, you're should of luck. But we also have lacked considerably in the education process of helping people understand how to make the right decision at the right time and set themselves up such that they don't make bad decisions. We're learning, we're learning, we're paying very expensive school fees, but we're learning. It's very interesting to see the amount of money that you out of centralized exchanges following what happened with FTX. It is like, people are like, ah, that's gonna happen to me. Ah, that's a very reputable exchange. I've seen these guys face everywhere. You know, it's nothing bad's gonna happen. So this was a good example in my view, like the fact that this topics change came down overnight with creditors, you know, begging to get the money back. That's like, oh, okay. We have all these nerves. Yeah, yeah, exactly. We had all these nerves telling us about self custody for years and we never really listened to them. It's not convenient, is it? Well, now we understand why they were waving about this thing for years now. And I certainly see it among my close friends circle, people that are outside of this space that are somewhat interested in crypto, kind of wanna hold some crypto. I've heard about what happened at FTX and they're like, maybe what should I do? Like, I'm ready. Like you've been telling me to move my money off exchanges, I haven't listened to you for years, but I'm listening. So yeah, we'll see, school fees, expensive school fees, that I wish we didn't have to pay, that I wish the community users didn't have to pay, but unfortunately in some cases, that's the only way to learn. Did you say school fees? Yeah, it's probably what might be an Australian saying. So it's like, you know, you pay the price to learn a life lesson. Oh, call that street lessons. You get school dates, son. That's right, yeah. That's right. I have a lot of questions. Please. I was trying to find the right ones. I'll start it easy. How do you remain so pure? Like if your job is like, I'm finding, it's crazy. Go into abilities. How do you not like, oh man, I could just grab some stacks real quick and take a good vacation. Nobody would know. I'll just, you know, how do you remain so pure? I'm not. So it's a great question. I've been doing this stuff for a while, right? Like I've been a white hacker for over 11 years now. And trust me, in my pre-blockchain times, the number of occasions where I was like, let's just go guys. So like testing banks, you know, you take like, test like big four banks here in Australia. You find a way that pop the whole bank basically, right? Like I'm like a one liner away from transferring. Just take a small percentage of your transaction. You'll never know. Yeah, I remember sitting there with my manager, which was seven years ago, and we were like, who, what are life choices? This is where you like look at each other, who's like, role-wearing suits, where I was working for a big four consulting firm, doing pen testing, and like you have to be presentable. And they were like, you know, we could, we could end this, you know, right now. So we've like that temptation I've had to leave with and deal with for over a decade now. So I'm fine. But it is a real challenge, I guess, for maybe people that come from Barker hats, background people come from the black hat industry, and I've been, I've interrupted with something. Very interesting. So I think it comes down to how we incentivize people to report things. Well, obviously I'll get paid, Sigma Prime gets paid to review the security posture of our clients work. Can you imagine if we were to not disclose some of the vulnerabilities we find, and just, you know, either exploit them in the wild, or report them through bug bounties? I mean, like, yeah, this company wouldn't last a week, I don't think, if we started doing this. So obviously the problem or the challenge is when you try and attract all these white hat hackers, all these gray hats, all these deaf black hats, and they're like, look, I've got crazy rewards. Just do the right thing. Please do the right thing. Don't pop this live on mainnet. Here are a set of steps that you can follow to responsibly disclose whatever you find. And guess what? If we deem that it is critical vulnerability, I've heard protocol give 10% of the TV out. Cap that one, like, I don't know, something not too crazy, like 20neil or 10neil, but they commit to giving 10% of their TV out. So if you find something juicy on a DeFi protocol, and you can get paid a couple of million dollars with, you know, clean money, in some cases, tax-free money, as opposed to having these tainted... And reputations? Exactly. Obviously, yeah, yeah. Oh, not even talking about reputations. So black hats usually don't care too much about reputations, because they like rotate through different handles and have capitalized. There you go. But you're absolutely right. So yeah, the alternative is to like, okay, I got two-meal clean in my bank account, or I pop the protocol and I get 20, 30, but every single exchange on earth, every single DEX has now my, you know, is now like tracking these funds. It'll be extremely difficult for me to actually be able to swap them to any sort of cash or value. So, and I think things like E-Munify are certainly heading towards that direction, or at least pointing people towards responsibly disclosing things as opposed to exploiting them in the wild. And now people are like, yeah, okay, cool. Two-meal is actually pretty good, right? And someone reported a bug on a bridge and got six-meal out of it. The bridge had $8 billion worth of value. So like, six-meal is decent, right? Because if you were to drain all those tokens, well, first of all, you're probably gonna dump them on any secondary market, which will affect the valuation of what, of your loot effectively. It's not gonna be 8 billion anymore. Probably gonna be like four, three, two, five, five, five. And more importantly, you'll find it extremely difficult to actually find liquidity to exchange those tainted points. And the event that you do, depending on the jurisdiction you live in, you're gonna get come after. So like, you're never gonna escape the fact that, like, you have to use this money in real-day life, and there's a lot of police that would love to catch you. Oh yeah, oh yeah, yeah. Was that wormhole, hyper deposited to the beacon chain? Didn't they? Mm-hmm. They're like, okay, I've got all these stolen ether. What could I do with it? Let's take it. The consensus governance framework called sensor use. So check it there. Interesting. I have too many questions. I'm gonna pass the rock to you, Jess. I actually don't have a question right now. Just enjoying listening. I'm gonna take the rock back. Oh, I could go forever. So like, do you have had it? There was this interesting content I was bringing up shortly before you joined, before the interview. And that's like, we're security overlaps safety. And what I mean by that is, I'm not, I know where near as much as no weeds as everyone here in the call, I'm more of a GPP. It's a phrase that Corey is writing his paper on, called general purpose person. I'm a tier three GPP though. I've been around for, I think, yeah, 10 years. So I know the things, you know, hardware wallets got a few. But as a tier three GPP, all the three layers below me, zero through two, they all come to me for advice. You know, like, hey, should I use this exchange? Hey, you know, should I buy this alien robot in FT? Hey, should I, and I'm like, as a tier three, I'm like, look, just get out of here. Go get out of here. But GPP's need that feeling of safety. And right now, crypto's a little far from that. I have, you know, tier one's coming to me every day. Hey, you know, I went to this website and met a mask, and now my stuff's gone. And I'm like, yeah, why would you do that? That's something that you shouldn't do. But they just don't know, right? So now they're like, screw crypto, screw all of it, screw you too, for getting me involved. And how can security start to pay just a little bit more attention to what makes users feel safe? Yeah, I think that's certainly pinpointed the massive, massive challenge that industry is facing. Yeah, there's no magic answer, I don't think. The key in my view is education, right? And you might scream at me, but also regulation. Hear me out, hear me out, hear me out. Oh, my man, we'll see you get drinks, hold up. Yeah, I'm going to download, but I like that you said that. So that's like education. We need to do a better job at explaining what Minimaster does, what like either. So there's two ways to go about this. Abstract the danger and complexity away from your users. That kind of defeats the purpose of what we're trying to achieve here. And that's like going back to what Corey was saying. This is what traditional businesses, you know, web to well, businesses have been offering their users, right? It's like leave that to us. We got you covered. Do not bother with these concepts. We are here for you. Moving away from this, now we kind of have to make sure that these people understand what they're dealing with in terms of custom-y private keys. What does that all mean? Right? Like if you're like talking back to your point D, like your tier one, tier two friend's date, first of all, they've never heard about these concepts. And most importantly, they don't get fuck about these things. Nope. Right? Like what they care about is not getting popped, right? Well, they also care about, you know, their ceiling investments and JPEG going through the moon. But yeah, let's say that's a different conversation. So we need to be able to explain these things. And the reality is we might even try it, right? Like bombarding them with like technical dog and expecting them to figure their way out of it. Like, that doesn't work. That's not how we should be doing this. Enter my second point, regulation. So, okay, let me out. I was called on to help out the Treasury Department here in Australia. He's coming up with a set of, not policies, but policy guidelines, right? Like, hey, policymakers, you may have to decide about these new crypto networks. And you've got no idea what they are, do you? So here, read this paper that is meant to explain to you relatively in detail what to expect, what are these networks, how they operate, etc. So following what has happened with FTX and even before that, Australian authorities were like, we need a framework. We need like something here to tell people what to do, tell businesses what to do, what not to do. And so I rocked up great, great people, clever cookies, great questions, a couple of hours back and forth questions. And at the very end, the head of that team was like, what's your view on regulation? But the, the offense of what I said was that education should be the priority of the regulatory framework. So like, you can't force uni swab to do something. You can't force Aave to comply with, like this, this way of thinking is so 2018. Like where you'd like speak to all these government officials, they're like, no, no, no, no, no, no, no, like these lending platforms, they should do this, this, this, this, and that. Oh, like these exchanges, you know, all their decentralized exchanges, doesn't matter, their exchanges, they need to comply with this, this, this, and that. They've really quickly, I think, understood that this is actually not doable, right? Like you can't, like Aave, who's going to, like, no. How about no? So what we should be focusing on in terms of regulation is education. So, hear me out. One of the suggestions I had was establishing a set of guidelines, right? And inviting all these protocols being built by and on people sometimes to follow these guidelines. Have you, do you have documentation? You know, do you have tests? Have you received an external security review? Do you have a better bounty program? Like, there's basic stuff, right? Like, Corey and I go through this stuff like multiple times a day, super basic stuff. But at least you can then go to your citizens and be like, hey, we care about you. We think that you shouldn't be interacting with any website, with any platform that doesn't prove to you that they've gone through this set of guidelines, right? And then we can probably take this one step further and have some sort of registry that's vetted by the community or the regulators on websites or protocols that are following these guidelines. So it gets to a tricky territory very quickly, like, who is actually issuing these certifications? Do we need a certification process? Or should we just make it, like, optional for people to maybe try to seek that accreditation from the government agency, right? Like, I'm Aave, I'm like, I'm legit, I've been spending billions of dollars in security. I want Australian people to know that, you know, I'm safe to interact with, and then I'll go and knock on the Treasury store, but like, hey, I represent Aave, we can prove to you that these things are met, can we get like some sort of tick from the government? Or like, can we end up on this registry of safe protocols? This is what regulation, in my opinion, should be focusing on. Education and incentivization for protocols to comply with a certain set of guidelines. And again, those guidelines, super basic, but I do believe that if they're followed, you're dramatically reducing your risk of exposure and exploitation. I'd like to follow up with an example of that that I think people can identify with, who's listening, and that is the little lockbox on your browser, your browsing internet. That, that UI and the warnings that you receive when that thing isn't present is an example of what it means to follow a specific type of like general guideline or regulation, right? We say like, it is safe to have TLS as a part of your like, your website offering. If you don't have these certificates to allow us to establish your connections to your website, then we're going to deem you unsafe. And we're going to warm the user and put red boxes and unlocks, right? And so if someone wants to make it easy for people to navigate their website, they do these things. And it's defining those types of things and the tools that alert users when people aren't doing it, that get us to a point of people making good decisions. And so like, I think that's a reasonable, normal example of what regulation and standardization looks like. And also then education, like focusing on education for that type of thing. It's like, oh, this is unsafe, so don't do it. And if you want to, you're more than welcome to, but it's at your own risk. Yeah, you're on your own. Yeah, agreed. So that's a great way to wrap up. How do we, as always, we like to ask the question like, is there something you wish we asked that we didn't? How is doing? I'm joking. No, no, I don't think so. How are you doing, Medi? I'm doing well. Thank you. No, it was great. Great chat to you guys. I don't think it was anything that I would want to specifically, explicitly cover. I think it's interesting. Maybe you have other questions. I've got a bit of time, but otherwise I can wrap it up. My effort was to not insert my opinions in most of these things. You've done well. You've done really well.