Managing Large Data Sets in Cybersecurity Platforms

Making Sense of ICS/OT Security Monitoring: A Framework That Actually Works Comprehensive visibility is not just about having more data — it’s about collecting the right data, safely and intelligently. OT systems demand precision, patience, and respect for operational continuity. A single misstep in data collection can cause downtime, disrupt production, or even impact safety. Every mature OT cybersecurity program needs a structured Collection Management Framework — one that aligns monitoring activities with both security and operational realities. 1️⃣ Planning — Building the Foundation Effective monitoring starts with strategy. Identify critical assets, understand your threat landscape, define collection requirements, and map them to compliance obligations. Without this step, data collection becomes guesswork — and guesswork in OT can be dangerous. 2️⃣ Data Sources — Knowing Where to Listen Industrial systems generate a wealth of telemetry: PLC, RTU, and DCS logs, HMI/SCADA events, network traffic (via SPAN or TAP), and asset inventories. Each tells a piece of the story. The challenge is correlating these diverse signals without overwhelming the network or the analysts. 3️⃣ Collection — Safely Capturing the Signal Collection in OT must be non-intrusive. Passive monitoring and protocol analysis (Modbus, DNP3, IEC 61850, Profinet, OPC UA, BACnet, and others) provide deep insights without interference. When active scanning is needed, it must be controlled, scheduled, and safety-approved. 4️⃣ Analysis — Turning Data into Detection Once collected, the focus shifts to enrichment and analytics. Combine anomaly detection, behavioral modeling, and threat intelligence with correlation rules to spot early indicators of compromise. The value isn’t in the raw data — it’s in the context you build around it. >> Supporting Layers of the Framework > Storage & Retention – Design for long-term forensic preservation, using Hot/Warm/Cold tiers and compliance-aligned data lakes. > Response & Action – Automate alert prioritization, playbook execution, and SIEM/SOAR integration for timely containment. > Governance – Anchor your program in standards like IEC 62443, NIST CSF, and NERC CIP, and continuously measure metrics and lessons learned. >> Critical Considerations for ICS/OT > Zero-impact monitoring: Never disrupt real-time operations. > Architecture Awareness: Respect secure architecture best practices such unidirectional gateways and isolated networks. > Legacy devices: Many lack native logging or encryption — plan accordingly. > Safety first: Cybersecurity controls must align with operational reliability. Industrial cybersecurity isn’t about collecting everything — it’s about collecting what matters, where it matters, and without breaking the process that keeps the plant running. A well-designed Collection Management Framework bridges the gap between data and defense, turning visibility into resilience. #OTSecurity #ICSsecurity #OTSOC

Ever wondered how cybersecurity incident correlation operates at an industry scale? 🤔 Our latest CIKM 2024 paper, led by myself and Amir G., dives deep into the sophisticated mechanisms enabling the Microsoft unified security operations platform (USOP) to correlate security incidents at the billion-scale 🎉 Here’s what sets the Microsoft USOP correlation engine apart: ✔️ Geo-distributed PySpark engine capable of handling large-scale data processing with unmatched efficiency ✔️ Graph mining algorithms that optimize the correlation process, making the system scalable to billions of alerts ✔️ Breaking the boundary between 1st and 3rd party alerts by profiling all detectors to ensure key correlation safety checks are met before enabling cross-detector correlation ✔️ Real-time threat intelligence combined with expert security insights to create highly contextualized and accurate incidents ✔️ Human-in-the-loop feedback system that continuously refines key correlation processes to adapt to the ever-evolving threat landscape 🌍 This research is deployed worldwide to all Microsoft USOP customers, maintaining a 99% accuracy rate across billions of correlations and hundreds of thousands of enterprises—a feat confirmed by customer feedback and extensive investigations by security experts. For more insights, check out the paper and blog links below. Title: "GraphWeaver: Billion-Scale Cybersecurity Incident Correlation" Paper: https://lnkd.in/gcEzubRh Blog: https://lnkd.in/gBesbXEM #Cybersecurity #Graphs #DataScience #Microsoft #XDR

In the machine world, handling the sheer volume and diverse formats of log data presents a constant, significant challenge. Getting all that valuable information into a standardized, immediately actionable format remains critical, especially as we build more sophisticated detection layers. Our latest blog post series collaboration with Chip (from CardinalHQ), "From VPC Logs to OCSF: A Streaming Pipeline with Kinesis and Zephflow," directly addresses this, using AWS VPC logs as a practical example. What makes this approach fundamentally different, and a core tenet of our platform, lies in its AI-native foundation. This does not involve simply applying a static mapping. Instead, we leverage intelligent capabilities to automate the often laborious and error-prone process of schema mapping and data ingestion. This also means we are not directly inferring an LLM on every event. With millions of events per second, and the absolute requirement for deterministic execution accuracy, direct LLM inference is simply not feasible. Our method provides the automation benefits of AI where it makes sense, without sacrificing the performance and reliability crucial for production security pipelines. This post serves as a production guideline for implementing real-time, scalable log transformation, moving beyond the limitations of traditional manual parsing. This is the first in a four-part series where we progressively build out a comprehensive security data pipeline. Here, we establish the groundwork, ensuring our raw log data transforms into the Open Cybersecurity Schema Framework (#OCSF). This makes the data universally compatible and ready to feed into advanced detection and analysis systems. We are eager to share how this foundational layer integrates with the broader detection ecosystem, and I hope this practical guide helps you streamline your own log engineering efforts. https://lnkd.in/gnHpyr5d Zachary Schmerber, Mike Radka, Marios Iliofotou, Ruchir Jha, Bo LEI, Ania Kacewicz, Ph.D. #cybersecurity #datainfra #kinesis #cloudtrail #streamprocessing #AIinfrastructure

As digital footprints expand and cyber threats become more sophisticated, organizations must adopt robust security data pipelines to ensure they are well-equipped to identify, understand, and mitigate risks effectively. A strong data foundation is not just beneficial for cybersecurity at scale; it's essential to ensure these downstream security platforms have the performant underlying queries to give the visibility required. The goal is to create a seamless flow of information that is both actionable and comprehensive, enabling security teams to react swiftly and decisively. 👉 Comprehensive Visibility: At its core, cybersecurity is about visibility. Without a complete view of what's happening across all systems and networks, security teams are blind to the actions of potential threat actors. A strong data foundation built through well-designed security data pipelines ensures that all relevant data is captured, normalized, and made readily available for analysis. This visibility is crucial for detecting correlated signs of compromise that could otherwise go unnoticed until it’s too late. 👉 Scalability: Cybersecurity threats evolve rapidly, and so too must the defenses. Security data pipelines facilitate scalability by automating data ingestion and analysis. As data volumes grow, these pipelines ensure data gets where it needs to go, in the format it needs to be, processing vast quantities of information efficiently. This scalability ensures that security measures can keep pace with expanding network perimeters and increasingly sophisticated attacks. 👉 Speed and Precision in Threat Detection and Response: In cybersecurity, speed is of the essence. The faster a potential threat can be identified and mitigated, the less damage it can do. Security data pipelines accelerate the detection process by leveraging advanced analytics, machine learning, and artificial intelligence to sift through mountains of data in real-time. They enable precise threat detection by correlating disparate data points, highlighting anomalies, and suggesting actionable insights. 👉 Regulatory Compliance and Risk Management: With increasing regulatory demands around data privacy and security, organizations must ensure they have robust mechanisms in place to protect sensitive information. A strong data foundation allows for the enforcement of compliance policies automatically. Being able to securely and efficiently get all your data to S3 or equivalent object storage, then rehydrate that data into SIEMs as needed, is extremely valuable. #otel #ocsf #securitypipelines #telemetrypipelines #siem Edge Delta #cybersecurity #security #splunk #crowdstrike #sentinel

In the fourth part of our series on Security Data Fabrics, we talk about how it enables scalability and flexibility in large organizations. As a quick refresher, a security data fabric is an architecture that centralizes data access while retaining distributed data processing, allowing multiple data sources to operate as independently managed systems yet cohesively available for all types of security analysts. A well-designed security data fabric allows organizations to manage large, detailed data sets efficiently while providing extra care for critical information. This balance ensures that data remains both scalable and flexible, ready to adapt to the ever-changing landscape of security threats. Scalability refers to a system’s ability to grow and handle increasing amounts of work or data. In security, this means being able to manage more data from more devices as an organization grows. Flexibility means being able to adapt quickly to new threats and changes in the environment. 💥 So, how does it work? A security data fabric works by allowing data to remain in its source systems while smaller, important data sets are copied and stored. These smaller sets include things like: ✨ Alerts: Notifications about potential security issues. ✨ User Data: Information about users, their roles, and their behavior. ✨ Device Data: Details about the devices connected to the network. ✨ Policies: Rules and guidelines that govern the organization’s security practices By keeping the detailed data in place and only copying necessary information, organizations save on storage costs and maintain a clear, comprehensive view of their security landscape. Security threats are always changing, so it’s crucial for security systems to be adaptable. A security data fabric helps organizations stay ahead of these threats by: ✨ Quick Adaptation: With a flexible data management system, organizations can quickly adjust their security measures in response to new threats. ✨ Comprehensive Analysis: By maintaining historical data, organizations can analyze past incidents to improve future security measures. By keeping large data sets in their original systems and only copying smaller, critical information, organizations can save costs and adapt quickly to new threats. If you'd like to read more on this topic and the entire series, the link is posted in the comments section. #SecurityDataFabric #Leen #DataFabric #Engineering #Security #UnifiedAPI