The standard enterprise SIEM pulls safety log knowledge from sources throughout the IT atmosphere, then normalizes it, analyzes it and retains it. However as a result of SIEM suppliers sometimes cost extra to carry extra knowledge, organizations usually should retain much less knowledge than they would favor and settle for the restrictions of subsequent analyses.
Moreover, SIEMs retain knowledge in their very own, usually proprietary codecs. In reality, how SIEM distributors parse and normalize knowledge is a method they differentiate themselves from opponents. Every seeks to make use of distinctive schemas, compression strategies and specialised databases to enhance each consequence high quality and velocity. Consequently, enterprises have restricted enter into how their knowledge is ingested and digested, and proprietary parsing and codecs could make it more durable to alter distributors.
Some CISOs — discovering the restrictions and trade-offs of information ingestion and retention in SIEM too constricting — are selecting to decouple their safety log knowledge feeds from their SIEMs. By doing so, they sometimes achieve freer entry to the information, improve management over retention timelines, enhance analytical capabilities, rein in SIEM prices and break freed from vendor lock-in. However decoupling knowledge from the SIEM additionally has its challenges and requires vital dedication, funding and planning.
How decoupling knowledge from the SIEM works
To decouple safety knowledge sources from the SIEM, safety groups insert programs that they management in the course of these knowledge flows. In follow, this implies establishing a separate, devoted knowledge retailer to carry the safety log knowledge, sometimes a knowledge lake residing in a relatively cheap cloud storage service. It additionally means establishing a brand new knowledge pipeline that takes in log knowledge, preprocesses and normalizes it after which dumps it within the knowledge lake. The enterprise then feeds its SIEM with knowledge from the lake.
Advantages of decoupling SIEMs from knowledge pipelines and storage
Establishing an unbiased, enterprise-controlled knowledge layer between the sources of safety log knowledge and the purposes that devour it — e.g., SIEMs and different instruments resembling consumer and entity habits and analytics — permits the enterprise to do the next:
- Dictate the information schema for log data.
- Fully management filtering of data and simply fluctuate it by vacation spot.
- Fully management the retention horizons for each sort of knowledge from every platform.
- Precisely and simply monitor all safety knowledge sources and all safety knowledge shoppers.
- Simply implement constant adherence to institutional polices on knowledge assortment and retention.
- Simply add new safety instruments that want entry to current knowledge feeds.
- Simply change — and even drop — SaaS and SIEM distributors with out dropping knowledge.
Buying and selling costlier SIEM-based storage for cheaper cloud bulk storage may even in all probability scale back the price of storing safety knowledge, per se. However — and that is vital to know — that price discount may not end in web financial savings, as new instruments or companies and workers time prices might overbalance these financial savings.
Challenges of decoupling SIEM from the information layer
In fact, together with its advantages, decoupling knowledge from SaaS or SIEM platforms additionally comes with challenges. These embody the next:
- Designing a robust, safe, scalable and cost-efficient knowledge lake and knowledge pipeline, together with deciding on acceptable knowledge change protocols and knowledge storage schemata.
- Engineering a robust, safe, scalable and cost-efficient knowledge lake and knowledge pipeline, together with deciding on instruments and companies with which to construct it and testing it adequately earlier than placing it into manufacturing.
- Migrating to the brand new structure with out knowledge loss or interruptions in safety scanning.
- Working and supporting the information lake and pipeline effectively, together with making certain backups and continuity of service within the face of disruptions.
- Dealing with latency created by interposing the brand new layer — requiring consideration within the design, engineering and operations phases, in addition to steady monitoring to make sure latency is inside acceptable limits.
- Dealing with compliance, as the brand new knowledge layer should respect and implement any relevant necessities — relying on firm kind, sector and geography — for knowledge at relaxation and in movement.
A decoupling toolbox
CISOs creating a brand new enterprise safety knowledge lake might want to decide their methods within the following areas.
SaaS knowledge extraction
SaaS knowledge extraction instruments may be inbuilt home utilizing SaaS APIs. Alternatively, third-party approaches embody such proprietary SaaS safety posture administration platforms as Obsidian Safety, NetSkope SSPM and AppOmni, in addition to open supply instruments resembling Mondoo and OpenASPM.
Knowledge pipeline
The info pipeline is the ingestion and pre-processing device that receives uncooked logs and spits out data for the information lake in standardized format(s). Business merchandise right here embody Cribl, DataDog and Splunk. Open supply choices embody Vector, Logstash and Fluentd.
Knowledge storage
Most bigger organizations have already got expertise with knowledge lakes, in addition to most popular distributors, resembling Snowflake and Google BigQuery, or open supply choices, resembling Apache HDFS or MinIO.
Enterprises even have to contemplate knowledge codecs. Open requirements needs to be everybody’s first alternative: Open Cybersecurity Schema Format for the log data heading out to SIEMs or elsewhere, for instance, and storage codecs resembling Apache Parquet or Delta Lake for the information lake correct.
By decoupling cybersecurity knowledge ingestion and retention from their SIEM platforms, CISOs can achieve management, flexibility and depth whereas doubtlessly decreasing prices. However they must make investments vital sources to seize these advantages.
John Burke is CTO and a analysis analyst at Nemertes Analysis. Burke joined Nemertes in 2005 with practically twenty years of know-how expertise. He has labored in any respect ranges of IT, together with as an end-user help specialist, programmer, system administrator, database specialist, community administrator, community architect and programs architect.
