What Amazon Security Lake is
Amazon Security Lake is a managed service that builds a security data lake in your own AWS account. It collects logs and findings from AWS services, and accepts data from on-premises systems, SaaS products and third-party security tools. It stores everything in Amazon S3 buckets that you own, converts it to Apache Parquet, and normalizes it to the Open Cybersecurity Schema Framework (OCSF).
The problem it solves is familiar to any security team. CloudTrail, VPC Flow Logs, DNS logs, firewall logs and endpoint alerts all arrive in different formats, in different accounts and Regions. Before you can investigate anything, someone has to collect, parse and join them. Security Lake does the collection and normalization so your analytics tools can start from one consistent dataset.
It is worth being precise about what it is not. Security Lake is a storage and normalization layer, not a detection engine or a SIEM. It does not analyze your data or raise alerts by itself. Detection happens in the tools that read from it: Amazon Athena queries, your SIEM, or a partner analytics product.
OCSF: one schema for everything
OCSF is an open source schema project started by AWS and security industry partners. It defines event classes such as API Activity, Network Activity, DNS Activity, Authentication and Security Finding, each with standard fields.
Because every source in Security Lake maps to an OCSF class, a query for a suspicious IP address can look at the same field names whether the record came from VPC Flow Logs, a firewall or a third-party EDR. For the current AWS source version (version 2), Security Lake writes OCSF 1.1.0.
Sources
Natively supported AWS sources
Security Lake can collect these AWS sources directly:
- AWS CloudTrail management events, and data events for Amazon S3 and AWS Lambda
- Amazon EKS audit logs
- Amazon Route 53 Resolver query logs
- AWS Security Hub CSPM findings (which carry findings from GuardDuty, Amazon Inspector, Amazon Macie and others)
- Amazon VPC Flow Logs
- AWS WAF (WAFv2) logs
Security Lake reads most of these through an independent stream, so you do not need to configure logging in those services first and your existing logging is not affected. CloudTrail management events are the exception: AWS notes that you must configure logging for them separately.
You choose which sources to collect, in which Regions and for which accounts. High-volume sources such as S3 data events and AWS WAF logs can add significant cost, and the default source selection for new accounts leaves them out for that reason.
Custom sources
Anything else, including on-premises firewalls, identity providers, endpoint tools and SaaS applications, comes in as a custom source. The source is responsible for converting its data to OCSF and Parquet before writing it. Security Lake gives each custom source its own S3 prefix, an IAM role to write with, an AWS Lake Formation table and an AWS Glue crawler to register partitions.
The requirements are strict, and they matter for query performance:
- One OCSF event class per custom source. A product that emits several kinds of events should be split into several sources.
- Parquet files, partitioned by Region, account and event day, with records sorted by time.
- Files delivered in batches between 5 minutes and one day apart.
- OCSF version 1.3 or earlier for custom sources.
Many security vendors now ship OCSF-ready integrations that write to Security Lake directly. AWS also publishes an OCSF validation tool to check your own pipelines.
Subscribers: how tools use the data
A subscriber is any service or tool that consumes data from the lake. You grant access per source, so a team or tool sees only the data it needs. There are two kinds:
- Data access subscribers are notified when new objects land in S3, either through an HTTPS endpoint they provide or by polling an Amazon SQS queue, and then read the Parquet files. This suits tools that ingest the data into their own platform, such as a SIEM.
- Query access subscribers query the data in place through Lake Formation tables, typically with Amazon Athena or other engines that integrate with AWS Glue. This suits tools that search the data without copying it.
Each subscriber can be given up to 10 sources. Security Lake creates and exchanges the credentials and permissions needed between your account and the subscriber.
AWS lists many third-party integrations as subscribers. Splunk, for example, can ingest Security Lake data through the Splunk Add-on for AWS or search it in place with federated analytics. IBM QRadar, Sumo Logic, Rapid7 InsightIDR, Palo Alto Networks and others are on the same list.
For your own analysis, Athena works well for investigation and threat hunting, because the OCSF field names stay the same across sources.
Regions, accounts and retention
- Per-Region storage. Security Lake stores data in a separate S3 bucket in each Region where it is enabled, and you can configure sources and retention per Region.
- Rollup Regions. You can designate a rollup Region and have other contributing Regions replicate their data to it, so subscribers can work from one place. Choose rollup Regions with data residency requirements in mind: if data must stay in a country or jurisdiction, keep it out of a rollup in another one.
- Organizations. The management account designates a delegated Security Lake administrator, which AWS recommends be your Log Archive account. The management account itself cannot be the administrator. The delegated administrator can enable sources across member accounts and set up automatic collection for new accounts.
- Retention and storage classes. You set an expiration period and S3 storage class transitions per Region, so older data can move to cheaper storage and eventually expire.
Costs to plan for
Security Lake charges for data ingestion from AWS sources and for normalizing data to OCSF and Parquet, with CloudTrail priced differently from other AWS logs. There is no Security Lake ingestion charge for custom sources. Amazon S3 storage, AWS Glue, Lambda, SQS and query costs are billed separately. New accounts get a 15-day free trial, and the console shows estimated usage costs. Check the pricing page for current rates before enabling high-volume sources.
Getting started
- Designate the delegated administrator and enable Security Lake in the Regions you use.
- Start with CloudTrail management events, VPC Flow Logs, Route 53 Resolver query logs and Security Hub CSPM findings.
- Add a query access subscriber for your security team and test a few investigation queries in Athena.
- Connect your SIEM as a subscriber, then add custom sources for on-premises and third-party data.
- Review retention and rollup settings against your compliance and residency requirements.
Getting help
9Acts is an AWS Advanced Tier Services Partner with the AWS Level 1 MSSP Competency. If you want help designing a security data lake or connecting your existing tools to it, see our security and compliance services.