Aws Streaming Analytics Pipeline

Fire TV-Style Streaming Engagement & Ad Analytics Pipeline

An end-to-end AWS data engineering pipeline that ingests, models, and reports on streaming-platform telemetry — video watches, ad impressions, and app installs/launches — the way a Fire TV / Appstore analytics platform would. Built as a portfolio project to demonstrate the AWS data stack (Kinesis, S3, Glue, Lambda, Redshift, IAM) applied with a real dimensional model, real data-quality controls, and a working GenAI self-service query layer, rather than a single-notebook demo.

**Data is synthetic**, generated by [`data_generator/generate_events.py`](data_generator/generate_events.py) — this is a portfolio/learning project, not a production system, and it's designed to be fully runnable on a laptop with no AWS account (see [Running it locally](#running-it-locally) below). The AWS resource provisioning, IAM policies, Glue jobs, and Redshift DDL are written the way they would actually be deployed in production.

Architecture

flowchart LR
    EV[Fire TV events\nwatch / ad / install] --> KDS[Kinesis Data Streams]
    KDS --> KDF[Kinesis Data Firehose]
    KDF --> RAW[(S3 raw zone)]
    RAW -- ObjectCreated --> LAMBDA[Lambda trigger]
    LAMBDA --> GLUE[Glue ETL + data quality]
    GLUE --> CURATED[(S3 curated zone, Parquet)]
    GLUE -- DQ failure --> SNS[SNS alert] --> CW[CloudWatch / on-call]
    CURATED -- COPY --> RS[(Redshift star schema)]
    RS --> BI[BI / Data Science / NL-to-SQL agent]

Sample output

Generated locally from synthetic data via `notebooks/exploratory_analysis.ipynb` — the same metrics a BI dashboard on top of the Redshift star schema would surface:


Full write-up, including why each service was chosen and a line-by-line map to the target job description: [`docs/architecture.md`](docs/architec