Real-Time Personalization Engine Using Kafka, AWS Lambda & DynamoDB

Summary

An e-commerce platform, needed to deliver personalized product recommendations in real time. Their existing system caused delays and outdated suggestions, hurting conversions. I worked remotely with their team to build a Kafka-Lambda-DynamoDB based recommendation pipeline that achieved sub-100ms latency and boosted conversions by 18%.

Business Challenge

SwiftCart’s personalization engine was entirely batch-driven, using Spark jobs to update recommendations every few hours. This resulted in:

• 5-second API latency during traffic surges.
• Stale, irrelevant recommendations during live sessions.
• Stagnant conversion rates during flash sales and campaigns.

The company wanted a real-time solution to detect user intent and adjust recommendations instantly during shopping sessions.

Solution and Architecture

I implemented a real-time personalization engine powered by Kafka, AWS Lambda, and DynamoDB. User clickstream data was streamed, enriched with historical preferences, and served back in under 100ms through a low-latency backend.

• Kafka: Collected real-time clickstream data (product views, clicks, scrolls).
• AWS Lambda: Enriched each event using user profiles, calculated relevance scores, and updated top picks.
• DynamoDB: Stored real-time recommendations per user, optimized for fast access via GSIs and DAX.
• Frontend API: Queried DynamoDB to fetch personalized results on every page load.
• Monitoring: Used CloudWatch, X-Ray, and alarms to maintain high reliability.

Steps Taken

• Developed Kafka consumers with event aggregation logic for interaction pattern detection.
• Created Lambda functions that merged behavior signals with user profile traits in DynamoDB.
• Defined scoring rules to rank product recommendations per user and wrote top 5 to DB.
• Optimized DynamoDB performance using DAX and GSIs for low-latency read/write operations.
• Implemented CloudWatch and X-Ray monitoring, and rollback mechanisms with blue/green deployment.

Impact and Results

• Recommendation latency dropped from 5 seconds to under 80ms (P95).
• Conversion rate increased by 18% within two weeks post-deployment.
• Users saw fresh, relevant product suggestions within seconds of interaction.
• Built a foundation for future ML-based recommendation integration with SageMaker pipelines.

This project blended real-time stream processing and low-latency backend design to directly enhance user experience and drive measurable business results for a growing e-commerce company.