AI-Powered Reddit Analytics Pipeline
Automated keyword-driven Reddit ingestion with LLM-based post insights, comment evaluations, and Excel reports.
The Reddit Brand Monitor Service is a backend analytics pipeline built on FastAPI. It allows brands to track mentions, measure public sentiment, detect high-intent buying signals, and evaluate community discussions in real time.
The system handles data ingestion by scanning subreddits for specific keywords using a rate-limit-aware HTTP fetch client with retries and exponential backoff. It can retrieve both main posts and comment trees down to several nesting levels. Raw data is cached in MongoDB, which stores collections for raw posts, raw comments, post insights, overall summaries, and comment evaluations.
The analysis engine batches text feeds and calls an LLM completion service (using LiteLLM or direct endpoints with JSON Schema formats). It extracts sentiment scoring, category classification (e.g. comparison, complaint, recommendation), and intent indicators. For comments, the engine analyzes whether they accurately answer the parent post and suggest custom rewrites if the post's relevance is weak. Analysts can stream multi-sheet Excel reports containing full evaluations directly from the service.
End-to-end data flow through the pipeline.
Robust Reddit Ingestion
An ingestion layer queries Reddit search endpoints for keywords. It features custom rate-limit prevention, retry limits, and exponential backoff to handle Reddit's API limits. De-duplicated posts and recursively parsed comment trees are mapped by keyword and indexed directly in MongoDB.
JSON Schema Post Classification
The pipeline groups posts into batches and uses structured LLM calls with JSON Mode to analyze brand intent. It evaluates sentiment, intent categories (complaint, comparison, purchase intent), and marks posts as 'high-intent questions' using key trigger phrases.
Comment Quality & Relevance Audits
The system matches parent posts against comment trees to verify comment relevance and evaluate answer quality. If a comment is marked as irrelevant, the LLM generates a suggested relevant comment rewrite, enabling automated moderation or response templates.
Multi-Sheet Excel Reporting
FastAPI generates a StreamingResponse using Pandas ExcelWriter and BytesIO, enabling the client to download a formatted, multi-sheet spreadsheet (`_analysis.xlsx`) matching the target keywords instantly.
Why each tool was chosen.
API Endpoint Engine
Provides highly efficient async API paths, automatic documentation, and request schema modeling via Pydantic.
Document Cache & Metrics Store
Allows flexible storage of unstructured Reddit JSON feeds and structured LLM analytics metrics under compound database indexes.
Structured LLM Orchestration
Enables uniform model routing, structured JSON outputs, and safe retry chains for classification and text generation.
Data Export Processing
Simplifies aggregation logic and outputs complex multi-sheet Excel files directly from memory without writing physical files to disk.