AI · Financial media · UAE
DRO: AI Financial News Intelligence
A scheduled AI pipeline that turns raw financial news into persona-tailored stories on UAE market impact, with editors kept in the loop.
80%less editorial effort
The challenge
Financial readers drown in duplicate coverage. Fifteen outlets report the same geopolitical event, and someone has to read all fifteen.
The editorial team wrote every persona summary by hand. Rule-based grouping of related articles was inaccurate and needed manual fixing, and tagging for companies, sectors and market relevance was thin.
The brief: a scalable aggregation framework that ingests news from many sources, structures it with AI, lets editors curate through a CMS, and serves each reader a feed shaped to their persona. New sources, personas and model improvements had to plug in without rework.
Our Founder's role
- Engineering manager for the CMS and frontend layers, leading the India-based team. A US-based AI/ML team owned collection, ETL and clustering.
- Owned the Sanity schema, the REST APIs that publish from the AI layer into the CMS, and the Next.js application.
- Owned end-to-end integration with the solution architect: every stage wired into one pipeline, tested so that what the AI layer produced reached the reader without failure, and documented for the client.
How it works
- 01
Ingest
Cloud Scheduler triggers a Cloud Run job that pulls RSS feeds and the AskNews API, normalises them and stages them as Parquet in Cloud Storage. Every article is SHA-256 hashed against Redis and duplicates are dropped. Hashes commit only after enrichment, so the pipeline restarts safely without reprocessing.
- 02
Enrich
A FastAPI service on Cloud Run uses Gemini on Vertex AI with DSPy structured prompting to tag each article with entities, industries, regions, stock tickers and persona-relevance scores, stored in Cloud Spanner.
- 03
Cluster into stories
Articles are embedded with a sentence transformer (all-MiniLM-L6-v2), reduced with UMAP and grouped by HDBSCAN, which finds the number of stories from the data rather than a fixed k. Maximal Marginal Relevance then picks the most representative, least redundant articles to feature per story.
- 04
Summarise and moderate
Gemini Flash generates a story title and description from the most relevant articles, then one summary per persona: general readers, financial advisors, market analysts and retail investors. Every summary is scored by a moderation gate: safe content publishes automatically, borderline content goes to a human review queue, and violations are blocked.
- 05
Publish and serve
Approved stories are pushed to Sanity through its REST API, where editors move reviewed items from In Review to Published through a custom admin plugin. A Next.js app on Vercel queries Sanity with GROQ. Content is pre-tagged by persona, edition (US, UK, UAE) and entity, so no AI runs at read time.
Engineering decisions
Model choice and prompt management
- Gemini Flash on Vertex AI: fast and cheap enough for high-volume daily news, with all data kept inside the client's Google Cloud project.
- DSPy defines entity extraction as typed inputs and outputs rather than free-text prompts, so results come back in a predictable shape.
- One prompt template per persona: the story stays constant and only the reader instructions change. The model name lives in configuration, so moving to a newer Gemini version is a config change, not a code change.
Humans review only what needs it
- The moderation gate publishes safe summaries automatically and sends only borderline items to editors. Nothing questionable goes live without a person checking it.
Caching designed around the read path
- Server-side rendering degraded as the archive grew into thousands of articles. Moved to Incremental Static Regeneration with background revalidation every 10 minutes.
- Each paginated slice is cached server-side with its own key. The next page is prefetched as soon as the current one loads, so “Load more” is a cache hit.
- All persona summaries ship with the first fetch. Switching persona is handled entirely on the client, with no new request.
Freshness without rebuilds
- Time-based revalidation alone can cache content that is already stale. Built and validated on-demand revalidation: a Sanity webhook hits /api/revalidate and the page cache is invalidated the moment a story publishes.
CMS ready before the AI
- Modelled the Sanity schema (stories, articles, persona summaries, tags and stock data) and seeded it with realistic fixtures, so the frontend shipped in parallel with the AI layer.
Outcomes
- Editorial work that took about 10 hours now takes 1–2: an 80–90% reduction, quoted conservatively as 80%. Measured by comparing the team's time before and after.
- Editors' remaining time goes to reviewing borderline items and spot-checking, not writing from scratch.
- Duplicate coverage collapses into a single story per event.
- Zero AI cost or latency at read time. All heavy lifting happens upstream.
- New sources, personas and model upgrades plug in without re-architecture.
Technology
- Google Cloud Platform
- Backend infrastructure
- Cloud Scheduler
- Triggers the ETL pipeline
- Cloud Run
- Batch ETL job and enrichment API
- Redis
- Content-hash deduplication
- Cloud Storage
- Pipeline artifacts between steps
- Cloud Spanner
- Structured article metadata
- Vertex AI · Gemini Flash
- Enrichment, summaries, moderation
- DSPy
- Structured prompting for entity extraction
- all-MiniLM-L6-v2 · UMAP · HDBSCAN
- Embeddings and story clustering
- MMR
- Diverse featured-article selection
- Python · FastAPI
- Aggregator API
- Sanity · GROQ
- Headless CMS and query layer
- Next.js · Vercel
- Persona-tailored web app
Contact
Have something you want to build?
Tell us what you're building, or what you'd like to automate. Let's figure out the next step.