Skip to content
MG
All work
DjangoPolyglot persistenceHybrid searchDocker

SociaLink

A unified operational data store for social presence

SociaLink pulls a person's activity across social platforms into one operational data store and makes it answerable in plain language. Built as a polyglot-persistence exercise that turned into a real system: layered ingestion pipelines, hybrid full-text and vector retrieval, and an assistant with awareness of the whole store rather than a single document.

Role
Full-stack Developer & Architect
Team
Solo
Timeline
2024 — 2025
Status
Prototype
At a glance
storage engines
3
Relational, document and vector, each where it wins
pipeline layers
3
Bronze, Silver, Gold
retrieval modes
2
Full-text and vector, fused

Why build it

Your own social history is technically yours and practically unsearchable. Every platform gives you a feed and no way to ask a question across years of it.

The engineering interest was the shape of the problem: the same record wants to be relational, document-shaped and semantic at once. That is a real polyglot persistence case rather than a contrived one.

What it taught me

That layered pipelines earn their cost the first time an enrichment step is wrong. Because Bronze is immutable, fixing the logic and replaying beats patching production data — and I learned that by needing it.

It is also where the retrieval instincts behind PROM's knowledge graph came from: pure vector search feels magical in a demo and fails on exact names, which is why hybrid fusion is the default there too.

System design

How it fits together

  1. Sources

    • Platform APIs

    • Exports

    • Webhooks

  2. Pipeline

    • Bronze

      raw, immutable

    • Silver

      normalised, deduped

    • Gold

      enriched, embedded

  3. Storage

    • PostgreSQL

      entities · relations

    • MongoDB

      raw documents

    • Vector store

      semantic index

  4. Query

    • Hybrid retrieval

      lexical + vector

    • NL assistant

      store-wide context

Raw in, enriched in the middle, queryable at the top. Each layer is rebuildable from the one below it.

What I built

The work, specifically

  1. 01

    Designed a polyglot persistence layer — Postgres for relations, MongoDB for raw documents, a vector store for semantics — rather than forcing one engine to do all three jobs.

  2. 02

    Built Bronze, Silver and Gold ingestion stages so raw payloads stay immutable and every enrichment is reproducible from source.

  3. 03

    Implemented hybrid retrieval fusing lexical and vector results, which handles both exact-name lookups and vague conceptual questions.

  4. 04

    Made the assistant aware of the store rather than a single document, so questions can span platforms and time ranges.

  5. 05

    Containerised the whole stack so a clean machine is one compose command from a working system.

Stack

What it is made of

Backend
PythonDjangoDjango REST FrameworkCeleryRedis
Data
PostgreSQLMongoDBVector storeBronze/Silver/Gold pipelines
AI
LangChainEmbeddingsHybrid retrievalRAG
Ops
DockerDocker Compose
Visuals
SociaLink landing page
Landing concept introducing the unified store and natural-language navigation.

Next case study

prom CLI & MCP bridge

Live product context inside a coding agent