Architecture of a RAG Pipeline with Confluence → Azure Synchronization
RAG (Retrieval-Augmented Generation) systems need structured, up-to-date data to work effectively. Yet enterprise documentation typically lives in Confluence, a system not optimized for LLM consumption. And a one-shot migration isn't enough: documentation keeps evolving, opening a gap between the migrated knowledge base and the RAG system that consumes it.
To address this challenge, we designed an automated pipeline that synchronizes Confluence to Azure daily, following a five-layer architecture: ingestion (the Confluence REST API selectively extracts pages and metadata, filtered by space and modification date), transformation (HTML is converted to structured Markdown that preserves semantic hierarchy), storage (Azure Blob Storage mirrors the Confluence space structure), indexing (Azure AI Search generates vector embeddings for semantic search), and orchestration (Azure Functions drives the whole pipeline via scheduled triggers). Across all layers, Azure Key Vault centralizes credentials and secrets to guarantee compliance with enterprise security standards.
This architecture eliminates manual maintenance while guaranteeing consistency between source documentation and the RAG system, enabling organizations like Adservio to deploy generative AI capabilities built on their proprietary knowledge.
Technical Requirements for the Pipeline
A high-performance Confluence-Azure synchronization pipeline rests on three requirements we systematically encounter at Adservio in enterprise contexts: operational efficiency, transformation quality, and architectural resilience.
Incremental Synchronization and Change Detection
In a context where Confluence bases contain between 10,000 and 100,000 pages, syncing the entire corpus on every run means hours of processing and significant API costs. Our change-detection strategy relies on three mechanisms: the last-modified timestamp exposed by the Confluence API (the version.when field), a metadata table in Azure Table Storage that serves as the registry of already-synced versions, and a delta sync that extracts, transforms, and indexes only the pages modified since the last run.
The result: daily execution time drops from 2 hours to 5 minutes for a 15,000-page corpus (-95%), Confluence API calls fall from 15,000 to an average of 150 (-99%), the monthly Azure Functions cost from €75 to €15 (-80%), and the maintenance window from 2-3 hours to 5-10 minutes (-96%), all while keeping data fresh for downstream RAG systems.
HTML-to-Markdown Transformation
Confluence's native HTML contains presentation markup (CSS classes, layout attributes, navigation structures) that dilutes the semantic signal for LLMs and generates 40 to 60% token overhead compared to pure Markdown, with no added informational value. Our transformation pipeline chains five steps: HTML parsing with BeautifulSoup4, extraction of meaning-bearing structures (hierarchical headers, lists, code blocks), removal of proprietary Confluence macros that don't export correctly, Markdown conversion via html2text, then post-processing (whitespace normalization, structural validation).
This transformation cuts vector embedding size by 45% while improving recall@5 by 23% in our internal RAG benchmarks: more compact, structured content produces better retrieval accuracy.
Serverless Architecture with Auto-Scaling
The Azure Functions architecture rests on four pillars: a Consumption Plan billed to the millisecond of execution (no fixed costs during idle periods), a Timer Trigger scheduled daily at 2am UTC to spare source systems, an adaptive 10-minute timeout per function with chunking for large batches, and an exponential backoff retry policy capped at 3 attempts to absorb transient failures. It delivers a throughput of 50 pages per minute, an 8-second cold start for the Python runtime, 99.95% availability (Azure SLA), and a monthly cost of about €15 for 15,000 pages synced daily.
Selective Migration by Space
Rather than migrating entire datasets, the pipeline supports selectively migrating only the relevant spaces to Azure Blob Storage, driven by three imperatives: contextual relevance (a sales assistant doesn't need engineering runbooks; filtering by space increases the useful signal for each persona), governance (sensitive spaces, HR data, commercial strategy, intellectual property, must never feed certain models, in GDPR compliance), and economic optimization (why pay to store and index data that will never be queried?).
Concretely, a full migration of 50 spaces (500 GB) costs about €250 per month. A migration limited to the 20 relevant spaces (200 GB) brings the cost down to €100 per month while improving recall by 15%. A persona-targeted migration (3 to 5 spaces, about 50 GB and €25 per persona) pushes the recall gain to 25%: finer-grained selection improves cost and response relevance at the same time.
How We Carried Out the Migration
The services used for this migration are Azure Blob Storage, Azure Functions, Azure Key Vault, Azure Identity, and Azure AI Search. Choosing the Azure ecosystem was strategic for our clients already invested in Microsoft, but the architectural principles apply to any cloud provider (AWS, GCP).
The Seven Phases of the Pipeline
Environment setup: loading configuration, initializing the Blob Storage client with managed identity authentication (zero plaintext credentials), OAuth 2.0 authentication against the Confluence API, and structured JSON logging to Azure Application Insights. Retrieving and filtering spaces: whitelist, blacklist, and regex patterns determine the scope, excluding temporary, archived, or irrelevant spaces. Content extraction: pages and blog posts are retrieved with pagination, along with their critical metadata, body.storage (the native HTML, more structured than body.view), version.number and version.when for change tracking, ancestors[] to rebuild the document tree, labels for semantic categorization, and restrictions to preserve ACLs.
Content transformation: the most critical step, converting HTML to clean Markdown, cleaning up paths and URLs, and preparing YAML front matter. The quality of this transformation directly determines the recall, precision, and relevance of the RAG systems built on top. Azure storage operations: creating new content, updating modified content, and deleting content removed from Confluence, while faithfully maintaining the space hierarchy.
Statistics and logging: the pipeline tracks the number of pages added, updated, and deleted, and the processing time per space. Automated alerts trigger on thresholds: a variation of more than 50% in the daily page delta, processing time exceeding twice the baseline, an error rate above 5%, or average document size growing more than 30% per month. Finalization: cleaning up orphaned blobs, generating detailed migration summaries (JSON plus a human-readable report), and sending error notifications to monitoring channels (Slack, Teams, PagerDuty).
The Business Value of an AI-Ready Knowledge Base
Migration is only a technical foundation: the real value comes from what you build on top of it. The first lever is accelerated knowledge discovery: by enriching extracted content with summarization, keywords, and semantic tags, we've observed 60 to 70% reductions in information search time across our clients.
The second lever is smarter enterprise search: with Azure AI Search embeddings, users ask questions in natural language ("What's our refund policy for international projects?") instead of guessing keywords, and get the most relevant answers directly rather than dozens of results to sift through. Incremental updates also avoid the "stale documentation" phenomenon that erodes teams' trust in internal systems.
Specialized AI Assistants and Onboarding
With a clean, structured base, you can build specialized assistants with targeted scopes: an HR assistant for company policies, a technical assistant backed by runbooks, a sales assistant for product information during calls. Grounded in proprietary knowledge via RAG, they reduce hallucinations by 80 to 95% compared to generic LLMs. For onboarding, an assistant powered by the migrated base instantly answers recurring questions: equipped organizations cut new-hire ramp-up time by 30 to 40% (from 8 to 5 weeks on average).
Centralization finally unlocks analyses that were previously impossible: which topics are best documented, where the critical knowledge gaps are, which documents are outdated or barely used, insights that guide documentation and continuous improvement efforts.
Key Takeaways
Deploying this pipeline across several Adservio clients produced three major lessons, spanning the technical, security, and data quality dimensions.
Use Markdown for documentation: human-readable, Git-versionable, and unambiguously machine-parsable, it preserves information hierarchy (headers, lists, code blocks) while staying compact. Where raw HTML (about 15 KB per page) drowns LLMs in presentation noise and PDF is neither parsable nor versionable, Markdown (about 8 KB) combines human readability, optimal LLM parsing, and frictionless Git versioning.
Secure the exchanges: connecting to Confluence with SSL certificates encrypts data exchange end-to-end, a compliance prerequisite in regulated environments (GDPR, HIPAA, PCI-DSS). We recommend rotating certificates at least every 90 days and actively monitoring their expiration with automated alerts.
Capture only relevant data: aggressively filtering out off-topic content cuts storage costs by 40 to 60% while improving AI response quality by 15 to 25%. The more irrelevant data the model receives, the more likely it is to produce confusing answers: data curation is an investment that pays off, in user experience and in economic efficiency alike.
The Importance of Governance in AI Pipelines
Governance is a frequently underestimated pillar of AI knowledge pipelines. At Adservio, we insist: it must be built in by design, not added afterward as a regulatory patch.
Traceability and audit: every operation is logged in a structured way, which page was extracted, when (UTC timestamp), by which managed identity, which transformations were applied, and to which blob path. In regulated sectors like finance or healthcare (MiFID II, HIPAA), demonstrating complete data lineage is a legal requirement; our JSON logs integrate with SIEMs like Splunk or Azure Sentinel, with configurable retention.
Versioning and rollback: every pipeline run creates a restore point in Azure Blob Storage, with a lifecycle policy to keep costs under control. A transformation corruption is restored in 5 minutes via automatic fallback, an accidental deletion in 10 minutes thanks to 30-day retention, and an Azure regional outage in 30 minutes via geo-redundancy, with zero data loss in all three cases.
Granular access control: a document restricted to a team in Confluence (Sales, Engineering, Executive) must stay restricted after migration. The pipeline syncs ACLs to Azure via metadata tags and IAM policies, so downstream RAG systems respect the same restrictions.
Conclusion
Migrating Confluence content to Azure Blob Storage requires careful planning covering the technical, security, and governance dimensions. The serverless Azure Functions architecture proved ideal for this use case: pay-per-execution scaling, reduced maintenance, and automatic overnight synchronization that guarantees teams always work with the latest Confluence data.
This pipeline is not a one-off project with an end date, but critical infrastructure that must evolve with AI models (new embedding models, longer contexts, agentic RAG) to preserve competitive advantage and maximize return on investment.
In summary, on average across our Adservio clients: the mean time to find information drops from 15 to 6 minutes (-60%), monthly infrastructure spending falls by 80% compared to a full sync, recall@5 improves by 23% compared to raw HTML, and full deployment of such a pipeline takes 2 to 4 weeks.
Note: The statements and opinions expressed in this article are those of the authors and don't necessarily reflect Adservio's positions.
STAY POSTED
Get our next analyses and field notes straight to your inbox.
