Section 11: Plugin Development Guide

Quantra is a general-purpose document processing pipeline platform whose capabilities are extended through a modular plugin system. Rather than modifying the core platform code, developers create self-contained plugin packages that the platform discovers, loads, and integrates at runtime. This guide provides a comprehensive walkthrough of plugin architecture, directory conventions, manifest configuration, and the step-by-step process for building each of the three plugin types: datasources, tools, and workbenches.

11.1 Plugin Architecture Overview

Every plugin in Quantra falls into one of three categories. Understanding the role of each category is essential before writing any code, because the category (referred to as the plugin kind) determines the directory the plugin lives in, the interfaces it must implement, and how the platform presents it to end users.

Datasources

A datasource plugin provides the platform with a way to ingest documents or data from an external system. Examples include connectors for file shares, cloud storage buckets, REST APIs, email inboxes, or database tables. Datasource plugins typically expose one or more pages that let a user configure connection parameters (credentials, paths, filters) and then pull documents into the Quantra pipeline for downstream processing. Datasource plugins live under /plugins/datasources/.

Tools

A tool plugin implements a discrete processing operation that can be composed into pipelines. Tools receive documents (or structured data) as input, perform a transformation, analysis, or enrichment step, and produce output that flows to the next stage. Examples include OCR extraction, NLP entity recognition, PDF splitting, data validation, redaction, and format conversion. Tool plugins live under /plugins/tools/.

Workbenches

A workbench plugin provides a rich, interactive user interface for reviewing, annotating, or manipulating documents and data. Workbenches are UI-focused: they render templates with custom JavaScript and CSS to give users a dedicated workspace for a particular task, such as side-by-side document comparison, manual quality-assurance review, or data labelling. Workbench plugins live under /plugins/workbenches/.

11.2 Plugin Directory Structure

All plugins follow a consistent directory layout. The top-level /plugins/ directory contains three subdirectories corresponding to the three plugin kinds. Within each subdirectory, every plugin occupies its own folder whose name matches the plugin identifier.

/plugins/
+-- datasources/
|   +-- plugin_name/
|       +-- plugin.json          # Plugin manifest (required)
|       +-- server.py            # Backend handler module (required)
|       +-- templates/
|       |   +-- plugin_name/
|       |       +-- index.html   # Primary template
|       |       +-- detail.html  # Additional templates
|       +-- static/
|           +-- plugin_name/
|               +-- css/
|               +-- js/
+-- tools/
|   +-- plugin_name/
|       +-- plugin.json
|       +-- server.py
|       +-- templates/
|       |   +-- plugin_name/
|       +-- static/
|           +-- plugin_name/
+-- workbenches/
    +-- plugin_name/
        +-- plugin.json
        +-- server.py
        +-- templates/
        |   +-- plugin_name/
        |       +-- index.html
        |       +-- review.html
        +-- static/
            +-- plugin_name/
                +-- css/
                +-- js/
                +-- img/

The templates/ subdirectory uses a nested folder named after the plugin to avoid template-name collisions when the Django/Jinja template loader aggregates templates across all plugins. The same namespacing convention applies to static/ assets.

11.3 Plugin Manifest (plugin.json)

Every plugin must include a plugin.json file at the root of its directory. This manifest tells the pluginhost everything it needs to know to register and render the plugin: its identity, its kind, its version, the routes it exposes, and descriptive metadata.

Full Example

{
    "id": "invoice_scanner",
    "kind": "tools",
    "name": "Invoice Scanner",
    "version": "1.0.3",
    "pages": [
        {
            "route": "/tools/invoice_scanner/",
            "template": "invoice_scanner/index.html"
        },
        {
            "route": "/tools/invoice_scanner/results/",
            "template": "invoice_scanner/results.html"
        }
    ],
    "meta": {
        "category": "Extraction",
        "description": "Scans uploaded invoice documents, extracts key fields (vendor, date, line items, totals), and outputs structured JSON suitable for downstream validation or ERP import."
    }
}

Field Reference

Field	Type	Required	Description
id	string	Yes	A unique, URL-safe identifier for the plugin. Must match the plugin directory name. Use lowercase letters, digits, and underscores only.
kind	string	Yes	One of "datasources", "tools", or "workbenches". Determines which subdirectory the plugin resides in and how the platform categorises it in the UI.
name	string	Yes	A human-readable display name shown in navigation menus, dashboards, and pipeline configuration screens.
version	string	Yes	Semantic version string (e.g., "1.0.3"). The final segment (build count) must be incremented every time the plugin code is modified.
pages	array	Yes	An array of page objects. Each has a route (URL path) and a template (path to template relative to plugin templates/ directory).
meta	object	Yes	Contains category (grouping label) and description (prose description of purpose and capabilities).

11.4 Creating a Datasource Plugin

This section walks through the creation of a datasource plugin from scratch. The example plugin, sharepoint_connector, ingests documents from a Microsoft SharePoint site.

Step 1: Create the Directory

Create a new folder under /plugins/datasources/ with the plugin identifier as its name.

mkdir -p /plugins/datasources/sharepoint_connector/templates/sharepoint_connector
mkdir -p /plugins/datasources/sharepoint_connector/static/sharepoint_connector/css
mkdir -p /plugins/datasources/sharepoint_connector/static/sharepoint_connector/js

Step 2: Write plugin.json

{
    "id": "sharepoint_connector",
    "kind": "datasources",
    "name": "SharePoint Connector",
    "version": "1.0.0",
    "pages": [
        { "route": "/datasources/sharepoint_connector/", "template": "sharepoint_connector/index.html" },
        { "route": "/datasources/sharepoint_connector/browse/", "template": "sharepoint_connector/browse.html" }
    ],
    "meta": {
        "category": "Cloud Storage",
        "description": "Connects to Microsoft SharePoint Online sites, enumerates document libraries, and ingests selected files into the Quantra pipeline."
    }
}

Step 3: Create server.py with Handler Functions

The server.py module is the backend entry point for your plugin. It must expose handler functions that the pluginhost calls when the corresponding routes are requested. Each handler receives the Django request object and returns a context dictionary that the template engine will render.

# /plugins/datasources/sharepoint_connector/server.py
import logging
from datetime import datetime

logger = logging.getLogger(__name__)
PLUGIN_VERSION = "1.0.0"

def index(request):
    """Render the main configuration page."""
    logger.info("sharepoint_connector: index page requested")
    return {
        "plugin_version": PLUGIN_VERSION,
        "page_title": "SharePoint Connector",
        "timestamp": datetime.utcnow().isoformat(),
    }

def browse(request):
    """List available document libraries for ingestion."""
    site_url = request.POST.get("site_url", "")
    tenant_id = request.POST.get("tenant_id", "")
    logger.info("sharepoint_connector: browsing site %s", site_url)
    documents = []  # placeholder for API results
    return {
        "plugin_version": PLUGIN_VERSION,
        "site_url": site_url,
        "tenant_id": tenant_id,
        "documents": documents,
    }

Step 4: Create Templates

Place your Jinja/Django template files inside templates/sharepoint_connector/. Templates have access to the context dictionary returned by the corresponding handler, as well as standard Django template variables injected by the pluginhost.

<!-- templates/sharepoint_connector/index.html -->
{%% extends "base.html" %%}
{%% block title %%}SharePoint Connector{%% endblock %%}
{%% block content %%}
<h1>SharePoint Connector</h1>
<p>Version: {{ plugin_version }}</p>
<form method="post" action="/datasources/sharepoint_connector/browse/">
    {%% csrf_token %%}
    <label for="site_url">SharePoint Site URL</label>
    <input type="text" name="site_url" id="site_url" required />
    <label for="tenant_id">Tenant ID</label>
    <input type="text" name="tenant_id" id="tenant_id" required />
    <button type="submit">Browse Libraries</button>
</form>
{%% endblock %%}

Step 5: Register the Service Endpoint

If your datasource plugin requires a dedicated backend service (e.g., a gRPC microservice for data fetching), register it in the platform ServiceEndpoint table. Before choosing a port, always query the ServiceEndpoint table to determine which ports are already in use. Microservice ports are allocated in the range 50051–50090.

# Check existing port allocations (Django shell)
from core.models import ServiceEndpoint
for ep in ServiceEndpoint.objects.all().order_by("port"):
    print(f"{ep.port}  {ep.name}")

# Pick the next available port and register:
ServiceEndpoint.objects.create(
    name="sharepoint_connector",
    host="localhost",
    port=50067,
    protocol="grpc",
    is_active=True,
)

11.5 Creating a Tool Plugin

Tool plugins follow the same structural conventions as datasource plugins. The principal differences are the kind value in the manifest and the nature of the handler logic (processing/transforming data rather than ingesting it).

Step 1: Create the Directory

mkdir -p /plugins/tools/pdf_splitter/templates/pdf_splitter
mkdir -p /plugins/tools/pdf_splitter/static/pdf_splitter/css

Step 2: Write plugin.json

{
    "id": "pdf_splitter",
    "kind": "tools",
    "name": "PDF Splitter",
    "version": "1.0.0",
    "pages": [
        { "route": "/tools/pdf_splitter/", "template": "pdf_splitter/index.html" }
    ],
    "meta": {
        "category": "Document Processing",
        "description": "Splits multi-page PDF documents into individual pages or page ranges."
    }
}

Step 3: Implement server.py

# /plugins/tools/pdf_splitter/server.py
import logging
logger = logging.getLogger(__name__)
PLUGIN_VERSION = "1.0.0"

def index(request):
    results = None
    if request.method == "POST":
        uploaded_file = request.FILES.get("pdf_file")
        split_mode = request.POST.get("split_mode", "single_pages")
        logger.info("pdf_splitter: processing %s in mode %s",
                     uploaded_file.name if uploaded_file else "none", split_mode)
        results = {"status": "complete", "pages_created": 0}
    return {"plugin_version": PLUGIN_VERSION, "page_title": "PDF Splitter", "results": results}

Step 4: Create Templates and Register

Follow the same template and registration pattern described in the datasource section above. If the tool requires a backing microservice, register its endpoint in the ServiceEndpoint table after verifying port availability.

11.6 Creating a Workbench Plugin

Workbench plugins are UI-heavy. They provide full-page interactive interfaces for reviewing, annotating, comparing, or otherwise working with documents and data. While the directory structure and manifest format are identical, workbench plugins tend to have more templates, more static assets, and richer client-side logic.

Key Differences from Datasources and Tools

• Multiple pages are common. A workbench might expose a listing/dashboard page, a detailed review page, a settings page, and an export page.
• Client-side interactivity is expected. Workbenches frequently include significant JavaScript for drag-and-drop, keyboard shortcuts, inline editing, canvas rendering, and AJAX calls.
• Review interfaces. Many workbenches implement a review pattern: a list of items to review, a detail view for each item, and actions (approve, reject, flag, annotate). Handlers must support both GET and POST.

Example Manifest

{
    "id": "doc_review_bench",
    "kind": "workbenches",
    "name": "Document Review Workbench",
    "version": "1.0.0",
    "pages": [
        { "route": "/workbenches/doc_review_bench/", "template": "doc_review_bench/index.html" },
        { "route": "/workbenches/doc_review_bench/review/", "template": "doc_review_bench/review.html" },
        { "route": "/workbenches/doc_review_bench/export/", "template": "doc_review_bench/export.html" }
    ],
    "meta": {
        "category": "Quality Assurance",
        "description": "Side-by-side document review interface for comparing extracted data against source documents, annotating discrepancies, and approving or rejecting results."
    }
}

11.7 Server Handler Pattern

The server.py file in every plugin serves as the bridge between HTTP requests and template rendering. The pluginhost imports the module dynamically and looks up handler functions by matching route paths to function names.

How It Works

1. When a request arrives for a URL matching a plugin route, the pluginhost identifies the plugin by its route prefix.
2. The pluginhost imports (or reuses a cached import of) the plugin server.py module.
3. The function name is derived from the route: the root route maps to index, sub-routes map to functions named after the final path segment (e.g., /tools/pdf_splitter/results/ maps to results).
4. The handler is called with the Django HttpRequest object as its sole argument.
5. The handler returns a Python dictionary which becomes the template context.
6. The pluginhost renders the specified template with the context and returns the HTML response.

Request / Response Pattern

def results(request):
    """Handler for /tools/pdf_splitter/results/"""
    job_id = request.GET.get("job_id")
    if request.method == "POST":
        action = request.POST.get("action")
    uploaded = request.FILES.get("document")
    user = request.user
    # Return a context dictionary - never an HttpResponse
    return {"job_id": job_id, "status": "complete", "rows": []}

Important: Handlers must return a plain dictionary, not a Quantra HttpResponse. The pluginhost handles response construction internally.

11.8 Templates and Static Assets

Template Engine

Quantra uses the Django template engine. Plugin templates should extend base.html to inherit standard page chrome (navigation, sidebar, footer, global CSS/JS). Override the content block to inject plugin-specific markup.

{%% extends "base.html" %%}
{%% load static %%}
{%% block title %%}My Plugin{%% endblock %%}
{%% block extra_css %%}
<link rel="stylesheet" href="{%% static 'my_plugin/css/style.css' %%}" />
{%% endblock %%}
{%% block content %%}
<div class="my-plugin-container">
    <h1>{{ page_title }}</h1>
</div>
{%% endblock %%}
{%% block extra_js %%}
<script src="{%% static 'my_plugin/js/main.js' %%}"></script>
{%% endblock %%}

Static Asset Conventions

• All static files must be placed inside static/<plugin_id>/ to avoid name collisions.
• Use the static template tag to generate URLs. Never hard-code paths.
• Keep CSS and JavaScript in separate subdirectories (css/, js/). Images go in img/.
• Minify production assets but keep un-minified source in the repository.

11.9 Plugin Discovery

The Quantra pluginhost is the subsystem responsible for discovering, loading, and integrating plugins at platform startup.

Discovery Process

1. Directory scan. On startup, the pluginhost iterates over datasources, tools, and workbenches subdirectories. For each child directory, it looks for plugin.json.
2. Manifest validation. The pluginhost parses and validates all required fields. Invalid plugins are skipped with a warning.
3. Kind verification. The kind field must match the parent subdirectory.
4. Route registration. Each page entry is registered as a Django URL pattern.
5. Navigation injection. The pluginhost builds navigation data for the base template sidebar.
6. Static file collection. Each plugin static directory is registered with Django static file finders.

Because discovery is dynamic, adding a new plugin requires no core platform changes. Place the directory, restart the platform, and the plugin appears.

11.10 Version Management

The version field uses MAJOR.MINOR.BUILD format. The build count must be incremented every time the plugin code is modified.

• Build count: Increment for every change. 1.0.3 becomes 1.0.4.
• Minor version: Increment for new backward-compatible functionality. Reset build to zero.
• Major version: Increment for breaking changes. Reset minor and build to zero.

See the Administration appendix for the bump_versions.py script.

11.11 Best Practices

• Keep plugins self-contained. Do not import from other plugins. Shared logic goes in /ms/libs/.
• Log generously. Use Python logging with a logger named after the plugin.
• Handle errors gracefully. Return error messages in context rather than raising unhandled exceptions.
• Namespace everything. Template dirs, static dirs, CSS classes, JS globals should all use the plugin ID prefix.
• Test in isolation. Mock the Django request object for unit tests.
• Check port availability. Always query ServiceEndpoint before registering a port.
• Document your plugin. Include a README.md in the plugin directory.
• Bump the version. Every modification must increment the build count in plugin.json.