Manifest Automation Platform

Overview

Problem and context

My contributions

Technical architecture

One contract, five pipelines

Every shipping provider has its own pipeline behind a shared interface: validate the input, process it, return a set of output files. The server owns the generic lifecycle around them, so providers differ only where their documents do.

Deterministic first, AI where it helps

Input files are parsed with ordinary, deterministic code. The language model is used for the two things rules handle badly: classifying free-text product descriptions and cleaning inconsistent names.

Jobs with live progress

Processing runs in background workers, and progress streams to the dashboard as it happens. Each job is scoped to the operator who started it and recorded for history.

Operations built in

The dashboard includes a review queue for uncertain classifications, an error feed from the server’s own logs, and an AI health indicator that shows when the model provider is rate limiting.

Deployment

The application ships as a Docker image to a private server behind a reverse proxy. Database migrations run when the container starts, and staging runs beside production as a separate environment.

Engineering challenges

1. 01

   Five providers without five codebases

   Challenge

   Each shipping provider sends a different layout with its own validation, business rules, and output set. A single generic parser would be fragile; five separate programs would be unmaintainable.

   Approach

   Pipelines implement one typed contract and are listed in an explicit registry that loads at startup, so a broken pipeline fails at deploy time instead of at upload. Providers with similar documents reuse a shared processing chain with their own configuration.

   Technology

   • Python protocols
   • Dataclasses
   • FastAPI

   Outcome

   One provider’s handling can change without touching the others, and three of the five share most of their code.

2. 02

   Classification that never ships a silent guess

   Challenge

   HS codes must come from an allowed list, but product descriptions are free text and a language model can answer differently each time, or answer off the list.

   Approach

   Classification is tiered: direct mappings, then keyword rules, then a cache of earlier confident answers, then the model, then a nearest-match fallback. Every result records which tier produced it. Only confident model answers are cached, and fallback results pause the job for operator input or go to a review queue that feeds corrections back into the cache.

   Technology

   • Rule engine
   • SQLite cache
   • OpenAI API
   • Human review queue

   Outcome

   Routine items are classified without a model call, and uncertain ones are surfaced to a person instead of reaching a customs document.

3. 03

   Keeping model output inside the lines

   Challenge

   Model answers arrive as plain text, the provider enforces rate limits across the whole account, and several jobs can run at once.

   Approach

   Answers are normalised and matched against the allowed list, with an explicit “none” option and one follow-up question before falling back. Items are classified in batches, a shared token bucket paces every call, and failed calls degrade to lexical matching with a flag instead of failing the job.

   Technology

   • Batching
   • Token-bucket rate limiting
   • Retries with backoff

   Outcome

   A rate limit or an outage at the model provider slows a job down but does not stop the operation.

4. 04

   Documents without a desktop

   Challenge

   The manual process relied on spreadsheet software to compute values and print PDFs. A Linux server has neither, and PDF rendering is slow enough to block everything else.

   Approach

   Excel files are written from templates in Python, with lookup formulas resolved in code. PDFs are rendered from HTML templates in a separate pool of worker processes that recovers if a worker dies, and the image includes fonts for Arabic, right-to-left text.

   Technology

   • openpyxl
   • Jinja2
   • WeasyPrint
   • Process pools

   Outcome

   Every output is produced on the server, in parallel with other jobs.

5. 05

   Changing compliance output safely, alone

   Challenge

   With one engineer and customs documents as the product, any refactor could quietly change an output that someone will submit to an authority.

   Approach

   An automated test suite runs against a throwaway database with no real AI calls. Before production, a golden-output check compares generated spreadsheets cell by cell and PDFs by text. CI runs the tests and builds on every push, and staging deploys before production.

   Technology

   • pytest
   • Golden-file comparison
   • GitHub Actions
   • Docker

   Outcome

   Changes are verified against known-good documents before operators see them.

Results and impact

Reduction in processing time per manifest

92.5%

Approximate, based on the before and after times below.

Processing time per manifest

20 → under 2 min

Manual process compared with the platform.

Manifests processed

1,750+

Shipping providers automated

5

Visuals