Skip to main content

Private Cloud OCR batch jobs

A job converts every document under a folder in your object storage. You name the folder; the deployment lists it, converts each file, writes the outputs next to it (or under another folder), and leaves a manifest and a report in the same bucket. The deployment reads and writes the bucket with its own credentials, so nothing is uploaded through the API and no key leaves your network.

Endpoint contract: reference: batch jobs.

1. Look before you run​

?dry_run=true lists the folder, counts files by type, samples PDFs for a page total and estimates GPU hours at the deployment's measured rate. Nothing is created.

curl -s 'http://pco.internal:8080/pco/v1/jobs?dry_run=true' -H 'Content-Type: application/json' -d '{"input": {"folder": "s3://example-docs/batch1/"}}'
{
  "dry_run": true,
  "files": {"discovered": 18240, "excluded": 12, "already_processed": 0, "by_type": {"pdf": 17900, "docx": 300, "epub": 40}},
  "pages": {"files_sampled": 20, "estimated_total": 233000},
  "estimated_gpu_hours": 2.1,
  "deployment": {"gpu_workers": 8, "measured_pages_per_minute": 1850.0}
}

2. Submit​

curl -s http://pco.internal:8080/pco/v1/jobs -H 'Content-Type: application/json' -d '{
  "name": "arxiv-2024-q3",
  "input": {"folder": "s3://example-docs/arxiv/2024-q3/", "recursive": true, "include": ["**/*.pdf", "**/*.docx"]},
  "output": {"folder": "s3://example-docs/arxiv/2024-q3/", "layout": "alongside", "on_existing": "skip"},
  "options": {
    "conversion_formats": {"md": true, "docx": true},
    "extra_outputs": [".lines.json"],
    "ocr": {"math_inline_delimiters": ["\\(", "\\)"], "include_smiles": true}
  },
  "policy": {"max_pages_per_document": 5000, "per_page_timeout_seconds": 15, "max_retries": 3},
  "callback_url": "https://hooks.example.internal/mathpix",
  "callback_events": ["job.completed", "file.error"]
}'

201 returns the job object with its job_id. The id is derived from the spec, so posting the same spec again returns the existing job with 200 instead of starting a second one; with on_existing: skip that is also how a job resumes after a cancel or a Redis loss.

What the fields mean:

  • input.folder is a folder URI the deployment's credentials can read: s3://bucket/path/, gs://bucket/path/, or https://ACCOUNT.blob.core.windows.net/container/path/. The trailing slash is required, so a typo never silently becomes a one-file job. Every supported document under it is part of the job (recursively unless recursive: false), narrowed by include and exclude globs. For an explicit list of documents, send input.files (up to 10,000 object URIs) instead of input.folder; exactly one of the two.
  • output.folder defaults to the input folder. layout is alongside (doc.pdf → doc.mmd, doc.docx next to it; requires the output folder to be the input folder), mirror (the input tree reproduced under another folder) or flat (one folder per document id). on_existing is skip, the default, or replace.
  • options.ocr is the same object POST /v3/pdf accepts, validated by the same code with the same error ids, so you can test options on one file first. options.conversion_formats are the hosted format names. .mmd and .lines.json are always written; extra_outputs may add .lines.mmd.json, a byte copy of .lines.json for pipelines that still read that name.
  • policy replaces per-worker flags: max_pages_per_document (default 5000), per_page_timeout_seconds (15), max_retries (3), priority (low, normal, high).
  • callback_url, callback_events, callback_headers are the hosted API's webhook fields, unchanged. Events: job.completed (default), file.error (recommended), file.completed (opt-in). Unlike the hosted API, a deployment may call private and in-cluster receivers, since it lives inside your network.

A job never overwrites your source file: when an output would land on the input's own URI, an alongside memo.docx with the docx format requested, it is written as memo.converted.docx. A rerun over the same folder never converts its own outputs: earlier reports under _mathpix/ tell the enumerator what to pass over.

3. Watch​

curl -s http://pco.internal:8080/pco/v1/jobs/j_7f3a9c1e4b
{
  "job_id": "j_7f3a9c1e4b",
  "name": "arxiv-2024-q3",
  "status": "processing",
  "enumeration": {"status": "completed", "files_listed": 18240, "files_excluded": 12},
  "files": {"discovered": 18240, "queued": 11200, "processing": 24, "completed": 6900, "failed": 96, "skipped": 20},
  "pages": {"completed": 231540, "failed": 412},
  "throughput_pages_per_minute": 1840.5,
  "eta_seconds": 5400,
  "output": {
    "folder": "s3://example-docs/arxiv/2024-q3/",
    "layout": "alongside",
    "manifest_uri": "s3://example-docs/arxiv/2024-q3/_mathpix/j_7f3a9c1e4b/manifest.json",
    "report_uri": "s3://example-docs/arxiv/2024-q3/_mathpix/j_7f3a9c1e4b/report.jsonl"
  },
  "versions": {"ocr_version": "SuperNet-200", "pdf_version": "SuperNet-200p2"}
}
statusmeaning
enumeratinglisting the folder; documents found so far are already being processed
processingenumeration finished, documents in flight
completedevery document completed or skipped; files.failed is 0
completed_with_failuresevery document reached a terminal state and some failed; retry is available
cancelledcancelled by you; in-flight documents finished, queued ones were skipped
errorthe job itself could not run; error_info says why

Documents are processed as enumeration finds them, so a million-file folder starts producing outputs within seconds. eta_seconds is null until enumeration is complete and enough pages have finished to mean something.

4. Failures, retry, cancel​

curl -s 'http://pco.internal:8080/pco/v1/jobs/j_7f3a9c1e4b/files?status=failed'

Each file carries its key, status, pages, attempts, error_id, error_message and the URIs of its outputs. A completed document with pages_failed above 0 finished with a page-level error inside it; GET .../files/{document_id} lists the pages. The error ids are the hosted API's (error handling).

Once the job is terminal, requeue what failed, all of it or only some error ids:

curl -s -X POST http://pco.internal:8080/pco/v1/jobs/j_7f3a9c1e4b/retry -H 'Content-Type: application/json' -d '{"error_ids": ["pdf_content_type"]}'

Stop a running job; in-flight documents finish, queued ones are marked skipped:

curl -s -X POST http://pco.internal:8080/pco/v1/jobs/j_7f3a9c1e4b/cancel

5. What lands in your bucket​

Under the output folder, _mathpix/{job_id}/ holds manifest.json (the spec with every default filled in, the deployment's versions, enumeration totals, timestamps) and report.jsonl, one line per file as it finishes, identical to what GET .../report streams. The report is the durable record: if the deployment's Redis is ever lost, every result and every error is still there, and a resubmission with on_existing: skip resumes from it.

arxiv/2024-q3/_mathpix/j_7f3a9c1e4b/manifest.json
arxiv/2024-q3/_mathpix/j_7f3a9c1e4b/report.jsonl
arxiv/2024-q3/2401.01235.pdf   2401.01235.mmd   2401.01235.md   2401.01235.docx   2401.01235.lines.json
arxiv/2024-q3/images/...       (figure and equation crops referenced from the Markdown)

6. Webhooks​

Pass callback_url on the job and the deployment POSTs a signed notification when a file fails (file.error), when the job finishes (job.completed), or when each file completes (file.completed, opt-in). The payloads, the Mathpix-Signature header, the retry ladder and the signing secret rotation are the hosted API's (webhooks), so an existing handler works unchanged. The signing secret is per deployment: read it at GET /pco/v1/webhook-config, rotate it at POST /pco/v1/webhook-config/secret, send a test delivery with POST /pco/v1/webhook-config/test, and list deliveries with GET /pco/v1/deliveries?job_id=....

7. From the command line​

The pco tool wraps all of the above:

pco convert s3://example-docs/arxiv/2024-q3/ --formats md,docx --dry-run     # the estimate
pco convert s3://example-docs/arxiv/2024-q3/ --formats md,docx               # submit and watch to the end
pco convert s3://example-docs/arxiv/2024-q3/ --formats md,docx --detach      # submit and return the job id
pco jobs status j_7f3a9c1e4b --watch
pco jobs files j_7f3a9c1e4b --status failed
pco jobs retry j_7f3a9c1e4b --error-id pdf_content_type
pco jobs report j_7f3a9c1e4b > report.jsonl

Coming from Mathpix SCS classic: pco migrate-scs --conversion-options conversion_options.json --ocr-options ocr_options.json --ext-list .lines.json --input-folder s3://bucket/folder/ > job.json turns your existing option files into a job spec, and pco convert --job-spec job.json --dry-run confirms the enumeration matches what you publish today.