Skip to main content

Private Cloud OCR batch jobs

A job converts every document under a folder in your object storage. You name the folder; the deployment lists it, converts each file, writes the outputs next to it (or under another folder), and leaves a manifest and a report in the same bucket. The deployment reads and writes the bucket with its own credentials, so nothing is uploaded through the API and no key leaves your network.

Endpoint contract: reference: batch jobs.

1. Look before you run​

?dry_run=true lists the folder, counts files by type, samples PDFs for a page total and estimates GPU hours at the deployment's measured rate. Nothing is created.

curl -s 'http://pco.internal:8080/pco/v1/jobs?dry_run=true' -H 'Content-Type: application/json' -d '{"input": {"folder": "s3://example-docs/batch1/"}}'
{
  "dry_run": true,
  "files": {"discovered": 18240, "excluded": 12, "already_processed": 0, "by_type": {"pdf": 17900, "docx": 300, "epub": 40}},
  "pages": {"files_sampled": 20, "estimated_total": 233000},
  "estimated_gpu_hours": 2.1,
  "deployment": {"gpu_workers": 8, "measured_pages_per_minute": 1850.0}
}

2. Submit​

curl -s http://pco.internal:8080/pco/v1/jobs -H 'Content-Type: application/json' -d '{
  "name": "arxiv-2024-q3",
  "input": {"folder": "s3://example-docs/arxiv/2024-q3/", "recursive": true, "include": ["**/*.pdf", "**/*.docx"]},
  "output": {"folder": "s3://example-docs/arxiv/2024-q3/", "layout": "alongside", "on_existing": "skip"},
  "options": {
    "conversion_formats": {"md": true, "docx": true},
    "extra_outputs": [".lines.json"],
    "ocr": {"math_inline_delimiters": ["\\(", "\\)"], "include_smiles": true}
  },
  "policy": {"max_pages_per_document": 5000, "per_page_timeout_seconds": 15, "max_retries": 3},
  "callback_url": "https://hooks.example.internal/mathpix",
  "callback_events": ["job.completed", "file.error"]
}'

201 returns the job object with its job_id. The id is derived from the spec, so posting the same spec again returns the existing job with 200 instead of starting a second one; with on_existing: skip that is also how a job resumes after a cancel or a Redis loss.

What the fields mean:

  • input.folder is a folder URI the deployment's credentials can read: s3://bucket/path/, gs://bucket/path/, or https://ACCOUNT.blob.core.windows.net/container/path/. The trailing slash is required, so a typo never silently becomes a one-file job. Every supported document under it is part of the job (recursively unless recursive: false), narrowed by include and exclude globs. For an explicit list of documents, send input.files (up to 10,000 object URIs) instead of input.folder; exactly one of the two.
  • output.folder defaults to the input folder. layout is alongside (doc.pdf → doc.mmd, doc.docx next to it; requires the output folder to be the input folder), mirror (each input gets its own folder under the output folder, named after the input and keeping the input tree: 2401/paper.pdf → 2401/paper/paper.mmd with that document's images/ inside), or flat (one folder per document id). on_existing is skip, the default, or replace.
  • options.ocr is the same object POST /v3/pdf accepts, validated by the same code with the same error ids, so you can test options on one file first. options.conversion_formats are the hosted format names. .mmd and .lines.json are always written; extra_outputs may add .lines.mmd.json, a byte copy of .lines.json for pipelines that still read that name.
  • policy replaces per-worker flags: max_pages_per_document (default 5000), per_page_timeout_seconds (15), max_retries (3), priority (low, normal, high).
  • callback_url, callback_events, callback_headers are the hosted API's webhook fields, unchanged. Events: job.completed (default), file.error (recommended), file.completed (opt-in). Unlike the hosted API, a deployment may call private and in-cluster receivers, since it lives inside your network.

A job never overwrites your source file: when an output would land on the input's own URI, an alongside memo.docx with the docx format requested, it is written as memo.converted.docx. A rerun over the same folder never converts its own outputs: earlier reports under _mathpix/ tell the enumerator what to pass over.

3. Watch​

curl -s http://pco.internal:8080/pco/v1/jobs/j_7f3a9c1e4b
{
  "job_id": "j_7f3a9c1e4b",
  "name": "arxiv-2024-q3",
  "status": "processing",
  "enumeration": {"status": "completed", "files_listed": 18240, "files_excluded": 12},
  "files": {"discovered": 18240, "queued": 11200, "processing": 24, "completed": 6900, "failed": 96, "skipped": 20},
  "pages": {"completed": 231540, "failed": 412},
  "throughput_pages_per_minute": 1840.5,
  "eta_seconds": 5400,
  "output": {
    "folder": "s3://example-docs/arxiv/2024-q3/",
    "layout": "alongside",
    "manifest_uri": "s3://example-docs/arxiv/2024-q3/_mathpix/j_7f3a9c1e4b/manifest.json",
    "report_uri": "s3://example-docs/arxiv/2024-q3/_mathpix/j_7f3a9c1e4b/report.jsonl"
  },
  "versions": {"ocr_version": "SuperNet-200", "pdf_version": "SuperNet-200p2"}
}
statusmeaning
enumeratinglisting the folder; documents found so far are already being processed
processingenumeration finished, documents in flight
completedevery document completed or skipped; files.failed is 0
completed_with_failuresevery document reached a terminal state and some failed; retry is available
cancelledcancelled by you; in-flight documents finished, queued ones were skipped
errorthe job itself could not run; error_info says why

Documents are processed as enumeration finds them, so a million-file folder starts producing outputs within seconds. eta_seconds is null until enumeration is complete and enough pages have finished to mean something.

4. Failures, retry, cancel​

curl -s 'http://pco.internal:8080/pco/v1/jobs/j_7f3a9c1e4b/files?status=failed'

Each file carries its key, status, pages, attempts, error_id, error_message and the URIs of its outputs. A completed document with pages_failed above 0 finished with a page-level error inside it; GET .../files/{document_id} lists the pages. The error ids are the hosted API's (error handling).

Once the job is terminal, requeue what failed, all of it or only some error ids:

curl -s -X POST http://pco.internal:8080/pco/v1/jobs/j_7f3a9c1e4b/retry -H 'Content-Type: application/json' -d '{"error_ids": ["pdf_content_type"]}'

Stop a running job; in-flight documents finish, queued ones are marked skipped:

curl -s -X POST http://pco.internal:8080/pco/v1/jobs/j_7f3a9c1e4b/cancel

5. What lands in your bucket​

Under the output folder, _mathpix/{job_id}/ holds manifest.json (the spec with every default filled in, the deployment's versions, enumeration totals, timestamps) and report.jsonl, one line per file as it finishes, identical to what GET .../report streams. The report is the durable record: if the deployment's Redis is ever lost, every result and every error is still there, and a resubmission with on_existing: skip resumes from it.

arxiv/2024-q3/_mathpix/j_7f3a9c1e4b/manifest.json
arxiv/2024-q3/_mathpix/j_7f3a9c1e4b/report.jsonl
arxiv/2024-q3/2401.01235.pdf   2401.01235.mmd   2401.01235.md   2401.01235.docx   2401.01235.lines.json
arxiv/2024-q3/images/...       (figure and equation crops referenced from the Markdown)

6. Webhooks​

Pass callback_url on the job and the deployment POSTs a signed notification when a file fails (file.error), when the job finishes (job.completed), or when each file completes (file.completed, opt-in). The payloads, the Mathpix-Signature header, the retry ladder and the signing secret rotation are the hosted API's (webhooks), so an existing handler works unchanged. The signing secret is per deployment: read it at GET /pco/v1/webhook-config, rotate it at POST /pco/v1/webhook-config/secret, send a test delivery with POST /pco/v1/webhook-config/test, and list deliveries with GET /pco/v1/deliveries?job_id=....

7. From the command line​

The Mathpix CLI's pco service wraps all of the above, so you rarely have to call the endpoints by hand. Set MPX_ENDPOINT to your deployment (or pass --endpoint, then point convert at a folder in your storage:

mpx pco convert s3://example-docs/arxiv/2024-q3/ --formats md,docx    # submit a server-side job and watch it to the end
mpx pco convert s3://example-docs/in/ s3://example-docs/out/          # write outputs to a different folder
mpx pco convert s3://example-docs/arxiv/ --include '**/*.pdf' --dry-run   # enumerate and estimate pages; create nothing
mpx pco convert s3://example-docs/arxiv/ --overwrite                  # reprocess everything, e.g. after upgrading the image
mpx pco convert s3://example-docs/arxiv/ --detach                     # submit and return; the job runs on without you

convert on a cloud folder creates the job, or reattaches to the existing one for an identical spec, and watches it, showing a live progress bar and one line per file as it finishes. A second folder argument sets output.folder; --include/--exclude narrow the enumeration; --options-json passes any /v3/pdf option; --overwrite sets on_existing: replace; --detach returns as soon as the deployment has the job.

Manage a running or finished job:

mpx pco jobs list                                                     # every job, newest first
mpx pco jobs get j_7f3a9c1e4b                                         # status and per-file counts
mpx pco jobs files j_7f3a9c1e4b --status error                       # only the files that failed
mpx pco jobs retry j_7f3a9c1e4b --error-id pdf_content_type          # retry just those
mpx pco jobs report j_7f3a9c1e4b > report.jsonl                      # the full per-file report
mpx pco jobs cancel j_7f3a9c1e4b                                     # stop a running job

Moving a batch over from SCS classic? mpx pco migrate-scs turns its conversion_options.json, ocr_options.json and extension list into a job spec, which convert --job-spec submits:

mpx pco migrate-scs --conversion-options conversion_options.json --ocr-options ocr_options.json --input-folder s3://example-docs/in/ > job.json
mpx pco convert --job-spec job.json --dry-run    # check the spec, then run it without --dry-run

See the CLI reference for every subcommand and flag. (convert also processes a local file or folder by looping over /v3/pdf from your workstation, with the same progress display and a --resumeable record; the quickstart shows that side.)