Private Cloud OCR operations

Monitoring

Liveness: GET /health

200 when Redis answers, the bucket answers, and at least one GPU worker has a live heartbeat; 503 otherwise. The body is the same shape in every case:

field	meaning
status	ok or degraded. The HTTP status follows it: 200 or 503
redis_connected	Redis answered a ping
storage_connected	the bucket exists and answered a head request
gpu_workers_alive, gpu_workers_expected	GPU workers with a live heartbeat, and the number the supervisor keeps running (VRAM / OCR_WORKER_MEMORY_GB). 0 alive when Redis is unreachable, since the heartbeats live there
document_workers_alive, document_workers_expected	CPU-side document workers with a live heartbeat, and the 2 per container the supervisor keeps running. With none alive, documents queue and nothing starts
text_requests_in_flight, text_requests_max	concurrent POST /v3/text requests now, and the cap (8 × GPU workers) above which the endpoint answers 429
license.status	verified, grace, blocked, not_enforced, or unknown. Not a live check: it is the verdict of the last background check (every 60 to 90 s), stored in Redis so the 24 h grace clock survives a restart. unknown means Redis cannot be read, so the last verdict is unavailable and no new one can be recorded until Redis is back; /health never calls Mathpix itself
license.last_verified_at	the last successful check, null before the first one

The license state never changes status: a blocked license still returns 200, because restarting the container is not a remedy for it. A document worker below expected is normally a blip, the supervisor respawns it within a second; below expected for minutes means it is crash-looping while status stays ok. Restart the container on a persistent 503.

Metrics: GET /metrics

The body is the Prometheus text exposition format: plain text, one sample per line. The deployment does not ship Prometheus or a dashboard; point the scraper you already run at http://HOST:8080/metrics every 15–60 seconds (Prometheus, Grafana Alloy, the Datadog agent's OpenMetrics check, the CloudWatch agent, VictoriaMetrics, New Relic all read it). Without any of them, the same numbers are in GET /pco/v1/status as JSON.

metric	watch for
pco_gpu_workers_alive vs pco_gpu_workers_expected, pco_document_workers_alive vs pco_document_workers_expected	alive below expected for more than a few minutes: a worker is crash-looping, read the container logs
pco_queue_depth{queue="pco-documents"}, {queue="pco-pages"}	growing for an hour: more load than GPUs; add a host, or accept the latency
pco_pages_completed_total, pco_pages_failed_total, pco_documents_failed_total	failures climbing: look at the failing documents' status bodies
pco_page_seconds histogram	p95 rising with a flat queue: GPU contention or a stalled worker
pco_license_status{status="..."}	one sample, value 1, labelled with the current state; alert on anything but verified or not_enforced
pco_license_seconds_since_verified	above 600 on a licensed build: the check is failing
pco_usage_report_age_seconds, pco_usage_days_pending	metered builds: age above 1800 or pending above 2 for hours means reports are not getting through
pco_redis_connected, pco_storage_connected	0: the dependency is down

Status: GET /pco/v1/status

Versions, the build identity (customer, modes, build id, deployment id), uptime, queue depths, worker counts, the license, metering and logging blocks. Always 200. This is the body to paste into a support request, together with the log lines of the document in question (below).

Logs, and shipping them to your logging tool

Every process in the pco container writes one line per event to standard output: the API, the supervisor and each document, page and job worker. That output is the log. There are two ways to get it into a logging tool, and you use one of them, never both, or every line arrives twice:

your situation	use	section
no logging tool, or not yet	nothing: the rotated local log and docker logs pco, or kubectl logs
your tool already collects container output on the host or cluster (a Docker logging driver, or a Fluent Bit, Promtail, Datadog, CloudWatch agent on the node)	that, with PCO_LOG_FORMAT=json; PCO needs no other setting	Compose, Kubernetes
no agent and no driver you can configure, for example an EC2 host where nothing but the instance role is available	PCO_LOG_SINK, the container sends its own copy	sink

CloudWatch appears in both routes because both reach it: the awslogs Docker driver when the Docker daemon has credentials, the cloudwatch:// sink when it does not. Pick whichever your host supports. Nothing leaves the container unless you set PCO_LOG_SINK.

The lines

Two shapes, chosen with PCO_LOG_FORMAT (.env on Compose, logging.format in the chart):

2026-09-17T12:03:04.512Z INFO pco-doc-1-g2 [PCO][DOC_WORKER] 5c8b8fe5 completed in 3.1s

text, the default, for reading docker logs by eye. json for a logging tool:

{"time": "2026-09-17T12:03:04.512Z", "level": "INFO", "process": "document-worker", "worker_id": "pco-doc-1-g2",
 "deployment_id": "example-prod-1", "event": "pco.doc_worker", "message": "[PCO][DOC_WORKER] 5c8b8fe5 completed in 3.1s",
 "document_id": "5c8b8fe5...", "duration_ms": 3100}

field	meaning
time, level	UTC, DEBUG INFO WARNING ERROR
process	api, supervisor, document-worker, page-worker, job-worker
worker_id	the worker that wrote the line; a restarted worker gets a new suffix
deployment_id	your PCO_DEPLOYMENT_ID
event	the line's [PCO][X] prefix as pco.x: pco.http, pco.doc_worker, pco.page_worker, pco.pipeline, pco.job, pco.license, pco.usage, pco.webhook, pco.supervisor, pco.log
document_id, job_id, request_id, page_index	present when the line is about one document, job, /v3/text request or page
http_method, http_path, http_status, duration_ms	on pco.http lines, one per request; /health and /metrics at DEBUG

PCO_LOG_LEVEL (logging.level) is INFO by default; DEBUG adds the shared OCR stack's per-page bookkeeping, useful only when Mathpix asks for it.

Docker Compose: the logging driver

By default nothing to do: docker-compose.yml rotates the local json-file log (5 files of 100 MB) and docker logs pco reads it. When your logging tool collects through a Docker driver instead (and then PCO_LOG_SINK stays unset), docker-compose.logging.yml in the deploy folder holds one logging: block each for syslog, fluentd (Fluentd and Fluent Bit), awslogs (CloudWatch; the Docker daemon, not your shell, needs the credentials, on EC2 the instance role), gelf (Graylog) and splunk. Keep the block for your tool, delete the others, and start with both files; docker-compose.yml itself is not edited, and the override sets PCO_LOG_FORMAT=json for you.

docker compose -f docker-compose.yml -f docker-compose.logging.yml up -d --wait

Kubernetes: your cluster's log agent

Nothing to install in the chart. Fluent Bit, Promtail, the Datadog agent, Vector or the CloudWatch agent read every pod's output; set logging.format: json and give the agent its selectors through the pod's labels and annotations:

logging:
  format: json
podLabels:
  app: mathpix-pco
podAnnotations:
  co.elastic.logs/enabled: "true"          # Elastic
  ad.datadoghq.com/pco.logs: '[{"source": "mathpix-pco", "service": "pco"}]'   # Datadog

No agent on the host: a sink from the container

PCO_LOG_SINK (logging.sink) sends a copy of every line from the container itself. It is the route for a host with no log agent and no Docker driver you can use; if your tool already collects the container's output, leave it unset, or the tool gets every line twice. Standard output continues either way, so docker logs and kubectl logs keep working.

sink	form	notes
syslog	syslog+udp://host:514, syslog+tcp://host:514, syslog+tls://host:6514	RFC 5424, the JSON line as the message; rsyslog, syslog-ng and every tool's syslog input
CloudWatch Logs	cloudwatch://GROUP?stream=NAME&region=us-east-1, or cloudwatch:///group/with/slashes	credentials from the instance or node role; needs logs:CreateLogGroup, CreateLogStream, PutLogEvents; one JSON event per line, so Logs Insights filters on the fields

The sink never slows processing: lines queue in memory and go out from a background thread, and a sink that is down drops them. GET /pco/v1/status shows the sink, lines_sent and lines_dropped, and a warning line on standard output reports drops as they happen.

Following one document

Every line about a document carries its document_id, the pdf_id you received from POST /v3/pdf. With JSON lines, one filter gives its whole life, from submitted through each page to completed and the downloads:

CloudWatch Logs Insights:   fields @timestamp, message | filter document_id = "5c8b8fe5..." | sort @timestamp
Grafana Loki:               {app="mathpix-pco"} | json | document_id = "5c8b8fe5..."
Splunk:                     index=pco document_id="5c8b8fe5..."
Elastic (KQL):              document_id : "5c8b8fe5..."
Datadog:                    @document_id:5c8b8fe5...

A pco.page_worker line per page with duration_ms shows where the time went; a document that stays processing shows its last page line and then nothing, which is what to send us together with GET /pco/v1/status.

Usage

The deployment counts what it processes, per UTC day, and keeps the counters in Redis with an hourly copy in the bucket (pco-usage/YYYY-MM-DD.json), so they survive a Redis restart.

curl -s 'localhost:8080/pco/v1/usage?from=2026-09-01&to=2026-09-30'    # a month, one row per day
curl -s 'localhost:8080/pco/v1/usage?from=2026-09-15&to=2026-09-15'    # a single day

Both bounds are inclusive UTC days; from equal to to returns that one day, from after to is a 400, and a range may cover at most 366 days. Without parameters you get the last 30 days.

Each day row has nine counters. Three are the billable ones, the rest are operational:

counter	meaning
pages_billable	pages of documents that completed with no failed page. A document with one failed page bills zero. A blank page is not a failure: it completes with no text and counts as a page
images_billable	/v3/text images that returned a result, one page each
documents_billable	documents that contributed to pages_billable
pages_completed, pages_failed, documents_completed, documents_failed, images_completed, images_failed	what ran, including partial documents

The invoice quantity for a period is pages_billable + images_billable. A metered build sends these rows to Mathpix every 15 minutes, so the numbers you see are the numbers Mathpix has. An unmetered or airgapped build sends nothing; if your contract is usage-based you export a statement, which carries the deployment id and build identity, and send it to Mathpix:

curl -s 'localhost:8080/pco/v1/usage/export?from=2026-09-01&to=2026-09-30' > usage-2026-09.json

GET /pco/v1/status shows metering.last_report_at and metering.last_report_error for a metered build; a persistent error means the report cannot reach global.gateway.mathpix.com.

License

A licensed build proves to Mathpix every 60–90 seconds that it is a build we issued and the contract behind it is active, by posting a one-time code derived from a secret compiled into the image. Nothing about your documents is sent. GET /pco/v1/license:

{
  "status": "verified",
  "license_mode": "licensed",
  "deployment_id": "example-prod-1",
  "last_verified_at": "2026-09-15T14:02:11+00:00",
  "last_checked_at": "2026-09-15T14:02:11+00:00",
  "consecutive_failures": 0,
  "grace_ends_at": "2026-09-16T14:02:11+00:00",
  "expires_at": null,
  "last_error": null
}

status	meaning	what to do
verified	the last check succeeded	nothing
grace	the last check failed, or none has completed yet; processing continues for 24 hours from the last success	read last_error. unreachable:... means egress to global.gateway.mathpix.com:443 is blocked; clock_skew:+118s means the host clock is off by more than 30 seconds, fix NTP; unauthorized with a correct clock means the contract has lapsed, contact Mathpix
blocked	24 hours without a successful check	POST /v3/pdf, POST /v3/text and POST /pco/v1/jobs answer 503 license_unverified; downloads, status and in-flight documents keep working. Restore connectivity or the contract; the next successful check clears it within 90 seconds, no restart needed
not_enforced	an airgapped build	nothing; there is no check

Restarting the container does not reset the 24-hour window: it is measured from the last successful check, which is stored in Redis.

Upgrades

Mathpix gives you a new tag. Docker: change PCO_IMAGE in .env and run docker compose up -d --wait. Kubernetes: helm upgrade pco ./helm/mathpix-pco -n pco -f pco-values.yaml --set image=REGISTRY/mathpix-pco:NEWTAG; pods are replaced one at a time, so with two or more there is no gap. What happens to work in progress, per container:

1. The old container gets SIGTERM. Each GPU worker finishes the page it is on and exits; the supervisor waits up to 25 seconds for them, writes the usage counters to the bucket, and exits. The stop takes about 30 seconds.
2. Documents in flight do not finish in the old container. Their queue message stays in Redis, and every page result already produced is in Redis and the bucket. The new container's workers take the message over and continue from the first page not yet done: a 100-page document that was at page 60 resumes at page 60, and at most one page per GPU worker is done twice. Clients polling GET /v3/pdf/{pdf_id} see processing throughout. This is what our worker-kill tests and single-host stop-and-start rehearsals show; a rolling upgrade of a multi-pod cluster on a release image has not been soaked yet, so treat a document that turns error during an upgrade as a bug and send us its pdf_id and the container logs.
3. Documents queued but not started wait in Redis and are picked up in order.
4. Between the old container's exit and the new one's /health turning 200, a few minutes while the models load, submissions are refused at the port. To upgrade without that gap, run two containers and upgrade one at a time; the Helm chart does this by itself.

GET /pco/v1/status shows ocr_version, pdf_version and build_id before and after; if you keep outputs, those two versions tell you whether a stored result would differ from a fresh one.

Backups

Two volumes matter: the Redis volume (queues, document status for 7 days, usage counters) and the storage volume (inputs, page images, outputs, usage copies, the deployment id); on Kubernetes the PersistentVolumeClaims data-pco-redis-0 and data-pco-storage-0. Snapshot them the way you snapshot any volume. Losing Redis loses the queue and up to an hour of usage counters, which the deployment refills from the bucket copy on the next start. Losing the bucket loses stored outputs and the deployment id copy; usage already reported to Mathpix is unaffected.

Retention

Every document's input, page images and outputs stay in the bucket under pco/<pdf_id>/ until removed. Document status in Redis expires after 7 days, so outputs older than that can only be reached by key, not through the API. Set a retention rule matching that on the pco/ prefix and nothing else: the deployment keeps its usage copies at pco-usage/ and the deployment id at pco-meta/, outside that prefix, precisely so a rule on pco/ cannot remove them.

• Included SeaweedFS, once after install (the TTL applies to files written after it is set, and space returns in steps as volumes age out rather than exactly at day seven):

  printf 'fs.configure -locationPrefix=/buckets/pco/pco/ -ttl=7d -apply\n' | docker compose exec -T storage weed shell -master=localhost:9333

  On Kubernetes:

  kubectl -n pco exec pco-storage-0 -- sh -c "echo 'fs.configure -locationPrefix=/buckets/pco/pco/ -ttl=7d -apply' | weed shell -master=localhost:9333"

• Your own S3, GCS or Azure bucket: a lifecycle rule expiring objects under the prefix pco/ after 7 days.

Scaling

One container is one GPU. For more throughput run more containers against the same Redis and bucket with the same PCO_DEPLOYMENT_ID; they share the queues and count as one installation. On Kubernetes that is replicas, one per GPU the cluster can give the deployment. Do not run two containers against one GPU.

Troubleshooting

symptom	cause	fix
container exits at start, log names a PCO_* variable	missing or contradictory configuration	set it in .env or the chart values
container exits at start, log shows connection refused to redis:6379 or storage:8333	a Compose profile removed without setting PCO_REDIS_URL or the PCO_STORAGE_* variables	set them, or put the profile back
/health 503, gpu_workers_alive: 0, log shows Failed to initialize NVML: Unknown Error or No CUDA GPUs are available	the container lost the GPU device after a host systemctl daemon-reload (an apt upgrade is enough)	recreate the container: docker compose up -d --force-recreate pco
/health 503, redis_connected: false, submissions answer 500	Redis down or unreachable	check Redis; the deployment reconnects by itself, no restart
/health 503, storage_connected: false, submissions answer 500	bucket unreachable or credentials wrong	.env and seaweedfs-s3.json must carry the same keys; the included SeaweedFS takes about 30 seconds after a start before it answers; the deployment reconnects by itself
documents stay processing, queue depth flat, document_workers_alive: 0	document workers crash-looping; health stays ok because the GPU workers are fine	container logs, [SUPERVISOR] restart lines and the crash before them
CUDA out of memory in the log	too many workers for the card	raise OCR_WORKER_MEMORY_GB
license grace with unreachable:	egress blocked	allow HTTPS to global.gateway.mathpix.com
license grace with clock_skew:	host clock off by more than 30 seconds	fix NTP; the next check succeeds
license blocked after connectivity is back	none; it clears itself	wait up to 90 seconds, check /pco/v1/license
503 license_unverified on submissions	license blocked	see License
429 http_max_requests on /v3/text	more concurrent image requests than 8 × GPU workers	retry with backoff, or add a GPU

When contacting Mathpix, include GET /pco/v1/status, GET /pco/v1/license, and the last 200 lines of the pco container's logs.