Introduction

git clone git@github.com:Mathpix/api-examples.git
cd api-examples/images

The MathpixOCR API is a JSON API for extracting text from images. Unlike other OCR API’s, MathpixOCR has 1rst class support for scientific notation, as used in chemistry, math, physics, computer science, economics, and other STEM subjects.

To use MathpixOCR, you must base64 encode your image, and send it to the MathpixOCR API, along with other JSON information.

If you have any problems, please send us an email at support@mathpix.com.

Authorization

The request headers must be set as follows:

{
  "content-type": "application/json",
  "app_id": "YOUR_APP_ID",
  "app_key": "YOUR_APP_KEY"
}

MathpixOCR uses API keys to allow access to the API. You can find your API keys on our dashboard website by signing up at https://dashboard.mathpix.com.

MathpixOCR expects for the API key to be included in all API requests to the server via HTTP Basic Auth. Expected set of HTTP headers is shown on the right.

Process image (v3/text) (beta)

Mathpix supports image recognition for jpg and png images.

The v3/text endpoint is in beta but already does a much better job for text heavy content, esp tall images with lots of text mixed with math.

The Latex in the image is returned inside inline delimiters $ ... $ and block mode delimiters \[ .... \]. Lines are separated with \n newline characters. In some cases (eg multiple choice equations) we will try flatten horizontally aligned content into different lines in order to keep the markup simple.

The main downside vs v3/text is that it lacks configuration options. The upside is that it provides sensible default behavior for various scenarios.

Request parameters

Send an images:

{
  "src": "data:image/jpeg;base64,..."
}

#!/usr/bin/env python
import sys
import base64
import requests
import json

# put desired file path here
file_path = 'limit.jpg'
image_uri = "data:image/jpg;base64," + base64.b64encode(open(file_path, "rb").read()).decode()
r = requests.post("https://api.mathpix.com/v3/text",
    data=json.dumps({'src': image_uri}),
    headers={"app_id": "YOUR_APP_ID", "app_key": "YOUR_APP_KEY",
             "Content-type": "application/json"})
print(json.dumps(json.loads(r.text), indent=4, sort_keys=True))

curl -X POST https://api.mathpix.com/v3/text \
-H 'app_id: YOUR_APP_ID' \
-H 'app_key: YOUR_APP_KEY' \
-H 'Content-Type: application/json' \
--data '{ "src": "data:image/jpeg;base64,'$(base64 -i limit.jpg)'" }'

POST https://api.mathpix.com/v3/text

Parameter	Type	Description
src	string	Image data, or public URL where image is located

Result objects

Get an API response:

{
    "text": "\\( \\lim _{x \\rightarrow 3}\\left(\\frac{x^{2}+9}{x-3}\\right) \\)"
}

Field	Type	Description
`text` (optional)	string	Recognized `text` format, if such is found

Process image (v3/latex)

The v3/latex is more mature than the v3/text endpoint, but doesn’t work as well on text heavy images.

The main advantage v3/latex has is the rich set of options for dealing with math and requesting various formats like mathml and asciimath.

Mathpix supports image recognition for jpg and png images.

Request parameters

You can request multiple formats for a single image:

{
  "src": "data:image/jpeg;base64,...",
  "ocr": ["math", "text"], 
  "skip_recrop": true,
  "formats": [
    "text",
    "latex_simplified",
    "latex_styled",
    "mathml",
    "asciimath",
    "latex_list"
   ]
}

#!/usr/bin/env python
import sys
import base64
import requests
import json

# put desired file path here
file_path = 'limit.jpg'
image_uri = "data:image/jpg;base64," + base64.b64encode(open(file_path, "rb").read()).decode()
r = requests.post("https://api.mathpix.com/v3/latex",
    data=json.dumps({'src': image_uri, 'formats': ['latex_normal']}),
    headers={"app_id": "YOUR_APP_ID", "app_key": "YOUR_APP_KEY",
             "Content-type": "application/json"})
print(json.dumps(json.loads(r.text), indent=4, sort_keys=True))

curl -X POST https://api.mathpix.com/v3/latex \
-H 'app_id: YOUR_APP_ID' \
-H 'app_key: YOUR_APP_KEY' \
-H 'Content-Type: application/json' \
--data '{ "src": "data:image/jpeg;base64,'$(base64 -i limit.jpg)'", "formats": ["latex_normal"] }'

POST https://api.mathpix.com/v3/latex

Parameter	Type	Description
src	string	Image data, or public URL where image is located
formats	string[]	String postprocessing formats (see Formatting section)
ocr (optional)	string[]	Process only math [“math”] or both math and text [“math”, “text”]
format_options (optional)	object	Options for specific formats (see Formatting section)
skip_recrop (optional)	bool	Force algorithm to consider whole image
confidence_threshold (optional)	number in [0,1]	Set threshold for triggering confidence errors
beam_size (optional)	number in [1,5]	Number of results to consider during recognition
n_best (optional)	number in [1,beam_size]	Number of highest-confidence results to return
region (optional)	object	Specify the image area with the pixel coordinates `top_left_x`, `top_left_y`, `width`, and `height`
callback (optional)	object	Callback request object

Formatting

The following formats can be used in the request:

Format	Description
text	text mode output, with math inside delimiters, eg. `test $x^2$`, inline math by default
text_display	same as `text`, except uses block mode math instead of inline mode when in doubt
latex_normal	direct LaTeX representation of the input
latex_styled	modified output to improve the visual appearance such as adding ’\left’ and ’\right’ around parenthesized expressions that contain tall expressions like subscript or superscript
latex_simplified	modified output for symbolic processing such as shortening operator names, replacing long division with a fraction, and converting a column of operands into a single formula
latex_list	output split into a list of simplified strings to help process multiple equations
mathml	the MathML for the recognized math
asciimath	the AsciiMath for the recognized math
wolfram	a string compatible with the Wolfram Alpha engine

Format options

To return a more compact latex_styled result, one could send the following request:

{
  "src":"data:image/jpeg;base64,...",
  "ocr": ["math", "text"], 
  "skip_recrop": true,
  "formats": [
    "text",
    "latex_simplified",
    "latex_styled",
    "mathml",
    "asciimath",
    "latex_list"
   ],
   "format_options": {
     "latex_styled": {"transforms": ["rm_spaces"]}
   }
}

The result for “latex_styled” would now be
"\\lim_{x \\rightarrow 3}\\left(\\frac{x^{2}+9}{x-3}\\right)"
instead of
"\\lim _ { x \\rightarrow 3 } \\left( \\frac { x ^ { 2 } + 9 } { x - 3 } \\right)"

The optional format_options request parameter allows a request to customize the LaTeX result formats using an object with a format as the property name and the options for that format as the value. The options value may specify the following properties:

Option	Type	Description
transforms	string[]	array of transformation names
math_delims	[string, string]	[begin, end] delimiters for math mode, for example `$` and `$`
displaymath_delims	[string, string]	[begin, end] delimiters for displaymath mode, for example `\[` and `\]`

The currently-supported transforms are:

Transform	Description
rm_spaces	omit spaces around LaTeX groups and other places where spaces are superfluous
rm_newlines	uses spaces instead of newlines between text lines in paragraphs
rm_fonts	omit mathbb, mathbf, mathcal, and mathrm commands
rm_style_syms	replace styled commands with unstyled versions, e.g., bigoplus becomes oplus
rm_text	omit text to the left or right of math
long_frac	convert longdiv to frac

Note that rm_fonts and rm_style_syms are implicit in latex_normal, latex_simplified, and latex_list. The long_frac transformation is implicit in latex_simplified and latex_list.

Result objects

{
    "detection_list": [],
    "detection_map": {
        "contains_chart": 0,
        "contains_diagram": 0,
        "contains_geometry": 0,
        "contains_graph": 0,
        "contains_table": 0,
        "is_inverted": 0,
        "is_not_math": 0,
        "is_printed": 0
    },
    "latex_normal": "\\lim _ { x \\rightarrow 3 } ( \\frac { x ^ { 2 } + 9 } { x - 3 } )",
    "latex_confidence": 0.86757309488734,
    "position": {
        "height": 273,
        "top_left_x": 57,
        "top_left_y": 14,
        "width": 605
    }
}

Field	Type	Description
`text` (optional)	string	Recognized `text` format
`text_display` (optional)	string	Recognized `text_display` format
`latex_normal` (optional)	string	Recognized `latex_normal` format
`latex_simplified` (optional)	string	Recognized `latex_normal` format
`latex_styled` (optional)	string	Recognized `latex_styled` format
`latex_list` (optional)	string[]	Recognized `latex_list` format
`mathml` (optional)	string	Recognized MathML format
`asciimath` (optional)	string	Recognized AsciiMath format
`wolfram` (optional)	string	Recognized Wolfram format
`position` (optional)	object	Position object, pixel coordinates
`detection_list` (optional)	string[]	Detects image properties (see image properties)
`error` (optional)	string	US locale error message
`error_info` (optional)	object	Error info object
`latex_confidence` (optional)	number in [0,1]	Estimated probability 100% correct
`candidates` (optional)	object[]	n_best results

Image properties

The API defines multiple detection types:

Detection	Definition
`contains_diagram`	Contains a diagram.
`is_printed`	The image is taken of printed math, not handwritten math.
`is_not_math`	No valid equation was detected.

Error info object

In addition to the error field that contains a string (en-us locale) describing an error, a Mathpix response has an error_info field with an object providing programmatic information. The fields of this object include id, uniquely specifying the error, message, containing a string similar to the top-level error field, and detail fields specific to the type of error. The table below lists the different errors Mathpix returns.

Error info fields:

Field	Type	Description
`id`	string	specifies the error id (see below)
`message`	string	error message
`detail` (optional)	string	Additional error info

Error id types

Here is a table with all the error_id possibilities:

Id	Description	Detail fields	HTTP Status
http_unauthorized	Invalid credentials		401
http_max_requests	Too many requests	count	429
json_syntax	JSON syntax error		200
image_missing	Missing URL in request body		200
image_download_error	Error downloading image	url	200
image_decode_error	Cannot decode the image data		200
image_no_content	No content found in image		200
image_not_supported	Image is not math or text		200
image_max_size	Image is too large to process		200
opts_bad_callback	Bad callback field(s)	post?, reply?, batch_id?	200
opts_unknown_ocr	Unknown ocr option(s)	ocr	200
opts_unknown_format	Unknown format option(s)	formats	200
opts_number_required	Option must be a number	name,value	200
opts_value_out_of_range	Value not in accepted range	name,value	200
math_confidence	Low confidence		200
math_syntax	Unrecognized math		200
batch_unknown_id	Unknown batch id	batch_id	200
sys_exception	Server error	batch_id?	200

Callback request object

Full request body example (includes callback field, shortened src field):

{  
   "src":"data:image/jpeg;base64,...",
   "ocr": ["math", "text"],
   "formats": ["text", "latex_styled"],
   "skip_recrop": true,
   "callback":{  
      "post":"http://RequestBin.com/10z3g561",
      "reply":"integral.jpg"
   }
}

Field	Type	Description
`post`	string	URL to which to make POST callback
`headers`	object	key value pairs of headers to make POST
`reply` (optional)	string	Sets values of `reply` field of callback response object (see callback response object)

Callback response object

{
  "reply": "integral.jpg",
  "result": {
    "detection_list": [
      "is_printed"
    ],
    "detection_map": {
      "contains_chart": 0,
      "contains_diagram": 0,
      "contains_geometry": 0,
      "contains_graph": 0,
      "contains_table": 0,
      "is_inverted": 0,
      "is_not_math": 0,
      "is_printed": 1
    },
    "error": "",
    "latex": "\\int \\frac { 4 x } { \\sqrt { x ^ { 2 } + 1 } } d x",
    "latex_confidence": 0.99817252453161,
    "position": {
      "height": 215,
      "top_left_x": 57,
      "top_left_y": 0,
      "width": 605
    }
  }
}

Field	Type	Description
`reply`	string	Request identifier
`result`	object	Result object

Process batch (v3/batch)

A batch request is made as follows:

{  
   "urls":{  
      "inverted":"https://raw.githubusercontent.com/Mathpix/api-examples/master/images/inverted.jpg",
      "algebra":"https://raw.githubusercontent.com/Mathpix/api-examples/master/images/algebra.jpg"
   },
   "formats": ["latex_simplified"]
}

curl -X POST https://api.mathpix.com/v3/batch \
-H "app_id: YOUR_APP_ID" \
-H "app_key: YOUR_APP_KEY" \
-H "Content-Type: application/json" \
--data '{ "urls": {"inverted": "https://raw.githubusercontent.com/Mathpix/api-examples/master/images/inverted.jpg", "algebra": "https://raw.githubusercontent.com/Mathpix/api-examples/master/images/algebra.jpg"},"formats":["latex_simplified"] }'


import requests
import json

base_url = 'https://raw.githubusercontent.com/Mathpix/api-examples/master/images/'

data = {
    'urls': {
        'algebra': base_url + 'algebra.jpg',
        'inverted': base_url + 'inverted.jpg'
    },
    'formats': ['latex_simplified']
}

r = requests.post(
    "https://api.mathpix.com/v3/batch", data=json.dumps(data),
    headers={
        'app_id': 'YOUR_APP_ID',
        'app_key': 'YOUR_APP_KEY',
        'content-type': 'application/json'
    },
    timeout=30
)
reply = json.loads(r.text)
assert reply.has_key('batch_id')

The response to the batch is a positive integer value for batch_id.

{
  "batch_id": "17"
}

The response to GET /v3/batch/17 is below when the batch has completed. Before completion the “results” field may be empty or contain only one of the two results.

{
  "keys": ["algebra", "inverted"],
  "results": {
    "algebra": {
      "detection_list": [], 
      "detection_map": {
        "contains_chart": 0, 
        "contains_diagram": 0, 
        "contains_geometry": 0, 
        "contains_graph": 0, 
        "contains_table": 0, 
        "is_inverted": 0, 
        "is_not_math": 0, 
        "is_printed": 0
      }, 
      "latex_simplified": "12 + 5 x - 8 = 12 x - 10", 
      "latex_confidence": 0.99640350138238, 
      "position": {
        "height": 208, 
        "top_left_x": 0, 
        "top_left_y": 0, 
        "width": 1380
      }
    }, 
    "inverted": {
      "detection_list": [
        "is_inverted", 
        "is_printed"
      ], 
      "detection_map": {
        "contains_chart": 0, 
        "contains_diagram": 0, 
        "contains_geometry": 0, 
        "contains_graph": 0, 
        "contains_table": 0, 
        "is_inverted": 1, 
        "is_not_math": 0, 
        "is_printed": 1
      }, 
      "latex_simplified": "x ^ { 2 } + y ^ { 2 } = 9", 
      "latex_confidence": 0.99982263230866, 
      "position": {
        "height": 170, 
        "top_left_x": 48, 
        "top_left_y": 85, 
        "width": 544
      }
    }
  }
}

The Mathpix API supports processing multiple images in a single POST request to a different endpoint: /v3/batch. The request body may contain all the /v3/latex parameters except src and must contain a urls parameter. The request may contain an additonal callback parameter to receive results after all the images in the batch have been processed.

Parameter	Type	Description
urls	object	key-value for each image in the batch where the value may be a string url or an object containing a url and image-specific request arguments such as `region` and `formats`.
callback (optional)	object	description of where to send the batch results
callback.post	string	url to post results
callback.reply (optional)	object	data to send in reply to batch POST
callback.body (optional)	object	data to send with results to callback.post
callback.headers (optional)	object	headers to use when posting results

The response contains only a unique batch_id value. Even if the request includes a callback, there is no guarantee the callback will run successfuly (because of a transient network failure, for example). The preferred approach is to wait an appropriate length of time (about one second for every five images in the batch) and then do a GET on /v3/batch/:id where :id is the batch_id value. The GET request must contain the same app_id and app_key headers as the POST to /v3/batch.

The GET response has the following fields:

Field	Type	Description
keys	string[]	all the url keys present in the originating batch request
results	object	an OCR result for each key that has been processed

Long division

Response for image on the left side:

{
  "detection_map": {
    "contains_chart": 0,
    "contains_diagram": 0,
    "contains_geometry": 0,
    "contains_graph": 0,
    "contains_table": 0,
    "is_inverted": 0,
    "is_not_math": 0,
    "is_printed": 1
  },
  "latex_normal": "8 \\longdiv { 7200 }"
}

The following <script> tags will take care of rendering the \longdiv markup.

<script type="text/javascript" src="//cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
<script type="text/x-mathjax-config">
    MathJax.Hub.Config({
        tex2jax: {inlineMath: [['$','$'], ['\\(','\\)']]},
        messageStyle: "none",
        TeX: {
            Macros: {
                longdiv: ["{\\overline{\\smash{)}#1}}", 1]
            }
        }
    });
</script>

We use the special markup \longdiv to represent long division; it is the only nonvalid Latex markup we return. Long division is used much like \sqrt which is visually similar.

Latency considerations

The biggest source of latency is image uploads. The speed of a response from Mathpix API servers is roughly proportional to the size of the image. Try to use images under 100kb for maximum speeds. JPEG compression and image downsizing are recommended to ensure lowest possible latencies.

Example: Mathpix Snipping Tool

Mathpix Snipping Tool uses the request arguments seen on the right:

{
   "format_options":{
      "text":{
         "displaymath_delims":[
            "\n$$\n",
            "\n$$\n"
         ],
         "transforms":[
            "rm_spaces"
         ],
         "math_delims":[
            "\\(",
            "\\)"
         ]
      },
      "latex_styled":{
         "transforms":[
            "rm_spaces"
         ]
      }
   },
   "skip_recrop":true,
   "ocr":[
      "math",
      "text"
   ],
   "formats":[
      "text",
      "text_display",
      "latex_styled"
   ]
}

Note: src image field is abbreviated to make the JSON more readable.

Vocabulary

Mathpix generates any of the following characters:

!	“	$	&	’
(	)	*	+	,
-	.	/	0	1
2	3	4	5	6
7	8	9	:	;
<	=	>	?	A
B	C	D	E	F
G	H	I	J	K
L	M	N	O	P
Q	R	S	T	U
V	W	X	Y	Z
[	\	]	^	_
a	b	c	d	e
f	g	h	i	j
k	l	m	n	o
p	q	r	s	t
u	v	w	x	y
z	{	\|	}	~
\%	\\	\{	\}
\alpha	\angle	\langle	\rangle	\approx
\because	\begin{array}	\beta	\bot	\cap
\cdot	\cdots	\chi	\circ	\cong
\cup	\dagger	\Delta	\delta	\div
\dot	\dots	\ell	\emptyset	\eta
\end{array}	\epsilon	\equiv	\exists	\kappa
\forall	\frac	\gamma	\Gamma	\geq
\hat	\hbar	\hline	\in	\infty
\int	\Lambda	\lambda	\lceil	\left(
\left.	\left[	\left\{	\left\|	\Leftrightarrow
\leftrightarrow	\lfloor	\leq	\longdiv	\mathcal
\mp	\mu	\nabla	\neq	\notin
\oint	\Omega	\omega	\operatorna	\oplus
\overline	\otimes	\parallel	\partial	\perp
\Phi	\phi	\pi	\pm	\prime
\prod	\propto	\Psi	\psi	\qquad
\quad	\rceil	\rfloor	\rho	\right)
\right.	\right\}	\right]	\Rightarrow	\rightarrow
\right\|	\sigma	\sim	\simeq	\sqrt
\square	\star	\sum	\subset	\supset
\subseteq	\supseteq	\tau	\text	\therefore
\Theta	\theta	\times	\tilde
\varphi	\vdots	\vec	\wedge	\vee
\Xi	\xi	\zeta

Example outputs

Examples PDF