the role of a lexer and parser

A lexer detects patterns in a sequence of characters (the input string). The patterns are specified as regex in a grammar file (e.g., ANTLR's .g4 file).

The lexer produces a stream of tokens consumed by the parser.

A parser assembles an AST
The nodes of the AST are tokens.
The parser then has a set of rules which are used to apply actions.
For example (in the context of the Physics Derivation Graph), the rules produce Sympy expressions.



For the Physics Derivation Graph, the lexer will be consuming math latex.

The math latex strings are expected to be agnostic to which Physics domain the .tex file is from.
In contrast, the parser rules may be domain-specific.