Hello A.,
Here is an example:
@startuml :This is a test with math <math>a*x^2+b*x+c</math>inside text; :Let the formula: <math>int_0^1f(x)dx</math>; @enduml
Not only the size, there is the alignment...
But the big deal: how to manage big formula?...Align on bottom, on middle...To debate...Regards,Th.
If you switch from asciimath to latexmath, you get more control over size.
(You can use site https://www.intmath.com/help/asciimath-input-latex-katex-output.php to convert asciimath into latexmath).
Then you can use \scriptsize (or other sizes listed at https://www.overleaf.com/learn/latex/Font_sizes%2C_families%2C_and_styles)
Or use \scalebox{.7}{text} to scale.
Unfortunately I could find no way to explicitly set the font name and size.
e.g.
@startuml :This is a test with math<latex>\scalebox{.7} {\displaystyle{a}\cdot{x}^{2}+{b}\cdot{x}+{c}}</latex>inside text; :Let the formula: <latex>\scriptsize {{ \int_{{0}}^{{1}}} f{{\left({x}\right)}}{\left.{d}{x}\right.}}</latex>; @enduml
Another example using tt font for 1st, and \tiny for 2nd :
(--------)
See some other answer element here: