merlin stands for Mixed Effects Regression for LInear, Non-linear and user-defined models. The merlin packages implement the methodology described in the paper:
Crowther MJ. Extended multivariate generalised linear and non-linear mixed effects models. arXiv:1710.02223.
You can get the methods preprint here (or the easier to read software preprint here), which I recommend you read before trying out the accompanying software (note, merlin used to be called megenreg, and the syntax is out of date in the methods pre-print!