Recently I was running into a problem with JSDock Toolkit where it would throw an error saying js: exception from uncaught JavaScript throw: java.lang.OutOfMemoryError: Java heap space.
The solution was to add the -s flag so that it would not try and generate a large, marked-up source file for viewing with the docs.
java -jar jsdoc-toolkit/jsrun.jar jsdoc-toolkit/app/run.js gamelib.js -d=docs -n -s
I also added the -n flag to prevent it from documenting methods that didn’t have comments, such as minified jQuery or Box2d, though this was irrelevant in preventing the js: exception from uncaught JavaScript throw: java.lang.OutOfMemoryError: Java heap space error.
Daniel X's Blog 