Configuring SKIL Notebooks for Large Neural Networks

More ambitious SKIL projects require larger neural networks. Neural networks are memory-hungry, so the default settings in Zeppelin notebooks may not suffice. Hitting memory limits produces errors like this:

Screen-Shot-2018-03-23-at-3.43.40-PM

To fix memory issues in Zeppelin, increase the amount of JVM RAM available. Zeppelin's default configuration provides 8 GB of RAM on the JVM, and 4 GB of off-heap space. You can expand these by changing the DEFAULT_ZEPPELIN_JVM_ARGS environment variable and deleting the Zeppelin interpreter process in the Processes tab.

To set the environment variable, either specify it when you launch the Docker container (with -e), or add it to /etc/skil/skil-env.sh (or /etc/profile.d/skil-env.sh if /etc/skil-env.sh doesn't exist)

The following line shows the maximum settings available if you have a K80 GPU:

DEFAULT_ZEPPELIN_JVM_ARGS=-Xmx16g -Dorg.bytedeco.javacpp.maxbytes=16G -Dorg.bytedeco.javacpp.maxphysicalbytes=16G -Dorg.nd4j.versioncheck=false -Dorg.deeplearning4j.config.custom.enabled=false

If you are training on CPUs alone, you are only limited to your RAM. After you set the environment variable, you'll need to stop and re-start the SKIL system with the commands:

sudo systemctl stop skil
sudo systemctl start skil