Once you start building larger models, make sure you are storing these model artifacts in HDFS, S3, or any other external storage. Do not use the project filesystem to store large output artifacts.
In general, any project files larger than 50 MB must be part of your
.gitignore file so that they are
not included in snapshots for future experiments/model builds. Note that in case your models
require resources that are stored outside the model itself, it is up to you to ensure that
these resources are available and immutable as model replicas may be restarted at any