jm + ooms   2

Spark memory tuning on EMR
'Best practices for successfully managing memory for Apache Spark applications on Amazon EMR', on the AWS Big Data blog.

'In this blog post, I detailed the possible out-of-memory errors, their causes, and a list of best practices to prevent these errors when submitting a Spark application on Amazon EMR.

My colleagues and I formed these best practices after thorough research and understanding of various Spark configuration properties and testing multiple Spark applications. These best practices apply to most of out-of-memory scenarios, though there might be some rare scenarios where they don’t apply. However, we believe that this blog post provides all the details needed so you can tweak parameters and successfully run a Spark application.'
spark  emr  aws  tuning  memory  ooms  java 
12 days ago by jm
How to stop Ubuntu Xenial (16.04) from randomly killing your big processes
ugh.
Unfortunately, a bug was recently introduced into the allocator which made it sometimes not try hard enough to free kernel cache memory before giving up and invoking the OOM killer. In practice, this means that at random times, the OOM killer would strike at big processes when the kernel tries to allocate, say, 16 kilobytes of memory for a new process’s thread stack — even when there are many gigabytes of memory in reclaimable kernel caches!
oom-killer  ooms  linux  ops  16.04 
march 2017 by jm

Copy this bookmark:



description:


tags: