jm + malloc   6

Fix glibc's MALLOC_ARENA_MAX variable
It seems that recent versions of glibc (up to glibc 2.25 at least) have some dysfunctional behaviour around malloc's arenas on multi-CPU systems, massively inflating the number of arenas allocated, which inflate reported VM sizes and (for multi-threaded Ruby services in particular) fragmenting memory badly.

See also https://devcenter.heroku.com/articles/testing-cedar-14-memory-use

Presto issue reported with glibc malloc arena-per-thread behaviour resulting in Presto OOMs: https://github.com/prestodb/presto/issues/8993

Hadoop affected by the inflated VM sizes reported as a side effect: https://issues.apache.org/jira/browse/HADOOP-7154

Good detailed writeup from IBM's WebSphere blog: https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage
ops  ruby  memory  malloc  allocation  arenas  tuning  fragmentation 
28 days ago by jm
What causes Ruby memory bloat? – Joyful Bikeshedding
Another likely suspect is the memory allocator. After all, Nate Berkopec and Heroku remarked that fiddling with the memory allocator (either replacing it altogether with jemalloc, or setting the magical environment variable MALLOC_ARENA_MAX=2) drastically lowers [Ruby] memory usage.
ruby  memory  rails  linux  malloc  bloat  jemalloc  ops 
28 days ago by jm
Native Memory Tracking
Java 8 HotSpot feature to monitor and diagnose native memory leaks
java  jvm  memory  native-memory  malloc  debugging  coding  nmt  java-8  jcmd 
september 2017 by jm
Debugging Java Native Memory Leaks (evanjones.ca)
Using jemalloc to instrument the contents of the native heap and record stack traces of each chunk's allocators, so that leakers can be quickly identified (GZIPInputStream in this case).

See also https://gdstechnology.blog.gov.uk/2015/12/11/using-jemalloc-to-get-to-the-bottom-of-a-memory-leak/ , https://github.com/jeffgriffith/native-jvm-leaks/blob/master/README.md .
debugging  memory  jvm  java  leaks  memory-leaks  leak-checking  jemalloc  malloc  native  heap  off-heap  gzipinputstream 
january 2017 by jm
Please grow your buffers exponentially
Although in some cases x1.5 is considered good practice. YMMV I guess
malloc  memory  coding  buffers  exponential  jemalloc  firefox  heap  allocation 
november 2014 by jm
memcached turns 10 years old
Well, apparently tomorrow, but close enough. Happy birthday to bradfitz' greatest creation and its wonderful slab allocator!
birthdays  code  via:alex-popescu  open-source  history  malloc  memory  caching  memcached 
may 2013 by jm

Copy this bookmark:



description:


tags: