jm + java   213

Squeezing blood from a stone: small-memory JVM techniques for microservice sidecars
Reducing service memory usage from 500MB to 105MB:
We found two specific techniques to be the most beneficial: turning off one of the two JIT compilers enabled by default (the “C2” compiler), and using a 32-bit, rather than a 64-bit, JVM.
32bit  jvm  java  ops  memory  tuning  jit  linkerd 
6 weeks ago by jm
ztellman/dirigiste
'centrally-planned object and thread pools' for java.

'In the default JVM thread pools, once a thread is created it will only be retired when it hasn't performed a task in the last minute. In practice, this means that there are as many threads as the peak historical number of concurrent tasks handled by the pool, forever. These thread pools are also poorly instrumented, making it difficult to tune their latency or throughput. Dirigiste provides a fast, richly instrumented version of a java.util.concurrent.ExecutorService, and provides a means to feed that instrumentation into a control mechanism that can grow or shrink the pool as needed. Default implementations that optimize the pool size for thread utilization are provided. It also provides an object pool mechanism that uses a similar feedback mechanism to resize itself, and is significantly simpler than the Apache Commons object pool implementation.'

Great metric support, too.
async  jvm  dirigiste  java  threadpools  concurrency  utilization  capacity  executors  object-pools  object-pooling  latency 
8 weeks ago by jm
Koloboke Collections
Interesting new collections lib for Java 6+; generates Map-like and Set-like collections at runtime based on the contract annotations you desire. Fat (20MB) library-based implementation also available
collections  java  koloboke  performance  coding 
8 weeks ago by jm
Why the Very Silly Oracle v. Google Trial Actually Matters
If it’s illegal to write clean room implementations of APIs, then no one has clean hands. The now-shelved open source project Apache Harmony, like Android, reimplemented Java SE, and tech giant IBM contributed code to that project. Oracle itself built its business off a proprietary implementation of SQL, which was created by IBM. The proposition “Reimplementations of APIs are infringements” creates a recursive rabbit hole of liability that spans across the industry. Even the very 37 Java APIs at issue in this trial contain reimplementations of other APIs. Google witness Joshua Bloch—who, while at Sun Microsystems, wrote many of the Java APIs—testified that specific Java APIs are reimplementations of other APIs from Perl 5 and the C programming language.
apis  fair-use  copyright  ip  android  java  google  oracle  law 
9 weeks ago by jm
uber-common/infer-plugin
Gradle plugin that allows easy integration with the infer static analyzer
infer  java  static-analysis  bugs  coding  null 
10 weeks ago by jm
In Oracle v. Google, a Nerd Subculture Is on Trial
“The G part stands for GNU?” Alsup asked in disbelief.
“Yes,” said Schwartz on the stand.
“That doesn’t make any sense,” said the 71-year-old Clinton appointee.
law  gnu  gpl  licensing  java  oracle  sun  apis  ip 
11 weeks ago by jm
JitPack
Publish JVM and Android libraries direct from github -- it'll build and package a lib on the fly, caching them via CDN
build  github  java  maven  gradle  dependencies  packaging  libraries 
april 2016 by jm
Hashed Wheel Timer
nice java impl of this efficient data structure, broken out from Project Reactor
scalability  java  timers  hashed-wheel-timers  algorithms  data-structures 
march 2016 by jm
Conversant ConcurrentQueue and Disruptor BlockingQueue
'Disruptor is the highest performing intra-thread transfer mechanism available in Java. Conversant Disruptor is the highest performing implementation of this type of ring buffer queue because it has almost no overhead and it exploits a particularly simple design.

Conversant has been using this in production since 2012 and the performance is excellent. The BlockingQueue implementation is very stable, although we continue to tune and improve it. The latest release, 1.2.4, is 100% production ready.

Although we have been working on it for a long time, we decided to open source our BlockingQueue this year to contribute something back to the community. ... its a drop in for BlockingQueue, so its a very easy test. Conversant Disruptor will crush ArrayBlockingQueue and LinkedTransferQueue for thread to thread transfers.

In our system, we noticed a 10-20% reduction in overall system load and latency when we introduced it.'
disruptor  blocking-queues  queues  queueing  data-structures  algorithms  java  conversant  concurrency  performance 
march 2016 by jm
Argon2 code audits - part one - Infer
A pretty viable way to run Facebook's Infer dataflow static analysis tool (which is otherwise quite a bear to run).
infer  facebook  java  clang  errors  static-analysis  lint  dataflow  docker 
february 2016 by jm
Netty @Apple: Large Scale Deployment/Connectivity [video]
'Norman Maurer presents how Apple uses Netty for its Java based services and the challenges of doing so, including how they enhanced performance by participating in the Netty open source community. Maurer takes a deep dive into advanced topics like JNI, JVM internals, and others.'
apple  netty  norman-maurer  java  jvm  async  talks  presentations 
january 2016 by jm
The Importance of Tuning Your Thread Pools
Excellent blog post on thread pools, backpressure, Little's Law, and other Hystrix-related topics (PS: use Hystrix)
hystrix  threadpools  concurrency  java  jvm  backpressure  littles-law  capacity 
january 2016 by jm
raskasa/metrics-okhttp
Metrics integration for OkHttp. looks quite nice
okhttp  java  clients  http  metrics  dropwizard 
december 2015 by jm
Introducing Netty-HTTP from Cask
netty-http library solves [Netty usability issues] by using JAX-RS annotations to build a HTTP path routing layer on top of netty. In addition, the library implements a guava service to manage the HTTP service. netty-http allows users of the library to just focus on writing the business logic in HTTP handlers without having to worry about the complexities of path routing or learning netty pipeline internals to build the HTTP service.

We've written something very similar, although I didn't even bother supporting JAX-RS annotations -- just a simple code-level DSL.
jax-rs  netty  http  cask  java  services  coding 
december 2015 by jm
Open-sourcing PalDB, a lightweight companion for storing side data
a new LinkedIn open source data store, for write-once/read-mainly side data, java, Apache licensed.

RocksDB discussion: https://www.facebook.com/groups/rocksdb.dev/permalink/834956096602906/
linkedin  open-source  storage  side-data  data  config  paldb  java  apache  databases 
october 2015 by jm
How Netty is used at Layer
pretty conventional HTTP/1.1, WebSockets and HTTP/2 front-end services with modern Netty practices
netty  http  api-services  coding  java  servers 
october 2015 by jm
librato/disco-java
Librato's service discovery library using Zookeeper (so strongly consistent, but with the ZK downside that an AZ outage can stall service discovery updates region-wide)
zookeeper  service-discovery  librato  java  open-source  load-balancing 
october 2015 by jm
Byteman
a tool which simplifies tracing and testing of Java programs. Byteman allows you to insert extra Java code into your application, either as it is loaded during JVM startup or even after it has already started running. The injected code is allowed to access any of your data and call any application methods, including where they are private. You can inject code almost anywhere you want and there is no need to prepare the original source code in advance nor do you have to recompile, repackage or redeploy your application. In fact you can remove injected code and reinstall different code while the application continues to execute. The simplest use of Byteman is to install code which traces what your application is doing. This can be used for monitoring or debugging live deployments as well as for instrumenting code under test so that you can be sure it has operated correctly. By injecting code at very specific locations you can avoid the overheads which often arise when you switch on debug or product trace. Also, you decide what to trace when you run your application rather than when you write it so you don't need 100% hindsight to be able to obtain the information you need.
tracing  java  byteman  injection  jvm  ops  debugging  testing 
september 2015 by jm
Time on multi-core, multi-socket servers
Nice update on the state of System.currentTimeMillis() and System.nanoTime() in javaland. Bottom line: both are non-monotonic nowadays:
The conclusion I've reached is that except for the special case of using nanoTime() in micro benchmarks, you may as well stick to currentTimeMillis() —knowing that it may sporadically jump forwards or backwards. Because if you switched to nanoTime(), you don't get any monotonicity guarantees, it doesn't relate to human time any more —and may be more likely to lead you into writing code which assumes a fast call with consistent, monotonic results.
java  time  monotonic  sequencing  nanotime  timers  jvm  multicore  distributed-computing 
september 2015 by jm
Stormpot
an object pooling library for Java. Use it to recycle objects that are expensive to create. The library will take care of creating and destroying your objects in the background. Stormpot is very mature, is used in production, and has done over a trillion claim-release cycles in testing. It is faster and scales better than any competing pool.


Apache-licensed, and extremely fast: https://medium.com/@chrisvest/released-stormpot-2-4-eeab4aec86d0
java  stormpot  object-pooling  object-pools  pools  allocation  gc  open-source  apache  performance 
september 2015 by jm
Large Java HashMap performance overview
Large HashMap overview: JDK, FastUtil, Goldman Sachs, HPPC, Koloboke, Trove – January 2015 version
java  performance  hashmap  hashmaps  optimization  fastutil  hppc  jdk  koloboke  trove  data-structures 
september 2015 by jm
Preventing Dependency Chain Attacks in Maven
using a whitelist of allowed dependency JARs and their SHAs
security  whitelisting  dependencies  coding  jar  maven  java  jvm 
august 2015 by jm
Implementing Efficient and Reliable Producers with the Amazon Kinesis Producer Library - AWS Big Data Blog
Good advice on production-quality, decent-scale usage of Kinesis in Java with the official library: batching, retries, partial failures, backoff, and monitoring. (Also, jaysus, the AWS Cloudwatch API is awful, looking at this!)
kpl  aws  kinesis  tips  java  batching  streaming  production  cloudwatch  monitoring  coding 
august 2015 by jm
Java lambdas and performance
Lambdas in Java 8 introduce some unpredictable performance implications, due to reliance on escape analysis to eliminate object allocation on every lambda invocation. Peter Lawrey has some details
lambdas  java-8  java  performance  low-latency  optimization  peter-lawrey  coding  escape-analysis 
july 2015 by jm
OkHttp
A new HTTP client library for Android and Java, with a lot of nice features:
HTTP/2 and SPDY support allows all requests to the same host to share a socket.

Connection pooling reduces request latency (if SPDY isn’t available).

Transparent GZIP shrinks download sizes.

Response caching avoids the network completely for repeat requests.

OkHttp perseveres when the network is troublesome: it will silently recover from common connection problems. If your service has multiple IP addresses OkHttp will attempt alternate addresses if the first connect fails. This is necessary for IPv4+IPv6 and for services hosted in redundant data centers. OkHttp initiates new connections with modern TLS features (SNI, ALPN), and falls back to TLS 1.0 if the handshake fails.

Using OkHttp is easy. Its 2.0 API is designed with fluent builders and immutability. It supports both synchronous blocking calls and async calls with callbacks.
android  http  java  libraries  okhttp  http2  spdy  microservices  jdk 
july 2015 by jm
sjk
a command line tool for JVM diagnostic troubleshooting and profiling.
java  jvm  monitoring  commandline  jmx  sjk  tools  ops 
june 2015 by jm
Facebook Infer
New static analysis goodnews, freshly open-sourced by Facebook:
Facebook Infer uses logic to do reasoning about a program's execution, but reasoning at this scale — for large applications built from millions of lines of source code — is hard. Theoretically, the number of possibilities that need to be checked is more than the number of estimated atoms in the observable universe. Furthermore, at Facebook our code is not a fixed artifact but an evolving system, updated frequently and concurrently by many developers. It is not unusual to see more than a thousand modifications to our mobile code submitted for review in a given day. The requirements on the program analyzer then become even more challenging because we expect a tool to report quickly on these code modifications — in the region of 10 minutes — to fit in with developers' workflow. Coping with this scale and velocity requires advanced mathematical techniques. Facebook Infer uses two such techniques: separation logic and bi-abduction.

Separation logic is a theory that allows Facebook Infer's analysis to reason about small, independent parts of the application storage, rather than having to consider the entirety of the memory potentially at every step. That would be a daunting task on modern processors with their large addressable virtual memories.

Bi-abduction is a logical inference technique that allows Facebook Infer to discover properties about the behavior of independent parts of the application code. By storing these properties between runs, Facebook Infer needs to analyze only the parts of the software that have changed, reusing the results of its previous analysis where it can.

By combining these approaches, our analyzer is able to find complex problems in modifications to an application built from millions of lines of code, in minutes.


(via Bryan O'Sullivan)
via:bos  infer  facebook  static-analysis  lint  code  java  ios  android  coding  bugs 
june 2015 by jm
Orbit Async
Orbit Async implements async-await methods in the JVM. It allows programmers to write asynchronous code in a sequential fashion. It was developed by BioWare, a division of Electronic Arts.


Open source, BSD-licensed.
async  await  java  jvm  bioware  coding  threading 
june 2015 by jm
Buck
A high-performance java build tool, from Facebook. Make-like
android  build  java  make  coding  facebook 
june 2015 by jm
Apache HTrace
a Zipkin-compatible distributed-system tracing framework in Java, in the Apache Incubator
zipkin  tracing  trace  apache  incubator  java  debugging 
may 2015 by jm
MappedBus
a Java based low latency, high throughput message bus, built on top of a memory mapped file; inspired by Java Chronicle with the main difference that it's designed to efficiently support multiple writers – enabling use cases where the order of messages produced by multiple processes are important. MappedBus can be also described as an efficient IPC mechanism which enable several Java programs to communicate by exchanging messages.
ipc  java  jvm  mappedbus  low-latency  mmap  message-bus  data-structures  queue  message-passing 
may 2015 by jm
Migration to, Expectations, and Advanced Tuning of G1GC
Bookmarking for future reference. recommended by one of the GC experts, I can't recall exactly who ;)
gc  g1gc  jvm  java  tuning  performance  ops  migration 
may 2015 by jm
The Injector: A new Executor for Java
This honestly fits a narrow niche, but one that is gaining in popularity. If your messages take > 100μs to process, or your worker threads are consistently saturated, the standard ThreadPoolExecutor is likely perfectly adequate for your needs. If, on the other hand, you’re able to engineer your system to operate with one application thread per physical core you are probably better off looking at an approach like the LMAX Disruptor. However, if you fall in the crack in between these two scenarios, or are seeing a significant portion of time spent in futex calls and need a drop in ExecutorService to take the edge off, the injector may well be worth a look.
performance  java  executor  concurrency  disruptor  algorithms  coding  threads  threadpool  injector 
may 2015 by jm
"Trash Day: Coordinating Garbage Collection in Distributed Systems"
Another GC-coordination strategy, similar to Blade (qv), with some real-world examples using Cassandra
blade  via:adriancolyer  papers  gc  distsys  algorithms  distributed  java  jvm  latency  spark  cassandra 
may 2015 by jm
Smarter testing Java code with Spock Framework
hmm, looks quite nice as a potential next-gen JUnit replacement for unit tests
java  testing  bdd  tests  junit  unit-tests  spock  via:trishagee 
may 2015 by jm
Cassandra moving to using G1 as the default recommended GC implementation
This is a big indicator that G1 is ready for primetime. CMS has long been the go-to GC for production usage, but requires careful, complex hand-tuning -- if G1 is getting to a stage where it's just a case of giving it enough RAM, that'd be great.

Also, looks like it'll be the JDK9 default: https://twitter.com/shipilev/status/593175793255219200
cassandra  tuning  ops  g1gc  cms  gc  java  jvm  production  performance  memory 
april 2015 by jm
_Blade: a Data Center Garbage Collector_
Essentially, add a central GC scheduler to improve tail latencies in a cluster, by taking instances out of the pool to perform slow GC activity instead of letting them impact live operations. I've been toying with this idea for a while, nice to see a solid paper about it
gc  latency  tail-latencies  papers  blade  go  java  scheduling  clustering  load-balancing  low-latency  performance 
april 2015 by jm
OG-Commons/Guavate.java
'Utilities that help bridge the gap between Java 8 and Google Guava. Guava has the {@link FluentIterable} concept which is similar to streams. In many ways, fluent iterable is nicer, because it directly binds to the immutable collection classes. However, on balance it seems wise to use the stream API rather than {@code FluentIterable} in Java 8.'
guava  java-8  java  fluentiterable  streams  fluent  coding 
april 2015 by jm
Optimizing Java CMS garbage collections, its difficulties, and using JTune as a solution | LinkedIn Engineering
I like the sound of this -- automated Java CMS GC tuning, kind of like a free version of JClarity's Censum (via Miguel Ángel Pastor)
java  jvm  tuning  gc  cms  linkedin  performance  ops 
april 2015 by jm
Cassandra remote code execution hole (CVE-2015-0225)
Ah now lads.
Under its default configuration, Cassandra binds an unauthenticated
JMX/RMI interface to all network interfaces. As RMI is an API for the
transport and remote execution of serialized Java, anyone with access
to this interface can execute arbitrary code as the running user.
cassandra  jmx  rmi  java  ops  security 
april 2015 by jm
The Four Month Bug: JVM statistics cause garbage collection pauses (evanjones.ca)
Ugh, tying GC safepoints to disk I/O? bad idea:
The JVM by default exports statistics by mmap-ing a file in /tmp (hsperfdata). On Linux, modifying a mmap-ed file can block until disk I/O completes, which can be hundreds of milliseconds. Since the JVM modifies these statistics during garbage collection and safepoints, this causes pauses that are hundreds of milliseconds long. To reduce worst-case pause latencies, add the -XX:+PerfDisableSharedMem JVM flag to disable this feature. This will break tools that read this file, like jstat.
bugs  gc  java  jvm  disk  mmap  latency  ops  jstat 
march 2015 by jm
ben-manes/caffeine
'Caffeine is a Java 8 based concurrency library that provides specialized data structures, such as a high performance cache.'
cache  java8  java  guava  caching  concurrency  data-structures  coding 
march 2015 by jm
Biased Locking in HotSpot (David Dice's Weblog)
This is pretty nuts. If biased locking in the HotSpot JVM is causing performance issues, it can be turned off:
You can avoid biased locking on a per-object basis by calling System.identityHashCode(o). If the object is already biased, assigning an identity hashCode will result in revocation, otherwise, the assignment of a hashCode() will make the object ineligible for subsequent biased locking.
hashcode  jvm  java  biased-locking  locking  mutex  synchronization  locks  performance 
march 2015 by jm
Glowroot
"Open source APM for Java" -- profiling in production, with a demo benchmark showing about a 2% performance impact. Wonder about effects on memory/GC, though
apm  java  metrics  measurement  new-relic  profiling  glowroot 
march 2015 by jm
JClarity's Illuminate
Performance-diagnosis-as-a-service. Cool.
Users download and install an Illuminate Daemon using a simple installer which starts up a small stand alone Java process. The Daemon sits quietly unless it is asked to start gathering SLA data and/or to trigger a diagnosis. Users can set SLA’s via the dashboard and can opt to collect latency measurements of their transactions manually (using our library) or by asking Illuminate to automatically instrument their code (Servlet and JDBC based transactions are currently supported).

SLA latency data for transactions is collected on a short cycle. When the moving average of latency measurements goes above the SLA value (e.g. 150ms), a diagnosis is triggered. The diagnosis is very quick, gathering key data from O/S, JVM(s), virtualisation and other areas of the system. The data is then run through the machine learned algorithm which will quickly narrow down the possible causes and gather a little extra data if needed.

Once Illuminate has determined the root cause of the performance problem, the diagnosis report is sent back to the dashboard and an alert is sent to the user. That alert contains a link to the result of the diagnosis which the user can share with colleagues. Illuminate has all sorts of backoff strategies to ensure that users don’t get too many alerts of the same type in rapid succession!
illuminate  jclarity  java  jvm  scala  latency  gc  tuning  performance 
february 2015 by jm
Proving that Android’s, Java’s and Python’s sorting algorithm is broken (and showing how to fix it)
Wow, this is excellent work. A formal verification of Tim Peters' TimSort failed, resulting in a bugfix:
While attempting to verify TimSort, we failed to establish its instance invariant. Analysing the reason, we discovered a bug in TimSort’s implementation leading to an ArrayOutOfBoundsException for certain inputs. We suggested a proper fix for the culprit method (without losing measurable performance) and we have formally proven that the fix actually is correct and that this bug no longer persists.
timsort  algorithms  android  java  python  sorting  formal-methods  proofs  openjdk 
february 2015 by jm
Sysdig Cloud’s JMX Metrics
Sysdig Cloud users have the ability to view and analyze Java Management Extensions (JMX) metrics out of the box with no additional configuration or setup required.
sysdig  jmx  java  jvm 
february 2015 by jm
Superfish: A History Of Malware Complaints And International Surveillance - Forbes
Superfish, founded and led by former Intel employee and ex-surveillance boffin Adi Pinhas, has been criticised by users the world over since its inception in 2006.
superfish  lenovo  privacy  surveillance  ads  java  windows  mac  firefox  pups  ssl  tls  ad-injection  komodia 
february 2015 by jm
OpenJDK: jol
'JOL (Java Object Layout) is the tiny toolbox to analyze object layout schemes in JVMs. These tools are using Unsafe, JVMTI, and Serviceability Agent (SA) heavily to decoder the actual object layout, footprint, and references. This makes JOL much more accurate than other tools relying on heap dumps, specification assumptions, etc.'

Recommended by Nitsan Wakart, looks pretty useful for JVM devs
java  jvm  tools  scala  memory  estimation  ram  object-layout  debugging  via:nitsan 
february 2015 by jm
RateLimitedLogger
Our latest open source release from Swrve Labs: an Apache-licensed, SLF4J-compatible, simple, fluent API for rate-limited logging in Java:

'A RateLimitedLog object tracks the rate of log message emission, imposes an internal rate limit, and will efficiently suppress logging if this is exceeded. When a log is suppressed, at the end of the limit period, another log message is output indicating how many log lines were suppressed. This style of rate limiting is the same as the one used by UNIX syslog; this means it should be comprehensible, easy to predict, and familiar to many users, unlike more complex adaptive rate limits.'

We've been using this in production for months -- it's pretty nifty ;) Never fear your logs again!
logs  logging  coding  java  open-source  swrve  slf4j  rate-limiting  libraries 
february 2015 by jm
google/error-prone
Nice looking static code validation tool for Java, from Google. I recognise a few of these errors ;)
google  static  code-validation  lint  testing  java  coding 
february 2015 by jm
TL;DR: Cassandra Java Huge Pages
Al Tobey does some trial runs of -XX:+AlwaysPreTouch and -XX:+UseHugePages
jvm  performance  tuning  huge-pages  vm  ops  cassandra  java 
february 2015 by jm
Google Java Style
A good set of basic, controversy-free guidelines for clean java code style
style  java  google  coding  guidelines  formatting  coding-standards 
february 2015 by jm
'Prometheus instrumentation library for JVM applications'
Good example of a clean java OSS release, from Soundcloud. will be copying bits of this myself soon...
prometheus  java  libraries  oss  github  sonatype  maven  releases 
february 2015 by jm
Maintaining performance in distributed systems [slides]
Great slide deck from Elasticsearch on JVM/dist-sys performance optimization
performance  elasticsearch  java  jvm  ops  tuning 
january 2015 by jm
JCTools
Java Concurrency Tools for the JVM. This project aims to offer some concurrent data structures currently missing from the JDK:

Bounded lock free queues
SPSC/MPSC/SPMC/MPMC variations for concurrent queues
Alternative interfaces for queues (experimental)
Offheap concurrent ring buffer for ITC/IPC purposes (experimental)
Executor (planned)
concurrency  lock-free  data-structures  queues  jvm  java 
january 2015 by jm
Hack workaround to get JVM thread priorities working on Linux
As used in Cassandra ( http://grokbase.com/t/hbase/dev/13bf9kezes/about-xx-threadprioritypolicy-42 )!
if you just set the "ThreadPriorityPolicy" to something else than the legal values 0 or 1, [...] a slight logic bug in Sun's JVM code kicks in, and thus sets the policy to be as if running with root - thus you get exactly what one desire. The operating system, Linux, won't allow priorities to be heightened above "Normal" (negative nice value), and thus just ignores those requests (setting it to normal instead, nice value 0) - but it lets through the requests to set it lower (setting the nice value to some positive value).
cassandra  thread-priorities  threads  java  jvm  linux  nice  hacks 
january 2015 by jm
Java for Everything
Actually, I'm really agreeing with a lot of this. Particularly this part:
Programmers will cringe at writing some kind of command dispatch list:

if command = "up":
up()
elif command = "status":
status()
elif command = "revert":
revert()
...

so they’ll go off and write some introspecting auto-dispatch cleverness, but that takes longer to write and will surely confuse future readers who’ll wonder how the heck revert() ever gets called. Yet the programmer will incorrectly feel as though he saved himself time. This is the trap of the dynamic language. It feels like you’re being more productive, but aside from the first 10 minutes of a new program, you’re not. Just write the stupid dispatch manually and get on with the real work.


I've also gone right off dynamic languages for any kind of non-toy work.

Mind you he needs to get around to ditching Vim for a proper IDE. That's the key thing that makes coding in a statically-typed language really pleasant -- when graphical refactoring becomes easy and usable, and errors are visible as you type them...
java  coding  static-typing  python  unit-tests 
november 2014 by jm
ExecutorService - 10 tips and tricks
Excellent advice from Tomasz Nurkiewicz' blog for anyone using java.util.concurrent.ExecutorService regularly. The whole blog is full of great posts btw
concurrency  java  jvm  threading  threads  executors  coding 
november 2014 by jm
Netty: Using as a generic library
Some cool stuff that comes along with Netty: an improved ByteBuffer, a thread-local object pool, a hashed-wheel Timer, and some nice mechanical-sympathy utils.
mechanical-sympathy  netty  java  bytebuffer  object-pools  data-structures  hashed-wheel-timer  algorithms  timers 
november 2014 by jm
"Aeron: High-Performance Open Source Message Transport" [slides, PDF]
a new networked pub/sub library from Martin "Disruptor" Thompson, based around a replicated, persistent log of messages, with exceptionally low latency. Apache-licensed. Very similar to the realtime messaging stack we've built in Swrve. ;)

https://github.com/real-logic/Aeron
realtime  messaging  pub-sub  ipc  queues  transports  martin-thompson  slides  latencies  open-source  java  libraries 
november 2014 by jm
Why We Didn’t Use Kafka for a Very Kafka-Shaped Problem
A good story of when Kafka _didn't_ fit the use case:
We came up with a complicated process of app-level replication for our messages into two separate Kafka clusters. We would then do end-to-end checking of the two clusters, detecting dropped messages in each cluster based on messages that weren’t in both.

It was ugly. It was clearly going to be fragile and error-prone. It was going to be a lot of app-level replication and horrible heuristics to see when we were losing messages and at least alert us, even if we couldn’t fix every failure case.

Despite us building a Kafka prototype for our ETL — having an existing investment in it — it just wasn’t going to do what we wanted. And that meant we needed to leave it behind, rewriting the ETL prototype.
cassandra  java  kafka  scala  network-partitions  availability  multi-region  multi-az  aws  replication  onlive 
november 2014 by jm
Jauter
This Java library can route paths to targets and create paths from targets and params (reverse routing). This library is tiny, without additional dependencies, and is intended for use together with an HTTP server side library. If you want to use with Netty, see netty-router.
java  jauter  scala  request-routing  http  netty  open-source 
october 2014 by jm
Tehuti
An embryonic metrics library for Java/Scala from Felix GV at LinkedIn, extracted from Kafka's metric implementation and in the new Voldemort release. It fixes the major known problems with the Meter/Timer implementations in Coda-Hale/Dropwizard/Yammer Metrics.

'Regarding Tehuti: it has been extracted from Kafka's metric implementation. The code was originally written by Jay Kreps, and then maintained improved by some Kafka and Voldemort devs, so it definitely is not the work of just one person. It is in my repo at the moment but I'd like to put it in a more generally available (git and maven) repo in the future. I just haven't had the time yet...

As for comparing with CodaHale/Yammer, there were a few concerns with it, but the main one was that we didn't like the exponentially decaying histogram implementation. While that implementation is very appealing in terms of (low) memory usage, it has several misleading characteristics (a lack of incoming data points makes old measurements linger longer than they should, and there's also a fairly high possiblity of losing interesting outlier data points). This makes the exp decaying implementation robust in high throughput fairly constant workloads, but unreliable in sparse or spiky workloads. The Tehuti implementation provides semantics that we find easier to reason with and with a small code footprint (which we consider a plus in terms of maintainability). Of course, it is still a fairly young project, so it could be improved further.'

More background at the kafka-dev thread: http://mail-archives.apache.org/mod_mbox/kafka-dev/201402.mbox/%3C131A7649-ED57-45CB-B4D6-F34063267664@linkedin.com%3E
kafka  metrics  dropwizard  java  scala  jvm  timers  ewma  statistics  measurement  latency  sampling  tehuti  voldemort  linkedin  jay-kreps 
october 2014 by jm
Troubleshooting Production JVMs with jcmd
remotely trigger GCs, finalization, heap dumps etc. Handy
jvm  jcmd  debugging  ops  java  gc  heap  troubleshooting 
september 2014 by jm
"Left-Right: A Concurrency Control Technique with Wait-Free Population Oblivious Reads" [pdf]
'In this paper, we describe a generic concurrency control technique with Blocking write operations and Wait-Free Population Oblivious read operations, which we named the Left-Right technique. It is of particular interest for real-time applications with dedicated Reader threads, due to its wait-free property that gives strong latency guarantees and, in addition, there is no need for automatic Garbage Collection.
The Left-Right pattern can be applied to any data structure, allowing concurrent access to it similarly to a Reader-Writer lock, but in a non-blocking manner for reads. We present several variations of the Left-Right technique, with different versioning mechanisms and state machines. In addition, we constructed an optimistic approach that can reduce synchronization for reads.'

See also http://concurrencyfreaks.blogspot.ie/2013/12/left-right-concurrency-control.html for java implementation code.
left-right  concurrency  multithreading  wait-free  blocking  realtime  gc  latency  reader-writer  locking  synchronization  java 
september 2014 by jm
Java tip: optimizing memory consumption
Good tips on how to tell if object allocation rate is a bottleneck in your JVM-based code
yourkit  memory  java  jvm  allocation  gc  bottlenecks  performance 
august 2014 by jm
OpenHFT/hftc · GitHub
This is a yet another Java collections library of primitive specializations. Java 6+. Apache 2.0 license. Currently only hash sets and hash maps are implemented.
openhft  performance  java  jvm  collections  asl  hashsets  hashmaps  data-structures 
august 2014 by jm
MinHash for dummies
A Java-oriented practical intro to the MinHash duplicate-detection shingling algo
shingling  algorithms  minhash  hashing  duplicates  duplicate-detection  fuzzy-matching  java 
august 2014 by jm
How to take over the computer of any JVM developer
To prove how easy [MITM attacking Mavencentral JARs] is to do, I wrote dilettante, a man-in-the-middle proxy that intercepts JARs from maven central and injects malicious code into them. Proxying HTTP traffic through dilettante will backdoor any JARs downloaded from maven central. The backdoored version will retain their functionality, but display a nice message to the user when they use the library.
jars  dependencies  java  build  clojure  security  mitm  http  proxies  backdoors  scala  maven  gradle 
july 2014 by jm
« earlier      
per page:    204080120160

related tags

3des  32bit  actors  ad-injection  adrian-cockroft  ads  adtuitive  aeron  akka  alexey-shipilev  algorithms  allocation  analysis  android  annotations  apache  apache-harmony  api  api-services  apis  apm  apple  applets  architecture  arrays  asl  asl2  async  atmosphere  atomic  availability  avro  await  aws  azul  backdoors  backpressure  batching  bazel  bdb  bdb-je  bdd  benchmarking  benchmarks  berkeley-db  biased-locking  binary  bintray  bioware  bitcoin  blade  blocking  blocking-queues  bloom-filters  bottlenecks  boundary  bugs  build  bytearrayoutputstream  bytebuffer  bytebuffers  byteman  c++  c10k  cache  caching  capacity  capnproto  cardinality  cas  cascading  cask  cassandra  censum  charles-nutter  charlie-hunt  chmod  chronicle  chronon  clang  classpath  cli  clients  cliff-click  clocks  clock_monotonic  clock_realtime  clojure  closures  cloudwatch  clustering  cms  code  code-validation  codecs  coding  coding-standards  collections  command-line  commandline  commercial  compare-and-set  compilation  compiler  compression  comsat  concurrency  concurrent  config  consensus  contracts  conversant  coordinated-omission  copyright  core  core-dump  cost  counters  cpu  cracking  crypto  cuda  curator  currency  currying  d2  dalvik  data  data-structures  databases  dataflow  debugging  demos  dependencies  deployment  des  design  dirigiste  disk  disks  disruptor  distcomp  distributed  distributed-computing  distsys  dns  docker  documents  doug-lea  dram  dropwizard  duplicate-detection  duplicates  dvr  dynamic-languages  dynamodb  dynect  ec2  eclipse  editors  eiffel  elasticsearch  encapsulation  encoding  erlang  errors  escape-analysis  estimation  etsy  eureka  event-processing  event-streams  events  ewma  exceptions  executable  executor  executors  exploits  facebook  fail  failover  fair-use  fast-data  fastutil  fault-tolerance  fauxflake  final  findbugs  firefox  flatbuffers  float  floating-point  fluent  fluent-interfaces  fluentiterable  fomats  formal-methods  formats  formatting  fp  framework  frameworks  frp  functional  functional-programming  fuzzy-matching  g1  g1gc  games  garbage-collection  gc  gc-scout  gcore  gdb  get  gil-tene  giraph  git  github  globbing  glowroot  gnu  go  google  gossip  gpl  gpu  gradle  groovy  guardian  guava  guidelines  hacking  hacks  hadoop  hardware  hash  hashcode  hashed-wheel-timer  hashed-wheel-timers  hashing  hashmap  hashmaps  hashsets  hbase  hdfs  hdrhistogram  head  heap  heap-dump  hive  hll  hosting  hotspot  hpc  hppc  http  http2  httpurlconnection  huge-pages  hystrix  idea  ides  ids  illuminate  impala  incubator  infer  infoq  injection  injector  instrumentation  intellij  interactive  interoperability  invariants  io  ios  ip  ipc  jackson  james-gosling  jar  jars  jauter  java  java-8  java8  jax-rs  jay-kreps  jclarity  jcmd  jdk  jdk6  jemalloc  jenkins  jersey  jetty  jit  jmh  jmm  jmx  joda-time  jpl  jruby  json  jsr-133  jsr166  jstat  junit  jvm  kafka  kamon  karsten-nohl  kinesis  koloboke  komodia  kpl  l2  lambda  lambdas  language  languages  larry-ellison  latencies  latency  law  lazy  lazy-computation  lazy-loading  lazyset  left-right  lenovo  libpfm  libraries  library  librato  licensing  lift  linkedin  linkerd  lint  linux  lisp  lists  littles-law  llc  lmax  load-balancing  lock-free  locking  locks  log4j  logging  logs  long-polling  longadder  low-latency  lucene  mac  mailinator  make  map  map-reduce  mappedbus  mapreduce  maps  marc-brooker  martin-fowler  martin-thompson  maven  maven-central  measurement  mechanical-sympathy  memory  memory-barriers  memstore  message-bus  message-passing  messaging  messing  metrics  microbenchmarks  microservices  migration  minhash  mission-control  mitm  mmap  money  monitoring  monotonic  mpsc  multi-az  multi-region  multicore  multithreading  murmurhash  mutex  mutexes  mvc  mythbusters  nanoseconds  nanotime  nasa  netflix  netty  network-partitions  networking  new-relic  nice  nio  nitsanw  non-blocking  nonblockinghashtable  norman-maurer  nsecs  ntp  ntpd  null  numeric  object-layout  object-model  object-pooling  object-pools  observable  off-heap  off-heap-storage  okhttp  olap  onlive  oo  oom  oop  open-source  openhft  openjdk  opensource  ops  optimization  oracle  oss  ota  outages  overflow  p99  packages  packaging  packet-loss  paldb  papers  patterns  paul-tyma  pdf  percentiles  perf  performance  persistence  persistent  peter-lawrey  pipelines  pipes  play  playframework  plumbr.eu  polyglot  pools  preconditions  presentation  presentations  privacy  prng  production  profiling  programming  prometheus  proofs  protobuf  protobufs  proxies  pthreads  pub-sub  pulsar  pups  putordered  python  qcon  quasar  querying  queue  queueing  queues  rackspace  raft  rails  rainbow-tables  ram  random  randomaccessfile  rate-limiting  reactive  reactive-programming  reactor  reader-writer  realtime  recordinality  refactoring  reference  references  relearning  release  releases  reliability  repl  replication  request-routing  resiliency  rest  rest.li  retries  retrying  reversing  ribbon  rmi  route53  ruby  rx  s3  sampling  sbe  scala  scalability  scalding  scaling  scheduling  scott-andreas  scraping  scripting  sdk  search  security  semaphores  sequencing  serialization  server  server-side  servers  service-discovery  service-metrics  services  servlets  set  sets  shell  shingling  side-data  signalling  sim-cards  sinatra  sjk  sketching  slab-allocator  slas  slew  slf4j  slides  smartcards  sms  snaptree  snowflake  soa  sockets  softreferences  software  sonatype  sorting  spark  spdy  speed  spinedbuffer  split-brain  spock  spray  spring  spsc  sql  ssd  sse  ssh  ssl  stack-size  startup  static  static-analysis  static-typing  statistics  step  stepping  storage  stormpot  streaming  streams  strong-typing  structs  style  sun  superfish  surveillance  swrve  sync  synchronization  sysadmin  sysdig  tail-latencies  talks  tcp  tech-talks  tehuti  telemetry  templates  templating  testing  tests  thread-priorities  thread-safety  threading  threadpool  threadpools  threads  time  time-series  time-synchronization  time-warp  timeouts  timers  timing  timsort  tips  tls  tools  trace  tracing  transports  trees  tries  troubleshooting  trove  tsd  tuning  twitter  types  uberjars  udp  ultradns  undocumented  unique  unique-ids  unit-tests  unix  urban-airship  urlconnection  usenix  utf  utf8  utilization  via:adriancolyer  via:b3n  via:bos  via:caro  via:cscotta  via:dehora  via:highscalability  via:janl  via:jbaruch  via:kellabyte  via:netflix  via:nitsan  via:normanmaurer  via:peakscale  via:sampullara  via:sbtourist  via:spyced  via:trishagee  via:twitter  video  vim  vm  volatile  voldemort  wait-free  walmart  war-stories  weak  weakhashmap  weakreferences  web  web-services  webapps  webdev  websockets  white-papers  whitelisting  windows  yammer  yourkit  youtube  zip  zipkin  zookeeper 

Copy this bookmark:



description:


tags: