jm + debugging   32

How-to Debug a Running Docker Container from a Separate Container
arguably this shouldn't be required -- building containers without /bin/sh, strace, gdb etc. is just silly
strace  docker  ops  debugging  containers 
4 weeks ago by jm
Debugging Java Native Memory Leaks (
Using jemalloc to instrument the contents of the native heap and record stack traces of each chunk's allocators, so that leakers can be quickly identified (GZIPInputStream in this case).

See also , .
debugging  memory  jvm  java  leaks  memory-leaks  leak-checking  jemalloc  malloc  native  heap  off-heap  gzipinputstream 
10 weeks ago by jm
The hidden cost of QUIC and TOU
The recent movement to get all traffic encrypted has of course been great for the Internet. But the use of encryption in these protocols is different than in TLS. In TLS, the goal was to ensure the privacy and integrity of the payload. It's almost axiomatic that third parties should not be able to read or modify the web page you're loading over HTTPS. QUIC and TOU go further. They encrypt the control information, not just the payload. This provides no meaningful privacy or security benefits.

Instead the apparent goal is to break the back of middleboxes [0]. The idea is that TCP can't evolve due to middleboxes and is pretty much fully ossified. They interfere with connections in all kinds of ways, like stripping away unknown TCP options or dropping packets with unknown TCP options or with specific rare TCP flags set. The possibilities for breakage are endless, and any protocol extensions have to jump through a lot of hoops to try to minimize the damage.
quic  tou  protocols  http  tls  security  internet  crypto  privacy  firewalls  debugging  operability 
december 2016 by jm
The Nyquist theorem and limitations of sampling profilers today, with glimpses of tracing tools from the future
Awesome post from Dan Luu with data from Google:
The cause [of some mystery widespread 250ms hangs] was kernel throttling of the CPU for processes that went beyond their usage quota. To enforce the quota, the kernel puts all of the relevant threads to sleep until the next multiple of a quarter second. When the quarter-second hand of the clock rolls around, it wakes up all the threads, and if those threads are still using too much CPU, the threads get put back to sleep for another quarter second. The phase change out of this mode happens when, by happenstance, there aren’t too many requests in a quarter second interval and the kernel stops throttling the threads. After finding the cause, an engineer found that this was happening on 25% of disk servers at Google, for an average of half an hour a day, with periods of high latency as long as 23 hours. This had been happening for three years. Dick Sites says that fixing this bug paid for his salary for a decade. This is another bug where traditional sampling profilers would have had a hard time. The key insight was that the slowdowns were correlated and machine wide, which isn’t something you can see in a profile.
debugging  performance  visualization  instrumentation  metrics  dan-luu  latency  google  dick-sites  linux  scheduler  throttling  kernel  hangs 
february 2016 by jm
How to inspect SSL/TLS traffic with Wireshark 2
turns out it's easy enough -- Mozilla standardised a debugging SSL session-key logging file format which Wireshark and Chrome support
chrome  ssl  browser  firefox  wireshark  debugging  tls 
december 2015 by jm
a tool which simplifies tracing and testing of Java programs. Byteman allows you to insert extra Java code into your application, either as it is loaded during JVM startup or even after it has already started running. The injected code is allowed to access any of your data and call any application methods, including where they are private. You can inject code almost anywhere you want and there is no need to prepare the original source code in advance nor do you have to recompile, repackage or redeploy your application. In fact you can remove injected code and reinstall different code while the application continues to execute. The simplest use of Byteman is to install code which traces what your application is doing. This can be used for monitoring or debugging live deployments as well as for instrumenting code under test so that you can be sure it has operated correctly. By injecting code at very specific locations you can avoid the overheads which often arise when you switch on debug or product trace. Also, you decide what to trace when you run your application rather than when you write it so you don't need 100% hindsight to be able to obtain the information you need.
tracing  java  byteman  injection  jvm  ops  debugging  testing 
september 2015 by jm
Benchmarking GitHub Enterprise - GitHub Engineering
Walkthrough of debugging connection timeouts in a load test. Nice graphs (using matplotlib)
github  listen-backlog  tcp  debugging  timeouts  load-testing  benchmarking  testing  ops  linux 
july 2015 by jm
Apache HTrace
a Zipkin-compatible distributed-system tracing framework in Java, in the Apache Incubator
zipkin  tracing  trace  apache  incubator  java  debugging 
may 2015 by jm
Combining static model checking with dynamic enforcement using the Statecall Policy Language
This looks quite nice -- a model-checker "for regular programmers". Example model for ping(1):

<pre>01 automaton ping (int max_count, int count, bool can_timeout) {
02 Initialize;
03 during {
04 count = 0;
05 do {
06 Transmit_Ping;
07 either {
08 Receive_Ping;
09 } or (can_timeout) {
10 Timeout_Ping;
11 };
12 count = count + 1;
13 } until (count &gt;= max_count);
14 } handle {
16 Print_Summary;
17 };</pre>
ping  model-checking  models  formal-methods  verification  static  dynamic  coding  debugging  testing  distcomp  papers 
march 2015 by jm
OpenJDK: jol
'JOL (Java Object Layout) is the tiny toolbox to analyze object layout schemes in JVMs. These tools are using Unsafe, JVMTI, and Serviceability Agent (SA) heavily to decoder the actual object layout, footprint, and references. This makes JOL much more accurate than other tools relying on heap dumps, specification assumptions, etc.'

Recommended by Nitsan Wakart, looks pretty useful for JVM devs
java  jvm  tools  scala  memory  estimation  ram  object-layout  debugging  via:nitsan 
february 2015 by jm
8 gdb tricks you should know (Ksplice Blog)
These are very good -- bookmarking for the next time I'm using gdb, probably about 3 years from now
c  debugging  gdb  c++  tips  coding 
january 2015 by jm
Solving the Mystery of Link Imbalance: A Metastable Failure State at Scale | Engineering Blog | Facebook Code
Excellent real-world war story from Facebook -- a long-running mystery bug was eventually revealed to be a combination of edge-case behaviours across all the layers of the networking stack, from L2 link aggregation at the agg-router level, up to the L7 behaviour of the MySQL client connection pool.
Facebook collocates many of a user’s nodes and edges in the social graph. That means that when somebody logs in after a while and their data isn’t in the cache, we might suddenly perform 50 or 100 database queries to a single database to load their data. This starts a race among those queries. The queries that go over a congested link will lose the race reliably, even if only by a few milliseconds. That loss makes them the most recently used when they are put back in the pool. The effect is that during a query burst we stack the deck against ourselves, putting all of the congested connections at the top of the deck.
architecture  debugging  devops  facebook  layer-7  mysql  connection-pooling  aggregation  networking  tcp-stack 
november 2014 by jm
Troubleshooting Production JVMs with jcmd
remotely trigger GCs, finalization, heap dumps etc. Handy
jvm  jcmd  debugging  ops  java  gc  heap  troubleshooting 
september 2014 by jm
Systemd: Harbinger of the Linux apocalypse
While there are many defensible aspects of Systemd, other aspects boggle the mind. Not the least of these was that, as of a few months ago, trying to debug the kernel from the boot line would cause the system to crash. This was because of Systemd's voracious logging and the fact that Systemd responds to the "debug" flag on the kernel boot line -- a flag meant for the kernel, not anything else. That, straight up, is a bug.

However, the Systemd developers didn't see it that way and actively fought with those experiencing the problem. Add the fact that one of the Systemd developers was banned by Linus Torvalds for poor attitude and bad design and another was responsible for causing significant issues with Linux audio support, but blamed the problem on everything else but his software, and you have a bad situation on your hands.

There's no shortage of egos in the open source development world. There's no shortage of new ideas and veteran developers and administrators pooh-poohing something new simply because it's new. But there are also 45 years of history behind Unix and extremely good reasons it's still flourishing. Tools designed like Systemd do not fit the Linux mold, to their own detriment. Systemd's design has more in common with Windows than with Unix -- down to the binary logging.

The link re systemd consuming the "debug" kernel boot arg is a canonical example of inflexible coders refusing to fix their own bugs. (via Jason Dixon)
systemd  linux  red-hat  egos  linus-torvalds  unix  init  booting  debugging  logging  design  software  via:obfuscurity 
august 2014 by jm
a single application IP packet sniffer that captures all TCP and UDP packets of a single Linux process. It consists of the following elements:

* ptrace monitor - tracks bind(), connect() and sendto() syscalls and extracts local port numbers that the traced application uses;
* pcap sniffer - using information from the previous module, it captures IP packets on an AF_PACKET socket (with an appropriate BPF filter attached);
* garbage collector - periodically reads /proc/net/{tcp,udp} files in order to detect the sockets that the application no longer uses.

As the output, tracedump generates a PCAP file with SLL-encapsulated IP packets - readable by eg. Wireshark. This file can be later used for detailed analysis of the networking operations made by the application. For instance, it might be useful for IP traffic classification systems.
debugging  networking  linux  strace  ptrace  tracedump  tracing  tcp  udp  sniffer  ip  tcpdump 
may 2014 by jm
open source, system-level exploration: capture system state and activity from a running Linux instance, then save, filter and analyze.
Think of it as strace + tcpdump + lsof + awesome sauce.
With a little Lua cherry on top.

This sounds excellent. Linux-based, GPLv2.
debugging  tools  linux  ops  tracing  strace  open-source  sysdig  cli  tcpdump  lsof 
april 2014 by jm
A cool-looking new debugging tool for C/C++ from Mozilla.
Many, many people have noticed that if we had a way to reliably record program execution and replay it later, with the ability to debug the replay, we could largely tame the nondeterminism problem. This would also allow us to deliberately introduce nondeterminism so tests can explore more of the possible execution space, without impacting debuggability. Many record and replay systems have been built in pursuit of this vision. (I built one myself.) For various reasons these systems have not seen wide adoption. So, a few years ago we at Mozilla started a project to create a new record-and-replay tool that would overcome the obstacles blocking adoption. We call this tool rr.

Low runtime overhead; easy deployability; targeted at 32-bit (?!) Linux; OSS. (via Bryan O'Sullivan)
via:bos  mozilla  debugging  coding  firefox  rr  record  replay  gdb  c++  linux 
march 2014 by jm
IntelliJ IDEA 13.1 will support Chronon Debugger
This, IMO, would be a really good reason to upgrade to the payware version of IDEA - Chronon looks cool.
Chronon is a new revolutionary tool keeping track of running Java programs and recording their execution process for later analysis, which can be helpful when you need to thoroughly retrace your steps when dealing with complicated bugs.
chronon  debugging  java  intellij  idea  ides  coding  time-warp  time 
march 2014 by jm
A sampling profiler for your daily browsing - Google Groups
via Ilya Grigorik: Chrome Canary now has a built-in, always-on, zero-overhead code profiler. I want this in my server-side JVMs!
chrome  tracing  debugging  performance  profiling  google  sampling-profiler  javascript  blink  v8 
january 2014 by jm
Make The Web Fast - The HAR Show: Capturing and Analyzing performance data with HTTP Archive format — Google Developers
Wow, I didn't know about this. Great idea.
Need a flexible format to record, export, and analyze network performance data? Well, that's exactly what the HTTP Archive format (HAR) is designed to do! Even better, did you know that Chrome DevTools supports it? In this episode we'll take a deep dive into the format (as you'll see, its very simple), and explore the many different ways it can help you capture and analyze your sites performance. Join Ilya Grigorik and Peter Lubbers to find out how to capture HAR network traces in Chrome, visualize the data via an online tool, share the reports with your clients and coworkers, automate the logging and capture of HAR data for your build scripts, and even adapt it to server-side analysis use cases
capturing  logging  performance  http  debugging  trace  capture  har  archives  protocols  recording 
december 2013 by jm
dumping a JVM heap using gdb
now this is a neat trick -- having been stuck having to flip to spares and do other antics while a long-running heap dump took place, this is a winner.
Dumping a JVM’s heap is an extremely useful tool for debugging problems with a J2EE application. Unfortunately, when a JVM explodes, using the standard jmap tool can take an inordinate amount of time to execute for lots of different reasons. This leads to extended downtime when a heap dump is attempted and even then, jmap regularly fails.
This blog post is intended to outline an alternate method using [gdb] to achieve a heap dump that only requires mere seconds of additional downtime allowing the slow jmap process to happen once the application is back in service.
heap-dump  gdb  heap  jvm  java  via:peakscale  gcore  core  core-dump  debugging 
march 2013 by jm
Peek and poke in the age of Linux
Neat demo of using ptrace to inject into a running process, just like the good old days ;)
Some time ago I ran into a production issue where the init process (upstart) stopped behaving properly. Specifically, instead of spawning new processes, it deadlocked in a transitional state. [...] What’s worse, upstart doesn’t allow forcing a state transition and trying to manually create and send DBus events didn’t help either. That meant the sane options we were left with were:
restart the host (not desirable at all in that scenario);
start the process manually and hope auto-respawn will not be needed.
Of course there are also some insane options. Why not cheat like in the old times and just PEEK and POKE the process in the right places? The solution used at the time involved a very ugly script driving gdb which probably summoned satan in some edge cases. But edge cases were not hit and majority of hosts recovered without issues.
debugging  memory  linux  upstart  peek  poke  ptrace  gdb  processes  hacks 
march 2013 by jm
Chronon DVR for Java
"record entire execution of your Java app; play it back on any machine". Other features: time-travelling debugger -- step backwards, jump to any point in execution, designed for long running programs; post-execution logging -- add log statements after the program has run, and see what it would have logged. Looks extremely nifty, but I wonder how big those recording files get...
debugging  via:peakscale  eclipse  chronon  dvr  java  coding  logging  jvm 
may 2012 by jm
DTrace and Erlang
from Basho, via istvan. DTrace is becoming more compelling as a deep instrumentation/monitoring API -- I didn't realise disabled DTrace probes were virtually 0-overhead (a "2 NOOP instruction placeholder", apparently), that's nifty. Wonder if they've fixed the licensing mess, though
dtrace  monitoring  instrumentation  debugging  tracing  unix  erlang  via:istvan 
november 2011 by jm
peak6/scala-ssh-shell - GitHub
'Backdoor that gives you a scala shell over ssh on your jvm. The shell is not sandboxed, anyone access the shell can touch anything in the jvm and do anything the jvm can do including modifying and deleting files, etc.' nifty!
scala  ssh  repl  interactive  debugging  coding  jvm  java 
october 2011 by jm
wraps strace(1) to summarise and aggregate I/O ops performed by a Linux process. looks pretty nifty (via Jeremy Zawodny)
via:jzawodny  io  strace  linux  monitoring  debugging  performance  profiling  sysadmin  ioprofile  unix  tools  from delicious
october 2010 by jm
JSON Format
'your online JSON Formatter'. useful. via JKeyes
via:jkeyes  json  formatting  tools  useful  format  debugging  from delicious
november 2009 by jm
Hijack: Get A Live IRB Prompt For Any Existing Ruby Process
injects via gdb. pretty cool, if it works; one comment notes that they couldn't use it on a Rails app
gdb  hijack  ruby  debugging  irb  live  coding  rubygems  debugger 
august 2009 by jm

related tags

aggregation  apache  architecture  archive  archives  benchmarking  blink  booting  browser  byteman  c  c++  canary-requests  capture  captures  capturing  chrome  chronon  cli  cloudflare  coding  connection-pooling  containers  conversion  core  core-dump  crypto  dan-luu  debugger  debugging  design  devops  dick-sites  distcomp  distributed-systems  docker  dtrace  dvr  dynamic  eclipse  egos  erlang  estimation  facebook  firefox  firewalls  formal-methods  format  formatting  gc  gcore  gdb  github  google  gzipinputstream  hacks  hangs  har  heap  heap-dump  hijack  http  http2  idea  ides  incubator  infrastructure  init  injection  instrumentation  intellij  interactive  internet  io  ioprofile  ip  irb  java  javascript  jcmd  jemalloc  jmeter  jmx  json  jvm  kernel  latency  layer-7  leak-checking  leaks  linus-torvalds  linux  listen-backlog  live  load-testing  logging  lsof  malloc  memory  memory-leaks  metrics  model-checking  models  monitoring  mozilla  mysql  native  networking  object-layout  off-heap  open-source  operability  ops  papers  peek  performance  ping  poke  privacy  processes  production  profiling  protocols  ptrace  quic  ram  record  recording  red-hat  repl  replay  rr  ruby  rubygems  sampling-profiler  scala  scheduler  security  sniffer  software  spdy  ssh  ssl  stack  static  strace  sysadmin  sysdig  systemd  tcp  tcp-stack  tcpdump  testing  throttling  time  time-warp  timeouts  tips  tls  tools  tou  trace  tracedump  tracer-requests  tracing  troubleshooting  twitter  udp  unix  upstart  useful  v8  verification  via:bos  via:istvan  via:jkeyes  via:jzawodny  via:nitsan  via:obfuscurity  via:peakscale  visualization  wireshark  zipkin 

Copy this bookmark: