scaling   16143

« earlier    

Locking, Little's Law, and the USL
Excellent explanatory mailing list post by Martin Thompson to the mechanical-sympathy group, discussing Little's Law vs the USL:
Little's law can be used to describe a system in steady state from a queuing perspective, i.e. arrival and leaving rates are balanced. In this case it is a crude way of modelling a system with a contention percentage of 100% under Amdahl's law, in that throughput is one over latency.

However this is an inaccurate way to model a system with locks. Amdahl's law does not account for coherence costs. For example, if you wrote a microbenchmark with a single thread to measure the lock cost then it is much lower than in a multi-threaded environment where cache coherence, other OS costs such as scheduling, and lock implementations need to be considered.

Universal Scalability Law (USL) accounts for both the contention and the coherence costs.
http://www.perfdynamics.com/Manifesto/USLscalability.html

When modelling locks it is necessary to consider how contention and coherence costs vary given how they can be implemented. Consider in Java how we have biased locking, thin locks, fat locks, inflation, and revoking biases which can cause safe points that bring all threads in the JVM to a stop with a significant coherence component.
usl  scaling  scalability  performance  locking  locks  java  jvm  amdahls-law  littles-law  system-dynamics  modelling  systems  caching  threads  schedulers  contention 
4 days ago by jm
[1709.02293] Analyzing huge pathology images with open source software
Backgr: Digital pathology images are increasingly used both for diagnosis and research, because slide scanners are nowadays broadly available and because the quantitative study of these images yields new insights in systems biology. However, such virtual slides build up a technical challenge since the images occupy often several gigabytes and cannot be fully opened in a computer's memory. Moreover, there is no standard format. Therefore, most common open source tools such as ImageJ fail at treating them, and the others require expensive hardware while still being prohibitively slow.
Res: We have developed several cross-platform open source software tools to overcome these limitations. The NDPITools provide a way to transform microscopy images initially in the loosely supported NDPI format into one or several standard TIFF files, and to create mosaics (division of huge images into small ones, with or without overlap) in various TIFF and JPEG formats. They can be driven through ImageJ plugins. The LargeTIFFTools achieve similar functionality for huge TIFF images which do not fit into RAM. We test the performance of these tools on several digital slides and compare them, when applicable, to standard software. A statistical study of the cells in a tissue sample from an oligodendroglioma was performed on an average laptop computer to demonstrate the efficiency of the tools.
Concl: Our open source software enables dealing with huge images with standard software on average computers. Our tools are cross-platform, independent of proprietary libraries, and very modular, allowing them to be used in other open source projects. They have excellent performance in terms of execution speed and RAM requirements. They open promising perspectives both to the clinician who wants to study a single slide and to the research team or data centre who do image analysis of many slides on a computer cluster.
image-processing  software-development-is-not-programming  rather-interesting  bioinformatics  medinformatics  user-experience  scaling  to-write-about 
5 days ago by Vaguery
Lessons Learned Scaling Airbnb 100X – Jonathan Golden – Medium
Jonathan Golden on how they scaled Airbnb during their first years, including some analysis on when/how/if to automate for manual work - "if you don't solve problems manually first, you could lose crucial insights into what's causing a problem and why".
airbnb  scale  scaling  automation  manual  edgecase  flow 
7 days ago by arrelid
Tripling an Engineering Team in Six Months - Part Three: Setting up for Success — Zac Sky
Tripling an Engineering Team in Six Months - Part Three: Setting up for Success This was the first cut. Broadly speaking the blue tasks are technology centric and the green tasks are people centric. Having this defined was useful for everyone to understand their role but it was absolutely crucial to have before we hired newly created roles. via Pocket
ifttt  pocket  article  engineering  hiring  jobs  scaling  startup 
12 days ago by jeremyday
Tripling an Engineering Team in Six Months - Part Two: Planning to Scale — Zac Sky
Tripling an Engineering Team in Six Months - Part Two: Planning to Scale Our teams were formed roughly into two parallel squads, one for each of the key subject matter domains we were tackling. The squads were fluid. via Pocket
ifttt  pocket  article  engineering  hiring  jobs  scaling  startup 
12 days ago by jeremyday

« earlier    

related tags

10  2017-09-03  3d  4k  5  advanced  advice  agile  airbnb  amazon  amdahls-law  app  architecture  article  asg  automatic  automation  autoscale  autoscaling  avoid  aws  balancing  baselines  billing  bioinformatics  bitcoin  blitzscaling  blockchain  blog  blogpost  branding  business  caching  case  challenges  cities  cloud  cluster  coding  community  comparison  complex  complexity  compliance  configuration  connect  contention  cost-saving  cropping  css  customers  database  databases  db  deployment  design  designsystem  devops  display  distributed  diy  dns  docker  dropbox  ec2  ecommerce  ecosystem  edgecase  elasticsearch  elixir  elm  engagement:  engineering  enterprise  entrepreneurship  erlang  ethereum  example  fail  feature  ffmpeg  firstround  flink  flow  gaming  general.electric  global  golang  google  graphics  guideline  ha  haproxy  hardware  heroku  hi-dpi  hickory  hidpi  high-availability  high  hiring  hn  horizontal.scaling  horizontal_scale  howto  humanize  hype  ifttt  image-processing  images  implementation  infrastructure  interesting  internet.of.shit  java  jobs  jonathangolden  jvm  kafka  kills  knowhow  kubernetes  large  largile  latency  laws  lb  leader  leadership  lessons  littles-law  loadbalancing  locking  locks  log  logging  loggly  machinelearning  management  manual  marketing  medinformatics  meh  memory  merchant.account  methodology  metrics  microservices  mobile  modelling  monitoring  more  multi-instance  multi-tenancy  multiple  mysql  nginx  nodejs  nodes  numpy  of  operations  ops  optimization  orgculture  paas  paulgraham  performance  physics  pocket  postgres  postgresql  premature  presentation  pricing  productmanagement  products  programming  prometheus  puma  rails  rather-interesting  rendering  reponsive  resizing  resolution  responsive  route  ruby  rubypro  rwd  s3  safe  scala  scalability  scalable  scale  scalingup  schedulers  science  scio  screen  screencast  scrum  security  segwit  services  shard  sharding  simonw  site  sites  slides  social  socket.io  software-development-is-not-programming  software  solution  spark  spot-fleet  spot-instances  sql  ssltermination  startup  startups  stateful-sets  storage  stories  strategies  svg  swarm  sysadmin  system-dynamics  systems  technology  tensorflow  theory  threads  tips  to-write-about  to  todo  tutorial  twitter  user-experience  usl  ux  video  virtualization  watched  webdev  websockets  windows  with  wordpress 

Copy this bookmark:



description:


tags: