papers   24398

« earlier    

Sincronia: Near-Optimal Network Design for Coflows
We present Sincronia, a near-optimal network design for coflows that can be implemented on top on any transport layer (for flows) that supports priority scheduling. Sincronia achieves this using a key technical result --- we show that given a "right" ordering of coflows, any per-flow rate allocation mechanism achieves average coflow completion time within 4X of the optimal as long as (co)flows are prioritized with respect to the ordering.

Sincronia uses a simple greedy mechanism to periodically order all unfinished coflows; each host sets priorities for its flows using corresponding coflow order and offloads the flow scheduling and rate allocation to the underlying priority-enabled transport layer. We evaluate Sincronia over a real testbed comprising 16-servers and commodity switches, and using simulations across a variety of workloads. Evaluation results suggest that Sincronia not only admits a practical, near-optimal design but also improves upon state-of-the-art network designs for coflows (sometimes by as much as 8X).
networking  papers  datacenter 
7 days ago by mikecb
B4 and After: Managing Hierarchy, Partitioning, and Asymmetry for Availability and Scale in Google’s Software-Defined WAN
Private WANs are increasingly important to the operation of
enterprises, telecoms, and cloud providers. For example, B4,
Google’s private software-defined WAN, is larger and growing
faster than our connectivity to the public Internet. In this
paper, we present the five-year evolution of B4. We describe
the techniques we employed to incrementally move from
offering best-effort content-copy services to carrier-grade
availability, while concurrently scaling B4 to accommodate
100x more traffic. Our key challenge is balancing the tension
introduced by hierarchy required for scalability, the partitioning
required for availability, and the capacity asymmetry
inherent to the construction and operation of any large-scale
network. We discuss our approach to managing this tension:
i) we design a custom hierarchical network topology for both
horizontal and vertical software scaling, ii) we manage inherent
capacity asymmetry in hierarchical topologies using
a novel traffic engineering algorithm without packet encapsulation,
and iii) we re-architect switch forwarding rules
via two-stage matching/hashing to deal with asymmetric
network failures at scale.
scale  sdn  networking  papers 
7 days ago by mikecb
Toward Predicting the Outcome of an A/B Experiment for Search Relevance
A standard approach to estimating online click-based metrics
of a ranking function is to run it in a controlled experiment
on live users. While reliable and popular in practice,
configuring and running an online experiment is cumbersome
and time-intensive. In this work, inspired by recent
successes of offline evaluation techniques for recommender
systems, we study an alternative that uses historical search
log to reliably predict online click-based metrics of a new
ranking function, without actually running it on live users.
To tackle novel challenges encountered in Web search,
variations of the basic techniques are proposed. The first
is to take advantage of diversified behavior of a search engine
over a long period of time to simulate randomized data
collection, so that our approach can be used at very low cost.
The second is to replace exact matching (of recommended
items in previous work) by fuzzy matching (of search result
pages) to increase data efficiency, via a better trade-off
of bias and variance. Extensive experimental results based
on large-scale real search data from a major commercial
search engine in the US market demonstrate our approach
is promising and has potential for wide use in Web search.
IR  ab-testing  counterfactual  papers 
7 days ago by foodbaby
Generalized neural network for nonsmooth nonlinear programming problems - IEEE Journals & Magazine
In 1988 Kennedy and Chua introduced the dynamical canonical nonlinear programming circuit (NPC) to solve in real time nonlinear programming problems where the objective function and the constraints are smooth (twice continuously differentiable) functions. In this paper, a generalized circuit is introduced (G-NPC), which is aimed at solving in real time a much wider class of nonsmooth nonlinear programming problems where the objective function and the constraints are assumed to satisfy only the weak condition of being regular functions. G-NPC, which derives from a natural extension of NPC, has a neural-like architecture and also features the presence of constraint neurons modeled by ideal diodes with infinite slope in the conducting region. By using the Clarke's generalized gradient of the involved functions, G-NPC is shown to obey a gradient system of differential inclusions, and its dynamical behavior and optimization capabilities, both for convex and nonconvex problems, are rigorously analyzed in the framework of nonsmooth analysis and the theory of differential inclusions. In the special important case of linear and quadratic programming problems, salient dynamical features of G-NPC, namely the presence of sliding modes , trajectory convergence in finite time, and the ability to compute the exact optimal solution of the problem being modeled, are uncovered and explained in the developed analytical framework.
papers  to-read  neural-networks  optimization  dynamical-systems  control-theory  stability 
9 days ago by mraginsky

« earlier    

related tags

16th  1960s  300  3d  ab-testing  abstraction  academia  academic  advertising  advice  after  ai  algorithm  algorithms  anthropology  are  articles  arvix  arxiv  at  audio  aug1  back.  big_data  bigdata  biology  bitting  blog  books  bret  business  butterfly  called  capitalism  category-theory  classical  classics  coding  collaboration  compiler  compsci  computers  computerscience  container  containers  control-theory  cooking  counterfactual  critique  crypto  cs  culture  cuts  cyberculture  data  datacenter  deep-learning  design  dev  devops  dijkstra  display  distributed  dl  docker  dsp  dynamical-systems  ebooks  economics  editing  editor  education  elsevier  enemy  engineering  ethics  exploits  extension  extraction  feminism  filetype:pdf  formalverification  frp  functional  functionalprogramming  functionalreactiveprogramming  future  gan  github  google  gradual-typing  graph  graphics  hadoop  handwriting  hardware  hilbert's  history  howto  illiberalleft  image  infrastructure  intepertable  intepretability  ir  journal  k8s  kubernetes  latex  layoffs  list  machinelearning  make  management  mask  memory  mendeley  microsoft  ml  monarch  more  ncp  ncpin  networking  networks  neural-networks  news  nine  nk  nlp  nn  now  nydn  of  on  open-source  openaccess  operating-systems  operations  optimization  os  oss  other  paper  paperpile  pay-wall  pdf  people.”  physics  pizza  policing  policy  power  press  privacy  problem  productivity  profession  programming  psychology  publishing  pushing  pytorch  quality  quantum  rand-corporation  reactiveprogramming  reading-materials  reading  real  recommendation  reference  reproducibility  research  research_methods  scalability  scale  science  sdn  search  security  serverless  software  sourceseparation  spark  speech  stability  staticanalysis  statistics  streaming  surveillance  sync  synthesis  sysadmin  testing  text  than  the  theory  time  tla  to-read  todo  toread  triggerwarnings  tronc  trump  type-safety  type-systems  uber  vae  via-mastodon  victor  will  workflow  writing  “the 

Copy this bookmark: