resnet   88

« earlier    

[1806.10909] ResNet with one-neuron hidden layers is a Universal Approximator
We demonstrate that a very deep ResNet with stacked modules with one neuron per hidden layer and ReLU activation functions can uniformly approximate any Lebesgue integrable function in d dimensions, i.e. ℓ1(ℝd). Because of the identity mapping inherent to ResNets, our network has alternating layers of dimension one and d. This stands in sharp contrast to fully connected networks, which are not universal approximators if their width is the input dimension d [Lu et al, 2017; Hanin and Sellke, 2017]. Hence, our result implies an increase in representational power for narrow deep networks by the ResNet architecture.
resnet  neural-net  universal-approximator 
5 days ago by arsyed
Under the hood: Facebook Marketplace powered by artificial intelligence - Facebook Code
"To understand the relationship between buyer activity and product content, the system also incorporates a model for the buyer, created with embeddings using the demographic information from the person’s Facebook profile and keywords from searches within Marketplace." - you know what this means, and it should not come as a surprise
facebook  ai  machinelearning  deeplearning  cnn  via:gnat  deeptext  lumos  resnet  marketplace  entityextraction 
9 weeks ago by danhon
[1605.07146] Wide Residual Networks
Deep residual networks were shown to be able to scale up to thousands of layers and still have improving performance. However, each fraction of a percent of improved accuracy costs nearly doubling the number of layers, and so training very deep residual networks has a problem of diminishing feature reuse, which makes these networks very slow to train. To tackle these problems, in this paper we conduct a detailed experimental study on the architecture of ResNet blocks, based on which we propose a novel architecture where we decrease depth and increase width of residual networks. We call the resulting network structures wide residual networks (WRNs) and show that these are far superior over their commonly used thin and very deep counterparts. For example, we demonstrate that even a simple 16-layer-deep wide residual network outperforms in accuracy and efficiency all previous deep residual networks, including thousand-layer-deep networks, achieving new state-of-the-art results on CIFAR, SVHN, COCO, and significant improvements on ImageNet. Our code and models are available at this https URL
resnet  wide-resnet  neural-net 
may 2018 by arsyed
[1708.07120] Super-Convergence: Very Fast Training of Residual Networks Using Large Learning Rates
In this paper, we show a phenomenon, which we named "super-convergence", where residual networks can be trained using an order of magnitude fewer iterations than is used with standard training methods. The existence of super-convergence is relevant to understanding why deep networks generalize well. One of the key elements of super-convergence is training with cyclical learning rates and a large maximum learning rate. Furthermore, we present evidence that training with large learning rates improves performance by regularizing the network. In addition, we show that super-convergence provides a greater boost in performance relative to standard training when the amount of labeled training data is limited. We also derive a simplification of the Hessian Free optimization method to compute an estimate of the optimal learning rate. The architectures and code to replicate the figures in this paper are available at github.com/lnsmith54/super-convergence.
neural-net  sgd  resnet  training  performance  leslie-smith  learning-rate 
april 2018 by arsyed
Decoding the ResNet architecture // teleported.in
A blog where I share my intuitions about artificial intelligence, machine learning, deep learning.
resnet  shortcut-connections  network-architecture  convolutional-neural-networks  cnns  deep-learning  fast.ai  from pocket
february 2018 by rishaanp
Yet Another ResNet Tutorial (or not) – Apil Tamang – Medium
The purpose of this article is to expose the most fundamental concept driving the design and success of ResNet architectures. Many blogs and articles go on and on describing how this architecture is…
ResNet  neural-networks  network-architecture  from pocket
february 2018 by rishaanp

« earlier    

related tags

ai  alexnet  ami  analysis  approximator  arch  architect  architecture  asr  batts  boltzmann  boosting  bpi  caffe  certification  cis  cnn  cnns  computer-vision  computervision  control  convnet  convolution  convolutional-nets  convolutional-neural-networks  database  deep-learning  deep  deeplearning  deeptext  dense-net  densenet  differential  dl  dnn  dog-breeds  efficiency  energy  energystar  ensemble  entityextraction  example  facebook  fast.ai  fiberglass  fusion  github  gradient  green  heat-map  heatmap  hers  hersh2o  highered  highway-networks  highway  hopfield  icml  identity-mapping  ilsvrc  imagenet  imagerecognition  insulation  internet  issues  kaggle  keras  kohonen  launchd  learning-rate  learning  lennar  leslie-smith  list  lstm  lua  lumos  machine-learning  machinecomprehension  machinelearning  marketplace  markov  microsoft  ml  nac  neon  net  netsquid  network-architecture  network  networkaccesscontrol  networks  neural-net  neural-networks  neural-turing-machine  nlp  nn-architecture  nn  one  open  paper  papers  patterns  people  performance  printing  purdue  pytorch  rater  rbm  readit  remrate  research  residual-learning  residual-networks  residual  review  rnn  sap  security  self-organizing-map  sgd  shortcut-connections  shot  snort  software  source  squid  svm  talks  tamu.edu  tamu  tech  tensorflow  theory  tinyimagenet  torch  training  transfer-learning  tutorial  tutorials  ucsd  universal-approximator  universal  usage  uw  vgg  vggcam  virus  viruses  visualization  vpn  waterloo  wide-resnet  zoo 

Copy this bookmark:



description:


tags: