arsyed + tensorflow   218

TensorSpace.js – Present tensor in space, neural network 3D visualization framework_Github - jishuwen(技术文)
TensorSpace is a neural network 3D visualization framework built by TensorFlow.js, Three.js and Tween.js. TensorSpace provides Keras-like APIs to build deep learning layers, load pre-trained models, and generate a 3D visualization in the browser.
neural-net  tensorflow  visualization 
28 days ago by arsyed
AutoGraph converts Python into TensorFlow graphs – TensorFlow – Medium
"AutoGraph takes in your eager-style Python code and converts it to graph-generating code."
tensorflow  python  graph 
july 2018 by arsyed
Override Tensorflow backward-propagation – Sirena – Medium
"In paper “Clothing Retrieval with Visual Attention Model” they describe attention network that generates Bernoulli series has to be multiplied with another feature map. Unfortunately Bernoulli is not differentiable, hence backward propagation will not flow.

Follow the above points I implement forward propagation that generates Bernoulli series with the given shape and in backprop function I implement custom gradient — it was just multiplication between intermediate layer (another feature map) and coming gradient."
tensorflow  custom-gradient 
june 2018 by arsyed
Google AI Blog: Realtime tSNE Visualizations with TensorFlow.js
"With this implementation, what used to take 15 minutes to calculate (on the MNIST dataset) can now be visualized in real-time and in the web browser. Furthermore this allows real-time visualizations of much larger datasets, a feature that is particularly useful when deep neural output is analyzed. One main limitation of our work is that this technique currently only works for 2D embeddings. However, 2D visualizations are often preferred over 3D ones as they require more interaction to effectively understand cluster results."
tsne  visualization  tensorflow  javascript 
june 2018 by arsyed
[1805.01772] Dynamic Control Flow in Large-Scale Machine Learning
"This paper presents a programming model for distributed machine learning that supports dynamic control flow. We describe the design of the programming model, and its implementation in TensorFlow, a distributed machine learning system. Our approach extends the use of dataflow graphs to represent machine learning models, offering several distinctive features. First, the branches of conditionals and bodies of loops can be partitioned across many machines to run on a set of heterogeneous devices, including CPUs, GPUs, and custom ASICs. Second, programs written in our model support automatic differentiation and distributed gradient computations, which are necessary for training machine learning models that use control flow. Third, our choice of non-strict semantics enables multiple loop iterations to execute in parallel across machines, and to overlap compute and I/O operations. "
programming  machine-learning  tensorflow  control-flow 
may 2018 by arsyed
Evaluated expressions of variables differ sometimes when using the GPU · Issue #2226 · tensorflow/tensorflow
"The GPU kernel for reduce_sum is known to be non-deterministic since it
uses the atomic operations. When the number of elements is large, the
difference could be quite large.

If your model is trained with dropout in it, it tends to be less likely to
be affected by the noise."

"It sounds like your 97.7% vs. 9.8% difference occurs for a model where different CPU systems can also produce 10%, and thus that you may have found an impressively unreliable set of hyperparameters. If using a smaller learning rate helps, it may just mean that the GPU version blows up slightly earlier than the CPU version."

"the problem could be attributed to small differences introduced in the non-deterministic behavior on GPU. Or some numerical instability due to the precision difference between CPU and GPU."
tensorflow  cpu  gpu  sgd  brittle  stability  numeric 
april 2018 by arsyed
How to use Dataset in TensorFlow – Towards Data Science
One shot. It can iterate once through a dataset, you cannot feed any value to it.
Initializable: You can dynamically change calling its initializer operation and passing the new data with feed_dict . It’s basically a bucket that you can fill with stuff.
Reinitializable: It can be initialised from different Dataset. Very useful when you have a training dataset that needs some additional transformation, eg. shuffle, and a testing dataset. It’s like using a tower crane to select different container.
Feedable: It can be used to select with iterator to use. Following the previous example, it’s like a tower crane that selects which tower crane to use to select which container to take. In my opinion is useless.
april 2018 by arsyed
TencentAILab/tf-adaptive-softmax-lstm-lm: The experiment result of LSTM language models on PTB (Penn Treebank) and GBW (Google Billion Word) using AdaptiveSoftmax on TensorFlow.
This repository shows the experiment result of LSTM language models on PTB (Penn Treebank) and GBW (Google One Billion Word) using AdaptiveSoftmax on TensorFlow.

Adaptive Softmax
The adaptive softmax is a faster way to train a softmax classifier over a huge number of classes, and can be used for both training and prediction. For example, it can be used for training a Language Model with a very huge vocabulary, and the trained languaed model can be used in speech recognition, text generation, and machine translation very efficiently.

Tha adaptive softmax has been used in the ASR system developed by Tencent AI Lab, and achieved about 20x speed up than full sotfmax in the second pass for rescoing.
tensorflow  language-model  asr  adaptive-softmax 
march 2018 by arsyed
Tensorflow on GPU worrking inefficiently - Google Groups
Three important things helped a lot:
1. replacing feed_dict with tensor variable
2. using float 32 instead of 64 (which speed everything up anyway, without loosing accuracy)
3. session tracing showed me other things to improve
tensorflow  gpu  preformance 
march 2018 by arsyed
GitHub - tryolabs/luminoth: Deep Learning toolkit for Computer Vision
Luminoth is an open source toolkit for computer vision. Currently, we support object detection and image classification, but we are aiming for much more. It is built in Python, using TensorFlow and Sonnet.
python  libs  computer-vision  deep-learning  object-detection  tensorflow  via:euler 
january 2018 by arsyed
[1711.10604] TensorFlow Distributions
"The TensorFlow Distributions library implements a vision of probability theory adapted to the modern deep-learning paradigm of end-to-end differentiable computation. Building on two basic abstractions, it offers flexible building blocks for probabilistic computation. Distributions provide fast, numerically stable methods for generating samples and computing statistics, e.g., log density. Bijectors provide composable volume-tracking transformations with automatic caching. Together these enable modular construction of high dimensional distributions and transformations not possible with previous libraries (e.g., pixelCNNs, autoregressive flows, and reversible residual networks). They are the workhorse behind deep probabilistic programming systems like Edward and empower fast black-box inference in probabilistic models built on deep-network components. TensorFlow Distributions has proven an important part of the TensorFlow toolkit within Google and in the broader deep learning community."
papers  tensorflow  probabilistic-programming  distribution 
november 2017 by arsyed
Eric Jang on Twitter: "TensorFlow trick: if you want a custom gradient
"TensorFlow trick: if you want a custom gradient
f'(x) + g'(x) for a layer f(x),
just do f(x) + g(x) - tf.stop_gradient(g(x))
and let autodiff do the rest."

"there are somewhat deep numerical reasons why this works (IEEE754). i learned this trick from josh dillon! "

"h(x) = f(x) + g(x) - stop_gradient(g(x)) = f(x) + g(x) - g(x) = f(x) when evaluating h(x), but grad(h(x)) = grad(f(x)) + grad(g(x)) - 0 by definition of stop_gradient"
tensorflow  automatic-differentiation  tricks 
november 2017 by arsyed
How to silence TensorFlow · Martin Thoma
I've added export TF_CPP_MIN_LOG_LEVEL=2 to my ~/.profile. Note that this only suppresses the warnings. The better solution would be to compile Tensorflow with SSE-X support.
tensorflow  warnings 
september 2017 by arsyed
Using Tensorboard Embeddings Visualiser with Numpy Arrays - nlml
"So I decided to throw together a function save_embeddings() that takes the hassle out of this, allowing you to go straight from numpy arrays to Tensorboard-visualised embeddings."
tensorboard  tensorflow  visualization  code 
august 2017 by arsyed
« earlier      
per page:    204080120160

related tags

acoustic-model  adaptive-softmax  adversarial-learning  agents  aggregator  api  art  asr  atari  audio  audioset  auditory  auomatic-differentiation  autoencoder  automatic-differentiation  autoregressive  baidu  basics  bayesian  began  benchmarks  best-practices  binaries  blogs  books  brittle  build  builder  builds  caffe  categorical  checkpoint  classification  cloud  code  collab  comparison  computer-vision  control-flow  convnet  convolution  courses  cpu  cross-entropy  ctc  cuda  custom  custom-gradient  cyclegan  data  dataset  dcgan  debugging  decay  decoding  decorator  deep-learning  deepdream  deepmind  deepspeech  demos  deterministic  dilation  dip  discrete  distcomp  distributed  distribution  dnn  docker  dynamic-batching  e2e  edward  eesen  embedding  encoder-decoder  estimator  examples  few-shot  fizzbuzz  fluent  forward-mode  fst  funny  gaming  gan  gaussian-processes  generative  generative-models  google  gpu  gradient  gradient-descent  gradient-reversal  graph  handwriting  image-completion  imagenet  imperative  inception  input  install  internals  interop  io  ios  ipynb  iterator  javascript  jeff-dean  jit  kaldi  keras  language-model  learning-rate  learning-to-learn  libs  links  linux  llvm  logit  logits  loss-functions  lstm  mac  machine-learning  masking  mixture-density-network  mixture-models  mnist  model-compression  model-zoo  models  moocs  mozilla  mxnet  neural-net  nips-2015  nlp  nlu  nnvm  nondeterminism  normlization  numeric  object-detection  ocr  one-shot  optimizer  papers  parallel  parsing  people  performance  phased-lstm  phone  pipeline  placeholder  prediction  predictron  preformance  probabilistic-programming  probability  profiling  programming  properties  python  pytorch  quantization  questions  queue  queues  redirection  ref  reinforcement-learning  restore  resume  richard-socher  rnn  save  scaling  scope  seq2seq  sequence  session  sgd  shape  shared-variable  shared-weights  shuffle  siamese-net  sigmoid  size  skflow  sklearn  slides  softmax  sonnet  sparse  spectral  speech  stability  stackOverflow  stanford  stdout  structure-learning  style-guide  style-transfer  summarization  swarch  talks  tensor  tensorboard  tensorflow  tensorflow.js  testing  text  tf-slim  tflearn  theano  tied-weights  time-series  timit  tips  torch  tranfer-learning  transfer-learning  transformation  tricks  tsne  tutorials  vae  validation  variable  variable-length  vgg  via:euler  via:nirum  videos  visualization  vq-vae  warning  warnings  wasserstein  wav2letter  wavenet  webapps  wfst  wgan  word2vec  zoo 

Copy this bookmark: