arsyed + tensorflow   104

AutoGraph converts Python into TensorFlow graphs – TensorFlow – Medium
"AutoGraph takes in your eager-style Python code and converts it to graph-generating code."
tensorflow  python  graph 
july 2018 by arsyed
Override Tensorflow backward-propagation – Sirena – Medium
"In paper “Clothing Retrieval with Visual Attention Model” they describe attention network that generates Bernoulli series has to be multiplied with another feature map. Unfortunately Bernoulli is not differentiable, hence backward propagation will not flow.

Follow the above points I implement forward propagation that generates Bernoulli series with the given shape and in backprop function I implement custom gradient — it was just multiplication between intermediate layer (another feature map) and coming gradient."
tensorflow  custom-gradient 
june 2018 by arsyed
Google AI Blog: Realtime tSNE Visualizations with TensorFlow.js
"With this implementation, what used to take 15 minutes to calculate (on the MNIST dataset) can now be visualized in real-time and in the web browser. Furthermore this allows real-time visualizations of much larger datasets, a feature that is particularly useful when deep neural output is analyzed. One main limitation of our work is that this technique currently only works for 2D embeddings. However, 2D visualizations are often preferred over 3D ones as they require more interaction to effectively understand cluster results."
tsne  visualization  tensorflow  javascript 
june 2018 by arsyed
[1805.01772] Dynamic Control Flow in Large-Scale Machine Learning
"This paper presents a programming model for distributed machine learning that supports dynamic control flow. We describe the design of the programming model, and its implementation in TensorFlow, a distributed machine learning system. Our approach extends the use of dataflow graphs to represent machine learning models, offering several distinctive features. First, the branches of conditionals and bodies of loops can be partitioned across many machines to run on a set of heterogeneous devices, including CPUs, GPUs, and custom ASICs. Second, programs written in our model support automatic differentiation and distributed gradient computations, which are necessary for training machine learning models that use control flow. Third, our choice of non-strict semantics enables multiple loop iterations to execute in parallel across machines, and to overlap compute and I/O operations. "
programming  machine-learning  tensorflow  control-flow 
may 2018 by arsyed
Evaluated expressions of variables differ sometimes when using the GPU · Issue #2226 · tensorflow/tensorflow
"The GPU kernel for reduce_sum is known to be non-deterministic since it
uses the atomic operations. When the number of elements is large, the
difference could be quite large.

If your model is trained with dropout in it, it tends to be less likely to
be affected by the noise."

"It sounds like your 97.7% vs. 9.8% difference occurs for a model where different CPU systems can also produce 10%, and thus that you may have found an impressively unreliable set of hyperparameters. If using a smaller learning rate helps, it may just mean that the GPU version blows up slightly earlier than the CPU version."

"the problem could be attributed to small differences introduced in the non-deterministic behavior on GPU. Or some numerical instability due to the precision difference between CPU and GPU."
tensorflow  cpu  gpu  sgd  brittle  stability  numeric 
april 2018 by arsyed
How to use Dataset in TensorFlow – Towards Data Science
One shot. It can iterate once through a dataset, you cannot feed any value to it.
Initializable: You can dynamically change calling its initializer operation and passing the new data with feed_dict . It’s basically a bucket that you can fill with stuff.
Reinitializable: It can be initialised from different Dataset. Very useful when you have a training dataset that needs some additional transformation, eg. shuffle, and a testing dataset. It’s like using a tower crane to select different container.
Feedable: It can be used to select with iterator to use. Following the previous example, it’s like a tower crane that selects which tower crane to use to select which container to take. In my opinion is useless.
april 2018 by arsyed
TencentAILab/tf-adaptive-softmax-lstm-lm: The experiment result of LSTM language models on PTB (Penn Treebank) and GBW (Google Billion Word) using AdaptiveSoftmax on TensorFlow.
This repository shows the experiment result of LSTM language models on PTB (Penn Treebank) and GBW (Google One Billion Word) using AdaptiveSoftmax on TensorFlow.

Adaptive Softmax
The adaptive softmax is a faster way to train a softmax classifier over a huge number of classes, and can be used for both training and prediction. For example, it can be used for training a Language Model with a very huge vocabulary, and the trained languaed model can be used in speech recognition, text generation, and machine translation very efficiently.

Tha adaptive softmax has been used in the ASR system developed by Tencent AI Lab, and achieved about 20x speed up than full sotfmax in the second pass for rescoing.
tensorflow  language-model  asr  adaptive-softmax 
march 2018 by arsyed
GitHub - tryolabs/luminoth: Deep Learning toolkit for Computer Vision
Luminoth is an open source toolkit for computer vision. Currently, we support object detection and image classification, but we are aiming for much more. It is built in Python, using TensorFlow and Sonnet.
python  libs  computer-vision  deep-learning  object-detection  tensorflow  via:euler 
january 2018 by arsyed
[1711.10604] TensorFlow Distributions
"The TensorFlow Distributions library implements a vision of probability theory adapted to the modern deep-learning paradigm of end-to-end differentiable computation. Building on two basic abstractions, it offers flexible building blocks for probabilistic computation. Distributions provide fast, numerically stable methods for generating samples and computing statistics, e.g., log density. Bijectors provide composable volume-tracking transformations with automatic caching. Together these enable modular construction of high dimensional distributions and transformations not possible with previous libraries (e.g., pixelCNNs, autoregressive flows, and reversible residual networks). They are the workhorse behind deep probabilistic programming systems like Edward and empower fast black-box inference in probabilistic models built on deep-network components. TensorFlow Distributions has proven an important part of the TensorFlow toolkit within Google and in the broader deep learning community."
papers  tensorflow  probabilistic-programming  distribution 
november 2017 by arsyed
Using Tensorboard Embeddings Visualiser with Numpy Arrays - nlml
"So I decided to throw together a function save_embeddings() that takes the hassle out of this, allowing you to go straight from numpy arrays to Tensorboard-visualised embeddings."
tensorboard  tensorflow  visualization  code 
august 2017 by arsyed
tensorflow/tensor2tensor: A library for generalized sequence to sequence models
"T2T is a modular and extensible library and binaries for supervised learning with TensorFlow and with support for sequence tasks. It is actively used and maintained by researchers and engineers within the Google Brain team."
python  libs  tensorflow  sequence  seq2seq 
june 2017 by arsyed
tensorflow/ at master · tensorflow/tensorflow
"This test constructs a graph with a relu op that's not used by the normal inference path, and then tests that the strip_unused transform removes it as expected."
tensorflow  code  graph  transformation 
april 2017 by arsyed
tensorflow/fold: Deep learning with dynamic computation graphs in TensorFlow
"TensorFlow Fold is a library for creating TensorFlow models that consume structured data, where the structure of the computation graph depends on the structure of the input data.
Fold implements dynamic batching. Batches of arbitrarily shaped computation graphs are transformed to produce a static computation graph. This graph has the same structure regardless of what input it receives, and can be executed efficiently by TensorFlow."
tensorflow  libs  structure-learning  dynamic-batching 
february 2017 by arsyed
[1701.03757] Deep Probabilistic Programming
"We propose Edward, a Turing-complete probabilistic programming language. Edward builds on two compositional representations---random variables and inference. By treating inference as a first class citizen, on a par with modeling, we show that probabilistic programming can be as flexible and computationally efficient as traditional deep learning. For flexibility, Edward makes it easy to fit the same model using a variety of composable inference methods, ranging from point estimation, to variational inference, to MCMC. In addition, Edward can reuse the modeling representation as part of inference, facilitating the design of rich variational models and generative adversarial networks. For efficiency, Edward is integrated into TensorFlow, providing significant speedups over existing probabilistic systems. For example, on a benchmark logistic regression task, Edward is at least 35x faster than Stan and PyMC3."
papers  edward  probabilistic-programming  tensorflow  deep-learning 
january 2017 by arsyed
igormq/ctc_tensorflow_example: CTC + Tensorflow Example for ASR
"A toy example showing how to get CTC cost function working with Tensorflow for automatic speech recognition."
python  tensorflow  ctc  asr 
january 2017 by arsyed
"Tensorflow implementation of the recently introduced Phased LSTM by Neil et. al @ NIPS 2016"
code  tensorflow  lstm  phased-lstm 
december 2016 by arsyed
Welcome to TensorLayer — TensorLayer 1.2.2 documentation
"TensorLayer is a Deep Learning (DL) and Reinforcement Learning (RL) library extended from Google TensorFlow. It provides popular DL and RL modules that can be easily customized and assembled for tackling real-world machine learning problems."
python  libs  tensorflow  deep-learning  reinforcement-learning 
september 2016 by arsyed
« earlier      
per page:    204080120160

related tags

adaptive-softmax  agents  api  asr  atari  auditory  auomatic-differentiation  autoencoder  automatic-differentiation  autoregressive  baidu  bayesian  began  benchmarks  best-practices  binaries  blogs  brittle  builder  builds  categorical  cloud  code  collab  comparison  computer-vision  control-flow  convnet  convolution  courses  cpu  ctc  custom  custom-gradient  cyclegan  dataset  dcgan  debugging  decorator  deep-learning  deepmind  deepspeech  demos  deterministic  dilation  discrete  distributed  distribution  dnn  docker  dynamic-batching  e2e  edward  embedding  estimator  examples  few-shot  fizzbuzz  fluent  forward-mode  funny  gaming  gan  gaussian-processes  generative  generative-models  google  gpu  gradient  graph  handwriting  image-completion  imagenet  imperative  inception  ipynb  javascript  keras  language-model  learning-to-learn  libs  links  linux  lstm  mac  machine-learning  mixture-density-network  mixture-models  mnist  model-compression  models  mozilla  neural-net  nlp  nlu  nondeterminism  numeric  object-detection  one-shot  papers  parallel  parsing  performance  phased-lstm  pipeline  probabilistic-programming  probability  profiling  programming  properties  python  pytorch  quantization  queue  ref  reinforcement-learning  restore  rnn  save  scope  seq2seq  sequence  sgd  shared-variable  sklearn  slides  sonnet  speech  stability  structure-learning  style-transfer  summarization  tensorboard  tensorflow  tensorflow.js  text  tf-slim  tflearn  theano  time-series  torch  transfer-learning  transformation  tsne  tutorials  vae  variable  via:euler  via:nirum  visualization  vq-vae  warning  wasserstein  wav2letter  wavenet  webapps  wgan  word2vec 

Copy this bookmark: