**arsyed + tensorflow**
104

AutoGraph converts Python into TensorFlow graphs – TensorFlow – Medium

july 2018 by arsyed

"AutoGraph takes in your eager-style Python code and converts it to graph-generating code."

tensorflow
python
graph
july 2018 by arsyed

mcdermottLab/kelletal2018: Parameters for the neural network used in Kell, Yamins, Shook, Norman-Haignere, and McDermott. "A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hier

code tensorflow neural-net auditory

june 2018 by arsyed

code tensorflow neural-net auditory

june 2018 by arsyed

Override Tensorflow backward-propagation – Sirena – Medium

june 2018 by arsyed

"In paper “Clothing Retrieval with Visual Attention Model” they describe attention network that generates Bernoulli series has to be multiplied with another feature map. Unfortunately Bernoulli is not differentiable, hence backward propagation will not flow.

Follow the above points I implement forward propagation that generates Bernoulli series with the given shape and in backprop function I implement custom gradient — it was just multiplication between intermediate layer (another feature map) and coming gradient."

tensorflow
custom-gradient
Follow the above points I implement forward propagation that generates Bernoulli series with the given shape and in backprop function I implement custom gradient — it was just multiplication between intermediate layer (another feature map) and coming gradient."

june 2018 by arsyed

Google AI Blog: Realtime tSNE Visualizations with TensorFlow.js

june 2018 by arsyed

"With this implementation, what used to take 15 minutes to calculate (on the MNIST dataset) can now be visualized in real-time and in the web browser. Furthermore this allows real-time visualizations of much larger datasets, a feature that is particularly useful when deep neural output is analyzed. One main limitation of our work is that this technique currently only works for 2D embeddings. However, 2D visualizations are often preferred over 3D ones as they require more interaction to effectively understand cluster results."

tsne
visualization
tensorflow
javascript
june 2018 by arsyed

[1805.01772] Dynamic Control Flow in Large-Scale Machine Learning

may 2018 by arsyed

"This paper presents a programming model for distributed machine learning that supports dynamic control flow. We describe the design of the programming model, and its implementation in TensorFlow, a distributed machine learning system. Our approach extends the use of dataflow graphs to represent machine learning models, offering several distinctive features. First, the branches of conditionals and bodies of loops can be partitioned across many machines to run on a set of heterogeneous devices, including CPUs, GPUs, and custom ASICs. Second, programs written in our model support automatic differentiation and distributed gradient computations, which are necessary for training machine learning models that use control flow. Third, our choice of non-strict semantics enables multiple loop iterations to execute in parallel across machines, and to overlap compute and I/O operations. "

programming
machine-learning
tensorflow
control-flow
may 2018 by arsyed

Evaluated expressions of variables differ sometimes when using the GPU · Issue #2226 · tensorflow/tensorflow

april 2018 by arsyed

"The GPU kernel for reduce_sum is known to be non-deterministic since it

uses the atomic operations. When the number of elements is large, the

difference could be quite large.

If your model is trained with dropout in it, it tends to be less likely to

be affected by the noise."

"It sounds like your 97.7% vs. 9.8% difference occurs for a model where different CPU systems can also produce 10%, and thus that you may have found an impressively unreliable set of hyperparameters. If using a smaller learning rate helps, it may just mean that the GPU version blows up slightly earlier than the CPU version."

"the problem could be attributed to small differences introduced in the non-deterministic behavior on GPU. Or some numerical instability due to the precision difference between CPU and GPU."

tensorflow
cpu
gpu
sgd
brittle
stability
numeric
uses the atomic operations. When the number of elements is large, the

difference could be quite large.

If your model is trained with dropout in it, it tends to be less likely to

be affected by the noise."

"It sounds like your 97.7% vs. 9.8% difference occurs for a model where different CPU systems can also produce 10%, and thus that you may have found an impressively unreliable set of hyperparameters. If using a smaller learning rate helps, it may just mean that the GPU version blows up slightly earlier than the CPU version."

"the problem could be attributed to small differences introduced in the non-deterministic behavior on GPU. Or some numerical instability due to the precision difference between CPU and GPU."

april 2018 by arsyed

How to use Dataset in TensorFlow – Towards Data Science

april 2018 by arsyed

One shot. It can iterate once through a dataset, you cannot feed any value to it.

Initializable: You can dynamically change calling its initializer operation and passing the new data with feed_dict . It’s basically a bucket that you can fill with stuff.

Reinitializable: It can be initialised from different Dataset. Very useful when you have a training dataset that needs some additional transformation, eg. shuffle, and a testing dataset. It’s like using a tower crane to select different container.

Feedable: It can be used to select with iterator to use. Following the previous example, it’s like a tower crane that selects which tower crane to use to select which container to take. In my opinion is useless.

tensorflow
tf.data
Initializable: You can dynamically change calling its initializer operation and passing the new data with feed_dict . It’s basically a bucket that you can fill with stuff.

Reinitializable: It can be initialised from different Dataset. Very useful when you have a training dataset that needs some additional transformation, eg. shuffle, and a testing dataset. It’s like using a tower crane to select different container.

Feedable: It can be used to select with iterator to use. Following the previous example, it’s like a tower crane that selects which tower crane to use to select which container to take. In my opinion is useless.

april 2018 by arsyed

TencentAILab/tf-adaptive-softmax-lstm-lm: The experiment result of LSTM language models on PTB (Penn Treebank) and GBW (Google Billion Word) using AdaptiveSoftmax on TensorFlow.

march 2018 by arsyed

This repository shows the experiment result of LSTM language models on PTB (Penn Treebank) and GBW (Google One Billion Word) using AdaptiveSoftmax on TensorFlow.

Adaptive Softmax

The adaptive softmax is a faster way to train a softmax classifier over a huge number of classes, and can be used for both training and prediction. For example, it can be used for training a Language Model with a very huge vocabulary, and the trained languaed model can be used in speech recognition, text generation, and machine translation very efficiently.

Tha adaptive softmax has been used in the ASR system developed by Tencent AI Lab, and achieved about 20x speed up than full sotfmax in the second pass for rescoing.

tensorflow
language-model
asr
adaptive-softmax
Adaptive Softmax

The adaptive softmax is a faster way to train a softmax classifier over a huge number of classes, and can be used for both training and prediction. For example, it can be used for training a Language Model with a very huge vocabulary, and the trained languaed model can be used in speech recognition, text generation, and machine translation very efficiently.

Tha adaptive softmax has been used in the ASR system developed by Tencent AI Lab, and achieved about 20x speed up than full sotfmax in the second pass for rescoing.

march 2018 by arsyed

GitHub - tryolabs/luminoth: Deep Learning toolkit for Computer Vision

january 2018 by arsyed

Luminoth is an open source toolkit for computer vision. Currently, we support object detection and image classification, but we are aiming for much more. It is built in Python, using TensorFlow and Sonnet.

python
libs
computer-vision
deep-learning
object-detection
tensorflow
via:euler
january 2018 by arsyed

[1711.10604] TensorFlow Distributions

november 2017 by arsyed

"The TensorFlow Distributions library implements a vision of probability theory adapted to the modern deep-learning paradigm of end-to-end differentiable computation. Building on two basic abstractions, it offers flexible building blocks for probabilistic computation. Distributions provide fast, numerically stable methods for generating samples and computing statistics, e.g., log density. Bijectors provide composable volume-tracking transformations with automatic caching. Together these enable modular construction of high dimensional distributions and transformations not possible with previous libraries (e.g., pixelCNNs, autoregressive flows, and reversible residual networks). They are the workhorse behind deep probabilistic programming systems like Edward and empower fast black-box inference in probabilistic models built on deep-network components. TensorFlow Distributions has proven an important part of the TensorFlow toolkit within Google and in the broader deep learning community."

papers
tensorflow
probabilistic-programming
distribution
november 2017 by arsyed

Using Tensorboard Embeddings Visualiser with Numpy Arrays - nlml

august 2017 by arsyed

"So I decided to throw together a function save_embeddings() that takes the hassle out of this, allowing you to go straight from numpy arrays to Tensorboard-visualised embeddings."

https://github.com/nlml/np-to-tf-embeddings-visualiser

tensorboard
tensorflow
visualization
code
https://github.com/nlml/np-to-tf-embeddings-visualiser

august 2017 by arsyed

tensorflow/tensor2tensor: A library for generalized sequence to sequence models

june 2017 by arsyed

"T2T is a modular and extensible library and binaries for supervised learning with TensorFlow and with support for sequence tasks. It is actively used and maintained by researchers and engineers within the Google Brain team."

python
libs
tensorflow
sequence
seq2seq
june 2017 by arsyed

tensorflow/transform_graph_test.py at master · tensorflow/tensorflow

april 2017 by arsyed

"This test constructs a graph with a relu op that's not used by the normal inference path, and then tests that the strip_unused transform removes it as expected."

tensorflow
code
graph
transformation
april 2017 by arsyed

tensorflow/fold: Deep learning with dynamic computation graphs in TensorFlow

february 2017 by arsyed

"TensorFlow Fold is a library for creating TensorFlow models that consume structured data, where the structure of the computation graph depends on the structure of the input data.

[...]

Fold implements dynamic batching. Batches of arbitrarily shaped computation graphs are transformed to produce a static computation graph. This graph has the same structure regardless of what input it receives, and can be executed efficiently by TensorFlow."

tensorflow
libs
structure-learning
dynamic-batching
[...]

Fold implements dynamic batching. Batches of arbitrarily shaped computation graphs are transformed to produce a static computation graph. This graph has the same structure regardless of what input it receives, and can be executed efficiently by TensorFlow."

february 2017 by arsyed

Tensorflow_py_func_with_grad.py

january 2017 by arsyed

function with custom gradient

tensorflow
gradient
custom
via:nirum
january 2017 by arsyed

[1701.03757] Deep Probabilistic Programming

january 2017 by arsyed

"We propose Edward, a Turing-complete probabilistic programming language. Edward builds on two compositional representations---random variables and inference. By treating inference as a first class citizen, on a par with modeling, we show that probabilistic programming can be as flexible and computationally efficient as traditional deep learning. For flexibility, Edward makes it easy to fit the same model using a variety of composable inference methods, ranging from point estimation, to variational inference, to MCMC. In addition, Edward can reuse the modeling representation as part of inference, facilitating the design of rich variational models and generative adversarial networks. For efficiency, Edward is integrated into TensorFlow, providing significant speedups over existing probabilistic systems. For example, on a benchmark logistic regression task, Edward is at least 35x faster than Stan and PyMC3."

papers
edward
probabilistic-programming
tensorflow
deep-learning
january 2017 by arsyed

igormq/ctc_tensorflow_example: CTC + Tensorflow Example for ASR

january 2017 by arsyed

"A toy example showing how to get CTC cost function working with Tensorflow for automatic speech recognition."

python
tensorflow
ctc
asr
january 2017 by arsyed

Enny1991/PLSTM

december 2016 by arsyed

"Tensorflow implementation of the recently introduced Phased LSTM by Neil et. al @ NIPS 2016"

code
tensorflow
lstm
phased-lstm
december 2016 by arsyed

Welcome to TensorLayer — TensorLayer 1.2.2 documentation

september 2016 by arsyed

"TensorLayer is a Deep Learning (DL) and Reinforcement Learning (RL) library extended from Google TensorFlow. It provides popular DL and RL modules that can be easily customized and assembled for tackling real-world machine learning problems."

python
libs
tensorflow
deep-learning
reinforcement-learning
september 2016 by arsyed

**related tags**

Copy this bookmark: