**arsyed + tensorflow**
218

TensorSpace.js – Present tensor in space, neural network 3D visualization framework_Github - jishuwen(技术文)

28 days ago by arsyed

TensorSpace is a neural network 3D visualization framework built by TensorFlow.js, Three.js and Tween.js. TensorSpace provides Keras-like APIs to build deep learning layers, load pre-trained models, and generate a 3D visualization in the browser.

neural-net
tensorflow
visualization
28 days ago by arsyed

AutoGraph converts Python into TensorFlow graphs – TensorFlow – Medium

july 2018 by arsyed

"AutoGraph takes in your eager-style Python code and converts it to graph-generating code."

tensorflow
python
graph
july 2018 by arsyed

mcdermottLab/kelletal2018: Parameters for the neural network used in Kell, Yamins, Shook, Norman-Haignere, and McDermott. "A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hier

code tensorflow neural-net auditory

june 2018 by arsyed

code tensorflow neural-net auditory

june 2018 by arsyed

Override Tensorflow backward-propagation – Sirena – Medium

june 2018 by arsyed

"In paper “Clothing Retrieval with Visual Attention Model” they describe attention network that generates Bernoulli series has to be multiplied with another feature map. Unfortunately Bernoulli is not differentiable, hence backward propagation will not flow.

Follow the above points I implement forward propagation that generates Bernoulli series with the given shape and in backprop function I implement custom gradient — it was just multiplication between intermediate layer (another feature map) and coming gradient."

tensorflow
custom-gradient
Follow the above points I implement forward propagation that generates Bernoulli series with the given shape and in backprop function I implement custom gradient — it was just multiplication between intermediate layer (another feature map) and coming gradient."

june 2018 by arsyed

Google AI Blog: Realtime tSNE Visualizations with TensorFlow.js

june 2018 by arsyed

"With this implementation, what used to take 15 minutes to calculate (on the MNIST dataset) can now be visualized in real-time and in the web browser. Furthermore this allows real-time visualizations of much larger datasets, a feature that is particularly useful when deep neural output is analyzed. One main limitation of our work is that this technique currently only works for 2D embeddings. However, 2D visualizations are often preferred over 3D ones as they require more interaction to effectively understand cluster results."

tsne
visualization
tensorflow
javascript
june 2018 by arsyed

[1805.01772] Dynamic Control Flow in Large-Scale Machine Learning

may 2018 by arsyed

"This paper presents a programming model for distributed machine learning that supports dynamic control flow. We describe the design of the programming model, and its implementation in TensorFlow, a distributed machine learning system. Our approach extends the use of dataflow graphs to represent machine learning models, offering several distinctive features. First, the branches of conditionals and bodies of loops can be partitioned across many machines to run on a set of heterogeneous devices, including CPUs, GPUs, and custom ASICs. Second, programs written in our model support automatic differentiation and distributed gradient computations, which are necessary for training machine learning models that use control flow. Third, our choice of non-strict semantics enables multiple loop iterations to execute in parallel across machines, and to overlap compute and I/O operations. "

programming
machine-learning
tensorflow
control-flow
may 2018 by arsyed

Evaluated expressions of variables differ sometimes when using the GPU · Issue #2226 · tensorflow/tensorflow

april 2018 by arsyed

"The GPU kernel for reduce_sum is known to be non-deterministic since it

uses the atomic operations. When the number of elements is large, the

difference could be quite large.

If your model is trained with dropout in it, it tends to be less likely to

be affected by the noise."

"It sounds like your 97.7% vs. 9.8% difference occurs for a model where different CPU systems can also produce 10%, and thus that you may have found an impressively unreliable set of hyperparameters. If using a smaller learning rate helps, it may just mean that the GPU version blows up slightly earlier than the CPU version."

"the problem could be attributed to small differences introduced in the non-deterministic behavior on GPU. Or some numerical instability due to the precision difference between CPU and GPU."

tensorflow
cpu
gpu
sgd
brittle
stability
numeric
uses the atomic operations. When the number of elements is large, the

difference could be quite large.

If your model is trained with dropout in it, it tends to be less likely to

be affected by the noise."

"It sounds like your 97.7% vs. 9.8% difference occurs for a model where different CPU systems can also produce 10%, and thus that you may have found an impressively unreliable set of hyperparameters. If using a smaller learning rate helps, it may just mean that the GPU version blows up slightly earlier than the CPU version."

"the problem could be attributed to small differences introduced in the non-deterministic behavior on GPU. Or some numerical instability due to the precision difference between CPU and GPU."

april 2018 by arsyed

How to use Dataset in TensorFlow – Towards Data Science

april 2018 by arsyed

One shot. It can iterate once through a dataset, you cannot feed any value to it.

Initializable: You can dynamically change calling its initializer operation and passing the new data with feed_dict . It’s basically a bucket that you can fill with stuff.

Reinitializable: It can be initialised from different Dataset. Very useful when you have a training dataset that needs some additional transformation, eg. shuffle, and a testing dataset. It’s like using a tower crane to select different container.

Feedable: It can be used to select with iterator to use. Following the previous example, it’s like a tower crane that selects which tower crane to use to select which container to take. In my opinion is useless.

tensorflow
tf.data
Initializable: You can dynamically change calling its initializer operation and passing the new data with feed_dict . It’s basically a bucket that you can fill with stuff.

Reinitializable: It can be initialised from different Dataset. Very useful when you have a training dataset that needs some additional transformation, eg. shuffle, and a testing dataset. It’s like using a tower crane to select different container.

Feedable: It can be used to select with iterator to use. Following the previous example, it’s like a tower crane that selects which tower crane to use to select which container to take. In my opinion is useless.

april 2018 by arsyed

TencentAILab/tf-adaptive-softmax-lstm-lm: The experiment result of LSTM language models on PTB (Penn Treebank) and GBW (Google Billion Word) using AdaptiveSoftmax on TensorFlow.

march 2018 by arsyed

This repository shows the experiment result of LSTM language models on PTB (Penn Treebank) and GBW (Google One Billion Word) using AdaptiveSoftmax on TensorFlow.

Adaptive Softmax

The adaptive softmax is a faster way to train a softmax classifier over a huge number of classes, and can be used for both training and prediction. For example, it can be used for training a Language Model with a very huge vocabulary, and the trained languaed model can be used in speech recognition, text generation, and machine translation very efficiently.

Tha adaptive softmax has been used in the ASR system developed by Tencent AI Lab, and achieved about 20x speed up than full sotfmax in the second pass for rescoing.

tensorflow
language-model
asr
adaptive-softmax
Adaptive Softmax

The adaptive softmax is a faster way to train a softmax classifier over a huge number of classes, and can be used for both training and prediction. For example, it can be used for training a Language Model with a very huge vocabulary, and the trained languaed model can be used in speech recognition, text generation, and machine translation very efficiently.

Tha adaptive softmax has been used in the ASR system developed by Tencent AI Lab, and achieved about 20x speed up than full sotfmax in the second pass for rescoing.

march 2018 by arsyed

Tensorflow on GPU worrking inefficiently - Google Groups

march 2018 by arsyed

Three important things helped a lot:

1. replacing feed_dict with tensor variable

2. using float 32 instead of 64 (which speed everything up anyway, without loosing accuracy)

3. session tracing showed me other things to improve

tensorflow
gpu
preformance
1. replacing feed_dict with tensor variable

2. using float 32 instead of 64 (which speed everything up anyway, without loosing accuracy)

3. session tracing showed me other things to improve

march 2018 by arsyed

GitHub - tryolabs/luminoth: Deep Learning toolkit for Computer Vision

january 2018 by arsyed

Luminoth is an open source toolkit for computer vision. Currently, we support object detection and image classification, but we are aiming for much more. It is built in Python, using TensorFlow and Sonnet.

python
libs
computer-vision
deep-learning
object-detection
tensorflow
via:euler
january 2018 by arsyed

[1711.10604] TensorFlow Distributions

november 2017 by arsyed

"The TensorFlow Distributions library implements a vision of probability theory adapted to the modern deep-learning paradigm of end-to-end differentiable computation. Building on two basic abstractions, it offers flexible building blocks for probabilistic computation. Distributions provide fast, numerically stable methods for generating samples and computing statistics, e.g., log density. Bijectors provide composable volume-tracking transformations with automatic caching. Together these enable modular construction of high dimensional distributions and transformations not possible with previous libraries (e.g., pixelCNNs, autoregressive flows, and reversible residual networks). They are the workhorse behind deep probabilistic programming systems like Edward and empower fast black-box inference in probabilistic models built on deep-network components. TensorFlow Distributions has proven an important part of the TensorFlow toolkit within Google and in the broader deep learning community."

papers
tensorflow
probabilistic-programming
distribution
november 2017 by arsyed

Eric Jang on Twitter: "TensorFlow trick: if you want a custom gradient

november 2017 by arsyed

"TensorFlow trick: if you want a custom gradient

f'(x) + g'(x) for a layer f(x),

just do f(x) + g(x) - tf.stop_gradient(g(x))

and let autodiff do the rest."

"there are somewhat deep numerical reasons why this works (IEEE754). i learned this trick from josh dillon! https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/bayesflow/python/ops/monte_carlo_impl.py#L338 "

"h(x) = f(x) + g(x) - stop_gradient(g(x)) = f(x) + g(x) - g(x) = f(x) when evaluating h(x), but grad(h(x)) = grad(f(x)) + grad(g(x)) - 0 by definition of stop_gradient"

tensorflow
automatic-differentiation
tricks
f'(x) + g'(x) for a layer f(x),

just do f(x) + g(x) - tf.stop_gradient(g(x))

and let autodiff do the rest."

"there are somewhat deep numerical reasons why this works (IEEE754). i learned this trick from josh dillon! https://github.com/tensorflow/tensorflow/blob/master/tensorflow/contrib/bayesflow/python/ops/monte_carlo_impl.py#L338 "

"h(x) = f(x) + g(x) - stop_gradient(g(x)) = f(x) + g(x) - g(x) = f(x) when evaluating h(x), but grad(h(x)) = grad(f(x)) + grad(g(x)) - 0 by definition of stop_gradient"

november 2017 by arsyed

How to silence TensorFlow · Martin Thoma

september 2017 by arsyed

I've added export TF_CPP_MIN_LOG_LEVEL=2 to my ~/.profile. Note that this only suppresses the warnings. The better solution would be to compile Tensorflow with SSE-X support.

tensorflow
warnings
september 2017 by arsyed

Using Tensorboard Embeddings Visualiser with Numpy Arrays - nlml

august 2017 by arsyed

"So I decided to throw together a function save_embeddings() that takes the hassle out of this, allowing you to go straight from numpy arrays to Tensorboard-visualised embeddings."

https://github.com/nlml/np-to-tf-embeddings-visualiser

tensorboard
tensorflow
visualization
code
https://github.com/nlml/np-to-tf-embeddings-visualiser

august 2017 by arsyed

**related tags**

Copy this bookmark: