deeplearning   12626

« earlier    

NLP's ImageNet moment has arrived
Word2vec and related methods are shallow approaches that trade expressivity for efficiency. Using word embeddings is like initializing a computer vision model with pretrained representations that only encode edges: they will be helpful for many tasks, but they fail to capture higher-level information that might be even more useful. A model initialized with word embeddings needs to learn from scratch not only to disambiguate words, but also to derive meaning from a sequence of words. This is the core aspect of language understanding, and it requires modeling complex language phenomena such as compositionality, polysemy, anaphora, long-term dependencies, agreement, negation, and many more. It should thus come as no surprise that NLP models initialized with these shallow representations still require a huge number of examples to achieve good performance.

In NLP, models are typically a lot shallower than their CV counterparts. Analysis of features has thus mostly focused on the first embedding layer, and little work has investigated the properties of higher layers for transfer learning. Let us consider the datasets that are large enough, fulfilling desideratum #1. Given the current state of NLP, there are several contenders.

Language modeling (LM) aims to predict the next word given its previous word. Existing benchmark datasets consist of up to 1B words, but as the task is unsupervised, any number of words can be used for training. See below for examples from the popular WikiText-2 dataset consisting of Wikipedia articles.

In light of this step change, it is very likely that in a year’s time NLP practitioners will download pretrained language models rather than pretrained word embeddings for use in their own models, similarly to how pre-trained ImageNet models are the starting point for most CV projects nowadays.
nlp  deeplearning 
1 hour ago by mike
open-source-for-science/TensorFlow-Course: Simple and ready-to-use tutorials for TensorFlow
Simple and ready-to-use tutorials for TensorFlow . Contribute to open-source-for-science/TensorFlow-Course development by creating an account on GitHub.
programming  ai  tensorflow  tutorial  deeplearning  python  machinelearning 
10 hours ago by lukecathie
open-source-for-science/TensorFlow-Course: Simple and ready-to-use tutorials for TensorFlow
Simple and ready-to-use tutorials for TensorFlow . Contribute to open-source-for-science/TensorFlow-Course development by creating an account on GitHub.
ai  programming  tutorial  tensorflow  datascience  machinelearning  deeplearning 
18 hours ago by dlutcat
GitHub - MagNet-DL/magnet: Deep Learning Projects that Build Themselves
Deep Learning Projects that Build Themselves. Contribute to MagNet-DL/magnet development by creating an account on GitHub.
pytorch  deeplearning 
yesterday by twleung

« earlier    

related tags

adventures  ai  algorithms  analytics  ann  art  artificialintelligence  audio  aws  badscience  bot  business  ceo  cnn  color  computationalmusic  computers  computervision  courses  crypto  cuda  cyber  dance  datascience  deeptext  development  differentiableprogramming  digitalmarketing  diy  dl  dlcpus  dropbox  edu  education  embeddings  entityextraction  facebook  fashion  fastai  fefe  fortalk  gpu  gpus  halide  hardware  health  hn  host  hpc  illustration  ilya  inflation  intelligence  interview  javascript  jeremyhoward  kaggle  keras  lang:en  language  lectures  lowcode  lumos  machinelearning  manga  marketing  marketplace  mathematics  maths  medicine  mit  ml  monads  music  networks  neural  neuralnetwork  neuralnetworks  neuroscience  nlp  nn  ocr  online  openai  painting  paper  papers  people  photography  productmanagement  programming  python  pytorch  r.deeplearning  r.machinelearning  r  recommendations  reinforcementlearning  research  resnet  rig  scaling  science  seo  seq2seq  server  servers  siggraph18  simulation  softwaredev  sound  speech  statistics  sutskever  talk  ted  tensorflow  tensorflowjs  todo  tool  tools  transferlearning  translate  translation  translator  tutorial  video  videos  vision  weather  webtools  word2vec  wordembedding  workflow 

Copy this bookmark:



description:


tags: