Training Word Embedding on Bloomberg News

NLP 9 minutes read (About 1350 words)

While working from home (stuck at home indeed), I get a bit more time, so revisited some old deep learning notes last month. Would it be awesome if I can come up with a project to play with? As we all know knowledge would go rusty if lack of use!

Initially, I was thinking about a machine translator using Attention Models, because I was once asked in an interview. The fund showed a particular interest in developing such a tool for the purpose of distributing research reports to non-English-speaking countries. But I was completely set back by the data sets asking for literally thousands of dollars… While if labeled dataset is that expensive, why not run some unsurprised learning? Training word2vec doesn’t sound like a bad idea. I’ve been writing some views on FICC (fixed income, currencies, and commodities) on my other non-geeky-at-all blog. Won’t it be fun to run word embeddings on Bloomberg news, see if people talk about the right things when they analyze inflation and gold prices?

So here we go, rock and roll!

First it’s a bit introduction to word embedding. The term refers to a set of techniques in natural language processing that can map words into high dimensional vectors of real numbers. The vectors/embeddings can be seen as featurized representations of words, which preserve semantic properties, hence are commonly used as inputs in NLP tasks.

Here is an example,

embedding vectors

While in real cases, the dimension of vectors is usually a lot higher.

We will use a tool called word2vec to convert words into embeddings, gensim has it implemented in a very easy-to-use way.

Word2vec is a technique for natural language processing. The word2vec algorithm uses a neural network model to learn word associations from a large corpus of text. Once trained, such a model can detect synonymous words or suggest additional words for a partial sentence.

More explanation on word2vec can be found here.

Creating Corpus

Scarping Addresses of Bloomberg Articles

I would like to run the embeddings only on the most recent posts, so first need to grab their urls through a search page.

Import the libraries,

1
2
3
4
5
6
7
import re
import requests
import string
import numpy as np
import gensim.models.word2vec as word2vec
from bs4 import BeautifulSoup
from time import sleep

A function that can scrape the urls of articles that show up in the top p pages related to a specific topic,

1
2
3
4
5
6
7
8
9
10
11
12
def scrape_bloomberg_urls(subject, maxPage, headers):
    urls = []
    for p in range(1,maxPage):
        searchUrl = 'https://www.bloomberg.com/search?query=' + subject + '&sort=relevance:desc' + '&page=' + str(p)
        response = requests.get(searchUrl, headers=headers)
        soup = BeautifulSoup(response.content, 'html.parser')
        regex = re.compile('.*headline.*')
        for tag in soup.findAll('a', {"class" : regex}, href = True):
            href = tag.attrs['href']
            if '/news/articles/' in href and href not in urls:
                urls.append(href)            
    return urls

Set up headers to pass to the request, websites have gone crazy about blocking robots,

1
2
3
4
5
6
7
headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36',
    'referrer': 'https://google.com',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8',
    'Accept-Encoding': 'gzip, deflate, br',
    'Accept-Language': 'en-US,en;q=0.9',
    'Pragma': 'no-cache'}

We get 155 articles here, not a large dataset so it may give us weird results, but should be good enough for an exercise,

1
2
In: len(urls)
Out: 155
1
2
3
4
5
6
7
In: urls[:5]
Out: 
['https://www.bloomberg.com/news/articles/2020-08-07/inflation-trend-a-friend-in-real-yield-contest-seasia-rates',
 'https://www.bloomberg.com/news/articles/2020-08-06/blackrock-joins-crescendo-of-inflation-warnings-amid-virus-fight',
 'https://www.bloomberg.com/news/articles/2020-08-06/china-inflation-rate-headed-for-0-on-cooling-food-prices-chart',
 'https://www.bloomberg.com/news/articles/2020-08-06/inflation-binds-czechs-after-fast-rate-cuts-decision-day-guide',
 'https://www.bloomberg.com/news/articles/2020-08-07/jpmorgan-rejects-threat-to-dollar-status-flagged-by-goldman']

Parsing Articles

This step is to parse articles through the urls and save the corpus into a local text file.

1
2
3
4
5
6
7
8
9
10
11
12
def parse_article(urls, headers):
    corpus = []
    for url in urls:
        response = requests.get(url, headers=headers)
        soup = BeautifulSoup(response.content,'lxml')
        for tag in soup.find_all('p'):
            content = tag.get_text()
            cleanedContent = content.tra nslate(str.maketrans('', '', string.punctuation)).lower()
            corpus.append(cleanedContent)
        seconds = np.random.randint(low=5, high=20)
        sleep(seconds)
    return corpus

A function serves as a writer,

1
2
3
4
5
def write_to_txt(outdir, subject, contentList):
    outputfile = f'{outdir}/{subject}.txt'
    with open(outputfile, 'a') as file:
        for i in contentList:
            file.write(f'{i}\n')

Run the functions,

1
2
corpus = parse_article(urls, headers)
write_to_txt('Documents/Blog/', 'corpus', corpus)

Training the Model

Now that we have created the corpus, let’s train the model using word2vec from gensim.

Gensim has a cool function LineSentence that can directly read sentences from a text file with one sentence a line. Now you probably get why I had the corpus saved this way.

1
2
sentences = word2vec.LineSentence('corpus.txt')
model = word2vec.Word2Vec(sentences, min_count=10, workers=8, iter=500, window=15, size=300, negative=50)

The model I choose is using the Skip-Gram algorithm with Negative Sampling. Hyper-parameters above are what I find work well, see below for their definitions, just in case you would like to tune them yourself,

  • size – Dimensionality of the word vectors.
  • window – Maximum distance between the current and predicted word within a sentence.
  • min_count – Ignores all words with total frequency lower than this.
  • workers – Use these many worker threads to train the model (=faster training with multicore machines).
  • negative – If > 0, negative sampling will be used, the int for negative specifies how many “noise words” should be drawn (usually between 5-20). If set to 0, no negative sampling is used.

Checking Out the Results

The goal of this exercise to find out what people talk about when they analyze inflation expectation (basically what currently drives the stock market, TIPS and metals to roar higher and higher) and gold (my favorite and probably the most promising asset in the next 10 years).

Word2vev model can find words that have been used in a similar context with the word we care about. Here we employ a method most_similar to reveal them.

Let’s first check out the words people usually mention when they talk about inflation. I print out the top 20 words. The meaningful ones are,

  • “target”, “bank’s”, “bank” - of course central banks target on inflation and their monetary policies shape expectations.
  • “nominal”, “negative”, “yields” - words usually describing rates, make sense.
  • “food” - measure of inflation.
  • “employment” - make a lot sense if you remember the Phillips curve.

The results are actually very good, authors did understand and explain inflations well.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
In: model.wv.most_similar('inflation',topn=20)
Out:
[('target', 0.25702178478240967),
 ('strip', 0.18441016972064972),
 ('securities', 0.18246109783649445),
 ('full', 0.16774789988994598),
 ('nominal', 0.16606125235557556),
 ('bank’s', 0.162990003824234),
 ('yields', 0.15935908257961273),
 ('negative', 0.15806841850280762),
 ('remain', 0.15371079742908478),
 ('levels', 0.14426037669181824),
 ('well', 0.1438026875257492),
 ('current', 0.13828279078006744),
 ('attractive', 0.13690069317817688),
 ('showed', 0.1277044713497162),
 ('bank', 0.12733221054077148),
 ('food', 0.12377716600894928),
 ('similar', 0.11955425888299942),
 ('the', 0.11911500245332718),
 ('employment', 0.11866464465856552),
 ('keep', 0.11818670481443405)]

Then let’s try it for gold. It’s good to see silver rank top, the two metals do hold a strong correlation despite the natures of them are very different, gold generally behaves as a bond of real rates, while silver should be viewed more as a commodity. (People make mistakes on this, see the story of Hunt Brothers)

Other words are just descriptive, focusing on telling readers what’s going on in the markets. While this is not helpful, Bloomberg! People want to know why, you may want to talk more about real rates, inflation expectations, debt and global growth in the future.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
In: model.wv.most_similar('gold',topn=30)
Out:
[('gold’s', 0.26054084300994873),
 ('silver', 0.25277888774871826),
 ('ounce', 0.2523342967033386),
 ('bullion', 0.2294640988111496),
 ('2300', 0.22592982649803162),
 ('spot', 0.2171008288860321),
 ('climbing', 0.19733953475952148),
 ('metal', 0.1939847469329834),
 ('comex', 0.19371576607227325),
 ('rally', 0.18643531203269958),
 ('exchangetraded', 0.1859540343284607),
 ('a', 0.18539577722549438),
 ('delivery', 0.17595867812633514),
 ('reaching', 0.1702871322631836),
 ('strip', 0.16905872523784637),
 ('posted', 0.16247007250785828),
 ('lows', 0.16047437489032745),
 ('as', 0.1585429608821869),
 ('analysis', 0.1577225774526596),
 ('2011', 0.15711694955825806),
 ('precious', 0.15656355023384094),
 ('threat', 0.1542907953262329),
 ('more', 0.1541929841041565),
 ('drive', 0.15310749411582947),
 ('every', 0.1524747610092163),
 ('analyst', 0.1517343521118164),
 ('managing', 0.14977654814720154),
 ('price', 0.1497490406036377),
 ('amounts', 0.14969849586486816),
 ('backed', 0.1450338065624237)]
Project Covid Admissions Based on Vaccine Progress, Part 1 -- From Simple Linear Regression to Weighted Time Series Regression
Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×