{"id":257,"date":"2020-02-03T04:23:03","date_gmt":"2020-02-03T04:23:03","guid":{"rendered":"http:\/\/martinsiron.com\/?p=257"},"modified":"2020-02-03T04:23:03","modified_gmt":"2020-02-03T04:23:03","slug":"using-named-entity-recognition-and-natural-language-processing-to-build-a-map-of-accumulated-infections-of-n-cov2019","status":"publish","type":"post","link":"http:\/\/martinsiron.com\/index.php\/2020\/02\/03\/using-named-entity-recognition-and-natural-language-processing-to-build-a-map-of-accumulated-infections-of-n-cov2019\/","title":{"rendered":"Using Named Entity Recognition and Natural Language Processing to build a map of accumulated infections of n-Cov2019"},"content":{"rendered":"\n

I begin by scraping the following website for all its information and translating the text using Google Translate API: https:\/\/ncov.dxy.cn\/ncovh5\/view\/pneumonia_timeline?whichFrom=dxy<\/a><\/p>\n\n\n\n

The data consists of a header, a description, a timestamp of some sort, and the source from where the information came from. Using the HTML library in Python I extracted these various sections of the page and place it into a Pandas Dataframe.<\/p>\n\n\n\n

After some cleaning, I used \u201cNamed Entity Recognition\u201d to extract locations from both the descriptions and the titles. Since location is such a common use for NER, I decided to use an already existing model: DeepPavlov. I loaded this model and extracted all the locations and put them in their own Pandas columns in the data frame. <\/p>\n\n\n\n

Next, I need to extract the latitude and longitude from the name of these locations. Using GeoPy, I extracted and averaged all the locations in both the header and description. For now, I opted to use only the location information in the header as from a quick glance, this tends to be the main region or entity where the cases happened, rather than the city levels or entities that might have tested the case in the descriptions.<\/p>\n\n\n\n

Now, I was left with my ultimate bottleneck, text in the\ndescription like this:<\/p>\n\n\n\n

“From 00:00 to 24:00 on January 31, 2020, Shanxi Province reported 8 new confirmed cases of pneumonia due to new coronavirus infection. As of 14:00 on January 31, 47 cases of pneumonia confirmed by new coronavirus infection have been reported in 11 cities in Shanxi Province (including 2 severe cases, 1 critical case, 1 discharged case, and no deaths). At present, 1333 close contacts have been tracked. 41 people were released from medical observation on the same day, and a total of 1101 people were receiving medical observation.”<\/p><\/blockquote>\n\n\n\n

From this I\u2019ve already extracted this information:<\/strong>
Time<\/span>: January 31, 2020
Location<\/span>: Shanxi Province
However, I still need to extract the following information<\/strong>
New cases:<\/span> 8 new confirmed cases of pneumonia (8)
Accumulated cases:<\/span> 47 cases of pneumonia confirmed (47)
New deaths:<\/span> no deaths (0)<\/p>\n\n\n\n

In order to approach this, I needed to train a custom NER model.\nI used TagTog\u2019s tagging and AI tool to do this work:<\/p>\n\n\n\n

\"\"
TagTog: Annotation Tool<\/figcaption><\/figure>\n\n\n\n

This makes it easy to train by selecting the various information in a document. I created 8 entities and 4 relationships:<\/p>\n\n\n\n
Entity <\/strong><\/td> Description <\/strong><\/td> Relations <\/strong><\/td><\/tr>
new_case_I <\/td>New case added in location <\/td>New cases and it\u2019s number <\/td><\/tr>
new_case_N <\/td>Number of new cases <\/td><\/tr>
acc_case_I <\/td>Accumulated case in region <\/td>Accumulated cases and its number <\/td><\/tr>
acc_case_N <\/td>Number of accumulated cases <\/td><\/tr>
new_death_I <\/td>New death recorded in region <\/td>New deaths and its number <\/td><\/tr>
new_death_N <\/td>\n Number of new deaths\n <\/td><\/tr>
acc_death_I <\/td>Accumulated death in region <\/td>Accumulated deaths and its number <\/td><\/tr>
acc_death_N <\/td>\n Number of accumulated deaths\n <\/td><\/tr><\/tbody><\/table>\n\n\n\n

I manually tagged 20% of the data to build the model. From this, I parsed the rest of the data through the model to extract the remaining information. From the initial document example, the model extracted the following information:<\/p>\n\n\n\n
\n Text\n <\/td>\n Entity\n <\/td>\n Probability\n <\/td><\/tr>
‘no’<\/td>“accumulated_death-N” <\/td>0.874<\/td><\/tr>
‘8’<\/td>“new_case-N” <\/td>0.838<\/td><\/tr>
‘no deaths’<\/td>“accumulated_death-I” <\/td>0.698<\/td><\/tr>
‘1’<\/td>\u201cnew_death-N\u201d <\/td>0.621<\/td><\/tr>
‘1’<\/td>\u201cnew_death-N\u201d <\/td>0.550<\/td><\/tr>
’47’<\/td>“accumulated_case-N” <\/td>0.813<\/td><\/tr>
’41’<\/td>“accumulated_case-N” <\/td>0.714<\/td><\/tr>
‘8 new confirmed cases’<\/td>“new_case-I” <\/td>0.842<\/td><\/tr>
’47 cases of pneumonia confirmed’<\/td>“new_case-I” <\/td>0.640<\/td><\/tr><\/tbody><\/table>\n\n\n\n

We can see that the model with 20% of the data did fairly well. Numerically, we were able to pick up the correct new case number (8) and the correct accumulated case number (47).<\/strong><\/p>\n\n\n\n

With the rest of the data parsed, I used Plotly to build a final heatmap (data from morning of 01\/28). We can see from this that the model has clearly picked up the epicenter and the surrounding virus activity nearby.<\/p>\n\n\n\n

\"\"
Heatmap of nCov2019 Accumulated Infections as of 01.28.2019<\/figcaption><\/figure><\/div>\n\n\n\n

I plotted the accumulated cases over time:<\/p>\n\n\n\n

\"\"
Accumulated cases over time for nCov2019 Infections<\/figcaption><\/figure><\/div>\n\n\n\n

Finally, I made a GIF heatmap of the virus activity over time (note the scale bar changes overtime):<\/p>\n\n\n\n

\"\"
Animated heatmap over time. Scalebar changes overtime.<\/figcaption><\/figure>\n\n\n\n

Future:<\/strong>
In the imminent future, I plan on releasing a Flask app on my website with the data, stay tuned! <\/strong>Additionally, I will update it with the most recent data released from 01.28.2020.<\/p>\n","protected":false},"excerpt":{"rendered":"

I begin by scraping the following website for all its information and translating the text using Google Translate API: https:\/\/ncov.dxy.cn\/ncovh5\/view\/pneumonia_timeline?whichFrom=dxy The data consists of a header, a description, a timestamp of some sort, and the source from where the information came from. Using the HTML library in Python I extracted these various sections of the…Read more Using Named Entity Recognition and Natural Language Processing to build a map of accumulated infections of n-Cov2019<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-257","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"http:\/\/martinsiron.com\/index.php\/wp-json\/wp\/v2\/posts\/257","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/martinsiron.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/martinsiron.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/martinsiron.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/martinsiron.com\/index.php\/wp-json\/wp\/v2\/comments?post=257"}],"version-history":[{"count":0,"href":"http:\/\/martinsiron.com\/index.php\/wp-json\/wp\/v2\/posts\/257\/revisions"}],"wp:attachment":[{"href":"http:\/\/martinsiron.com\/index.php\/wp-json\/wp\/v2\/media?parent=257"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/martinsiron.com\/index.php\/wp-json\/wp\/v2\/categories?post=257"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/martinsiron.com\/index.php\/wp-json\/wp\/v2\/tags?post=257"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}