Martin Siron, Ph.D.

Mapping the evolution of the most dangerous areas of Los Angeles for pedestrians and bicyclists

Thu, 24 Nov 2022 19:30:39 GMT

I recently started biking…in Los Angeles. Los Angeles could be one of the most bike-friendly cities in the United States: it boasts plenty of annual sunshine, it has a mostly flat terrain, and quite mild weather. Unfortunately, many parts of the city lack proper bike infrastructure, making it dangerous when it comes to traffic collisions and traffic violence. In 2021, over 230 people lost their lives to the streets of Los Angeles.

With this in the background, I wanted to figure out the most dangerous streets in Los Angeles. I had been following the Streets For All Twitter account for a while and knew that they had a bot which posted any traffic violence involving pedestrians or bikes. This account had records dating back to just over one year. So I started with that (old results here). I used the Twitter API to extract all of the Tweets from the bot account, and then used the Google Maps API to convert the text-based location to a series of latitude and longitude.

I soon stumbled upon LA City’s records, dating back 2010-Present. This uses MO codes to detail the accident. After converting the MO codes for vehicle collision with pedestrian, and bikes, I had all the data needed to plot all collisions:

Overall traffic collisions, 2010-2020. Source: LA City.

For plotting, I am using Plotly with the Mapbox API.

How does this look year over year since the data started? First, a quick decline occurs after 2013 followed by similarly looking collision patterns year over year. We can see some streets always have a dense amount of collisions. Venice, Figueroa, Pico, Olympic, Wilshire, Santa Monica and Vermont appear to be big incident streets year over year.

Animated year over year traffic collision patterns.

Of course, many have pointed out that the pandemic has switched typical traffic flows for the city, as such, here are the pre-pandemic vs. post-pandemic collision map densities:

2017-2020	2020-2022

No big visual difference in pre- vs post-pandemic traffic collision patterns.

Just because a street has more absolute collision does not mean it has increased probability of collisions. To capture the probability of collisions, I’m going to make an assumption: that streets with more vehicular traffic should have more chances of vehicle-pedestrian interaction, so if I normalize the collision by the traffic flow, I should arrive at a value that is more representative of the probability of collision.

Luckily, LADOT has data on such traffic volume. Unfortunately, this data appears to start in 2011 and stop in 2016. It does not survey the same street year over year and seems sporadic in the streets surveyed over time. It has not been updated since 2016 giving a mixed snapshot of the traffic throughout Los Angeles.

The first plot will assume that the traffic from 2011-2016 more or less is constant throughout the collision data from 2010-2020:

Overall collision probability map: Is Silver Lake one of the most dangerous places to bike in the last decade? (arbitrary units, higher means higher probability)

To make a plot like this I had to engineer a trick: the data from the traffic flow does not overlay perfectly from the data from collisions, naturally. In order to be able to divide both data I needed to figure out an interpolation scheme for each. For traffic flow data I did a ‘nearest’ interpolation grid approach. I did this for two reasons, first I thought of the traffic flow data as a ‘graph’ in that the flow from one point to its nearest neighbor must be related: cars must travel between these points to an extent. The other reason for this interpolation, is that unlike the collision data, the traffic data was rather dense (in my opinion). For the collision data I opted for a Kernel Density Estimation (KDE) approach. Here, I made the assumption that an area near a collision point must be less safe, and thus a collision within a certain radius of another collision, is more likely to occur.

In the second plot, realizing that we have a snapshot of a traffic count done late 2016 (600+ intersections), and we also have collisions for just 2016, we will repeat the plot with just the snapshot for the 2016 year:

Just the 2016 data: dangerous centers appear in Calabasas, Koreatown, South LA, Venice, Boyle Heights, North Hollywood. (arbitrary units, higher means higher probability)

We will do the same thing with 2015, 2014, 2013, 2012, and 2011, but animated to see how the most dangerous areas changed over time:

We can see year over year variation in probability of collisions, however some areas are highlighted throughout most of the year: Koreatown, Silverlake, South LA, East LA.

Using Flask and Plotly to build a Web App of n-Cov2019 Cases

Sun, 09 Feb 2020 02:38:41 GMT

>>> Please check out the web app here.

>>> And read the blog post on the science behind the model here.

Screenshot of web app as of 02.08.2020

Using Named Entity Recognition and Natural Language Processing to build a map of accumulated infections of n-Cov2019

Mon, 03 Feb 2020 04:23:03 GMT

I begin by scraping the following website for all its information and translating the text using Google Translate API: https://ncov.dxy.cn/ncovh5/view/pneumonia_timeline?whichFrom=dxy

The data consists of a header, a description, a timestamp of some sort, and the source where the information came from. Using the HTML library in Python I extracted these various sections of the page and placed them into a Pandas Dataframe.

After some cleaning, I used “Named Entity Recognition” to extract locations from both the descriptions and the titles. Since location is such a common use for NER, I decided to use an already existing model: DeepPavlov. I loaded this model and extracted all the locations and put them in their own Pandas columns in the data frame.

Next, I need to extract the latitude and longitude from the name of these locations. Using GeoPy, I extracted and averaged all the locations in both the header and description. For now, I opted to use only the location information in the header as from a quick glance, this tends to be the main region or entity where the cases happened, rather than the city levels or entities that might have tested the case in the descriptions.

Now, I was left with my ultimate bottleneck, text in the description like this:

“From 00:00 to 24:00 on January 31, 2020, Shanxi Province reported 8 new confirmed cases of pneumonia due to new coronavirus infection. As of 14:00 on January 31, 47 cases of pneumonia confirmed by new coronavirus infection have been reported in 11 cities in Shanxi Province (including 2 severe cases, 1 critical case, 1 discharged case, and no deaths). At present, 1333 close contacts have been tracked. 41 people were released from medical observation on the same day, and a total of 1101 people were receiving medical observation.”

From this I’ve already extracted this information:
Time: January 31, 2020
Location: Shanxi Province
However, I still need to extract the following information
New cases: 8 new confirmed cases of pneumonia (8)
Accumulated cases: 47 cases of pneumonia confirmed (47)
New deaths: no deaths (0)

In order to approach this, I needed to train a custom NER model. I used TagTog’s tagging and AI tool to do this work:

TagTog: Annotation Tool

This makes it easy to train by selecting the various information in a document. I created 8 entities and 4 relationships:

Entity	Description	Relations
new_case_I	New case added in location	New cases and its number
new_case_N	Number of new cases
acc_case_I	Accumulated case in region	Accumulated cases and its number
acc_case_N	Number of accumulated cases
new_death_I	New death recorded in region	New deaths and its number
new_death_N	Number of new deaths
acc_death_I	Accumulated death in region	Accumulated deaths and its number
acc_death_N	Number of accumulated deaths

I manually tagged 20% of the data to build the model. From this, I parsed the rest of the data through the model to extract the remaining information. From the initial document example, the model extracted the following information:

Text	Entity	Probability
‘no’	“accumulated_death-N”	0.874
‘8’	“new_case-N”	0.838
‘no deaths’	“accumulated_death-I”	0.698
‘1’	“new_death-N”	0.621
‘1’	“new_death-N”	0.550
’47’	“accumulated_case-N”	0.813
’41’	“accumulated_case-N”	0.714
‘8 new confirmed cases’	“new_case-I”	0.842
’47 cases of pneumonia confirmed’	“new_case-I”	0.640

We can see that the model with 20% of the data did fairly well. Numerically, we were able to pick up the correct new case number (8) and the correct accumulated case number (47).

With the rest of the data parsed, I used Plotly to build a final heatmap (data from morning of 01/28). We can see from this that the model has clearly picked up the epicenter and the surrounding virus activity nearby.

Heatmap of nCov2019 Accumulated Infections as of 01.28.2020

I plotted the accumulated cases over time:

Accumulated cases over time for nCov2019 Infections

Finally, I made a GIF heatmap of the virus activity over time (note the scale bar changes over time):

Animated heatmap over time. Scalebar changes over time.

Future:
In the imminent future, I plan on releasing a Flask app on my website with the data, stay tuned! Additionally, I will update it with the most recent data released from 01.28.2020.

Predicting a US Primary Election

Sat, 30 Apr 2016 20:15:26 GMT

This primary season has been rather interesting to say the least. There is a former TV reality star running a show, spewing words that would be normally an automatic disqualifier if mentioned by previous candidates in previous election cycles. There exists a particularly angry voter base that scapegoats their fears and economic insecurities – often with understandable sentiments – at a ruling class they perceive as the establishment. You have people fixated on the character and honesty (or appearance of honesty) of candidates rather than their policies. And, as with the ’08 election cycle, you have a sense of hope in some candidates, as well as a sense of unsettledness about the possible nominees from the two major parties. The character of this unsettledness could perhaps be best described by these recent polls:

1 in 4 affiliated Republicans say they could not vote for the GOP nominee if said nominee was Trump[1]. 33% of Sanders supporters, according to a McClatchy-Marist poll would not vote for Hillary if she was the nominee[2], and 25% of voters would abstain from voting for one of the major party candidates if the nominees were Trump or Clinton[3]. It sure is an election of unique perspective, and uneasiness.

For me personally, this election was the first time I realized how much math is actually involved during an election year. However, for me, it’s not the polling which is of actual interest but rather some of the commentaries made by political pundits following each contest:

Bernie does better in primarily white, non-diverse states
Hillary does better among older and richer voters
Bernie does better among independents, and in open/same day registration contests
Bernie does better in caucuses.

What if based on the last 38 contests that happened in U.S. states (excluding territories, democrats abroad, etc) we could provide a model to predict not just whether Bernie or Hillary wins one of the next 14 contests, but perhaps also the margin by which each candidate will win. (This will be focused on the Democratic primary).

This is exactly what I propose in this post.

The first weight factor I will propose is one that is based entirely on the logistics of the contest:

Is it a primary or caucus?
Is there same day registration?
Is it an open or closed contest?
What region is the contest held in?

For each of these factors I assign certain value of “pro-Bernie” points, or logistical information which I believe would provide Bernie a contest where he would perform better:

Factor	Comment and Value
Primary or Caucus?	Because Bernie does much better in caucuses, I will arbitrarily assign a caucus an additive value of 4.
Same day registration or registration with a deadline?	Same day registration drastically helped Bernie. I assigned this an arbitrary, additive value of 3.
Open or closed contest?	It was noticed that this had no effect on how well Bernie did in previous contests.
Region (South, West, N.E., Midwest)	South – (-1) NE – (2) Midwest – (3) West – (5)

Here are the 39 contests summarized by election factor, to understand why I picked these values:

Region	Count	Average of Bernie vote
South	13	31.38462
West	3	74.7
Midwest	14	57.65
N.E	9	52.66667

Primary Type	Count	Average of Bernie vote
Caucus	13	64.87692
Primary	26	41.14615
Open	23	48.26522
Closed	16	50.19375
Same day	14	61.42857
Deadline	25	42.128

There is a clear correlation between the region the contest is held in, as well as whether it’s a caucus or primary, and whether there is same day registration or not. However, whether or not the primary was open or closed did not appear to have a significant effect (perhaps it’s more of a bundled factor with whether or not the registration was on the same day and the primary was open vs. the opposite).

Here is how I divided each state by region in the US:

Each of these components would then be added into an “Election” Factor (EF), which when plotted against actual percentage of votes Bernie gathered in the state would yield the following curve:

This gave a positive correlation with rather decent R² value.

Now let’s get into the demographics of the state

The first “trend” that I often hear by political pundits is that Bernie does better in rural states. So I tested this hypothesis directly by comparing a graph of population density by state vs. the percentage of votes Bernie received in that state:

This was a rather weak trend, with a rather low R² value that I decided to dismiss this factor into my final calculations. I guess the pundits aren’t always right?

Next demographic of interest is race, which surprisingly shows a rather strong correlation with the percentage of votes Bernie received. Often during the primary race I heard that “Bernie won that state because it’s a mostly white rural state.” Well we already threw out the rural hypothesis, but maybe the white one has some truth to it. So I decided to plot the percentage of white population of a state vs. the percentage of vote Bernie received:

This demonstrated a slightly weak, but none the less non-ignorable trend between the lack of racial diversity in a state and the amount of votes Bernie received.

Now let’s do the same with the black population per state:

Now that’s a rather strong correlation…

Let’s do the same with the Asian population:

Again, not a huge correlation, although it’s worth noting that Bernie is not uniquely strong in diversity-lacking states, he did very well in Hawaii, a state that’s only 26% white.

What about the Hispanic vote?

With such a low correlation, it is safe to say that the Hispanic population in a state does not show a noticeable effect on the percentage of vote Bernie wins in that state. I’m going to rule this out of my model.

Now I will put all of the racial demographic variables together into a “Race” Factor:

Where m_i, b_i, R² is the slope, intercept and R² of the specific population segment i (white, black, asian). P_i is the actual population percentage of the segment i of interest. a_i is an arbitrary coefficient of influence I designated based on how much influence each population segment had on the overall trend (1 for white, ½ for asian and 2 for black).

I then scaled the values so that they would be the same order of magnitude as the election factor (1-10). The highest value of the E.F. was 12, so I wanted the highest value of the RF to also be 12.

This yields an RF value vs. Bernie vote curve of:

This gives a rather positive and strong correlation with R² of 0.75.

Now let’s combine both factors into a final score per state (FS):

In this S_i is one of the two factors of interest (EF, RF). I decided to remove the data point for Vermont as it was an outlier in my data (confirmed by a simple Q-test). The most likely explanation for Vermont is that as Bernie’s home state it gave him a much higher advantage unaccounted for in the E.F.

This gave me a final FS vs. percentage of vote for Bernie curve of:

This shows a very strong correlation between our engineered FS factor and the vote that Bernie received.

Now, we can test the remainder of the states:

States	Total Delegates	Registration	Type	Region	EF	% White	Black	Asian	RF2	FS	Bernie
California	475	Deadline	Primary	West	5	79.07	7.45	13.47	9.0	42.8	53.2
DC	20	Same day	Primary	N.E	5	37	47	4	0.0	14.9	34.9
Indiana	83	Deadline	Primary	Midwest	3	84.3	9.1	1.6	8.2	34.5	47.7
Kentucky	55	Deadline	Primary	Midwest	3	84	8	3	8.7	36.0	48.7
Montana	21	Same day	Primary	Midwest	6	90	1	0	12.2	55.9	61.8
N Dakota	18	Same day	Caucus	Midwest	10	88	2	2	11.7	66.1	68.6
NJ	126	Deadline	Primary	N.E	2	58	13	9	6.6	26.4	42.4
NM	34	Deadline	Primary	South	-1	40	2	0	11.7	33.3	47.0
Oregon	61	Deadline	Primary	West	5	76	2	6	11.7	51.2	58.7
SD	20	Deadline	Primary	Midwest	3	85	2	0	11.7	45.2	54.8
West Virginia	29	Deadline	Primary	South	-1	92	3	0	11.1	31.7	45.9

The ones highlighted in green are the ones I predict he will have a definite victory. The yellow could go either way and the red I predict he will perform poorly in. This is entirely based on election logistics and demographics of the state.

I guess we will find out May 3^rd when Indiana is up. Right now, according to polls, Bernie is behind by 6.6% points[4]. My model predicts he will lose by 4.6%. This agrees pretty well with the polls. Check back this chart May 3^rd to see how my model holds up!

Resources:

[1]http://www.rasmussenreports.com/public_content/politics/elections/election_2016/no_trump_no_show_for_33_of_gop_voters

[2] http://www.wsj.com/video/poll-33-of-sanders-supporters-wouldnt-vote-for-clinton/69C05055-85FE-4320-8D02-3EAC972CACD0.html

[3]http://www.rasmussenreports.com/public_content/politics/elections/election_2016/24_opt_out_of_a_clinton_trump_race

[4] http://www.realclearpolitics.com/epolls/2016/president/in/indiana_democratic_presidential_primary-5807.html