I recently started biking…in Los Angeles. Los Angeles could be one of the most bike-friendly cities in the United States: it boasts plenty of annual sunshine, it has a mostly flat terrain, and quite mild weather. Unfortunately, many parts of the city lack proper bike infrastructure, making it dangerous when it comes to traffic collisions and traffic violence. In 2021, over 230 people lost their lives to the streets of Los Angeles.

With this in the background, I wanted to figure out the most dangerous streets in Los Angeles. I had been following the Streets For All Twitter account for a while and knew that they had a bot which posted any traffic violence against involving pedestrians or bikes. This account had records dating back to just over one year. So I started with that (old results here). I used the Twitter API to extract all of the Tweets from the bot account, and then used the Google Maps API to convert the text-based location to a series of latitude and longitude.

I soon stumbled upon LA City’s records, dating back 2010-Present. This uses MO codes to detail the accident. After converting the MO codes for vehicle collision with pedestrian, and bikes, I had all the data needed to plot all collisions:

For plotting, I am using Plotly with the Mapbox API.

How does this look year over year since the data started? First, a quick decline occurs after 2013 followed by similarly looking collision patterns year over year. We can see some streets always have a a dense amount of collisions. Venice, Figueroa, Pico, Olympic, Wilshire, Santa Monica and Vermont appear to be big incident street year over year.

Of course, many have pointed out that the pandemic has switched typical traffic flows for city, as such, here is the pre-pandemic vs. post-pandemic collision map densities:

2017-2020	2020-2022

Just because a street has more absolute collision does not mean it has increased probability of collisions. To capture the probability of collisions, I’m going to make an assumption: that streets with more vehicular traffic should have more chances of vehicle-pedestrian interaction, so if I normalize the collision by the traffic flow, I should arrive at a value that is more representative of the probability of collision.

Luckily, LADOT has data on such traffic volume. Unfortunately, this data appears to start in 2011 and stop in 2016. It does not survey the same street year over year and seems sporadic in the streets surveyed over time. It has not been updated since 2016 giving a mixed snapshot of the traffic throughout Los Angeles.

The first plot will assume that the traffic from 2011-2016 more or less is constant throughout the collision data from 2010-2020:

To make a plot like this I had to engineer a trick: the data from the traffic flow does not overlay perfectly from the data from collisions, naturally. In order to be able to divide both data I needed to figure out an interpolation scheme for each. For traffic flow data I did a ‘nearest’ interpolation grid approach. I did this for two reasons, first I thought of the traffic flow data as a ‘graph’ in that the flow from one point to its nearest neighbor must be related: cars must travel between these points to an extent. The other reason for this interpolation, is that unlike the collision data, the traffic data was rather dense (in my opinion). For the collision data I opted for a Kernel Density Estimation (KDE) approach. Here, I made the assumption that an area near a collision point must be less safe, and thus a collision within a certain radius of another collision, is more likely to occur.

In the second plot, realizing that we have a snapshot of a traffic count done late 2016 (600+ intersection), and we also have collisions for just 2016, we will repeat the plot with just the snapshot for the 2016 year:

We will do the same thing with 2015, 2014, 2013, 2012, and 2011, but animated to see how the most dangerous areas changed over time:

We can see year over year variation in probability of collisions, however some areas are highlighted throughout most of the year: Koreatown, Silverlake, South LA, East LA.

>>> Please check out the web app here.

>>> And read the blog post on the science behind the model here.

I begin by scraping the following website for all its information and translating the text using Google Translate API: https://ncov.dxy.cn/ncovh5/view/pneumonia_timeline?whichFrom=dxy

The data consists of a header, a description, a timestamp of some sort, and the source from where the information came from. Using the HTML library in Python I extracted these various sections of the page and place it into a Pandas Dataframe.

After some cleaning, I used “Named Entity Recognition” to extract locations from both the descriptions and the titles. Since location is such a common use for NER, I decided to use an already existing model: DeepPavlov. I loaded this model and extracted all the locations and put them in their own Pandas columns in the data frame.

Next, I need to extract the latitude and longitude from the name of these locations. Using GeoPy, I extracted and averaged all the locations in both the header and description. For now, I opted to use only the location information in the header as from a quick glance, this tends to be the main region or entity where the cases happened, rather than the city levels or entities that might have tested the case in the descriptions.

Now, I was left with my ultimate bottleneck, text in the description like this:

“From 00:00 to 24:00 on January 31, 2020, Shanxi Province reported 8 new confirmed cases of pneumonia due to new coronavirus infection. As of 14:00 on January 31, 47 cases of pneumonia confirmed by new coronavirus infection have been reported in 11 cities in Shanxi Province (including 2 severe cases, 1 critical case, 1 discharged case, and no deaths). At present, 1333 close contacts have been tracked. 41 people were released from medical observation on the same day, and a total of 1101 people were receiving medical observation.”

From this I’ve already extracted this information:
Time: January 31, 2020
Location: Shanxi Province
However, I still need to extract the following information
New cases: 8 new confirmed cases of pneumonia (8)
Accumulated cases: 47 cases of pneumonia confirmed (47)
New deaths: no deaths (0)

In order to approach this, I needed to train a custom NER model. I used TagTog’s tagging and AI tool to do this work:

This makes it easy to train by selecting the various information in a document. I created 8 entities and 4 relationships:

I manually tagged 20% of the data to build the model. From this, I parsed the rest of the data through the model to extract the remaining information. From the initial document example, the model extracted the following information:

We can see that the model with 20% of the data did fairly well. Numerically, we were able to pick up the correct new case number (8) and the correct accumulated case number (47).

With the rest of the data parsed, I used Plotly to build a final heatmap (data from morning of 01/28). We can see from this that the model has clearly picked up the epicenter and the surrounding virus activity nearby.

I plotted the accumulated cases over time:

Finally, I made a GIF heatmap of the virus activity over time (note the scale bar changes overtime):

Future:
In the imminent future, I plan on releasing a Flask app on my website with the data, stay tuned! Additionally, I will update it with the most recent data released from 01.28.2020.

Background:

In my first year of undergrad, I worked at iD Tech Camp. I taught game design, Java programming and a few other technical and computer science classes to primarily middle school-aged students. Consequently, with a computer at their fingertips, many of these students would wander off on the internet or play computer games. To overcome this, I built a Java-based application that would connect to, at the time, this new service called Firebase and load various commands that I could insert on my version of the app. I could at the touch of a button close anyone who had a browser window open, blackout a computer screen, send a file to a computer and more.

Main concept:

In order to understand some of the concepts behind blockchain a bit better, I decided to make my own Python-based blockchain, with Flask as a REST API package, for peers on my network to communicate to each other.

I wanted to attempt the same concept as I did at iD Tech Camp, but using an immutable, distributed, database system (ie. Blockchain) to store my commands.

The framework behind the code is that if the seed URL does not currently host a blockchain (ie a valid REST request cannot return a blockchain JSON) then the genesis chain is created, and the peer gets a “TEACHER” role. The password for this role (using a sha256 library) is saved at that peer. Only a person with this username and password combination and this teacher role can add blocks to the chain.

Anyone who connects to this blockchain automatically gets assigned a “STUDENT” role, and anyone who connects to any peers on this chain also gets assigned a “STUDENT” role. Anyone can point a REST request at any peer on this chain to add a block (with the right user/password combination) with a list of commands and more.

When a block sits unconfirmed on a peer, the peer keeps “solving” it until it can be added to the blockchain. Here, since the point is not to solve a complex problem, like traditional blockchains, rather, the problem is simple as there is enough built-in security to make the chain immutable. Proof of Work: the sha256 library creates a hexadecimal string from the block object with a dynamic variable that changes until the hexadecimal string matches certain requirements. Since this problem is not too complex, it will not take more than one second. Once the peer has solved the block, it sends the block and the answer to everyone who is connected to it and so forth:

Built-in security and drawback:

The way authentication works on the chain is that every time a new peer (N) connects to an existing peer (E), that existing peer (E) now hosts the password for the new peer (N) on the chain. Whenever N needs to authenticate again, it can connect to any peer on the network, but these peers pass on the requests for authentication to the other peers until it reaches E. E then decides if the authentication is correct. This has built-in security because it means the password is delocalized – it is not stored in some central peer or select few peers. Furthermore, the password is never transmitted over the network, it is only known by E and the N.

The drawback is obvious, if E every goes offline, the peer N can no longer authenticate on the chain and must get a new identity. In future versions, the credentials could be stored in E as well as a trusted or random backup peer.

A modular approach to commands:

The code approaches commands in a modular way. A command can simply be written as a Python package and dropped in the ‘mods’ folder. This allows the opportunity to decide the level of complexity as well as ‘access’ each peer has on the network. It also allows peers to block the ability of the chain to run a command locally. It also makes it easier to add functionality and maintain the codebase. As long as a command with a module name matches a Python package in the ‘mods’ folder, the command can be executed.

Future:

In the future, I plan to make use of back up peers for hosting the password in case a peer goes offline. Additionally, I plan on using the ‘inspect’ features of Python to decide if a peer is trusted enough to connect. This can be used to ensure that the codebase that connects to the network is the same that the network is running.

The code is available on my GitHub: https://github.com/mhsiron/Crypto_CControl/

I analyzed over 12000+ job approval polls split between adult democrats and adult republicans to extrapolate how ‘polarized’ the US became over time.

I propose a simple ‘dis-unity’ ratio that is the job approval percentage by the party in charge over the job approval percentage by the opposing party. The lower the number, the more united each party members are in their view of the president, while the higher the number, the more dis-united the party members are in their view of the current president.

Here is the result from Obama’s 2 terms and Trump’s 1st up to February:

Some key findings:

• Obama started with a more united America in his first term than Trump’s first term or Obama’s second term.
• During Obama’s first term, the country became more divided
• During Obama’s second term, the country became more united

Caveat: This ratio does not allude whether the current president is generally liked or disliked at any time, it only alludes the disparity between both party member’s perception of the current president, and thus gives an insight into the polarization of the nation during a presidency.

I am awaiting data from other presidencies to see if there is a general trend between presidents or over time. I will also add major events in the timeline that might account for some spikes.

Background

After a semester at UC Berkeley learning various machine learning and data science tools, I’ve decided to re-examine the model I built half a year ago to predict the remainder of the primary elections at the time.

I will be using the same key data:

• Geographic Region
• Election type (primary vs. Caucus, open vs. Closed)
• Demographics

In the previous model, I was using overall state-based demographic data since I did not have the computational skills at the time to handle more than 50 rows of data. However, with the Python skills I acquired over the semester, I decided to improve my model by adding more demographic and election data by using county level information provided by the US Census Bureau.

Instead of manually deciding which variables I think would exert the most influence on my model, I decided to let the model figure it out. I tried using both TensorFlow’s Convolutional Neural Network (CNN) using the Keras wrapper, as well as SKLearn’s Decision Tree Regressor.

Explanations of Algorithms

There is a key difference between the two algorithms:

Decision Trees	Convolutional Neural Networks
How it works: Decision trees can be thought as a collection of ‘if-then’ statements. They take an input data set and try to match the output data-set through a tree-like structure of if-then statements. Each node on the tree is known as a ‘leaf’ and each leaf assigns a value in a regression tree. The algorithm finds the best place to create a split in order to minimize the loss function (error) in the actual output vs. the output the decision tree creates. This is very similar to the game of 20 questions – you have to find the best questions in order to optimize the tree for new, unseen data.	Neural networks are fundamentally different from decision trees. Rather than being ‘linear’ in their progression, from starting input to ending output, the data goes back and forth between ‘neurons,’ before returning to the output layers. However, having very large inputs will create very large number of hidden neurons between the input and output layer. To reduce the number of neurons, we create a convolutional neural network. In a CNN, an input layer is reduced to one neuron as it progresses through each layer. Additionally certain variables might add weight on others in the network.
Problems: One major problems with decision trees is over-fitting your training data. While over-fitting might results in 100% accuracy with your training data this could leave to catastrophic results with unseen data. One way to limit overfitting is by limiting the depth of the nodes, or pruning (removing random leaves) after overfitting	Problems: CNNs are an active area of research and are still poorly understood. They are often over specified for very specific data and might not work well on new data. This is because its hard to predict or figure out which type of layer, or activation function might work best for certain applications. Often people will build two completely different architecture of CNN that will work well with some data sets but not up to par with others.

I decided to do a CNN instead of Recurrent Neural Network (RNN) because I believed my input data to not have much inter-correlation between each features. However, I will be testing an RNN in the future because I am still curious about the possible results.

Model Summary

I began by creating a data set that combines the county vote results with the demographic and election data. I then separate the data into states that had already had their elections by March 22^nd, and states that had yet to hold an election. I only took into consideration the democratic primary results. I further split the data into an 80% train-test ratio to not overfit both models.

For the CNN model, I built the model using 4 dense layers, with a sigmoid, softmax, and hyperbolic tangent activation layers. These layers are friendly to continuous, regression data. This created a model with almost 70 thousand parameters.

For the decision tree, regression model, I set the max depth to be 30 leafs, so as to not over-fit, and set the maximum features to be the square-root of the input features. I also used sklearn’s AdaBoostRegressor. This helps with continuous data as it provides a smoother output instead of a step-function output, by superimposing multiple decision tree (in my model, 1000 decision trees).

Results

To visualize the results, I created an output graph for each model of the predicted vs actual election results. The more accurate, the more the slope would approach unity:

Decision Tree Regressor	Convolutional Neural Network
Mean Error	Mean Error
+1.19%	-6.54%

Here is the state by state prediction error for both models:

While the error for the DTR model was more centered about zero, it provided more catastrophic results (above 5% error) than the CNN model. If the CNN model was linearly calibrated by 6% at the very end, it would have had two less catastrophic results, and would have been significantly better. Overall, both of these models resulted in more problems than the state-wide analysis. I attribute more data to more error as perhaps a case of Simpson’s Paradox.

However, perhaps by combining a linear combination of these two models, an even better model could be made than the previous model with just state-wide data. There are many more variables that I could further explore in the DTR and CNN library as well, that could perhaps optimize this model further.

*This calculation was achieved by weighing each county’s population with their votes, which may not be the same results from the published voting results by the state but is more accurate for the data used in these models.

Interesting connection – GOP vs. DNC

Just for curiosity, I decided to run the same CNN and DTR model on the GOP:

Decision Tree Regressor	Convolutional Neural Network
Mean Error	Mean Error
-18.02%	-17.4%

Both of these models outputted predictions with significantly greater error. I interpret this to mean that democratic voters fit more ‘neatly’ into specific demographic groups outlined by the census data than GOP voters.

Interesting connection – weights

I decided to further analyze in the DTR Model which variables were most prominent in calculating the percentage of votes received to each candidate in the DNC primary.

I outputted the weights from the trained DTR Model:

The region of the voter has a significantly greater impact (by one order of magnitude) on the results. As expected, the geographic south and west voted very differently in the democratic primary. Perhaps less expected, was the importance of the election type (open, closed, caucus, primary) as well as the racial make-up and certain odd economic factors (retail sales, median value of housing) of each county.

If you would like to see my Jupyter notebook or data set, please contact me!