I encountered a hurdle where I needed to evaluate pairs of analytical tests data for hundreds of separate comparisons. Each test produced around 130 numerical features that vary in both presence and amplitude. Manual excel comparisons was considered for individual analysis; however, proved far too cumbersome to perform hundreds of times over. To solve this problem I utilized python to automate the analysis and iterate over each comparison.
Data Cleansing SQL Data Manipulation Analytics Iteration
Jarrad McKay
September 2022
Python Pandas Numpy Matplot lib PyoDBC SQL
Initially the analysis looked to compare two analytical results results with one another. I needed to know how well features compare between two tests, and how well did the magnitude of the features correlate.
Though comparison across the entire feature set was important, each test was composed of several different sub categories that needed to be compared in isolation as well.
Data was stored in a SQL database, and the list of comparisons was held in a basic excel file. The below code allowed for me to query appropriate data from SQL and excel,
iterate over the list of comparisons, perform all the necessary calculations, and produce a dataframe that holds every calculated comparison variable across every evaluated pair inclusive of the subcategories important to each comparison.
I then wanted to evaluate how the metrics changed when features under a specific magnitude were removed from the evaluation. The below function LimitTest()
allowed iteration over threshold values and the set average of each metric was captured through each threshold limit.
The below commented code reveals how these objectives were achieved.
Countries Worked In
Data Points
Dollars Saved
Coffee Drinked
Some of the most widely consumed chemicals within the Oil and Gas industry are drag reducing agents. Often the industry simplifies physical characteristics of these agents to small-scale bench-top laboratory friction reduction tests. I authored this paper to tie together decades of academic research on drag reducing agents to their practical application within well completions.
This work highlights meaningful physical traits of drag reducing agents, and their practical use applications to allow operators to make better informed decisions on their chemical selections and capital allocations.
Read moreClients spend millions of dollars annually stimulating wells to enhance production with the ultimate goal of getting the best return on investment. Well stimulation requires injecting fluid at high pressure and velocity into the reservoir to produce conductive fractures. This analysis was used to investigate the best return on investment when selecting a fluid composition for completions.
Read moreDeriving insights with Python. Automating multivariate comparative analysis of diagnostic results composed of samples comprised of 130 independent features for hundreds of comparisons.
Read moreWhile learning python programming I decided to dive into web development. This site is the product of my continual learning, and drive to expand my technical knowledge.
Read more