Seediscussions,stats,andauthorprofilesforthispublicationat:https://www.researchgate.net/publication/290428432
ApplicationofBigDatainPetroleumIndustry
Article·February2016
READS
206
1author:
HamedHamzeiy
IstanbulSehirUniversity
4PUBLICATIONS0CITATIONS
SEEPROFILE
Availablefrom:HamedHamzeiy
Retrievedon:20July2016
Application of Big Data in PetroleumIndustry
Hamed HamzehDepartment of Electronics and Computer Engineering
Istanbul Sehir [email protected]
January 12, 2016
Abstract
Oil and Gas industry is a scope that is full of dangers. Most of the compa-nies in this area deal with huge amount of data from different phases such asexploration, drilling production that are increasing dramatically over the time.Hence, by increasing the data from these kinds of things, they have to use thestate-of-the-art Big Data methodologies and technologies to analyse data toachieve better performance and reduce their costs effectively, improve businessefficiency and performance and also make technical decisions. It gains if theycan use real-time data that collect from the wells in drilling operations. At thispaper I will do investigate on different methods of Big Data Analysis that areused in Oil and Gas companies, such as Hadoop, Microsoft MURA platform,IBM InfoSphere, Oracle and their applications.
1 Introduction
Data is one of the important aspects in Oil and Gas industry. Every day, most of thecompanies deal with the large scale of the data. They are trying to find novel solu-tions to analyse them. They setup multiple sensors and RFID infrastructure on thesurface of the earth to collect data. Collected data consist of structured, unstructuredand also semi-structured data. The advantages of these analysis is improvement intheir productions from 6% to 8%. By integrating the historical data and also real-time data from different sensors, they can deal with the massive amount of data. Oilproducers can get more detailed data in real time at lower costs from unreachable
1
places, to improve oilfield performance. Oil and gas companies will need to improvetheir analytics abilities in order to participate in an industry.
First of all, we will discuss the structure of the oil and gas companies and we willsee that how they should set up Big Data infrastructure in their industry. Then, wewill talk about basics of Big Data that are used in this industry and how they canuse it to analyse the data and also decision making. To help rapid decision-makingin essential areas such as reservoir modelling, drilling, production optimization, andreal-time consumer marketing, analysis and insights which should be passed in realtime. After that, we will talk about different approaches that Oil and Gas companiesuse them to analyse their data such as Hadoop, MURA and IBM InfoSphere andOracle platforms. We will have more concentration on Upstream data, that consistsof Exploration, Development and Production. Also, we will review the Midstream andDownstream Parts of the Petroleum industry.
2 Related Works:
HDS’ property [9] of BlueArc proposed in September 2011 to get the better of differ-ent challenges of companies in terms of the storage. It has been served Oil and Gascompanies near 20 years. This methodology uses for maintaining high performanceof computing. HDS provides storage and also servers to process data and it can beprovide sustainable solutions.The other approach is DCA(Decline curve analysis)[9] that uses non-linear patternsto predict future outputs of the wells by using the historical data. Because of thatproduction from a well is not continuous and always it decreases by removal works.Therefore, Oil and Gas companies use different Machine Learning algorithms to anal-yse their data.
3 Deal with Big data challenges in Oil and Gas
companies:
Oil and Gas companies, collect and analyse their data from different resources[1][4][8]as follows:
1. Collecting data from different sensors during drilling stage.
2. Traditional enterprise data from operational systems.
3. Social Media.
2
4. Web searching designs.
5. Demographic data.
6. Historical oil and gas exploration, delivery, and pricing data.
By Increasing data from the sources that are mentioned above and also Accordingto figure 1, it’s clear that by using these infrastructures and also raising the volumeof data, these are not responsible for today’s needs in the petroleum industry.In this point of view, data management(Using Big data platforms), integration, col-laboration and performance management will be too much important.
Figure 1: Current IT infrastructure of the Oil and Gas companies without using theModern IT technologies related to Big Data.
First of all lets look at the different opportunities [1][2][8][10] of Using Big Datain Petroleum industry as follows:
1. Exploration: Using the state-of-the-art tools such as pattern recognition toanalyse data from different wells via sensors that are seismic data .They can use other methods to enhance exploration attempts. For example,geologists can improve their expectations by using historical data and also pro-duction data.
2. Real-Time decision making: Getting real-time data from multiple sensorsand using predictive methods to analyze data.
3. Drilling: Deciding abnormalities through the drilling functioning is very vitalin terms of improving the exactness of drilling phase.(The levels of Analysis ofdata from drilling is shown in figure 3. )
3
Figure 2: Upstream Big Data.
4. Production Operations: Using decision making by using real-time data col-lected from wells and sensors to decide which wells can produce more produc-tion. In this case, Oil and Gas companies will be capable to decrease their costseffectively.
5. Maintenance: Oil and Gas companies can combine the information such aspressure, size and temperature in order to forecast possible crashes. In addition,many upstream works take place in distant places, so being able to suggestconservation on critical benefits is too much important, mainly, if the workneeds to purchase of skilled tools.
6. Reservoir Engineering: Oil companies need to combine and integrate real-time data into the mechanical earth models. In this case, they are able tomore accurately forecast future oil accessibility and recognize oil reservoirs. Toprocess and analyze the real-time data, they can use MAP-R data program.
The MapR (Converged Data Platform) [2][9] can help process and analyze real-time data and help to expand a better consideration of the subsurface of theearth to promote more better drilling operations.
7. Enterprise: In terms of business, petroleum companies encounter to theskilled employees problem. In continue, we will see that how companies dealwith this problem by using Microsoft MURA platform [5][10] to fill the gapbetween new and experienced people.
4
8. Security: Oil and gas companies require to identify different events that couldspecify a crucial security menace or hacking operations in order to keep theirstaffs and tools in secure state. To recognize different events, they can usepredictive analytics that can help to detect these menaces in progress. TheMapR Converged Data Platform can help in recognizing threats in real-time byusing machine learning algorithms and anomaly detection methodologies andreduce the likelihood of such occurrences.
Monitoring and Prediction within Real-Time Oil and Gas Sensor Net-works:
The purpose is to Create an integrated platform, based on “real-time” data ana-lytics and predictive modeling to [5][7][8]:
1. To better control and secure serious real-time data streams, data engineeringand also process the control systems for the Oil and Gas industry.
2. Improving risk management and reducing the severity from the seismic modelto engineering and facilities operations.
3. Integration of different data sources.
4. Continuous monitoring and assessment of marginal circumstances.
5. Support for decision making when marginal conditions are not contented.
6. Upgrading prediction models automatically when they are required.
3.1 Big Data Use Cases In Oil and Gas Industry:
We have 3 different use cases in Oil industry concludes Upstream, Midstream andDownstream. Each of them has 3 subsets as they have shown in Figure 3.
Figure 3: Different Big Data use cases in Oil and Gas industry.
5
As you can see in Figure 3, Upstream Big data use case is included of 3 importantparts:
Upstream:
1. Exploration: In exploration part, they are using Hadoop infrastructure tostore, analyse and also visualize seismic data.
2. Production and Operations: Determining different abnormalities derivedfrom the wells to decrease crashes to achieve more production.
Midstream:
1. Environmental Monitoring: By using this strategy, they can forecast main-tenance(conservation) according to the stage of pollution outflow. It’s basicallydone by real-time consumption of data from the sensors.
2. Predictive Maintenance: It’s useful for recognizing the non-productive timeand also storing conservation logs and examination data in order to futurecharge risk. Because of the sensitivity of new sensor tools, they should beprepared to analyze the huge amount of data in the short time interval.
Downstream:
1. Refining: To get more appropriate outputs in refining phase, Oil and Gascompanies should enhance real-time sensor monitoring tools that joins all themachines to inspect the conservation based on their needs [1][4].
2. Retail: Providing a link between different types of tools like as sensors andRFIDs to improve services and run the everyday activity of gas terminals. Also,Improving RFID tools and recognition to enhance services to the customers.[5][11]
3.2 Advantages of using Big Data Platforms in Oil and Gascompanies:
1. Reducing costs and implementations: Most of the companies do not knowabout the Big Data infrastructures. Hence, they should have investments on
6
this area and also buy different new solutions to improve the efficiency of theirproducts to content better at this industry.Therefore, by using Big Data programs, they can decrease the number ofhardware-based systems because of using the new IT technologies that are basedon the cloud.
2. Less resources, more delivery: One of the interesting things for an Oil andGas Company is to reduce resources and getting more throughput. Using morehardware-based tools, need more engineering works; and in this case, spendingmore time to data analysis is not possible to companies. A we discussed, byusing cloud infrastructures, they can spend their time to analyse the data.
3. Integrated Views: As we mentioned later, data collected from wells andsensors, are in different formats. Hence they should use Big Data Platforms tointegrate them.
4. Improve information management to deal with the explosion of in-formation throughout the Upstream process.
5. Speed up all analytics performed during Exploration and Produc-tion.
6. Improve quality and time to visualize information by improving ac-curacy and also allowing the sharing.
3.3 Big Data infrastructures in Petroleum industry:
Using Big data can support the companies to promote the “digital oilfield” combinedworks that join operational technology (OT) [8] including information technology (IT)to intensify decision making and enhance operational and business performance.
Actually, Many oil and gas companies will need to select the state-of-the-art ITsolutions designed to tackle the particular difficulties of big data. They need technol-ogy that can collect, manage and analyze large and rapidly growing massive amountof data. In addition, they need solutions that can analyze a wide variety of datafrom different tools such as drilling sensors and unstructured data from logs. Newsolutions must help combine or integrate trading data with scientific data.Big Data consists of different concepts which are shown in figure 1. We want todescribe them to see how they can be involved in Oil and Gas industry. [1][11].
1. Volume: It refers to seismic data or how much data that a company has.
7
Figure 4: To achieve a wider range of business-improvement opportunities throughanalysis and optimization, field OT and corporate IT systems must be integrated.
2. Variety: It means different models of data that collected from wells via mul-tiple sensors, that can be structured , unstructured such as images, videos andsemi-structured data.
3. Velocity: it refers to real-time streaming data that are collected from drillingtools.
4. Veracity: Improve the quality of data by applying different combined(integrated)models or combining data from different phases such as drilling, seismic andproduction.
3.4 Structure of a petroleum company that is using Big Data:
As we discussed before, one of the major problems in oil and gas companies is thehuge volume of data. So that an industry and especially companies in the petroleumindustry should improve their infrastructure to analyse this amount of data.According to figure 2, a company should have this kind of architecture to use big data[8]. This strategy and architecture are based on exploration and production. In thisstructure, we have Data driven analytics and physical modeling driven.In physical modeling driven, companies should provide different physical models suchas optimizations, weather predictions, collecting seismic data, convert them and alsoanalyse.
In Data-driven analytics, companies should use different analytical models such as
8
Figure 5: Analysis of data from Drilling.
intelligent models to get real-time data, and also the important thing is to integratevarious data in a coherent manner.
3.5 Example of a company that is using Big Data Tools(ShellCompany):
Shell company By using fiber optic cables is able to transfer data to different serversfrom the sensors that are established on the surfaces. These servers are preserved byAmazon Web Services(AWS).To guarantee better working of machines in company, Shell uses Big Data by spendinglittle time in contrast to offline mode in order to failure detection [12]. By collectingreal-time data from different sensors and comparing the performance of differentdrilling tools, to reduce overheads in terms of working of drilling equipment, the restof the operations transported to the other drilling tools that are idle.
To understand all of the data in an appropriate way, it’s necessary to visualizethem. Hence, to provide more relevant information, Shell is working with IBM andDream-Works Hollywood. All of the data collected from different sensors are analysedby Artificial Intelligence tools expanded by Shell company and performed in 3D and4D maps of the oil pools. All analysis are performed in the cloud and also thevisualizations are quickly available to the attendants who are working at the localfactory.
In Shell company, there are 70 expert people who are working on data analysis andalso the other people in other branches in all over the world. The IT centers of thiscompany consist of different experienced people in mathematics, physics, Information
9
Figure 6: Big Data infrastructure for the Oil and Gas company.
Technology.
4 Big data Platforms used in Petroleum industry:
In this section we will introduce different applications and methodologies that oil andgas companies use them to analyse their data.
4.1 Hadoop infrastructure
Hadoop [1][4][6] is one of the most applicable means to analyse the huge amount ofdata. It gets the data as an input and splits them into multiple chunks and then eachcluster can process them by using many maps and reduce workers. The architectureof the Hadoop that is specially created to petroleum industry in Big Data Upstreamcase is shown in figure 4.
Basically, engineers collect data according to the status of the wells and pumpsvia physical examinations(frequently in distant places). This means that examineddata was sparse and difficult to access, especially pending the high value of the toolsand the possible health and security impacts of events.Data from the sensors, move to the Hadoop distributed file system through the wells,
10
pumps and also other tools, by considering the minimum cost.
As we said before, the main purpose of using Big data, is to combine the historicaldata and real-time data. Based on figure 4, we have historical sources(historical data)and non-historical sources(can be Real-time data). All these kinds of data transferto the distributed file system in the Hadoop. Into the Hadoop, different operationsperform on the data by using Data operating system(yarn). Also, they need to storethe old data into the data containers. At the end, data can be visualized by usingdifferent platforms which are shown in the visualization part in the figure 7.
Figure 7: Architecture of Hadoop in petroleum industry.
4.2 IBM implementations:
IBM is provided diverse solutions to help oil and gas companies to optimize opera-tions [8], improve business performance, tactical decision making.The greater number of data in oil and gas industries are based on XML that are notrelational. Due to this, they need to a way to convert these data to the relationalformat.IBM provides an open source platform InfoSphere [8], that analysis the data in its na-tive format(diversity of data). Most of the companies have a problem in collecting thedata from different formats, because of this, InfoSphere BigInsights lets organizationsto control any changes required to prepare that data for modeling and simulation.
11
Figure 8: The IBM big data platform offers an array of integrated capabilities toaddress the tremendous volume, variety, velocity and veracity of big data.
By using this model, companies can get the data and analyse them quickly in real-time to make the better decisions. One of the interesting things in this methodologyis that companies do not need to store the data to analyse. It’s one of the advantagesof this system that is a cost-effective approach.
4.2.1 IBMPureData
This innovative tool enables the Oil and Gas companies to have a deep and complexanalytics on the huge amount of data. Also, it capables to quickly analyse of terabytesof data.
4.2.2 IBM InfoSphere data exploration
Analysing and researching on the huge amount of data is too much difficult and itrequires complex computations.Also, It takes weeks or even months to do the searchon these amount of data. To tackle this issue, IBM has provided InfoSphere dataexploration to decrease this time interval.
12
4.3 Microsoft MURA Platform:
Microsoft has provided the united business-based platform to improve the perfor-mance of workers [5][10]. Also, this methodology is created especially for Oil and Gascompanies to overcome business-related data.
Figure 9: Guiding Principles of the Microsoft Upstream Reference Architecture(MURA).
Important Parts of the MURA architecture:
1. Real-Time Analytics: It provides different packages and facilities for dataanalysis.
2. People, Process and Information Integration:This part is one of theimportant parts of the system that creates an integrated program which avoidsthe users to send and export their works from one device to another. Hence,to combine different formats, the integrated software should be established intheir machines.
3. Self-Serve Business Intelligence:It provides users to have more delibera-tions into the raising number of relevant data that is collected.
4. Rich Interactive User Experience:Providing tools to share the experiencesof knowledge people. In another word, it’s useful for filling the knowledge gapbetween new and experienced people.
5. Mobility:Providing facilities to the administrators to work on different systemsin different locations by considering their needs.
13
4.4 Oracle Architecture Development Process (OADP):
In this section we will discuss about the Oracle platform [4] that is created for dataanalysis in Oil and Gas companies. This methodology is full of different products andeach product in this structure is created for a specific purpose.
Figure 10: How Key Oracle Products Fit in the Generic Architecture.
According to the figure 7, after collecting the data from sensors, they move to theNo-SqlDB and after that they go to the different parts of the architecture such asReal-Time decisions to analyse real time data and also CRM, ERP that are customer-based products.
Let’s review some concepts in this architecture:
1. ODI:Oracle Data Integrator is a complete data integration program that over-lays all data integration needs.
2. Oracle Real-time Decisions:It provides a real-time recommendation engine.
3. Oracle Big Data Discovery:It’s a Hadoop-based information discovery tool.
4. Endeca:It’s an information recognition tool and engine.
5. Exadata: It uses for enhancing the performance in Oracle database workloadsby combining the server, storage and network infrastructure.
6. NoSQL Database: These kind of databases have not schema and they aredesigned in order to rapid writes. These are useful to help high consumptionworkloads.
14
5 Summary and Conclusion:
In this paper, we discussed the application of Big Data in Oil and Gas industry andwe saw that how different methodologies can affect on it. There is mentioned thatOil and gas companies collect their data from different resources which are multi-structured, hence, they should use novel tools to integrate and analyse them. Wetalked about different opportunities that can be provided by Big Dat tools such as,exploration, maintenance, security and so on. Then, there is mentioned that there are3 different Big Data use cases(Upstream, Downstream and Midstream). After that,we talked about different methodologies such as Hadoop, IBM InfoSphere, Oracleand Microsoft MURA. We saw that all of these infrastructures have specific effectsin different parts of the Oil and Gas companies.In reality, the limited number of companies are using those novel methods, and mostof them as it said, do not know or they don’t want to use them. All in All, by usingBig Data and by considering the huge amount of data that are producing every dayat this industry, it’s necessary to all companies to use these technologies to collectand analyse their data.
6 Future Works:
Until know, we discussed the latest methods of Big Data that are used in Petroleumindustry. Actually, by increasing the requirements and also developing the othermethodologies, companies will tend to other new infrastructures to enhance theirproduction and reduce the costs.
15
References
[1] S. Singh and S. Pandey, R. Shankar, A. Dumka ”Application of Big Data An-alytics to Optimizethe Operations in the Upstream Petroleum Industry”. 20152nd International Conference on Computing for Sustainable Global Development(INDIACom).
[2] R. Bertocco and V. Padmanabhan and ”Big Data analytics in oil and gas Con-verting”.
[3] A. Baaziz and L. Quoniam and V. Vasilak ”Enhancements Case of Early Kick De-tection while drilling of the oil or gas wells.”. International Journal of Innovationand Applied Studies (IJIAS), Vol. 4 No. 1, Sep. 2013.
[4] ”Improving Oil and Gas Performance with Big Data Architect’s Guide and Ref-erence Architecture Introduction”. ORACLE ENTERPRISE ARCHITECTUREWHITE PAPER, APRIL 2015
[5] A. Hems and A. Soofi and E. Prez ”Drilling for New Business Value How inno-vative oil and gas companies are using big data to outmaneuver the competition.”.Microsoft, May 2013.
[6] ”Hadoop is Transforming Oil and Gas”. http://hortonworks.com/industry/oil-and-gas/
[7] A. Baaziz and L. Quoniam ”HOW TO USE BIG DATA TECHNOLOGIES TOOPTIMIZE OPERATIONS IN UPSTREAM PETROLEUM INDUSTRY”. In-ternational journal of innovation, 2013.
[8] ”Tapping the power of big data for the oil and gas industry”. IBM Company,2013, IMW14680-USEN-01.
[9] J. Feblowitz and L. Quoniam ”The Big Deal About Big Data in Upstream Oil andGas”. IDC Energy Insights, Sponsored by: Hitachi Data Systems, October 2012.
[10] ”The Microsoft Upstream Reference Architecture”. Microsoft Corporation, 2011.
[11] M. Ferguson ”Architecting a big data platform for analytics”. Intelligent Busi-ness Strategies, October 2012
[12] B. Marr ”Big Data in Big Oil: The amazing way shell uses to drive businesssuccess”. http://www.smartdatacollective.com/bernardmarr/358203/big-data-big-oil-amazing-ways-shell-uses-analytics-drive-business-success
16
