Data Science Overview

Data science is the ability to capture and process raw data, and then analyze, visualize and communicate the processed information as insights to the stakeholders that brings in the value by enabling businesses in making critical decisions.

The Data Science Life Cycle

Now that we have a high-level understanding on what exactly is data science, it becomes extremely important for us to also understand the data science life cycle, and this will help us in building our knowledge on data exploration as we progress. This is more in line with the data analytics using Azure Data Explorer, but overall the underlying concept remains the same.

At the first place, we should have the data to be used for analysis. This is where the data is captured in volumes from different sources. This can be from a streaming data source, or a CSV or even programmatically generated based on different business processes set forth using different programming languages.

The data captured is then cleansed and maintained as per the architecture defined. The data architecture is created as per business needs to bring out the most relevant information at the most appropriate time, quickly. There are definitely data retention and other policies, which comes into the picture. Also, the performance and permissions on the database play a crucial role. As the next step, exploration is performed on the maintained data to identify anomalies and outliers, discover patterns, and other statistical analysis.

Once the data analysis is completed, it is finally time to visualize and share the results with the business stakeholders so that they could make key business decisions based on the insights. Sharing results can be done in multiple ways. We can share the data in the tabular format, or in the .csv format, or as a Power BI dashboard, and so on and so forth. It all depends on you as a data scientist, on how you want to present your data to be easily understood. You should ensure that the information you have created to be shared as reports, or as dashboards answer all relevant questions from the business stakeholders to drive strategy in the organization using relevant information.

Data Exploration and Visualization

Let us now take a moment to understand why exploration and visualization of a large volume of data important. And the answer simply is to resolve analytically complex issues.

Data Science is about how intelligently a data scientist can creatively mine the data in order to maximize the profitability for the business. At the core is certainly the data. Businesses have been investing heavily on data science in order to bring value to the business, keep it abreast as per current market trends and be future-ready.

If we look at some examples – When you visit an online store, you will see recommendations as per the current trends or suggestions based on your recent history. These are all smart data products developed to generate algorithmic results. These products can do predictive analysis, analyze end-user sentiments, etc. In other cases – it can be used in finding out anomalies, fraud detection, finding out patterns, and statistical data.

So now you can imagine the plethora of opportunities for a data scientist, and these opportunities are only going to increase. Data scientists are considered to be the backbone of any business as they are the ones who bring in the real-time information, which becomes the basis for making critical business decisions to drive growth, tap and grab new business opportunities, identifies areas of improvements, and so on and so forth.

Data Mining Phases

Data scientists should have the analytical ability as well as the business acumen so that they can effectively mine the data, clean all irrelevant data, and create meaningful outputs on a large volume of data. They should be able to bring in the structured or unstructured data from all relevant sources and then act on it.

There are different phases or you can say steps for mining the data. They are –

1. Define Critical Problems – This is where, as a data scientist, you will have to zero-in on the critical questions, which needs to be answered, which can be prudent in driving the business forward.
2. Identify Data Sources – To have the critical questions identified in step – 1 above answered, you have to identify the sources from where you can get all the relevant information.
3. Prepare Data – There are large volumes of data coming in from different sources and needs cleansing, formatting, and structuring as per the defined architecture, and this is what is exactly done in this phase.
4. Data Modelling – Once the data has been structured, and formatted, it is then time to model the data, wherein, as a data scientist you need to select or define algorithms, which performs analysis on the data.
5. Test Model – This is the fifth phase, where we use sample data, which is near real-time to test the data model defined in the previous step. These tests are performed multiple times on different datasets to confirm the results achieved are correct and the model defined are working appropriately.
6. Verify and Deploy – In the final step, after the model has been verified, data visualizations are done and the model is deployed to start providing business values.

---

Part – 2: Understanding Azure Data Explorer

Part – 3: Azure Data Explorer Features

Part – 4: Azure Data Explorer Service Capabilities

Part – 5: Creating the ADX Environment

Part – 6: The Kusto Query Language

Part – 7: Data Obfuscation in Kusto Query Language

Part – 8: Data Ingestion Preparation: Schema Mapping

Part – 9: Overview of data ingestion in Azure Data Explorer

Part – 10: Managing Azure Data Explorer Cluster