Skip to main content

Polybase Blog - Introduction


Overview:

This Polybase blog series is all about the use of Polybase Technology in today’s era to be able to take advantage of the Data(Relational and Non-Relational) by using T-SQL only. Data whether Big or not is the lifeline to many different sectors to cope up with Production, Maintenance, Predictions, Taking Precautionary Measures, Customer Satisfaction, Customer Retention, Sales, Revenue Generation and many more.

In this Polybase Tutorial blog we will learn about what is Polybase and in the succeeding blogs you will discover how it is being implemented, how to do Polybase Configuration, what are Polybase Limitations, Polybase External Table and how to use Polybase Scale-Out Group to distribute the T-SQL queries.

Introduction and Use Case:

Many industries are migrating towards Cloud based set up. The reason is quite simple less cost and scalability. But some tends to keep data in both Relational (Tabular) as well as Non-Relational (Non-Tabular) databases, and hence a need to merge at some point of time the both as per the requirements. So, here is the good news for all those who have MS SQL Server 2016 or above in their set up along with the data in the Cloud.

With the introduction of Polybase Technology Microsoft gives us an extension to be able to use the data in the Cloud and merge Big Data Sources with the Tables in SQL Server just like any other Table, to get more out of the data enhancing customer experience. With some little effort and WITHOUT making any change to the existing application you will be able to take advantage of the data of both the worlds Relational and Non-Relational using T-SQL only which are one of the many Polybase benefits. Let’s have a look at the figure to have a functional understanding of the Polybase.

Polybase, Polybase Blog, Polybase Tutorial, What is Polybase

Fig 1. How SQL Server uses Polybase Technology to Interact with External Big Data Sources

If you look at the figure above, it will give you a high level view of how Polybase is making it easy to access External Data Source by using T-SQL only. PolyBase enables your SQL Server instance to process Transact-SQL queries that read data from external big data sources. Polybase in SQL Server 2016 and higher can access external data in Hadoop and Azure Blob Storage,and Polybase SQL Server 2019 can be used to access big data sources like SQL Server, Oracle, Teradata, and MongoDB. Polybase is a simple idea that you only need to use simple T-SQL rather than learning a completely different language. With the use of T-SQL you will be able to merge both Relational and Non-Relational Databases.

So, I hope this blog will help you get a clear understanding of what is Polybase and why it is being used, in the coming blogs we will learn about what are the requirements to Install and Implement Polybase and how it is being implemented. Also, we will learn about the Polybase Scale-Out Group and how it is used to increase the computation power of existing T-SQL queries.

About Amlgo Labs : Amlgo Labs is an advanced data analytics and decision sciences company based out in Gurgaon and Bangalore, India. We help our clients in different areas of data solutions includes design/development of end to end solutions (Cloud, Big Data, UI/UX, Data Engineering, Advanced Analytics and Data Sciences) with a focus on improving businesses and providing insights to make intelligent data-driven decisions across verticals. We have another vertical of business that we call - Financial Regulatory Reporting for (MASAPRAHKMAEBAFEDRBI etc) all major regulators in the world and our team is specialized in commonly used regulatory tools across the globe (AxiomSL Controller ViewOneSumX DevelopmentMoody’s RiskIBM Open Pages etc).We build innovative concepts and then solutions to give an extra edge to the business outcomes and help to visualize and execute effective decision strategies. We are among top 10 Data Analytics Start-ups in India, 2019 and 2020.

Please feel free to comment or share your views and thoughts. You can always reach out to us by sending an email at info@amlgolabs.com or filling a contact form at the end of the page.

 



 

 

 

 


Comments

More Popular Posts

Amlgo Blog - Experience The Experiments

Amlgo Labs Blog  is a step towards our vision to share knowledge and experiences, Amlgoites accept every challenge very enthusiastically. We do experiments, we fail but we learn and build complex solutions to help our clients solve their problems in Data, Analytics, Prediction, Forecasting, Reporting, Designing and Development area. During this process we enjoy immense learning everyday and we have decided to share our thoughts, learnings, experiments and experiences so that we don't work in silos and contribute the best of our knowledge towards community and learn more by views and reviews. This website is maintained and brough to you by  Amlgo Labs Professionals .   Our Strong Basics -  1)   KISS (Keep It Simple and Straightforward) :  We believe most of the problems can be solved by keeping things simple and straight. This is the learning we had in past, sometimes we try to solve technical problems using high end algorithms and complex codes but this results into complications.

Polybase : Polybase Scale-Out Group

In the last blog, we discussed the Introduction of the Polybase and the Implementation process of Polybase in SQL Server . PolyBase Scale-out Group consists of multiple virtual machines, each having its own SQL server instances which help in parallel processing and distribution of data. Data loading and query performance can increase in the direct proportion of the number of SQL server instances on each virtual machine.

Financial Regulatory Reporting

This blog is an introduction to the Regulatory Reporting. Regulatory reporting is mandatory activity banks have to perform with the coordination of Treasury, Group Finance, IT, and business lines. Regulators across the globe depend on accurate and timely submission of various Risk and non-risk reports by banks to measure the overall health of the banking sector.

Polybase Installation for Scale-Out process

This part is the continuation of the previous blog about the introduction of  Polybase Scale-Out Group . As we have discussed in our earlier blog PolyBase enables your SQL Server instance to process Transact-SQL queries that read data from external data sources. SQL Server 2016 and higher can access external data in Hadoop and Azure Blob Storage. Starting in SQL Server 2019, PolyBase can be used to access external data in SQL Server, Oracle, Teradata, and MongoDB.

Qlikview tool for Financial Data Analytics

QlikView is a Business Intelligence and Data Visualization tool used for getting relevant, actionable, and timely data that help companies in taking the right decisions. Other competitor tools are Tableau, SAP Business Objects,  Microsoft Power BI, IBM Cognos Analytics. Amid uncertain economic conditions, changing dynamics, and a crisis of confidence in the financial markets, customer focus and risk management continue to be key drivers for profitability in banking. The urgent need for information to help address these priorities compels banks to attempt complex data integration and warehouse initiatives.QlikView in-memory analysis helps in faster data integration of data coming from disparate data sources and provides analytical capabilities to business users. The use of the Qlikview tool for financial data analytics is explained as below: Day On Day Variance :  The data analytics team within the Finance department needs to do DoD ,  Month-over-month, Quarter-over-Quarter, YT