Shreenath Nair, Head of Strategy- APACThis is where WhereScape steps in. The company provides data warehouse automation tools that cut the time to value for data infrastructure projects by as much as 80 percent. These tools are also tailored for Microsoft SQL Server, Microsoft Azure SQL Database, Microsoft Azure SQL Data Warehouse, Azure Synapse, and Microsoft Analytics Platforms System (PDW). “Data teams can automate the design, development, deployment, and operation of high-quality data infrastructures in the cloud, such as data warehouses, data vaults, data lakes, and data marts to significantly lower the cost and risk associated with enterprise data warehousing projects,” says Shreenath Nair, Head of Strategy-APAC at WhereScape. “We have been adding tremendous value to each customer by supporting their cloud migration initiatives.”
WhereScape’s applications build powerful data warehouses rapidly and cost-effectively. They automate 95 percent of the hand-coding typically required in data infrastructure development and automatically produce high-quality technical documentation with a touch of a button. “With traditional ETL tools, two developers would take 3-6 months to bring a (small-size) data warehouse from initial design through implementation, testing, and into production. But with WhereScape’s applications, developers can accomplish a task of similar magnitude in 9-10 days,” adds Nair.
Further, WhereScape provides Data Vault Express that automates data vault implementation at previously unseen levels of speed and quality of data warehouses. With changing business requirements, the company offers solutions for automating both dimensional and data vault modeled data warehouses. Companies that move WhereScape’s applications to the cloud
Data teams can automate the design, development, deployment, and operation of high-quality data infrastructures in the cloud, such as data warehouses, data vaults, data lakes, and data marts to significantly lower the cost and risk associated with enterprise data warehousing projects
can have several advantages.It’s a pretty straightforward process as there is no rewriting involved—most of it being just conversions. As a result, all the logics, data warehouse structures, and everything else migrate into the cloud.“We have made enhancements to facilitate the multi-platform compatibility of our products. For instance, code generation is decoupled from the software engine itself and is now done through the use of templates,” states Nair. There are templates available for major PaaS vendors, including Microsoft Azure SQL DB and Microsoft Azure SQL DW. The company also provides templates for exporting data to and reading from Azure Blob Storage. “Our engineers have worked hard to ensure that our data warehouse automation applications are future-proof and always relevant,” he prides.
WhereScape has a partnership with Microsoft that enables customers to harness big data and analytics on-premises, in the cloud, and in hybrid environments. As such, WhereScape’s products ensure that customer database systems operate efficiently, secure data from unauthorized access, and categorize data so that it can be translated into business insights.
With such a unique approach to data infrastructure projects, the company envisions being the market leader in data automation in the Asia Pacific region, especially Southeast Asia. There has been substantial growth in data automation adoption rates within financial services, manufacturing, and telecommunications industries, specifically in Southeast Asia. Much of it is attributed to the global pandemic and technology advancements.
“Manufacturing and/or financial services businesses need the data now more than ever to reduce their operational costs and maximize profit. Telecommunications companies are vying to capture major market share due to the advancements in 5G and the infrastructure needed to support the imminent growth; they need to derive insights from data faster than ever before to squash competition,” says Nair.
High-growth companies are increasingly employing the computing power of the cloud (such as Microsoft Azure) to harness the resources required to support bold initiatives around Artificial Intelligence and Machine Learning. The primary step in setting up a modern data infrastructure, in such a scenario, is to deliver a sustainable Microsoft Azure SQL Data Warehouse in record time, which is impossible without data warehouse automation tools like WhereScape. Going forward, the company has planned to enhance the enablement packs or automation templates for Google Big Query, AWS S3 native sourcing, Azure Blob native sourcing, Azure SQL DB, and Snowflake task and streams support to unleash the power of automation on popular cloud platforms. “Our product roadmap is tied directly to the growing needs of our 1000+ customer base,” concludes Nair.
WhereScape News
Just as in the realm of autonomous vehicles, DWA isn... n’t a binary state but rather exists on a spectrum of capabilities. In this article, I’ll explore the various levels of DWA, demystifying how they function and what each level means for businesses and technologists alike. This journey through the layers of automation will provide insights into not only how DWA is evolving but also its potential impact on our data-driven future.
Data Warehouse Automation Levels: From Cars to Coding
The United States National Highway Transportation Safety Administration has defined six levels of automated driving, primarily differentiated by the degree of driver involvement. This gradation of automation ranges from no automation (Level 0) where the driver performs all tasks, to full automation (Level 5), where no driver is needed at all.
This concept of varying levels of automation isn’t limited to the world of vehicular transport. It extends into my daily work realm – Data Warehouse Automation (DWA). Similar to the automotive industry, DWA encompasses a spectrum of capabilities. These capabilities can be categorized into levels, much like the automated driving system. Though not formally recognized, if we were to define levels 0-5 for DWA, they would represent a progression from manual data handling and analysis (Level 0) to a fully autonomous data management system (Level 5) where human intervention is minimal or unnecessary.
This gradation hints at an evolving landscape in data management, where efficiency, accuracy, and speed are continually enhanced through increasing levels of automation. I hadn’t thought about it formally, but if we had levels of 0-5 for DWA, what would they be?
Level 0 – No Automation
Everything is done manually. Design work is done on paper, or with minimal tools. Requirements and data profiling are done manually. Development of the structures and pipelines for storing and moving data are written by hand and then executed in a database administrative tool.
Level 1 – Developer Assistance
An entity-relationship tool is used to create the logical model of the data warehouse. Templates are used to write the DDL and DML code. Deployment and documentation are still manual processes.
Level 2 – Partial Automation
DDL and DML are generated from metadata, but no knowledge of data warehouse design or relationships. The code is manually deployed. Any data validation or transformation is done manually. Deep technical knowledge and experience is required.
Level 3 – Conditional Automation
Design, development, and deployment are created with an understanding of data warehouse architecture. Rules-based systems automatically apply attributes and transform data as needed. Profiling, design, creation of ELT processes, and linking various object types (such as facts and dimensions) are all done automatically. Deployments, documentation, and task scheduling are all handled automatically.
Level 4 – High Automation
DDL and DML generated and deployed. Design is fully automated, and relationships and data warehouse models are implemented automatically through data catalogs. Very, very little code is hand-written. All of the development and deployment of the data warehouse is managed by analysts, rather than technical staff.
Level 5 – Full Automation
Free-form queries are entered and the required data is gathered from multiple sources and processed in real-time. Does the data warehouse even exist anymore? Artificial intelligence might be involved in caching and calculating data before it is needed, but the idea of a dedicated online analytics system with batch jobs loading data for later analysis has been rendered obsolete. Why do I need a data warehouse if I can simply get the answer to any question I ask?
Why Automated Driving?
With automated driving, the first reaction of most people is “Great, I can relax while the car drives itself.” But fully automated driving is going to drastically change how we use cars, probably in ways we can’t predict. Some benefits would be faster commutes, less congestion meaning more sustainability, better use of intersections, fewer accidents, lower maintenance costs, greatly increased gas mileage, lower insurance premiums, and so on. Perhaps even the private ownership of cars may go away, and they will simply be available on demand.
Why Data Warehouse Automation?
Faster building of a data warehouse is a key benefit of DWA. But, as with driving, there are unexpected benefits:
Enhanced Development and Operations Workflows
WhereScape RED transforms the development landscape with its drag-and-drop approach, significantly shortening the time needed for data infrastructure development, deployment, and operations. This automation leads to a streamlined workflow that not only saves time but also ensures consistency across projects.
Platform-Native Code Generation
One of the most striking features of WhereScape RED is its ability to eliminate up to 95% of manual coding. By automatically generating SQL and other codes native to your target platform, it adheres to platform-specific best practices, boosting productivity and reducing the risk of inconsistencies.
Automatic Documentation and Metadata Management
Keeping documentation updated is a cumbersome task, but with WhereScape RED, this happens automatically. The tool not only maintains comprehensive documentation but also manages metadata efficiently. This feature ensures an up-to-date, transparent view of your data infrastructure, essential for both IT and business stakeholders.
Built-In Best Practices and Streamlined Workflow Management
WhereScape RED is embedded with best practices for various data warehousing methodologies like 3NF, Data Vault, and dimensional modeling. This integration reduces complexity and accelerates development. Furthermore, its integrated scheduling and workflow engine simplifies the management of decision support infrastructure, eliminating the need for manual scripting.
Advanced SQL Code Generation and Comprehensive Lifecycle Management
The tool excels in generating native SQL code, leveraging database-specific features and applications. Additionally, it automates the entire data warehousing life cycle, from design to operation, with its integrated metadata repository and support for agile methodologies.
Rapid Prototyping and Big Data Integration
WhereScape RED empowers users to move swiftly from source data to a populated schema, facilitating rapid prototyping. It also excels in integrating big data infrastructure, such as data lakes, with enterprise data, thus creating a comprehensive understanding across the business landscape.
Full ELT Support and Data Lineage
Offering complete extraction, load, and transformation capabilities, WhereScape RED includes integrated dependency management and scheduling. Its data lineage visualization aids in understanding the flow of data and the impact of changes, a crucial aspect of modern data management.
Where Does the Industry Stand Today?
There are several companies offering data warehouse automation tools today. WhereScape 3D and WhereScape RED, probably the most advanced tools, are around Level 3. With the adoption of data fabric and more advanced data cataloging, I expect Level 4 automation to come about in the next three to five years. Full automation? I think you’re going to find that’s going to be much easier in a car than in an open technical environment that requires data analytics.
Data Warehouse Automation is a necessity. If you are looking at DWA tools, think about how advanced each tool is. How flexible is it? How does it work in my current (or future) technical stack? How abstractly am I working? Am I telling the tool WHAT I want to do or HOW I want to do it? A good tool will know the HOW. You should simply provide the WHAT.
Embracing Automation in Data Warehousing with WhereScape
Data Warehouse Automation is more than a trend. It’s a significant shift in data handling and processing. The progression from Level 0 to Level 5 in data warehouse automation reflects our move towards a more automated and intelligent future. This evolution brings substantial benefits of data warehouse automation, including reduced costs, enhanced data processing speed and accuracy, and a transformative approach to business decision-making.
While reaching the pinnacle of Level 5 in DWA presents its challenges, it opens up a world of possibilities in data management. For businesses, adapting to these changes and choosing the right tools, like WhereScape’s offerings, are crucial steps in leveraging the potential of DWA. Let’s move forward into this automated future, recognizing that the journey is as much about visionary thinking as it is about technological advancement.
Read more