By Vassil Dimov, Mateo Tošić and Sowmya Kameswaran
In the previous blog post we have introduced Db2 for z/OS Data Gate and described it in detail. In this blog we will discuss the integration between Db2 Data Gate and Watson Knowledge Catalog and highlight the business value of it.
In the current business world, modernization of data and use of AI is the key to success. The guiding principles of the AI ladder defined by IBM, help organizations with business transformation based on the four key areas mentioned below:
- Collect — Make data simple and accessible (All data sources contribute to this pillar)
- Organize — Create a business-ready analytics foundation (Data governance services like Watson Knowledge Catalog)
- Analyze — Build and scale AI with trust and transparency (Watson Studio)
- Infuse — Operationalize AI throughout the business (this is what customers do with the data in their own products)
In this blog we will discuss how Db2 Data Gate and Watson Knowledge Catalog, representing the first two pillars of the AI Ladder, can help organizations to unlock the huge value of their Z data in the cloud.
About Db2 Data Gate
Db2 Data Gate enables modern high-volume, high-frequency hybrid cloud applications that need read-only access to valuable enterprise data from Db2 for z/OS. It plays a key role in the Collect pillar by enabling movement of data from Db2 for z/OS into the Cloud Pak for Data platform. With data synchronization between source Db2 for z/OS and target IBM Db2 and IBM Db2 Warehouse, applications are able to get access to current data. To learn more about IBM Db2 for z/OS Data gate, please read “What is Db2 Data Gate? Db2 Data Gate Blog Series Part 1”
About Watson Knowledge Catalog
Watson Knowledge Catalog (WKC) is an enterprise data catalog management platform that forms the core of the Organize pillar of the Cloud Pak for Data platform. A catalog connects people to the data and knowledge that they need. It is the key enabler to building the enterprise data catalog on Cloud Pak for Data that enables platform users to find, prepare, understand, and use the data as needed. The data governance framework ensures that data access and data quality are compliant with your business rules and standards.
WKC unites all information assets into a single metadata-rich catalog, based on Watson’s understanding of relationships between assets and how they’re being used and socialized among users in existing projects. It is integrated with an enterprise data governance platform that merges the analytics capabilities of Watson Studio. The data catalog assists data scientists in easily finding, preparing, understanding and using the data as needed.
Data protection has gained importance in recent years. That is why it is so important that WKC protects data from misuse and enables sharing of assets with automated, dynamic masking of sensitive data elements. This avoids violating various data protection regulations. For instance, when handling healthcare data in the USA, companies need to be aware of HIPAA (Health Insurance Portability and Accountability Act), a set of rules on how personally identifiable information maintained by the healthcare and healthcare insurance industries should be protected from fraud and theft. Moreover, any company based in the EU or offering services to people in the EU must comply with GDPR (General Data Protection Regulation), which has a much broader scope and governs the use of all personal data.
Db2 Data Gate — Watson Knowledge Catalog integration highlights
Watson Knowledge Catalog provides fine-grained control of data from various sources to users who need access to them. While administrators have the most permissions, data scientists and developers can only access data that is published to catalogs. Business analysts can, in addition to that, view data quality and access information asset views, while data engineers and data stewards can discover assets, import metadata, and access governance artifacts. The benefits are numerous for different user personas.
With the combination of Data Gate and WKC, data scientists and software engineers can explore the most important enterprise data coming from the mainframe and use all tools they are familiar with in the cloud for analysis and modeling and prototyping. They can benefit from tools, like schema structure discovery, to further accelerate the development of models and application. They do not even need to look for connection metadata since all assets are cataloged and accessible in just a few clicks.
Data stewards, on the other hand, can easily work on data quality using governance artifacts, such as business terms, business glossary, classifications, and automatic data profiling in WKC. They can define which columns from Db2 for z/OS are visible for whom in the cloud. More importantly, they can take care of regulations mentioned above (GDPR, HIPAAA, etc.). This has significant importance for data coming from Db2 for z/OS, as a data store containing the most sensitive customers’ data. On top of that, they can use rules, such as automatic data deletion, triggered once data on Z is deleted (e.g., customer related analysis for some customer that needs to be deleted once they leave the company).
In addition to the above, one of the other key benefits is the ability to understand and track data lineage — the journey made by the data from source through any transformations all the way till usage. Data lineage is very important when it comes to making sure the data is coming from the right source, being handled by the right people, undergoing the right transformations and landing in the right target. When Db2 for z/OS data is brought into the platform by Db2 Data Gate and then discovered and imported into the catalog, the data lineage can easily be maintained allowing data custodians to keep track of data all the way from the source. Last but not least, the usage of Db2 Data Gate allows to discover schema changes which can be maintained in the data linage of the data asses in WKC.
Steps to connect DG with WKC
Create a new catalog
Give your catalog a name and, optionally, a description. This catalog will be used to add the connections and assets to it.
Create the source connection
Choose Db2 for z/OS and type in the credentials and other parameters (host, port, etc.). WKC will use this metadata to access your database when you add some assets. You can click on “Test Connection” before creating it.
Add a source data asset
Choose the schema and the table you want from the newly created source connection. In the Assets tab you can see a preview of the data.
Create the target connection and add a target asset
Repeat the process from two previous steps. Instead of Db2 for z/OS choose Db2 or Db2 Warehouse according to your target database.
Add relationships to the data assets and connections
IsSameAs can be used to mark the source connection to be same as the target connection, but also to mark the source data asset to be same as the replicated, target data asset. IsContainedIn can be used to mark a data asset as contained in a connection (or Contains in the opposite direction).
Create Data Profile
(Benefit 1 — for Data Steward)
Data Profiles include generated metadata and statistics about the content of a data asset. An asset profile helps data stewards understand what actions to take to improve the data quality.
Use the data assets in Watson Studio
Create a project and add data assets
(Benefit 2 — Data Scientist / Software Engineer)
If you go to Watson Studio and create a new project (or use an existing one), you can add data assets from this catalog to it.
Create a notebook and load an asset into a data frame
Then you can use that asset for data analysis and modeling. In a Python/R notebook you will get an automatically generated block of code. Watson Studio will ask WKC for data and WKC will use the connection metadata to retrieve the data from the database. You can use the loaded data as a data frame.
We have shortly described Db2 Data Gate and introduced Watson Knowledge Catalog. We pointed out the benefits of their integration. By following the step by step video you were instructed how to get the integration yourself. To better materialize the benefits we went through a couple of example scenarios likely to be relatable in your usage flow.
For further reading, please check Daniel's blog here on Use Current Db2 for z/OS Data on Cloud, Without Direct Mainframe Access and Without Loosing Control Over Your Data
In part one of this blog James Cockayne looked at what might happen to a DB2 database that was attacked by ransomware encryption.
In part two James shares four dos and four don’ts to help protect DB2 databases from ransomware attacks.
With an uptake of more and more people being vaccinated against Covid-19, countries have opened their borders for travel over the past few months. This has opened the doors for IDUG committees to conduct their meetings face to face after almost two years. Face to face interaction is important for all of us at our workplace and more so for IDUG volunteers who give up some of their out of work time to help IDUG conduct it’s many activities. It helps rejuvenate what we believe in and keeps us motivated. A big thank you to all our volunteers for adapting to an online world over the past couple of years and continuing to help and support IDUG.
This month, the first ever IBM Community Festival 2021 is happening over November 8-10 and November 16-17. It’s online gathering for IBM user groups, clients, technology experts, distinguished engineers, & everyone else—to exchange technical knowledge & information about IBM products & services. See here for more details. I will be presenting on Monday, November 8. Please join me to learn all about IDUG – your organization! Check out the full agenda here.
On a high note, our IDUG Virtual 2021 Db2 Tech Conference is just round the corner over December 6-10. The Conference Planning Committee has been working hard to assemble a great agenda of sessions for you. Please register for a conference not to be missed!
And to end on another high note, I am excited to announce that the IDUG 2022 NA will be held in Boston on July 9-14, 2022. This will be our first face to face event in 2 years! Call for speakers is now open. Please consider submitting an abstract.
As always, please reach out to me with any questions, suggestions, or concerns. I am just an email away at firstname.lastname@example.org
Iqbal Goralwalla, IDUG President
This is the fourth in a four-part series of articles addressing Db2 for z/OS and modern development utilizing an Agile methodology and DevOps processes. To read the first three articles please follow these links:
In this article we address the critical needs of the Db2 for z/OS professional and the way these needs are met with the IBM Db2 DevOps Experience for z/OS. In addition, it will look at the needs of the modern Application Developer and the enterprise advantage to keeping your systems of engagement and innovation close to your system of record.
To read the entire article follow this link:
This is the third article of a four-part series addressing Db2 for z/OS and modern development utilizing an Agile methodology and DevOps processes. To read the first two articles please follow these links:
There has undoubtedly been substantial confusion concerning the various IBM software products that provide mainframe connectivity for web applications, what they do, what they cost, and what needs are they trying to serve. These include, but are not limited to; z/OSMF, z/OS Connect, Zowe (not IBM software, but open-source software that runs on z/OS), CICS Transaction Gateway, and now The IBM Unified Management Server for z/OS. Are all these servers related? No, they are not! Can some of the functionality of these servers overlap? Perhaps, but that is not the focus here. The question to be answered is what is the IBM Unified Management Server (UMS) and what benefits can be derived from installing it on a z/OS system?
To read the entire article follow this link:
By Mark Gillis
There are easily accessible means of checking what your Stored Procedure needs in the way of dependent objects (SYSCAT.ROUTINEDEP, basically). So, what if you find a, or a number of, Stored Procs that are marked as needing a REBIND and then, when you do that rebind, you get an SQL0440 indicating that “something” is missing. How do you go about checking that situation out? Find out here
By James Cockayne
By now I’m sure everyone has heard of the malicious practice known as ransomware attacks, where miscreants break into a corporate network and encrypt data before demanding huge sums of money to provide a method to decrypt that data and make it accessible again. The attacks tend to be insidious – sometimes the attacker is in the network for months before they gain access to the systems they are interested in, and they are known to target backup servers as well as the primary systems to cause maximum inconvenience to the target organisation.
Find out what an attack on a DB2 Database would look like. Continue reading James Cockayne's latest blog.
Hurry time is running out register for this live webcast including client interview Register now
Earlier this year, I hosted a client panel with several of our Db2 12 clients about the sucesses and migration of Db2 12, this event is now available on replay.
Don't miss this live webcast on 2nd November 2021 - 11:00AM EST
Tom Ramey will highlight some of the key challenges facing Db2 for z/OS clients and how AI is a breakthrough technology, that when applied to Db2 for z/OS performance management and resiliency can have a major impact. Tom will be joined by Benny Van Straten from Rabobank and Tom Beavin from IBM. Tom Beavin will share Db2 AI use cases and host a live Db2 AI demo.Rabobank is a Dutch multinational banking and financial services company; Benny will share first-hand experiences and lessons learnt around Db2 AI for z/OS and the power of in-database AI.
What will you learn by attending this webcast?
- Hear first-hand client feedback and experiences
- Learn how Db2 AI improves SQL performance using machine learning based on unique patterns found when executing the queries in a production environment.
- Learn how Db2 AI automatically detects SQL access path performance regressions and automatically restores the performance back to its optimal level
- Learn how Db2 AI automatically stabilizes dynamic queries with their optimal access path, reducing prepare overhead
Tom Ramey IBM WW Director, Data and AI on IBM Z
Benny van Straten IT Specialist/DB2 Rabobank
Akiko Hoshikawa IBM Distinguished Engineer
Tom Beavin IBM Db2 AI for z/OS Development Machine Learning and Optimization
September is usually a "back to" month. Back from holiday, back to work, back to school, and more. The "back from holiday" was missing for most of us last year as we adapted to living under COVID-19 restrictions and I hope everyone had a chance to take a much-needed break with their families and loved ones this summer.
As we closed in on summer, ripples of sadness were felt throughout the Db2 community as we grappled with the unexpected loss of a dear, long standing Db2 member from COVID-19 on August 18. Scott Hayes was a Db2 community leader, technical expert, mentor, speaker, and Gold consultant. He was indeed a Db2 legend (and so much more). His sudden passing away is a major loss for our community. Scott will surely be missed by us all.
In my last letter, I talked about my excitement in having the IDUG 2021 EMEA conference onsite--finally-in Edinburgh and in meeting with many of you in Scotland. However, due to uncertainties still prevailing around the COVID-19 situation, we could no longer hold this year’s EMEA event in person. The conference will now be a virtual event on December 6-10, 2021. I would like to thank our members and partners for their continued flexibility and support during these still-challenging times. I know the IDUG EMEA Conference Planning Committee is already working hard to deliver an excellent conference for you. Please watch out for the registration details which will be out soon. As for IDUG 2022 EMEA—that has been scheduled as an onsite event for October 22-26, 2022, in Edinburgh. As they say: third time’s a charm!
As always, please reach out to me with any questions, suggestions, or concerns about IDUG. I am always just an email away at email@example.com.
Thanks – Iqbal
REGISTER NOW starting at 11am EST 4pm UK
Live Webcast 21st September 11:00 AM EST live Q&A panel
Title: Get ready for Vnext - become AI enterprise!
Get a sneak preview into what around the corner in the World of Db2, and get a head-start for planning your migration to Db2 VNext. This is an event you do not want to miss.
IBM Db2 for z/OS is investing for leadership in the AI, cloud, and analytics era, extending our Z platform heritage of 24x7 availability, security, scalability & performance, while simplifying & modernizing to maintain Db2 as a top choice for next-gen IT professionals without requiring deep Z skills. This webcast will provide guidance on how to ensure that you get a head-start and are ready to upgrade to Db2 Vnext
Cris Molaro - IBM Gold Consultant will be giving an independent user view regarding challenges clients are facing during the current technological industry trends.
Apple’s Macs have been a popular development platform for many years now, but IBM have never really committed to supporting DB2 servers on macOS it seems. There was a version of DB2 Express-c v9.7 made available some years ago, but it was lacking features and is obviously way too far out of date to consider these days. A useful solution was to run Docker containers, or full virtual machines with Linux or Windows to make DB2 available locally, but the switch to the new Apple Silicon processors, or the M1 chip, means the Mac now uses the ARM64 instruction set rather than sharing the Intel/AMD x86-64 processors common to Windows and Linux platforms – and there is no option to download a copy of DB2 that runs on this processor.
So is that the end of having DB2 installed locally? Not quite.
IBM Db2 12 for z/OS Function Level Activation and Management
This technical paper follows on from an IBM Gold Consultants survey of business strategies for handling CI/CD in Db2 for z/OS, and in particular for advancing Function Levels.
Subject: Advancing Db2 12 for z/OS Function Levels safely.
Target audience: Executive-level decision makers and Db2 for z/OS system administrators.
Db2 12 z/OS represents a significant change in the way new features and functions are delivered with the introduction of continuous delivery: new features are introduced in the maintenance stream; and the activation of those new features is user-scheduled.
Db2 12 for z/OS has built on the capability introduced in Db2 11 to separate system migration from application migration using Application Compatibility, where application migration is under user control. It can be delayed until after system migration and scheduled on an application-by-application basis.
However, some users are still concerned about managing the introduction of new features and functions, particularly system-level features; for them, it is imperative that this be done without impacting the production service.
This IIBM Gold Consultants Technical Paper on Function Level activation with Db2 12 for z/OS continuous delivery is aimed at exactly that audience, and in particular it is targeted at decision makers and executives responsible for the production service. It describes how to turn off those features, and provides detailed guidance on eliminating or managing the impact of any incompatible or unavoidable changes even when advancing function levels. This allows Db2 customers to position themselves for the next release of Db2 for z/OS, which requires that the latest Db2 12 function level be activated before starting the migration process.
Note: this page contains paid content.
Please, subscribe to get an access.