All Posts (438)

Sort by

By Vassil Dimov, Mateo Tošić and Sowmya Kameswaran

Introduction

In the previous blog post we have introduced Db2 for z/OS Data Gate and described it in detail. In this blog we will discuss the integration between Db2 Data Gate and Watson Knowledge Catalog and highlight the business value of it.

AI Ladder

In the current business world, modernization of data and use of AI is the key to success. The guiding principles of the AI ladder defined by IBM, help organizations with business transformation based on the four key areas mentioned below:

  1. Collect — Make data simple and accessible (All data sources contribute to this pillar)
  2. Organize — Create a business-ready analytics foundation (Data governance services like Watson Knowledge Catalog)
  3. Analyze — Build and scale AI with trust and transparency (Watson Studio)
  4. Infuse — Operationalize AI throughout the business (this is what customers do with the data in their own products)

In this blog we will discuss how Db2 Data Gate and Watson Knowledge Catalog, representing the first two pillars of the AI Ladder, can help organizations to unlock the huge value of their Z data in the cloud.

About Db2 Data Gate

Db2 Data Gate enables modern high-volume, high-frequency hybrid cloud applications that need read-only access to valuable enterprise data from Db2 for z/OS. It plays a key role in the Collect pillar by enabling movement of data from Db2 for z/OS into the Cloud Pak for Data platform. With data synchronization between source Db2 for z/OS and target IBM Db2 and IBM Db2 Warehouse, applications are able to get access to current data. To learn more about IBM Db2 for z/OS Data gate, please read “What is Db2 Data Gate? Db2 Data Gate Blog Series Part 1

About Watson Knowledge Catalog

Watson Knowledge Catalog (WKC) is an enterprise data catalog management platform that forms the core of the Organize pillar of the Cloud Pak for Data platform. A catalog connects people to the data and knowledge that they need. It is the key enabler to building the enterprise data catalog on Cloud Pak for Data that enables platform users to find, prepare, understand, and use the data as needed. The data governance framework ensures that data access and data quality are compliant with your business rules and standards.

WKC unites all information assets into a single metadata-rich catalog, based on Watson’s understanding of relationships between assets and how they’re being used and socialized among users in existing projects. It is integrated with an enterprise data governance platform that merges the analytics capabilities of Watson Studio. The data catalog assists data scientists in easily finding, preparing, understanding and using the data as needed.

Data protection has gained importance in recent years. That is why it is so important that WKC protects data from misuse and enables sharing of assets with automated, dynamic masking of sensitive data elements. This avoids violating various data protection regulations. For instance, when handling healthcare data in the USA, companies need to be aware of HIPAA (Health Insurance Portability and Accountability Act), a set of rules on how personally identifiable information maintained by the healthcare and healthcare insurance industries should be protected from fraud and theft. Moreover, any company based in the EU or offering services to people in the EU must comply with GDPR (General Data Protection Regulation), which has a much broader scope and governs the use of all personal data.

Db2 Data Gate — Watson Knowledge Catalog integration highlights

Watson Knowledge Catalog provides fine-grained control of data from various sources to users who need access to them. While administrators have the most permissions, data scientists and developers can only access data that is published to catalogs. Business analysts can, in addition to that, view data quality and access information asset views, while data engineers and data stewards can discover assets, import metadata, and access governance artifacts. The benefits are numerous for different user personas.

With the combination of Data Gate and WKC, data scientists and software engineers can explore the most important enterprise data coming from the mainframe and use all tools they are familiar with in the cloud for analysis and modeling and prototyping. They can benefit from tools, like schema structure discovery, to further accelerate the development of models and application. They do not even need to look for connection metadata since all assets are cataloged and accessible in just a few clicks.

Data stewards, on the other hand, can easily work on data quality using governance artifacts, such as business terms, business glossary, classifications, and automatic data profiling in WKC. They can define which columns from Db2 for z/OS are visible for whom in the cloud. More importantly, they can take care of regulations mentioned above (GDPR, HIPAAA, etc.). This has significant importance for data coming from Db2 for z/OS, as a data store containing the most sensitive customers’ data. On top of that, they can use rules, such as automatic data deletion, triggered once data on Z is deleted (e.g., customer related analysis for some customer that needs to be deleted once they leave the company).

In addition to the above, one of the other key benefits is the ability to understand and track data lineage — the journey made by the data from source through any transformations all the way till usage. Data lineage is very important when it comes to making sure the data is coming from the right source, being handled by the right people, undergoing the right transformations and landing in the right target. When Db2 for z/OS data is brought into the platform by Db2 Data Gate and then discovered and imported into the catalog, the data lineage can easily be maintained allowing data custodians to keep track of data all the way from the source. Last but not least, the usage of Db2 Data Gate allows to discover schema changes which can be maintained in the data linage of the data asses in WKC.

Steps to connect DG with WKC

Create a new catalog

Give your catalog a name and, optionally, a description. This catalog will be used to add the connections and assets to it.

1*-TYXylG1oublSAn4RHDOqQ.gif

Create the source connection

Choose Db2 for z/OS and type in the credentials and other parameters (host, port, etc.). WKC will use this metadata to access your database when you add some assets. You can click on “Test Connection” before creating it.

1*wZrF-BSIrVLm8qTmRFkvsQ.gif

Add a source data asset

Choose the schema and the table you want from the newly created source connection. In the Assets tab you can see a preview of the data.

1*XjTJMd1VjZZQ3rg8DEwoqw.gif

Create the target connection and add a target asset

Repeat the process from two previous steps. Instead of Db2 for z/OS choose Db2 or Db2 Warehouse according to your target database.

1*IHm-RfspA7LX8gQ7GRP7Sw.gif

Add relationships to the data assets and connections

IsSameAs can be used to mark the source connection to be same as the target connection, but also to mark the source data asset to be same as the replicated, target data asset. IsContainedIn can be used to mark a data asset as contained in a connection (or Contains in the opposite direction).

1*XyWfyIZjKro04D-XGnhr9Q.gif

Create Data Profile

(Benefit 1 — for Data Steward)

Data Profiles include generated metadata and statistics about the content of a data asset. An asset profile helps data stewards understand what actions to take to improve the data quality.

1*3McA-3HLClQJ4AEAFWPFRg.gif

Use the data assets in Watson Studio

Create a project and add data assets

(Benefit 2 — Data Scientist / Software Engineer)

If you go to Watson Studio and create a new project (or use an existing one), you can add data assets from this catalog to it.

1*Agu_qKMniI2OkGgXaLQhfA.gif

Create a notebook and load an asset into a data frame

Then you can use that asset for data analysis and modeling. In a Python/R notebook you will get an automatically generated block of code. Watson Studio will ask WKC for data and WKC will use the connection metadata to retrieve the data from the database. You can use the loaded data as a data frame.

1*3VuYnQofeF3R9XBlSc5Yvw.gif

Conclusion

We have shortly described Db2 Data Gate and introduced Watson Knowledge Catalog. We pointed out the benefits of their integration. By following the step by step video you were instructed how to get the integration yourself. To better materialize the benefits we went through a couple of example scenarios likely to be relatable in your usage flow.

For further reading, please check Daniel's blog here on Use Current Db2 for z/OS Data on Cloud, Without Direct Mainframe Access and Without Loosing Control Over Your Data

Read more…

In part one of this blog James Cockayne looked at what might happen to a DB2 database that was attacked by ransomware encryption.  

In part two James shares four dos and four don’ts to help protect DB2 databases from ransomware attacks.

https://www.triton.co.uk/ransomware-and-the-db2-database-part-two/

9860513277?profile=RESIZE_710x

Read more…

Read Now! IDUG President's Letter: November 2021

9788203482?profile=RESIZE_400xWith an uptake of more and more people being vaccinated against Covid-19, countries have opened their borders for travel over the past few months. This has opened the doors for IDUG committees to conduct their meetings face to face after almost two years. Face to face interaction is important for all of us at our workplace and more so for IDUG volunteers who give up some of their out of work time to help IDUG conduct it’s many activities. It helps rejuvenate what we believe in and keeps us motivated. A big thank you to all our volunteers for adapting to an online world over the past couple of years and continuing to help and support IDUG. 

This month, the first ever IBM Community Festival 2021 is happening over November 8-10 and November 16-17. It’s online gathering for IBM user groups, clients, technology experts, distinguished engineers, & everyone else—to exchange technical knowledge & information about IBM products & services. See here for more details. I will be presenting on Monday, November 8. Please join me to learn all about IDUG – your organization! Check out the full agenda here

On a high note, our IDUG Virtual 2021 Db2 Tech Conference is just round the corner over December 6-10. The Conference Planning Committee has been working hard to assemble a great agenda of sessions for you. Please register for a conference not to be missed!

 And to end on another high note, I am excited to announce that the IDUG 2022 NA will be held in Boston on July 9-14, 2022. This will be our first face to face event in 2 years! Call for speakers is now open. Please consider submitting an abstract. 

As always, please reach out to me with any questions, suggestions, or concerns. I am just an email away at iqbal@idug.org

Iqbal Goralwalla, IDUG President

https://www.idug.org/blogs/iqbal-goralwalla/2021/11/02/november-news-upcoming-events-conference

Visit IDUG's website for more information

 

Read more…

This is the fourth in a four-part series of articles addressing Db2 for z/OS and modern development utilizing an Agile methodology and DevOps processes. To read the first three articles please follow these links:

https://community.ibm.com/community/user/hybriddatamanagement/blogs/daniel-luksetich/2021/06/21/traditional-db2-development-vs-modern-development

https://community.ibm.com/community/user/hybriddatamanagement/blogs/daniel-luksetich/2021/08/09/devops-and-modern-it-defined

https://community.ibm.com/community/user/hybriddatamanagement/blogs/daniel-luksetich/2021/09/14/the-ibm-unified-management-server-and-db2-for-zos

In this article we address the critical needs of the Db2 for z/OS professional and the way these needs are met with the IBM Db2 DevOps Experience for z/OS. In addition, it will look at the needs of the modern Application Developer and the enterprise advantage to keeping your systems of engagement and innovation close to your system of record.

To read the entire article follow this link:

https://community.ibm.com/community/user/hybriddatamanagement/blogs/daniel-luksetich/2021/11/03/ibm-db2-devops-experience-for-zos

Read more…

This is the third article of a four-part series addressing Db2 for z/OS and modern development utilizing an Agile methodology and DevOps processes. To read the first two articles please follow these links:

https://community.ibm.com/community/user/hybriddatamanagement/blogs/daniel-luksetich/2021/06/21/traditional-db2-development-vs-modern-development

https://community.ibm.com/community/user/hybriddatamanagement/blogs/daniel-luksetich/2021/08/09/devops-and-modern-it-defined

There has undoubtedly been substantial confusion concerning the various IBM software products that provide mainframe connectivity for web applications, what they do, what they cost, and what needs are they trying to serve. These include, but are not limited to; z/OSMF, z/OS Connect, Zowe (not IBM software, but open-source software that runs on z/OS), CICS Transaction Gateway, and now The IBM Unified Management Server for z/OS. Are all these servers related? No, they are not! Can some of the functionality of these servers overlap? Perhaps, but that is not the focus here. The question to be answered is what is the IBM Unified Management Server (UMS) and what benefits can be derived from installing it on a z/OS system?

To read the entire article follow this link:

https://community.ibm.com/community/user/hybriddatamanagement/blogs/daniel-luksetich/2021/09/14/the-ibm-unified-management-server-and-db2-for-zos

 

Read more…
The OMEGAMON Db2 Enhanced 3270 User Interface main workspace displays statistical values which are totals since the start of Db2. Because a DB2 subsystem runs for a long time the statistics may not be relevant to more recent problems. Situations and highlighting of fields can become almost meaningless because the values are too large. With this enhancement (APAR PH35707 PTF UI76291) you will be able to display Db2 Statistics for a recent interval for example the last 24 hours. Now situations and thresholds can be based on recent values like deadlocks that occurred today.
Read more…

By Mark Gillis

There are easily accessible means of checking what your Stored Procedure needs in the way of dependent objects (SYSCAT.ROUTINEDEP, basically). So, what if you find a, or a number of, Stored Procs that are marked as needing a REBIND and then, when you do that rebind, you get an SQL0440 indicating that “something” is missing. How do you go about checking that situation out? Find out here

9756322693?profile=RESIZE_584x

Read more…

By James Cockayne

By now I’m sure everyone has heard of the malicious practice known as ransomware attacks, where miscreants break into a corporate network and encrypt data before demanding huge sums of money to provide a method to decrypt that data and make it accessible again.  The attacks tend to be insidious – sometimes the attacker is in the network for months before they gain access to the systems they are interested in, and they are known to target backup servers as well as the primary systems to cause maximum inconvenience to the target organisation. 

Find out what an attack on a DB2 Database would look like. Continue reading James Cockayne's latest blog. 

https://www.triton.co.uk/ransomware-and-the-db2-database-part-1/

 

 

Read more…

In-Database AI Client Experiences with Db2 for z/OS + Demo 

Don't  miss this live webcast on 2nd November 2021 - 11:00AM EST
Tom Ramey will highlight some of the key challenges facing Db2 for z/OS clients and how AI is a breakthrough technology, that when applied to Db2 for z/OS performance management and resiliency can have a major impact. Tom will be joined by Benny Van Straten from Rabobank and Tom Beavin from IBM.  Tom Beavin will share Db2 AI use cases and host a live Db2 AI demo.Rabobank is a Dutch multinational banking and financial services company; Benny will share first-hand experiences and lessons learnt around Db2 AI for z/OS and the power of in-database AI.

 What will you learn by attending this webcast?

  • Hear first-hand client feedback and experiences
  • Learn how Db2 AI improves SQL performance using machine learning based on unique patterns found when executing the queries in a production environment. 
  • Learn how Db2 AI automatically detects SQL access path performance regressions and automatically restores the performance back to its optimal level
  • Learn how Db2 AI automatically stabilizes dynamic queries with their optimal access path, reducing prepare overhead

    9739408684?profile=RESIZE_400x

Tom Ramey IBM WW Director, Data and AI on IBM Z

Benny van Straten IT Specialist/DB2 Rabobank

Akiko Hoshikawa IBM Distinguished Engineer

Tom Beavin IBM Db2 AI for z/OS Development Machine Learning and Optimization

 

Read more…

By Sowmya Kameswaran and Jens Müller

 

As we all know, Db2 for z/OS has one of the largest footprints in the enterprise database world. Your organization may have all or most of its' business critical data on Db2 for z/OS (you are not alone and we wouldn't recommend you to change a thing about that!). We realize though that many organizations are experimenting with or, rearchitecting and extending their infrastructure to take advantage of hybrid cloud.
With the changing environment and importance of hybrid cloud, cloud-based applications need easy, secure access to this data for modern applications. IBM Db2 Data Gate for z/OS makes data from Db2 for z/OS readily accessible on the IBM Cloud Pak for Data platform for business users and application developers.
 
As-is scenario
We have embarked on an exciting era of compelling modern application development. There is a surge in both reporting applications requiring read-only access to transactional data as well as data-intensive analytics applications requiring access to historical data. Organizations are developing new compelling applications for differentiating services delivered to their customers. Since much of this data originates in Db2 for z/OS, many organizations have built custom ETL (extract, transform, load) jobs to extract and load this data into other databases to support their application needs. While this approach may work in the short term, some of the problems with this approach are:
 
  1. Expensive to create and maintain over the course of time (due to complexity, and the costs of synchronizing source and target databases, and ensuring transactional consistency if necessary)
  2. Data security concerns once data is moved from where it originates
  3. Increased operational processes and cost on IBM Z
 
Why Db2 Data Gate
  • It is an integrated solution to securely access data from Db2 for z/OS on the cloud without the need for direct access to Db2 for z/OS.
  • Avoids significant investment in building and maintaining custom ETL solutions to move Db2 for z/OS data.
  • Provides better data currency via the Integrated Synchronization feature that replicates data from Db2 for z/OS to IBM Cloud Pak for Data.
  • Significantly reduces (operation) cost of data replication on mainframe since 96 % of the underlying data synchronization technology is zIIP eligible.
  • Enables modernization and transformation in your enterprise's Journey to Cloud.
  • With Integrated Synchronization, the data availability for applications accessing data from the source is not affected (source tables are fully online for reading and writing) while data is replicated to the target.
  • HA/DR is built directly into IBM Cloud Pak for Data 

 

Architecture highlights

9712629474?profile=RESIZE_710x

 

Db2 Data Gate is based on Db2 (row store) or Db2 Warehouse (column store) as the target databases within IBM Cloud Pak for Data. This model makes it suitable for supporting  applications that require row level access  as well as analytical applications that benefit from column based data store. Only one Db2 for z/OS database can be used as the data source. The key aspect of the architecture is the Integrated Synchronization feature which is optimized to replicate data from Db2 for z/OS to Db2 running under IBM Cloud Pak for Data.
 
With Db2 Data Gate and the target database running on the IBM Cloud Pak for Data platform, the solution works wherever the platform is able to run – private, public or hybrid cloud implementations, thus making relevant data readily available to application developers and business users where they need it.
 
Lab performance benchmarks
Db2 Data Gate boasts unrivaled performance when compared to any other data synchronization tool synchronizing data from Db2 for z/OS to Db2 (Warehouse).
 
With IBM Cloud Pak for Data and Db2 Data Gate installed on Linux on IBM Z, using Db2 Warehouse as the target database using hostPath data storage:
 
  • Peak load performance (for making initial copy): 2.1 TB/h
  • Peak synchronization performance: 200k rows/sec at 1.2 secs peak latency
 
Db2 Data Gate does not:
  • Replace traditional transactional systems since it only provides read-only data access to source data
  • Serve as a replacement for other data replication technologies because it supports only one source and one target and also does not support bidirectional replication or data transformation
  • Support data versioning and hence is not an operational data store
  • Guarantee data currency for high-volume transactional workloads. It is not a carbon copy of source data. Meaning, applications requiring absolute currency should access data at its point of origin.
 
User Interface snapshots
 
Db2 Data Gate provisioning
Select the target database type and deployment, resource allocation and network routing to proceed with creation of the Db2 Data Gate instance.
 
9712621054?profile=RESIZE_710x
Setting up source
Once the instance is created, the first step is to point the Db2 Data Gate instance to the Db2 for z/OS subsystem to be used as data source.
 
9712599073?profile=RESIZE_710x9712580674?profile=RESIZE_710x
 
 
Select and add tables from source to target
The next step is to select the tables to synchronize data from source to target.
 
9712580064?profile=RESIZE_710x
 
Db2 Data Gate dashboard
Overview of status and activities associated with the provisioned Db2 Data Gate instance.
 
9712578895?profile=RESIZE_710x 
 
Db2 Data Gate in action
The video below demonstrates the Db2 Data Gate end user experience. One of the key aspects to note is that when the source tables are added, loaded and setup for synchronization with Db2 Data Gate, there is no impact to concurrent workloads executing on the source tables. The source tables are fully online for reading and writing while Db2 Data Gate makes the copy and starts synchronizing.
 
 
Our next blog ...
Next time we will look at Db2 for z/OS Data Gate and Watson Knowledge Catalog integration.
Read more…

IDUG President's Letter: September 2021

September is usually a "back to" month. Back from holiday, back to work, back to school, and more. The "back from holiday" was missing for most of us last year as we adapted to living under COVID-19 restrictions and I hope everyone had a chance to take a much-needed break with their families and loved ones this summer.

As we closed in on summer, ripples of sadness were felt throughout the Db2 community as we grappled with the unexpected loss of a dear, long standing  Db2 member from COVID-19 on August 18. Scott Hayes was a Db2 community leader, technical expert, mentor, speaker, and Gold consultant. He was indeed a Db2 legend (and so much more). His sudden passing away is a major loss for our community. Scott will surely be missed by us all. 

 In my last letter, I talked about my excitement in having the IDUG 2021 EMEA conference onsite--finally-in Edinburgh and in meeting with many of you in Scotland. However, due to uncertainties still prevailing around the COVID-19 situation, we could no longer hold this year’s EMEA event in person. The conference will now be a virtual event on December 6-10, 2021. I would like to thank our members and partners for their continued flexibility and support during these still-challenging times. I know the IDUG EMEA Conference Planning Committee is already working hard to deliver an excellent conference for you. Please watch out for the registration details which will be out soon.  As for IDUG 2022 EMEA—that has been scheduled as an onsite event for October 22-26, 2022, in Edinburgh. As they say: third time’s a charm!

As always, please reach out to me with any questions, suggestions, or concerns about IDUG. I am always just an email away at iqbal@idug.org.

Thanks – Iqbal

Read more…

REGISTER NOW starting at 11am EST 4pm UK
https://ibm.biz/BdfanX
Live Webcast 21st September 11:00 AM EST live Q&A panel
Title: Get ready for Vnext - become AI enterprise!

Get a sneak preview into what around the corner in the World of Db2, and get a head-start for planning your migration to Db2 VNext. This is an event you do not want to miss. 
IBM Db2 for z/OS is investing for leadership in the AI, cloud, and analytics era, extending our Z platform heritage of 24x7 availability, security, scalability & performance, while simplifying & modernizing to maintain Db2 as a top choice for next-gen IT professionals without requiring deep Z skills. This webcast will provide guidance on how to ensure that you get a head-start and are ready to upgrade to Db2 Vnext 

Cris Molaro - IBM Gold Consultant will be giving an independent user view regarding challenges clients are facing during the current technological industry trends. 

9644149095?profile=original
#db2z #ai #scalability #cloud #security #analytics #IDUG #Db212#machinelearning #agile #automation #ibm #IBMZ #innovation #event

Read more…

DB2 on Apple Silicon

By James Cockayne

Apple’s Macs have been a popular development platform for many years now, but IBM have never really committed to supporting DB2 servers on macOS it seems.  There was a version of DB2 Express-c v9.7 made available some years ago, but it was lacking features and is obviously way too far out of date to consider these days.  A useful solution was to run Docker containers, or full virtual machines with Linux or Windows to make DB2 available locally, but the switch to the new Apple Silicon processors, or the M1 chip, means the Mac now uses the ARM64 instruction set rather than sharing the Intel/AMD x86-64 processors common to Windows and Linux platforms – and there is no option to download a copy of DB2 that runs on this processor.

So is that the end of having DB2 installed locally?  Not quite.

Click to continue reading

9644149272?profile=original

Read more…

IBM Db2 12 for z/OS Function Level Activation and Management

This technical paper follows on from an IBM Gold Consultants survey of business strategies for handling CI/CD in Db2 for z/OS, and in particular for advancing Function Levels.

Subject: Advancing Db2 12 for z/OS Function Levels safely.

Target audience: Executive-level decision makers and Db2 for z/OS system administrators.

Db2 12 z/OS represents a significant change in the way new features and functions are delivered with the introduction of continuous delivery: new features are introduced in the maintenance stream; and the activation of those new features is user-scheduled.

Db2 12 for z/OS has built on the capability introduced in Db2 11 to separate system migration from application migration using Application Compatibility, where application migration is under user control. It can be delayed until after system migration and scheduled on an application-by-application basis.

However, some users are still concerned about managing the introduction of new features and functions, particularly system-level features; for them, it is imperative that this be done without impacting the production service.

This IIBM Gold Consultants Technical Paper on Function Level activation with Db2 12 for z/OS continuous delivery is aimed at exactly that audience, and in particular it is targeted at decision makers and executives responsible for the production service. It describes how to turn off those features, and provides detailed guidance on eliminating or managing the impact of any incompatible or unavoidable changes even when advancing function levels. This allows Db2 customers to position themselves for the next release of Db2 for z/OS, which requires that the latest Db2 12 function level be activated before starting the migration process.

Read more…