SQL Server 2005 Integration Services Matthew Stephen IT Pro Evangelist (SQL Server) Microsoft Ltd.

What is SQL Server Integration Services? A new Microsoft SQL Server Business Intelligence application The successor to Data Transformation Services The platform for a new generation of high performance data integration technologies

Data acquisition from source systems and integration Data transformation and synthesis Data enrichment, with business logic, hierarchical views Data discovery via data mining Data presentation and distribution Data access for the masses Integrate Analyze Report SQL Server Business Intelligence

Example: before Integration Services Call centre data: semi structured Legacy data: binary files Application database ETL Warehouse Reports Mobile data Data mining Alerts & escalation Integration and warehousing require separate, staged, operations. Preparation of data requires different, often incompatible, tools. Reporting and escalation is a slow process, delaying smart responses. Heavy data volumes make this scenario increasingly unworkable. Hand coding Staging Text Mining ETL Staging Cleansing & ETL Staging ETL

Example: with Integration Services Call centre: semi-structured data Legacy data: binary files Application database Alerts & escalation Integration and warehousing are a seamless, manageable, operation. Sourced, prepare and load data in a single, auditable process. Reporting and escalation can be parallelized with the warehouse load. Scales to handle heavy and complex data requirements. SQL Server Integration Services Text mining components Custom source Standard sources Data cleansing components Merges Data mining components Warehouse Reports Mobile data

How SSIS Operates Data sources can be diverse, including custom or scripted adapters Transformation components shape and modify data in many ways. Data is routed by rules or error conditions for cleansing and conforming. Flows can be as complex as your business rules, but highly concurrent. And finally data can be loaded in parallel to many varied destinations.

Customer benefits of SSIS Performance Data flows process large volumes of data efficiently Facility Many prebuilt adapters and transformations reduce hand coding Extensible object model Highly productive visual environment Data cleansing features Data mining imputation of incomplete data

Feature drilldown: Data Integration Traditional data sources XML Custom data sources Integrate diverse sources Parallel loading of diverse destinations

Feature drilldown: Data Warehousing Sorting and aggregation during loading Multicast and partition Slowly changing dimensions Load and process Analysis Services cubes

Feature drilldown: Intelligent Data Handling Capture error rows Fuzzy lookup and grouping Data mining on the data flow Text mining

Feature drilldown: Large Data Volumes Efficient data sources High performance processing > 700% faster than DTS 2000 Unique SQL Server Destination > 8% faster than Bulk Insert Advanced data flow architecture Enables flexible concurrent processing

Feature drilldown: Development Visual studio integration Visual designer Visual debugging Build and deploy Custom code integration

Enabling new architectures … Integration process simply conforms data and loads the database server The database performs aggregations, sorting and other operations Database competes for resources from user queries This solution does not scale very well Traditional (DTS) warehouse loading

Enabling new architectures … SQL Server Integration Services conforms the data But also aggregates and sorts, and loads the database This frees-up the database server for user queries Warehouse loading with SQL Server Integration Services

Life Cycle tools Design Business Intelligence Designer Migration wizard for pre SQL 2005 packages Execute DTS 2000 package Task Visual Source Safe Integration Deployment Deployment Utility Command Line execution Flexible Configuration Options Supportability Rich per package Logging SSIS service SQL Management Studio. Checkpoint - Restart ability

Sample Server Layout Destination data Source data SSIS package error rows SSIS package Logging SSIS packages stored in SQL Integration Services Package Execution SSIS support Server(s) Packages on file system Source Flat Files

Sample Server Layout Destination data Source data SSIS packages in SQL SSIS Parent Package Execution via SQL Agent (scheduled) Error Rows Logging SSIS support Server(s) Parent calls children via SQL Agent on other machines

Demo

Summary SQL Server Integration Services is an exceptionally high performance integration and transformation tool Some processes benefit more from parallelism, some from memory Many new tasks and transforms Separation of control flow and data flow

Resources Microsoft SQL Team blogs Microsoft SQL Server community on the web SQL Server Integration Services on the web SQL Server Developer Center Microsoft SQL Server 2005 website SQL Newsgroups SQL Server 2005 Datamining

© 2003 Microsoft Corporation. All rights reserved.