Keywords:  BizTalk Server 2004, EDI, SOA, Communications Subsystem, Web Services, Common XML Schemas, Custom Pipeline Components, Custom Adapters, Orchestration, Smart Client, Transmission Control Console, Event Log Monitoring

In July 2004 I was contracted by Unitek Information Systems to design and build a comprehensive BizTalk-based system for the IT Services division of the Pasha Group (www.pashagroup.com). Pasha is an international transportation and logistics company, and is involved with the worldwide freight management of new vehicles, general cargo, and household goods. The Pasha Automotive Services (PAS) group handles the importation, port processing, and dealer distribution of new Honda, Acura, VW, Audi, Mitsubishi, Isuzu, Porsche and Lotus vehicles. These vehicles arrive by ship at Pasha’s terminal at the Port of San Diego, and are dispatched by rail and truck to dealers in the western US. The vehicles are inspected on arrival, repaired if damaged in transit, and have option packages installed, before being loaded onto trains and trucks. Pasha Hawaii Transport Lines (PHTL) is a sister company that ships new Chrysler and Mazda vehicles through the Port of San Diego to the Hawaiian Islands.

I was tasked with architecting and implementing a new communications subsystem for exchanging electronic business documents between PAS/PHTL and their large number of trading partners. These organizations include vehicle manufacturers, trucking and rail companies, and other logistics firms. The documents involved are vehicle shipping manifests, dealer allocations, accessory installation orders, damage reports, vehicle arrival, dispatching, and departure notifications, vehicle status and location reports, billing notices, and vehicle window labels. In all there are more than 90 inbound and outbound document exchanges.

After a few weeks of analyzing these transmission requirements, I architected a system based on BizTalk Server 2004. This system is designed to function as the communications engine in a Service-Oriented Architecture (SOA), and communicates internally with Pasha’s line-of-business applications using web services. The main objective of the BizTalk system is to provide a very robust interface to Pasha’s trading partners, as the dependable exchange of the documents is mission critical. In addition, Pasha’s staff needed to view ad-hoc communication reports, to manually receive and transmit documents, and to have all inbound and outbound documents fully archived for possible internal re-processing or re-transmission. Another challenge was the variety of trading partner document formats. Most of these are EDI or flat-files, and a major part of the design process involved the specification of a few common, or generic, XML documents that can be used internally, and which can be mapped from and to inbound and outbound EDI formats. I managed the process of specifying the mapping (gap analysis) of each EDI document format to the common XML formats, along with the task of writing the BizTalk schemas to process all the EDI documents, using my own custom “Schema Studio” application for automating the verification of hundreds of sample EDI files against each EDI schema. I was also responsible for developing most of the BizTalk mapping files for converting EDI documents into common XML ones, and vise versa.

As part of the BizTalk 2004 solution, I designed and developed two custom pipeline components: one for archiving inbound and outbound documents, and one for splitting received documents into multiple files based on regular expression scans of each line. I also developed two custom BizTalk 2004 adapters. One adapter is used by BizTalk for communicating with Pasha’s line-of-business applications. The other is a very sophisticated FTP adapter which was written to overcome shortcomings in the standard BizTalk FTP adapter, and to enable complex schedule-based inbound FTP polling, very specific filename generation for transmitted documents, and detailed FTP session logging. I designed and developed a web service that allows XML documents to be sent to BizTalk by the line-of-business applications. This web service pre-validates all received XML documents against their BizTalk schemas, and also archives copies of them. The BizTalk orchestration that processes outbound documents was also my responsibility. To reduce complexity it uses “dynamic mapping” for converting common XML documents into EDI files; the BizTalk map required to produce a specific EDI file is determined at runtime, and the map is then loaded and invoked. This orchestration also calls a custom .NET assembly that generated notification emails based on configured templates, so that trading partners and Pasha personnel can keep track of important documents, events, and transmission failures. All these customizations are .NET-based and were written in C#.

Another major component of my solution is a Transmission Control Console (TCC). This is a Windows forms-based .NET application that is used by Pasha operations and support staff to monitor inbound and outbound trading partner document transmissions, to view the contents of archived EDI and XML documents, to resubmit inbound and outbound documents, to view and update communication parameters (such as FTP access credentials, polling schedules, etc.), and to view the detailed log files produced by the BizTalk web service, custom pipeline components, and custom adapters. The TCC application is designed as an auto-updating .NET “smart client”; once it is installed on a user’s machine it periodically monitors a central repository for updates to its .NET assemblies, which it automatically downloads and installs. The TCC is the window into all of Pasha’s document transmission and reception. It is integral to the successful day-to-day management and monitoring of the trading partner communications, and has become an especially visible and successful part of my overall solution.

Finally, I installed and configured my own custom Windows Event Log Monitor at Pasha. This is a lightweight service that monitors the application event log, and generates email notifications to specified staff when events occur that match a set of configured patterns. These events include BizTalk runtime failures such as document schema violations and mapping failures, and also FTP communication, network, and web service problems.

Pasha’s new BizTalk system went “live” in May 2005 with support for all the Honda/Acura and VW/Audi inbound and outbound documents. Support for the remaining manufacturers was added over the next two months.

Pasha Automotive Services and Pasha Hawaii Transport Lines now have a powerful and reliable BizTalk-based system for exchanging electronic documents with their numerous trading partners. The use of common internal XML documents allows the easy addition of new document exchanges as the companies expand their business.

For information about BizTalk or .NET implementation services, please contact me at simon@chadwickinfo.com.