The strength of the ESB Toolkit is that it’s highly extensible so you can create your own services and components to suit your specific requirements. For example, you may want to change the behavior of an existing component, create a custom messaging service to use in a pipeline or add providers and adapters.
The only downside is that there is little documentation available on MSDN about how to modify and extend the ESB Toolkit which also is not complete. Fortunately there are more and more blog posts, wikis and samples about it.

Overview of the main components that can be customized:

Creating a Custom Itinerary Service Using a BizTalk Orchestration

The Itinerary Framework that is part of the ​ESB Toolkit supports execution of Itinerary steps using Orchestrations. You can implement a custom itinerary service as a Microsoft BizTalk Server Orchestration based on your functional requirements, which may include the following:

  - Multiple service invocations
  - Protocol mediation and message correlation
  - Complex routing decisions based on message enrichment from external data sources
  - Business processing logic

Custom Orchestration Itinerary Service

 
For more information on how to create a custom Itinerary Orchestration Service see:

 

Creating a Custom Itinerary Messaging Service

The Itinerary Framework also supports execution of Itinerary steps using classes that are called by the Dispatcher pipeline component. You can implement a custom Messaging Service when you want the service to be responsible for the following:

  - Custom message validation configured in the itinerary
  - Custom message transformation configured in the itinerary
  - Custom processing of the message
 

Custom Itinerary Messaging Service

 
For more information on how to create a custom Itinerary Messaging Service see:

 

Creating a Custom Resolver

In an Itinerary Service a Resolver is used for dynamically resolving endpoint information and BizTalk Maps. The ESB Toolkit includes the following resolvers: STATIC, UDDI, UDDI3, XPATH, BRE, BRI, ITINERARY, ITINERARY-STATIC and LDAP but you can also create a custom Resolver.

Custom Resolver

 
For more information on how to create a custom Resolver see:

 

Creating a Custom Adapter Provider

After a Resolver component resolves an endpoint, an Adapter Provider component sets specific properties of registered BizTalk Server Adapters. The ESB Toolkit includes the following built-in adapter providers: FILE, FTP, SMTP,MQSeries, WCF-BasicHttp, WCF-WSHttp, and WCF-Custom. These can also be manually created.

Resolver Properties

 
For more information on how to create a custom Adapter Provider see:

 

Creating a Custom Extender for an Orchestration-Based Itinerary Service

The Itinerary Designer in Visual Studio allows you to create custom extenders for Itinerary Service model elements that can be used to add properties to the property bag for use by Orchestration-based Itinerary Services. Through this you no longer need to use a Resolver to feed your Orchestration with dynamic data because you can set the properties directly on the Itinerary Service.

Validate And Route Itinerary

Itinerary Service Properties

 
For more information on how to create a custom Extender see: