You can simply view Windows Server AppFabric as a hosting environment for NET 4 services using Windows Communication Foundation (WCF) and Windows Workflow Foundation (WF) . In more detail, it extends Windows Server to provide enhanced hosting, management, and caching capabilities for Web applications and middle-tier services. The AppFabric hosting features add service management extensions to Internet Information Services (IIS), Windows Process Activation Service (WAS), and the .NET Framework 4. This includes Hosting Services and Hosting Administration tools that make it easier to deploy, configure, and manage WCF- and WF-based services.
The definition of a pure Enterprise Service Bus (ESB) falls more into the technology of the Azure Service Bus. This ESB allows a cloud service to register itself in a common location and be discoverable and accessible across any network topology. But this is for services hosted in the Azure cloud. BizTalk Server and Windows Server AppFabric are for on-premise solutions. For more info on the difference between Windows Azure AppFabric and Windows Server AppFabric, see "Which AppFabric am I Looking For?" on the TechNet AppFabric Wiki site.
With BizTalk, you can use the WCF adapters to expose BizTalk orchestrations as "services" through WCF endpoints. There are two types of WCF BizTalk adapters. The WCF receive adapter receives incoming WCF service requests, creates a BizTalk Message object, and promotes the associated properties to the message context. The WCF send adapter is used to call a WCF service from a BizTalk orchestration using a typeless contract. So when you compare BizTalk WCF functionality to AppFabric fuctionality, you are really discussing the receive adapter since this is similar to hosting a WCF service in Windows Server AppFabric. But you can't host and expose a "pure" (say written in C# or VB.Net) WCF service (or WorkFlowService) assembly in BizTalk as you would host and expose a BizTalk orchestration using a WCF adapter. A WCF adapter converts outgoing and incoming calls into the WCF protocol on your behalf, shielding you from the details of WCF programming and configuration files. A BizTalk WCF receive adapter makes your orchestration appear to be a pure WCF service to external WCF clients. For more information on BizTalk WCF adapters, see WCF Adapters.
Similarly, in Windows Server AppFabric, any WF or WCF services it hosts communicates using WCF. And similarly to simplify configuring a WCF adapter for an orchestration, Windows Server AppFabric offers similar abstraction with its tooling and configuration inheritance for your pure WCF and WF service. AppFabric provides the host and scalability and support for building out your middle tier application components. It also simplifies deployment, configuration, management and scalability of composite applications. The goal of AppFabric is to provide a server infrastructure, on which business logic developed in a NET 4 WCF or WF service can be executed and exposed via WCF endpoints, without the need to design, develop, and support the infrastructural code.
Since this article is focusing on the hosting issue, it helps to give some history in Microsoft component hosting. Prior to .NET was released, developers used COM+ to host their middle tier components. The COM+ host provided instance management (just-in-time activation), role-based security, thread management, and automated transaction management. It also provided better memory and processor management, distributed transactions and a number of other services. For more information on COM+ and NET Enterprise Services, see my article on .NET Enterprise Services and COM+ 1.5 Architecture.
When .NET was introduced, we moved away from COM components and from hosting in COM+. This left a vacuum that so far had gone unfilled. In attempts to fill this vacuum, individual developers had to create their own hosts. These hosts typically were Windows Services. However, these services usually didn’t include multi-threading, scale out capabilities, tracking, monitoring, etc. – the types of functionality that has been in BizTalk for years. As Microsoft looked at BizTalk, as well as the .NET Framework, it was determined that there are many great features in BizTalk that if a scaled down version was available (and was offered free with the operating system instead of a separate purchase) it would address gaps in the framework. Not all service developers had the money to spend or the need for all the infrastructure support of BizTalk Server. Thus, Windows Server AppFabric evolved out of this thought process.
For .NET developers writing applications using WCF and WF, AppFabric fills the middle tier hosting vacuum. AppFabric provides the host and scalability support for building out your middle tier NET 4 WCF and WF services. It simplifies deployment, configuration, management, and scalability of composite applications. The goal of AppFabric is to provide a server infrastructure, on which business logic developed in WF can be executed and exposed via WCF endpoints, without the need to design, develop, and support the infrastructural code.
Windows Server AppFabric can't host non-service Workflows. This is because AppFabric needs a message to create and start an instance of a IIS-hosted Workflow. The Windows Server AppFabric Autostart feature pre-loads the service host and the workflow definition when IIS starts. This reduces the time it takes to invoke the service for the first time. But it does not create an instance of the workflow. However, AppFabric can host WorkflowService workflows. BizTalk Server has no capabilities to host NET 4 workflows (or pure WCF services for that matter). A BizTalk orchestration is a workflow-type model but is specific to BizTalk and cannot be hosted in AppFabric.
So when you ask yourself, "Which should I use for my NET 4 services that uses WCF/WF"?, they are really two different solutions for hosting and using WCF and WF services. BizTalk WCF receive adapters use the WCF protocols to expose BizTalk orchestrations as "WCF services". Windows Server AppFabric uses WCF protocols to expose pure NET 4 WCF and WF services. A WCF client does not know the difference as to whether it is accessing a pure WCF service or a BizTalk orchestration that looks like a WCF service since both expose WCF endpoints.
So just like one would choose a Phillips screwdriver over a straight-slot screwdriver for a job, you will make a similar decision.
Good question one needs to ask when one has to decide between the two.