A transaction is an action or series of actions that transform a system from one consistent state to another and as we all know, transactions follow ACID properties.Microsoft BizTalk Server supports two primary types of transactions,Atomic and the Long-running.
Atomic transactions are those which follows ACID properties while Long-Running transactions follow CD leaving Atomicity and Isolation. Long-Running transactions are used when the transaction can take hours or days to complete and when isolation is not apt for such a long time.
Transaction Boundaries
From an architecture perspective, BizTalk Server has three possible transaction boundaries which are explained as follows.
1. Adapter,Receive Pipeline and the Messagebox : 
Adapters can participate in transactions only when its transport type supports transactions. When a transaction is bounded between an Adapter(that supports transaction),Receive Pipeline and the Messagebox, the source system will also be included in the boundary of the transaction(refer to Figure 1 below) i.e., a message from the source system will be removed only when the message is committed to the messagebox. 

Receive pipelines can support transactions.  A pipeline component does not participate in a transaction automatically. It has to fetch the transaction from IPipelineContextEx.GetTransaction. For a transacted adapter, the GetTransaction gets the transaction created by the adapter and passes it to the pipeline component. When a non-transacted adapter is used, a new transaction is created and returned to the pipeline component by the GetTransaction call.If the adapter does not support transactions, then the boundary shrinks to the pipeline component and the messagebox only.

                            Figure 1: Boundary spanning the Adapter,Receive Pipeline and the MessageBox

  2. MessageBox and Orchestration : 

There are two different levels of transaction boundaries when it comes to Orchestrations and the messagebox as shown in Figure 2 below. The First level is the in-built BizTalk engine transaction in which a message is returned back to the messagebox  when an orchestration fails to trigger for some reason.The next level is within the orchestration, where transactions can be defined on scope shapes and the entire unit of work will be committed or rolled back automatically in case of an atomic transaction failure or can be compensated(manual way of rolling back) during a long-running transaction.

                                                              Figure 2: Boundary spanning the MessageBox and the Orchestrations

While using an atomic transaction in an orchestration that includes a transactable adapter, delivery notifications(ACK and NACKs) are to be used to complete the atomic transaction. This is because, there is no direct transaction boundary between an orchestration and a send adapter. Instead, it happens with two messagebox hops i.e., one between the orchestration and the messagebox and the other between the messagebox and the adapter. The atomic scope in the orchestration does a commit only when an ACK is received.
3. MessageBox and send adapter:

In this boundary also, the transaction between the messagebox and the send adapter can be extended to the destination system when the adapter’s transport supports transactions.When such a boundary is created, the message will be removed from the messagebox only when the message is delivered to the destination as shown in Figure 3.

Send pipelines does not support transactions. Even though a transaction can be created in a sendpipeline component, it cannot participate in the
transaction that BizTalk creates between the messagebox and the send adapter.
                                         Figure 3: Boundary spanning the MessageBox and the Orchestrations

There is no single transaction that involves an orchestration, messagbox, pipeline and the adapter. There is a clear line between the orchestration-messagebox and the messagebox-pipeline-adapter. When a long-running transaction fails, the unit of work that has already committed will not be rolled back.This can be done using Compensation. Compensation is used to undo the partial work that was committed as part of a long-running transaction.Compensation is not limited to long-running transactions alone, it can go with atomic transactions as well in scenarios where some action has to be performed when an atomic transaction rollbacks.

Some Good Reads on Transactions:
--> MSDN
--> An Old Article but good one
--> Transactions Section of BizTalk Server Professional 2006
--> Atomic Transaction and Rollback
--> Compensation

See Also

Another important place to find a huge amount of BizTalk related articles is the TechNet Wiki itself. The best entry point is BizTalk Server Resources on the TechNet Wiki.