> For the complete documentation index, see [llms.txt](https://fastapi-docs.duonghuuphuc.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://fastapi-docs.duonghuuphuc.com/unit-22-end-to-end-microservice-integration-and-testing.md).

# Unit 22: End-to-End Microservice Integration and Testing

Integrate multiple microservices into a complete business workflow and test both successful and failure scenarios.

### 1. End-to-End Scenario

So far, we have developed and deployed individual microservices.

In this unit, we integrate three services:

* **Product Service** — provides product information.
* **Order Service** — creates and manages orders.
* **Notification Service** — sends customer notifications.

The complete architecture is:

```
                     ┌─────────────────┐
                     │ Product Service │
                     └────────▲────────┘
                              │
                              │ REST
                              │
Client ──────► Order Service ─┤
                              │
                              │ REST
                              ▼
                    ┌──────────────────────┐
                    │ Notification Service │
                    └──────────────────────┘
```

The business workflow is:

```
Create Order
     ↓
Validate Product
     ↓
Check Stock
     ↓
Create Order
     ↓
Send Notification
     ↓
Return Result
```

Our goal is to test this **complete workflow**, not only individual endpoints.

***

## 2. Extend the Order Model

Use the Order Service from Unit 19.

Add the customer's email address to `OrderCreate`:

```python
class OrderCreate(BaseModel):
    customer_name: str
    email: str
    product_id: int
    quantity: int = Field(gt=0)
```

A request now contains:

```json
{
    "customer_name": "Nguyen Van An",
    "email": "an@example.com",
    "product_id": 1,
    "quantity": 2
}
```

The email will be used when Order Service calls Notification Service.

***

## 3. Configure Service Addresses

Order Service now depends on two other services.

Add:

```python
import os
```

Configure the service addresses:

```python
PRODUCT_SERVICE_URL = os.getenv(
    "PRODUCT_SERVICE_URL",
    "http://127.0.0.1:8000"
)

NOTIFICATION_SERVICE_URL = os.getenv(
    "NOTIFICATION_SERVICE_URL",
    "http://127.0.0.1:8002"
)
```

When running locally:

```
Product Service       → localhost:8000
Order Service         → localhost:8001
Notification Service  → localhost:8002
```

When running with Docker Compose, these addresses will be replaced by container service names.

***

## 4. Call Notification Service

After successfully creating an order, prepare the notification:

```python
notification = {
    "recipient": order.email,
    "subject": "Order Confirmation",
    "message": (
        f"Your order #{new_order['order_id']} "
        f"has been created successfully."
    )
}
```

Then call Notification Service:

```python
try:
    notification_response = httpx.post(
        f"{NOTIFICATION_SERVICE_URL}/notifications/email",
        json=notification,
        timeout=3.0
    )

except httpx.RequestError:
    raise HTTPException(
        status_code=503,
        detail="Notification Service is unavailable"
    )
```

Check the response:

```python
if notification_response.status_code != 200:
    raise HTTPException(
        status_code=502,
        detail="Notification Service failed"
    )
```

The workflow becomes:

```
POST /orders
      │
      ▼
Order Service
      │
      ├────► Product Service
      │          │
      │◄─────────┘
      │
      │ Create Order
      │
      ├────► Notification Service
      │          │
      │◄─────────┘
      │
      ▼
Return Response
```

***

## 5. Notification Service

For this unit, we only need a simple Notification Service.

Create:

```
notification_service/main.py
```

with:

```python
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI(title="Notification Service")


class EmailNotification(BaseModel):
    recipient: str
    subject: str
    message: str


@app.post("/notifications/email")
def send_notification(
    notification: EmailNotification
):
    print(
        f"Sending email to {notification.recipient}: "
        f"{notification.subject}"
    )

    return {
        "status": "sent",
        "recipient": notification.recipient
    }
```

In a real system, this service could integrate with an external email provider as demonstrated in Unit 20.

***

## 6. Complete Docker Compose Configuration

The project structure is:

```
unit22-integration/
│
├── product_service/
│   ├── main.py
│   ├── requirements.txt
│   └── Dockerfile
│
├── order_service/
│   ├── main.py
│   ├── requirements.txt
│   └── Dockerfile
│
├── notification_service/
│   ├── main.py
│   ├── requirements.txt
│   └── Dockerfile
│
├── tests/
│   └── test_integration.py
│
└── docker-compose.yml
```

Create:

```
docker-compose.yml
```

with:

```yaml
services:

  product-service:
    build: ./product_service
    ports:
      - "8000:8000"

  order-service:
    build: ./order_service
    ports:
      - "8001:8000"
    environment:
      PRODUCT_SERVICE_URL: http://product-service:8000
      NOTIFICATION_SERVICE_URL: http://notification-service:8000
    depends_on:
      - product-service
      - notification-service

  notification-service:
    build: ./notification_service
    ports:
      - "8002:8000"
```

Inside the Docker network:

```
Order Service
     │
     ├──► http://product-service:8000
     │
     └──► http://notification-service:8000
```

The client accesses:

```
Product Service       → http://localhost:8000
Order Service         → http://localhost:8001
Notification Service  → http://localhost:8002
```

***

## 7. Run the Complete System

Build and start all services:

```bash
docker compose up --build
```

Check their status:

```bash
docker compose ps
```

You should see three running services:

```
product-service
order-service
notification-service
```

The complete system is now ready for end-to-end testing.

***

## 8. End-to-End Testing

An **end-to-end test** verifies a complete workflow across multiple components or services.

Instead of testing only:

```
Does POST /orders work?
```

we verify:

```
Does the complete order workflow work?
```

Test:

```
POST http://localhost:8001/orders
```

Request:

```json
{
    "customer_name": "Nguyen Van An",
    "email": "an@example.com",
    "product_id": 1,
    "quantity": 2
}
```

Verify that:

1. Order Service receives the request.
2. Product Service returns the product.
3. Product stock is validated.
4. The order is created.
5. Notification Service receives the notification.
6. The client receives a successful response.

This verifies the interaction of the **complete system**.

***

## 9. Test Failure Scenarios

Testing only the successful case is not sufficient in distributed systems.

We should also verify how the system behaves when dependencies fail.

### Scenario 1 — Invalid Product

Send:

```json
{
    "customer_name": "Nguyen Van An",
    "email": "an@example.com",
    "product_id": 999,
    "quantity": 1
}
```

Expected result:

```
400 Bad Request
Invalid product
```

No order should be created.

***

### Scenario 2 — Insufficient Stock

Request a quantity greater than the available stock.

Expected result:

```
400 Bad Request
Insufficient product stock
```

***

### Scenario 3 — Product Service Unavailable

Stop Product Service:

```bash
docker compose stop product-service
```

Try to create an order.

Expected result:

```
503 Service Unavailable
Product Service is unavailable
```

Restart it:

```bash
docker compose start product-service
```

***

### Scenario 4 — Notification Service Unavailable

Stop Notification Service:

```bash
docker compose stop notification-service
```

Create a valid order.

Observe what happens when the order can be created but the notification cannot be sent.

Restart the service:

```bash
docker compose start notification-service
```

This demonstrates an important property of distributed systems:

> One service can succeed while another dependent service fails.

This situation is called a **partial failure**.

***

## 10. Should Notification Failure Cancel the Order?

Consider:

```
Product validated
       ↓
Order created
       ↓
Notification failed
```

Should the order be cancelled?

Usually, these operations do not have the same business importance.

```
Order Creation
    → Core business operation

Email Notification
    → Supporting operation
```

A more resilient design may keep the successfully created order even when notification fails.

For example:

```python
notification_status = "sent"

try:
    response = httpx.post(
        f"{NOTIFICATION_SERVICE_URL}/notifications/email",
        json=notification,
        timeout=3.0
    )

    if response.status_code != 200:
        notification_status = "failed"

except httpx.RequestError:
    notification_status = "failed"
```

The order can still be returned:

```python
return {
    "order": new_order,
    "notification_status": notification_status
}
```

Now:

```
Order succeeds
      │
      ├── Notification succeeds → sent
      │
      └── Notification fails    → failed
```

This is a simple example of designing for **partial failure**.

***

## 11. Automated Integration Testing

Manual testing is useful during development, but important workflows should also be tested automatically.

Create:

```
tests/test_integration.py
```

Use `pytest` and `httpx`.

### Test Successful Order

```python
import httpx


ORDER_SERVICE_URL = "http://localhost:8001"


def test_create_order():
    response = httpx.post(
        f"{ORDER_SERVICE_URL}/orders",
        json={
            "customer_name": "Nguyen Van An",
            "email": "an@example.com",
            "product_id": 1,
            "quantity": 2
        },
        timeout=5.0
    )

    assert response.status_code == 200
```

Run:

```bash
pytest
```

The test sends a real HTTP request through the running microservice system.

***

## 12. Test an Invalid Product

Add:

```python
def test_invalid_product():
    response = httpx.post(
        f"{ORDER_SERVICE_URL}/orders",
        json={
            "customer_name": "Nguyen Van An",
            "email": "an@example.com",
            "product_id": 999,
            "quantity": 1
        },
        timeout=5.0
    )

    assert response.status_code == 400
    assert response.json()["detail"] == "Invalid product"
```

We now automatically test both:

```
Valid workflow   → Success
Invalid workflow → Correct failure
```

***

## 13. Test Validation

Add another test:

```python
def test_invalid_quantity():
    response = httpx.post(
        f"{ORDER_SERVICE_URL}/orders",
        json={
            "customer_name": "Nguyen Van An",
            "email": "an@example.com",
            "product_id": 1,
            "quantity": 0
        },
        timeout=5.0
    )

    assert response.status_code == 422
```

This verifies that invalid input is rejected before the business workflow continues.

***

## 14. Trace Requests Through Logs

When several services participate in one operation, identifying where a failure occurs becomes more difficult.

View all logs:

```bash
docker compose logs
```

Follow them continuously:

```bash
docker compose logs -f
```

Or inspect one service:

```bash
docker compose logs order-service
```

For one order request, try to trace:

```
Client
   ↓
Order Service
   ↓
Product Service
   ↓
Order Service
   ↓
Notification Service
   ↓
Order Service
   ↓
Client
```

Logs help answer questions such as:

* Which service received the request?
* Which service failed?
* What HTTP status was returned?
* Did the request reach the downstream service?

More advanced distributed tracing and centralized logging are covered in observability topics.

***

## 15. Testing Levels

Microservice applications can be tested at different levels.

| Testing Level    | Main Purpose                        |
| ---------------- | ----------------------------------- |
| Unit Test        | Test a small function or component  |
| Service/API Test | Test one service independently      |
| Integration Test | Test interactions between services  |
| End-to-End Test  | Test the complete business workflow |

For example:

```
Function
   ↓
Unit Test

Product Service
   ↓
API Test

Order ↔ Product
   ↓
Integration Test

Client → Order → Product → Notification
   ↓
End-to-End Test
```

Different testing levels detect different types of problems.

***

## 16. Practice Tasks

### Task 1 — Verify Notification Result

Extend `test_create_order()` to verify that the response contains:

```
notification_status
```

and that its value is:

```
sent
```

***

### Task 2 — Test Another Invalid Request

Create an automated test where:

```
quantity = -1
```

Verify the expected HTTP status code.

***

### Task 3 — Test Partial Failure

Stop Notification Service:

```bash
docker compose stop notification-service
```

Create an order.

Verify that:

```
Order creation        → succeeds
Notification          → fails
notification_status   → failed
```

Restart Notification Service afterward.

***

### Task 4 — Investigate a Failure Using Logs

Stop Product Service and send an order request.

Use:

```bash
docker compose logs -f
```

Identify:

1. Which service receives the client request.
2. Which dependency is unavailable.
3. Which service returns the final error.

***

## 17. Key Takeaways

* End-to-end testing verifies complete business workflows across multiple services.
* Integration testing verifies communication between services.
* Successful and failure scenarios should both be tested.
* Distributed systems can experience partial failures.
* Failure of a supporting service should not always cause the core business operation to fail.
* Automated tests make integration behavior repeatable and verifiable.
* Docker Compose provides a convenient environment for running and testing multiple services together.
* Logs help identify failures across service boundaries.