“In today’s digital landscape, efficient communication between software systems is crucial for business success.”
This opening sentence establishes the fundamental business problem being addressed. Modern businesses rely on multiple software systems that need to exchange information efficiently. For example:
- CRM systems need to know about new customer inquiries
- Support ticket systems need to track email conversations
- Marketing automation tools need to monitor customer responses
- Workflow systems need to initiate processes based on incoming communications
When these systems can’t communicate efficiently, businesses experience delays, lost information, and missed opportunities. The statement frames the technical discussion that follows in terms of business value.
In this post, we’ll explore how Pub/Sub works for email integration, step-by-step implementation guidance, and best practices to optimize deliverability and performance.
1. What Is Pub/Sub?
Publish-subscribe is a messaging pattern where senders (publishers) categorize messages into classes without knowledge of which receivers (subscribers) will receive them. Similarly, subscribers express interest in specific message classes and only receive messages that match their interests, without knowledge of which publishers are generating the messages.
This decoupling offers several advantages:
- Scalability: Systems can handle large volumes of messages without becoming bottlenecks
- Flexibility: New publishers and subscribers can be added without disrupting existing components
- Resilience: Failure in one component doesn’t necessarily affect others
- Publisher sends an “EmailEvent” with payload (recipient, template ID, data).
- Pub/Sub Service persists and routes the event.
- Email Subscriber picks up events and calls your SMTP or email API provider (e.g., SendGrid).
- Analytics Subscriber logs events for metrics and monitoring.
The Email Integration Challenge
Email remains a critical communication channel for businesses, but integrating email functionality with modern applications presents several challenges:
- Managing high volumes of incoming and outgoing messages
- Ensuring reliable delivery despite network issues
- Processing emails asynchronously without blocking user interactions
- Tracking email statuses (delivered, opened, bounced)
- Implementing complex workflows triggered by email events
Pub/Sub architectures address these challenges elegantly by providing a robust foundation for email processing pipelines.
Implementing Pub/Sub for Email Integration
Core Components
A typical pub/sub-based email integration system consists of:
- Publishers: Components that generate email-related events (user registrations, order confirmations, marketing campaigns)
- Message Broker: The central hub that manages message routing (Kafka, RabbitMQ, Google Cloud Pub/Sub, AWS SNS/SQS)
- Subscribers: Services that consume email events and perform specific actions (sending emails, updating databases, triggering workflows)
- Email Service Provider (ESP): The external service responsible for actual email delivery (SendGrid, Mailchimp, Amazon SES)
Gmail Integration Using Pub/Sub
Gmailfs API with pub/sub notifications provides a powerful way to integrate email functionality into your applications. Lets explore how this works in practice:
1. Google Cloud Setup
- Open the Google Cloud Console.
- Click on IAM and Admin
- Create a project and name it (e.g., “Your Project Name”)
2. Enable Required APIs
Go to APIs & Services > Library and enable the following:
- Gmail API
- Cloud Pub/Sub API
3. Create Service Account and Credentials:
- Go to IAM & Admin > Service Accounts > Create Service Account
- Grant access to this service account with the role: Pub/Sub Publisher.
- Go to the Keys tab and create a new key.
- Select Key type, we choose JSON
- Download the JSON key file (keep it safe!). Please note, if this is lost, it can create a security risk.
4. We need to Grant Gmail Access via OAuth2
Please note: Google does not allow service accounts to access Gmail unless you are a G Suite/Workspace admin using domain-wide delegation. For personal accounts, we need to use OAuth access.
- Open APIs & Services > Credentials and click on Create Credentials
- Choose OAuth client ID
- Application Type, choose based on your application. (We use Web application in this example.)
- Either you can add JavaScript Origins or Redirect URIs for your application. We will use the Redirect URI as we need to configure a webhook as below:
https://yourdomain.com/oauth2callback (adjust as needed) - Please save the Client ID and Client Secret.
Now open the Cloud Shell.
5. Create a Pub/Sub Topic in Google Cloud
First, you’ll need to create a topic in Google Cloud Pub/Sub that will receive notifications from Gmail. Run the following command:
1
gcloud pubsub topics create gmail-notificationsPlease note: gmail-notifications is the topic name — you can choose your own name.
Now, you need to subscribe to the topic.
1
gcloud pubsub subscriptions create email-worker-sub --topic=email-notifications6. IAM Role Permissions for Pub/Sub
Ensure the service account has the correct roles and permissions assigned to it.
- Go to IAM & Admin > IAM
- Provide access (grant the following roles) to the service account:
- Pub/Sub Publisher (for production)
- Pub/Sub Subscriber (for manual testing)
- Pub/Sub Editor (optional for development)
7. Handling Gmail Push Notifications with FastAPI + Pub/Sub
If you also need to consume Gmail-Watch notifications (the Gmail API can publish a Pub/Sub message every time a mailbox changes), drop the following FastAPI endpoint into the same service that processes your email-events topic. It receives the HTTP push that Cloud Pub/Sub sends to your webhook, decodes the base-64 payload, and prints the change record.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
from fastapi import FastAPI, Request, status
import base64
import json
app = FastAPI()
@app.post("/gmail/webhook", status_code=status.HTTP_200_OK)
async def gmail_webhook(request: Request):
"""
Endpoint for Cloud Pub/Sub push subscriptions that forward Gmail ‘watch’
notifications. Pub/Sub wraps the notification in a JSON envelope:
{
"message": {
"data": "base64-encoded-json",
"messageId": "1234567890",
...
},
"subscription": "projects/-/subscriptions/your-sub"
}
We need to:
1. Decode the 'data' field (base-64).
2. Parse the inner JSON to read Gmail’s historyId / email metadata.
"""
envelope = await request.json()
message = envelope.get("message", {})
encoded = message.get("data")
if not encoded:
# Pub/Sub might occasionally send test messages with no data.
return {"status": "ignored – no data"}
decoded_bytes = base64.b64decode(encoded)
decoded_str = decoded_bytes.decode("utf-8")
gmail_event = json.loads(decoded_str)
# 👉 Replace with your own processing logic
# (e.g., push historyId into another Pub/Sub topic or queue a worker)
print("📬 Gmail push notification:", json.dumps(gmail_event, indent=2))
return {"status": "success"}Note:
Gmail watch() expires every 7 days
Set a cron job or schedulercron job or scheduler to refresh it automatically.
You have to set up the webhook URL for push notification as below
Gmail Subscriber
When moving your Gmail Pub/Sub integration to production, follow these best practices:
- 1. Secure Your Webhook Endpoint
Always use HTTPS and implement proper authentication for your webhook. - 2. Implement Comprehensive Logging
Track all important events in your integration. - 3. Create a Resilient Email Processing Pipeline
Design your system to handle failures gracefully.
Common Challenges and Solutions
- 1. Authentication Failures
Challenge: OAuth2 tokens expire or become invalid. Gmail only allows API calls with user-granted scopes; access tokens expire after 1 hour.
Solution: Implement token refresh logic and handle authentication errors gracefully. OR
Use offline access (access_type=offline) so Google issues a refresh token. - 2. Push Notification Delivery Issues
Challenge: Webhook endpoint unreachable or returning errors.
Solution: Implement health checks and monitoring for your webhook endpoint. - 3. Processing High Email Volume
Challenge: Heavy email traffic overwhelming your processing capacity.
Solution: Implement rate limiting and batching. - 4. Watch Expiration (push notifications)
Challenge: users.watch expires after ~7 days; if you forget to renew, you miss events.
Solution: Schedule a Cloud Scheduler / cron job to renew every 6 days; persist the latest historyId so you can replay missed events with users.history.list. - 5. Duplicate or Out-of-Order Webhooks
Challenge: Pub/Sub guarantees at-least-once delivery; Gmail can emit several notifications for the same change.
Solution: Deduplicate by the combination (historyId, messageId) before processing. Use a Redis set, Firestore document, or SQL unique index as a bloom.
Conclusion
Integrating your email workflows with a Pub/Sub messaging system brings robustness, scalability, and flexibility to your communications infrastructure. By decoupling event production from email delivery, you can handle varying workloads, ensure deliverability, and add new consumers (analytics, archiving) with minimal changes.
