Using SNS for S3 Observer Event Fan-Out
Use Amazon Simple Notification Service (SNS) to send S3 event notifications to multiple destinations, including Borneo's Observer service.
The "Observer" service is part of the Borneo for AWS Data Infrastructure solution. It uses Amazon S3 Event Notifications to detect when new objects are added to monitored S3 buckets, and ensures the new objects get scanned for sensitive infotypes. Amazon S3 only supports a single destination for each supported event type, but the Amazon Simple Notification Service (SNS) can be used to fan out notifications to multiple destinations.
Here's how to set up S3 event fan-out to multiple destinations using SNS.
Update the Observer queue to allow SNS to publish messages
Update the access policy of the Borneo Observer service SQS queue to allow Amazon SNS to publish messages to the queue in addition to S3. (Note: This step is required for Borneo versions v1.7.x and earlier.)
Find the Observer queue with the name
prod-s3-observer-queue in the SQS Console and update its access policy to add an sns.amazonaws.com to the list of service principals:
Required changes to Observer queue access policy
Create a new SNS topic for fan-out
Create a new SNS topic in the same AWS account where the S3 bucket is located. This topic will receive S3 event notifications, so the topic's access policy must allow S3 to publish messages to the topic:
1
{
2
"Sid": "s3-allow-publish",
3
"Effect": "Allow",
4
"Principal": {
5
"Service": "s3.amazonaws.com"
6
},
7
"Action": "SNS:Publish",
8
"Resource": "arn:aws:sns:<AWS-Region>:<AWS-Account-ID>:<Topic-Name>",
9
"Condition": {
10
"StringEquals": {
11
"aws:SourceAccount": "<AWS-Account-ID>"
12
}
13
}
14
}
Copied!
Update the AWS region, account ID, and topic name for the new topic.
Next, subscribe the Borneo Observer service queue to the new topic, using the SQS protocol. Ensure you enable raw message delivery, for the subscription.
Observer service queue subscription to the fan-out SNS topic
Subscribe any other services that should receive S3 events.
​
Subscribe the SNS topic to S3 event notifications
Now you can update the S3 event notifications for the S3 bucket to send object create events to the new SNS topic:
Fan-out SNS topic subscription to S3 event notifications
Last modified 6mo ago