Amazon Connect Call Recordings in S3: Ingesting Audio and Metadata Together

Amazon Connect does not write a call to S3 in one piece, which is the first thing anyone connecting a recording bucket to a redaction pipeline discovers. The audio, the contact record and any analytics output arrive at different moments and by different routes. A pipeline therefore has to reassemble each call and pick it up exactly once before any detection can run, and a call that goes missing or turns up twice usually traces back to that step.

Getting calls in is the first of the failure points in our guide to what breaks in call recording redaction, and it is the one integration engineers own. Connect makes a useful worked example because AWS documents its storage behavior in unusual detail. The same pattern holds for any contact center platform that writes recordings and metadata files separately, so the practices below carry over even where the paths and field names differ.

What Connect writes to S3, and when

An agent call recording is placed in the instance's S3 bucket shortly after the contact disconnects, according to AWS's page on when, what and where for contact recordings. Connect uploads recordings with either PutObject or a multipart upload, so AWS advises enabling notifications for all object create events, or for both the Put and CompleteMultipartUpload event types. The recording's path under the configured prefix includes the year, month and day it was made, as AWS notes in its walkthrough of S3 Object Lock for call recordings.

The contact record, which lists the recording's S3 location among its fields, travels by a different route altogether. With data streaming enabled, contact records go to a Kinesis stream or Firehose rather than into the bucket. AWS's contact record data model says records are delivered at least once and might be delivered again, for example when new information arrives after the first delivery. Consumers are told to check LastUpdateTimestamp for newer data and to deduplicate on ContactId. The recording also appears in the contact record only after the contact has left the After Contact Work state.

Conversational analytics, where it is enabled, adds a third set of files that arrive later still. A post-call analysis job starts after each contact completes, and AWS's service quotas page suggests planning on about 40% of the call length for it to finish. The analysis file and a redacted copy of the audio then land at their own paths, keyed by contact ID and by the time the agent connected.

File names add a further complication that AWS itself flags on the recording behavior page. Many recordings are named with the contact ID as a prefix, but AWS says there is no guarantee that the contact ID and the file name always match. Audio and metadata therefore reach a pipeline at different moments and through different routes, and a pipeline that assumes otherwise will eventually lose track of a call.

Pairing a call's audio with its metadata

Only a few signals reliably tie a call's files together, and most of them live in file names and folder paths. Files can be grouped on a shared part of their names, such as an identifier at a fixed position or one picked out by a pattern, or on the folder they sit in. Each file in a group then needs a role, so the pipeline knows which one is the recording, which is a caption file and which is supporting metadata.

We met both halves of the ingest problem in a deployment handling thousands of short calls a day. Each call's metadata file landed in storage at a different moment from its audio. Taking the audio as soon as it appeared would have created items with no metadata and left the later files as orphans or second items. Long pickup runs were also cut off now and then, by a network error or by someone disabling the connector partway through a sync. Starting again from the top would have taken every call a second time, and a name clash would have produced a copy called "call (1)" beside the original.

Redactor's answer is to take a call in only once every expected file is present, leaving an incomplete call for the next run. Each item also records the key of the source object it came from. A restart then skips every file whose key the library already holds, so no second copy appears. A genuine name clash between different files is resolved rather than failing the item, and an item whose creation fails is retried. The cost is a short delay on a call whose files straggle in, which is far cheaper than a clean-up exercise across thousands of items.

A metadata file that never arrives needs a policy, or its call will wait indefinitely. After a set interval the pipeline should alert someone or ingest the audio alone, and the right choice depends on what the metadata is used for downstream. Pairing by name has a quieter failure mode, because a recording whose file name does not carry its contact ID will never match its metadata and may never be noticed. Where contact records arrive through a stream instead of as files in the bucket, pairing becomes a join between two systems, and that join inherits the stream's delivery rules, duplicates included.

Events or a schedule

Every signal a pipeline might use to notice a new call can repeat, arrive late or stop partway, which makes exactly-once pickup harder than it sounds. AWS says S3 event notifications are designed to be delivered at least once, usually within seconds but sometimes after a minute or longer. Its page on event ordering and duplicate events adds that notifications can arrive out of order and that, on rare occasions, a retry produces a duplicate for the same object event. Bucket listings are no simpler, since ListObjectsV2 returns keys a page at a time and a long run through a busy bucket can be interrupted anywhere.

Pipelines usually either react to storage events or poll the bucket on a schedule, and the choice trades latency against robustness. Reacting to events picks up a call within seconds, but it inherits the at-least-once delivery, the occasional delay and the rare duplicate, so the consumer has to be idempotent anyway. Polling picks a call up on the next run instead of the moment it lands, and a late or repeated notification cannot cause double ingestion because notifications play no part. Redactor's connector polls on a schedule, with idempotent pickup as the backstop for everything else.

A recording bucket rarely holds recordings alone after a few years of use, so scoping matters as much as timing. An extension allowlist keeps the pipeline to audio and its companion files, and excluded folders keep it out of paths that should not be ingested, such as analytics output. The folder hierarchy can then be preserved or flattened to suit whoever searches the library later.

Whether call metadata needs to be searchable, or only needs to travel with the audio, is a decision to make before configuring anything. Fields inside a metadata file, such as the queue, the agent and custom contact attributes, differ from metadata written on the S3 object itself. Turning them into searchable fields is separate work, and it only makes sense if someone will actually search on them.

The source file after ingest

Once a call is ingested, the pipeline can leave the source object where it is, move it elsewhere, or delete it, and each option suits a different retention policy. Whichever is chosen, the clean-up must never run ahead of the ingest it depends on. A connector that moves or deletes a file whose item failed to create loses the recording outright, with no copy left anywhere. The post-ingestion action should therefore run only for files whose item is confirmed in the library.

Connect's analytics raises a separate retention question, because with redaction enabled it leaves a redacted and an unredacted version of the same call side by side. AWS's output locations page is direct about the consequence: to fully remove a recording, you must delete both files, and deleting only one leaves the other accessible. A retention policy for the recording bucket has to cover both versions, whatever the redaction pipeline does with its own copies.

Sending redacted calls back out

Redacted calls often have to return to storage for another system to collect, and the receiving system usually matches them to its own records by folder and file name. The return trip should therefore keep the original folder layout and file names, carry the metadata files alongside, and export each call once. An interruption should resume where it stopped rather than force a full re-export, and every call should carry a status showing whether it has gone. Clean-up after export should wait for a confirmed export, and a completion event tells the next system there is something to collect.

What the redacted files contain matters as much as how they travel back to the systems that collect them. Our article on chain of custody for redacted call audio shows how an uncompressed WAV can come back identical to its source outside the redacted spans. Sizing this flow for a full day of calls is the subject of our article on bulk call recording redaction.

Pointing Redactor at a Connect recording bucket

A Connect recording bucket can be read directly by VIDIZMO Redactor through its Amazon S3 connector, scoped by an extension allowlist and excluded folders, with the folder hierarchy kept or flattened. Related files are grouped into one item with each file assigned its role, metadata written on the S3 object becomes searchable attributes of the item, and the source is left, moved or deleted after ingestion as configured.

On the way out, a stored query drives export to a bucket you name, existing content can be backfilled, the folder structure is recreated at the destination and every export is logged. Webhooks tell the next system when a workflow completes or fails, and each delivery is logged with its retries. The connectors, export and notifications all come with the platform Redactor runs on, so the whole round trip runs inside one deployment.

For the whole bucket-to-bucket flow with no operator opening a file, see the S3 to S3 redaction pipeline.