\n<\/p>\n
Amazon SageMaker Characteristic Retailer<\/a> is a totally managed, purpose-built repository to retailer, share, and handle options for machine studying (ML) fashions. It offers low-latency on-line serving for real-time inference, an offline retailer for historic retention and coaching characteristic knowledge, and helps each streaming<\/a> and batch<\/a> ingestion patterns.<\/p>\n As ML platforms mature, two operational gaps floor repeatedly. First, groups operating high-throughput characteristic pipelines should name PutRecord<\/a> (which writes a single characteristic file to the web retailer) in a loop. This implies one API name per file, per characteristic group, which creates connection overhead and poor throughput. A fraud-detection pipeline ingesting 10,000 data per second throughout 5 characteristic teams should maintain 50,000 particular person API calls per second solely to maintain options present. A second problem is that groups utilizing the In-Reminiscence storage tier haven’t any strategy to browse or enumerate data saved within the on-line retailer. If file identifiers are misplaced by way of a bug or pipeline failure, these data turn into completely unrecoverable. There isn’t any offline retailer for the In-Reminiscence tier to fall again on, no Amazon Athena<\/a> question to run, and no API to find what exists.<\/p>\n As we speak, we’re saying two new APIs<\/a> for Amazon SageMaker Characteristic Retailer:<\/p>\n On this put up, we stroll by way of every API with code examples you should utilize to get began.<\/p>\n To comply with together with the examples on this put up, you want:<\/p>\n The BatchWriteRecord API tackles the throughput limits of single-record ingestion. The next sections clarify the issue it solves and the way it works.<\/p>\n The present PutRecord<\/a> API in Characteristic Retailer writes one file to 1 characteristic group per name. Every name performs a conditional write: the file is continued because the \u201cnewest\u201d model provided that its EventTime, included within the request, is newer than the present file. If the situation fails, the file remains to be written as a historic model for the offline retailer.<\/p>\n This design offers robust ordering ensures, however at scale it forces an N\u00d7M calling sample (N data \u00d7 M characteristic teams), creating connection overhead and tail latency that restrict throughput.<\/p>\n BatchWriteRecord<\/a> accepts as much as 25 entries in a single request, focusing on a number of characteristic teams concurrently. Every file succeeds or fails independently. It is a partial-success API, which means particular person file failures don’t fail your complete request.<\/p>\n The API preserves the identical EventTime-based ordering as PutRecord<\/a>:<\/p>\n The response returns solely the data that failed:<\/p>\n Information not listed in Errors<\/a> or UnprocessedEntries<\/a> succeeded. Your software ought to retry solely the failed data utilizing exponential backoff for retriable errors.<\/p>\n You possibly can goal a number of characteristic teams in a single request. Information are grouped by characteristic group and processed independently:<\/p>\n A failure in a single characteristic group doesn’t have an effect on data destined for different characteristic teams.<\/p>\n BatchWriteRecord<\/a> helps TTL<\/a> at three ranges of priority, proven within the following precedence order:<\/p>\n Most 25 entries per request. This restrict applies to the whole variety of entries throughout all characteristic teams in a single request.<\/p>\n Partial-success semantics: Not like transactional APIs, BatchWriteRecord doesn’t roll again profitable writes if some data fail. Design your retry logic to re-submit solely the data returned in Errors.<\/p>\n Comparable IAM mannequin as PutRecord: The caller will need to have sagemaker:BatchWriteRecord and sagemaker:PutRecord permission on the Amazon Useful resource Identify (ARN) of every goal characteristic group. Per-feature-group authorization is checked earlier than processing.<\/p>\n EventTime ordering is preserved: BatchWriteRecord makes use of conditional writes to keep up the identical latest-record-wins semantics as PutRecord. A stale file can’t overwrite a more moderen one within the on-line retailer.<\/p>\n TargetStores flexibility: Every entry can independently goal OnlineStore, OfflineStore, or each (defaults to the characteristic group\u2019s enabled shops), providing you with fine-grained management over the place every file lands.<\/p>\n The ListRecords API closes the hole in file discovery for each storage tiers. The next sections clarify the issue it solves and the way it works.<\/p>\n Characteristic Retailer helps PutRecord<\/a>, GetRecord<\/a>, and DeleteRecord<\/a>, however all require the caller to know the precise file identifier. There isn’t any API to browse or enumerate data inside a characteristic group.<\/p>\n For the Normal tier, the workaround is querying the offline retailer through the use of Amazon Athena<\/a>. This requires offline retailer configuration, provides value, and isn’t real-time.<\/p>\n For the In-Reminiscence tier, the state of affairs is crucial. There isn’t any corresponding offline retailer by default. If file identifiers are misplaced, these data are fully unrecoverable. You can’t uncover them, and you can not delete them. This results in phantom knowledge, wasted storage prices, and potential compliance dangers when knowledge topics request deletion.<\/p>\n ListRecords<\/a> enumerates file identifiers inside a characteristic group utilizing pagination. It returns solely energetic, non-deleted, non-expired data which are prepared for use with GetRecord<\/a> or DeleteRecord<\/a>.<\/p>\n The API works with each storage tiers:<\/p>\n Request physique:<\/p>\n Preliminary name<\/p>\n Or<\/p>\n Response:<\/p>\n When NextToken is absent within the response, pagination is full.<\/p>\n A typical use case is figuring out and deleting data which are now not wanted. That is crucial for In-Reminiscence tier characteristic teams, the place orphaned data persist indefinitely:<\/p>\n Report identifiers solely. The present launch returns file identifiers with out characteristic values. Use GetRecord<\/a> or BatchGetRecord<\/a> to retrieve full data for the identifiers you want.<\/p>\n Automated filtering. The API excludes soft-deleted data, expired data (Normal tier TTL), and inner system keys (In-Reminiscence tier). You see solely energetic, retrievable data.<\/p>\n IAM permission. The caller will need to have sagemaker:ListRecords permission on the characteristic group ARN.<\/p>\n Each tiers supported. ListRecords works identically from the caller\u2019s perspective no matter whether or not the characteristic group makes use of Normal or In-Reminiscence storage.<\/p>\n These two APIs complement one another naturally. Think about a compliance workflow that verifies full knowledge deletion for a person throughout a number of characteristic teams:<\/p>\n To keep away from ongoing costs, delete characteristic teams you created whereas following this walkthrough. For In-Reminiscence tier characteristic teams, use ListRecords<\/a> to enumerate data and DeleteRecord<\/a> to take away them earlier than deleting the characteristic group.<\/p>\n BatchWriteRecord<\/a> and ListRecords<\/a> present key enhancements within the knowledge aircraft of Amazon SageMaker Characteristic Retailer. BatchWriteRecord reduces the API name quantity for high-throughput ingestion by as much as 25x whereas preserving the EventTime-based ordering ensures that hold your on-line retailer appropriate. ListRecords unlocks file discovery and lifecycle administration. That is crucial for In-Reminiscence tier prospects who beforehand had no strategy to enumerate or clear up their knowledge.<\/p>\n Collectively, these APIs assist patterns that have been beforehand tough or unimaginable: bulk ingestion pipelines with fewer connections and decrease latency, compliance workflows that may confirm full knowledge deletion, and operational tooling that may browse characteristic group contents in actual time.<\/p>\n For extra data, see the Characteristic Retailer documentation<\/a>, the Characteristic Retailer API reference<\/a>, the offline retailer configuration documentation<\/a>, and the What\u2019s New announcement<\/a>.<\/p>\n For background on Characteristic Retailer capabilities, discover these associated posts:<\/p>\n\n
Stipulations<\/h2>\n
BatchWriteRecord<\/h2>\n
The problem with single-record ingestion<\/h3>\n
How BatchWriteRecord works<\/h3>\n
\n
Request construction<\/h3>\n
{\n \"Entries\": [\n {\n \"FeatureGroupName\": \"click-features\",\n \"Record\": [\n {\"FeatureName\": \"user_id\", \"ValueAsString\": \"user-123\"},\n {\"FeatureName\": \"event_time\", \"ValueAsString\": \"2026-06-05T12:00:00Z\"},\n {\"FeatureName\": \"click_count\", \"ValueAsString\": \"42\"}\n ],\n \"TargetStores\": [\"OnlineStore\", \"OfflineStore\"],\n \"TtlDuration\": {\"Unit\": \"Days\", \"Worth\": 7}\n },\n {\n \"FeatureGroupName\": \"login-features\",\n \"Report\": [\n {\"FeatureName\": \"user_id\", \"ValueAsString\": \"user-456\"},\n {\"FeatureName\": \"event_time\", \"ValueAsString\": \"2026-06-05T12:00:01Z\"},\n {\"FeatureName\": \"login_count\", \"ValueAsString\": \"18\"}\n ],\n \"TargetStores\": [\"OnlineStore\", \"OfflineStore\"]\n }\n ]\n}<\/code><\/pre>\n<\/p><\/div>\n{\n \"Errors\": [\n {\n \"Entry\": {\n \"FeatureGroupName\": \"string\",\n \"Record\": [\n {\n \"FeatureName\": \"string\",\n \"ValueAsString\": \"string\",\n \"ValueAsStringList\": [\"string\"]\n }\n ],\n \"TargetStores\": [\"string\"],\n \"TtlDuration\": {\n \"Unit\": \"string\",\n \"Worth\": quantity\n }\n },\n \"ErrorCode\": \"string\",\n \"ErrorMessage\": \"string\"\n }\n ],\n \"UnprocessedEntries\": [\n {\n \"FeatureGroupName\": \"string\",\n \"Record\": [\n {\n \"FeatureName\": \"string\",\n \"ValueAsString\": \"string\",\n \"ValueAsStringList\": [\"string\"]\n }\n ],\n \"TargetStores\": [\"string\"],\n \"TtlDuration\": {\n \"Unit\": \"string\",\n \"Worth\": quantity\n }\n }\n ]\n}<\/code><\/pre>\n<\/p><\/div>\nCode instance: Batch ingestion with Boto3<\/h3>\n
import boto3\n\nfeaturestore_runtime = boto3.consumer(\"sagemaker-featurestore-runtime\")\n\nresponse = featurestore_runtime.batch_write_record(\n Entries=[\n {\n \"FeatureGroupName\": \"click-features\",\n \"Record\": [\n {\"FeatureName\": \"user_id\", \"ValueAsString\": \"user-123\"},\n {\"FeatureName\": \"event_time\", \"ValueAsString\": \"2026-06-05T12:00:00Z\"},\n {\"FeatureName\": \"click_count\", \"ValueAsString\": \"42\"},\n ],\n \"TargetStores\": [\"OnlineStore\", \"OfflineStore\"],\n },\n {\n \"FeatureGroupName\": \"login-features\",\n \"Report\": [\n {\"FeatureName\": \"user_id\", \"ValueAsString\": \"user-456\"},\n {\"FeatureName\": \"event_time\", \"ValueAsString\": \"2026-06-05T12:00:01Z\"},\n {\"FeatureName\": \"login_count\", \"ValueAsString\": \"18\"},\n ],\n \"TargetStores\": [\"OnlineStore\", \"OfflineStore\"],\n },\n ]\n)\n\nif response[\"Errors\"]:\n for error in response[\"Errors\"]:\n print(f\"Report {error['Entry']}, ErrorCode: {error['ErrorCode']} Failed: {error['ErrorMessage']}\")\n\nif response[\"UnprocessedEntries\"]:\n for unprocessed in response[\"UnprocessedEntries\"]:\n print(f\"Unprocessed: {unprocessed['FeatureGroupName']}\")\n\nif not response[\"Errors\"] and never response[\"UnprocessedEntries\"]:\n print(\"All data written efficiently.\")<\/code><\/pre>\n<\/p><\/div>\nCode instance: Writing throughout a number of characteristic teams<\/h3>\n
featurestore_runtime = boto3.consumer(\"sagemaker-featurestore-runtime\")\n\nresponse = featurestore_runtime.batch_write_record(\n Entries=[\n {\n \"FeatureGroupName\": \"user-profile-features\",\n \"Record\": [\n {\"FeatureName\": \"user_id\", \"ValueAsString\": \"user-123\"},\n {\"FeatureName\": \"event_time\", \"ValueAsString\": \"2026-06-05T12:00:00Z\"},\n {\"FeatureName\": \"age\", \"ValueAsString\": \"34\"},\n {\"FeatureName\": \"region\", \"ValueAsString\": \"us-west-2\"},\n ],\n \"TargetStores\": [\"OnlineStore\"],\n },\n {\n \"FeatureGroupName\": \"click-features\",\n \"Report\": [\n {\"FeatureName\": \"user_id\", \"ValueAsString\": \"user-123\"},\n {\"FeatureName\": \"event_time\", \"ValueAsString\": \"2026-06-05T12:00:00Z\"},\n {\"FeatureName\": \"click_count\", \"ValueAsString\": \"42\"},\n ],\n \"TargetStores\": [\"OnlineStore\", \"OfflineStore\"],\n },\n ]\n)<\/code><\/pre>\n<\/p><\/div>\nTTL (Time-to-Stay) assist<\/h3>\n
\n
Key concerns<\/h3>\n
ListRecords<\/h2>\n
The problem with file discovery<\/h3>\n
How ListRecords works<\/h3>\n
\n
Request and response construction<\/h3>\n
POST \/FeatureGroup\/{FeatureGroupName}\/ListRecords<\/code><\/pre>\n<\/p><\/div>\n{\n \"MaxResults\": 50,\n \"NextToken\": \"eyJjdXJzb3IiOi4uLn0=\"\n}<\/code><\/pre>\n<\/p><\/div>\n{\n \"RecordIdentifiers\": [\n \"user-001\",\n \"user-002\",\n \"user-003\"\n ],\n \"NextToken\": \"eyJuZXh0IjoiLi4ufQ==\"\n}<\/code><\/pre>\n<\/p><\/div>\nCode instance: Enumerate all data in a characteristic group<\/h3>\n
import boto3\n\nfeaturestore_runtime = boto3.consumer(\"sagemaker-featurestore-runtime\")\n\nall_identifiers = []\nnext_token = None\n\nwhereas True:\n params = {\n \"FeatureGroupName\": \"user-profile-features\",\n \"MaxResults\": 100,\n }\n if next_token:\n params[\"NextToken\"] = next_token\n\n response = featurestore_runtime.list_records(**params)\n all_identifiers.prolong(response[\"RecordIdentifiers\"])\n\n next_token = response.get(\"NextToken\")\n if not next_token:\n break\n\nprint(f\"Discovered {len(all_identifiers)} energetic data.\")<\/code><\/pre>\n<\/p><\/div>\nCode instance: Clear up orphaned data<\/h3>\n
import boto3\n\nfeaturestore_runtime = boto3.consumer(\"sagemaker-featurestore-runtime\")\n\n# Step 1: Enumerate all file identifiers\nall_ids = []\nnext_token = None\nwhereas True:\n params = {\"FeatureGroupName\": \"session-features\", \"MaxResults\": 100}\n if next_token:\n params[\"NextToken\"] = next_token\n response = featurestore_runtime.list_records(**params)\n all_ids.prolong(response[\"RecordIdentifiers\"])\n next_token = response.get(\"NextToken\")\n if not next_token:\n break\n\n# Step 2: Examine towards your software's energetic session checklist\nactive_sessions = get_active_sessions() # Your software logic\norphaned = [rid for rid in all_ids if rid not in active_sessions]\n\n# Step 3: Delete orphaned data\nfor record_id in orphaned:\n featurestore_runtime.delete_record(\n FeatureGroupName=\"session-features\",\n RecordIdentifierValueAsString=record_id,\n EventTime=\"2026-06-05T12:00:00Z\",\n )\n\nprint(f\"Deleted {len(orphaned)} orphaned data.\")<\/code><\/pre>\n<\/p><\/div>\n\n
Key concerns<\/h3>\n
Placing it collectively<\/h2>\n
import boto3\n\nfeaturestore_runtime = boto3.consumer(\"sagemaker-featurestore-runtime\")\n\nfeature_groups = [\"user-profiles\", \"click-history\", \"purchase-signals\"]\nuser_to_delete = \"user-789\"\n\n# Step 1: Discover and delete the person throughout all characteristic teams\nfor fg_name in feature_groups:\n all_ids = []\n next_token = None\n whereas True:\n params = {\"FeatureGroupName\": fg_name, \"MaxResults\": 100}\n if next_token:\n params[\"NextToken\"] = next_token\n response = featurestore_runtime.list_records(**params)\n all_ids.prolong(response[\"RecordIdentifiers\"])\n next_token = response.get(\"NextToken\")\n if not next_token:\n break\n\n if user_to_delete in all_ids:\n featurestore_runtime.delete_record(\n FeatureGroupName=fg_name,\n RecordIdentifierValueAsString=user_to_delete,\n EventTime=\"2026-06-05T23:59:59Z\",\n )\n print(f\"Deleted '{user_to_delete}' from {fg_name}\")\n\n# Step 2: Log the deletion occasion utilizing BatchWriteRecord\nfeaturestore_runtime.batch_write_record(\n Entries=[\n {\n \"FeatureGroupName\": \"deletion-audit-log\",\n \"Record\": [\n {\"FeatureName\": \"request_id\", \"ValueAsString\": \"del-001\"},\n {\"FeatureName\": \"event_time\", \"ValueAsString\": \"2026-06-05T23:59:59Z\"},\n {\"FeatureName\": \"user_id\", \"ValueAsString\": user_to_delete},\n {\"FeatureName\": \"status\", \"ValueAsString\": \"completed\"},\n {\"FeatureName\": \"feature_groups_cleaned\", \"ValueAsString\": \"3\"},\n ],\n \"TargetStores\": [\"OnlineStore\", \"OfflineStore\"],\n }\n ]\n)<\/code><\/pre>\n<\/p><\/div>\nCleanup<\/h2>\n
Conclusion<\/h2>\n
\nConcerning the authors<\/h2>\n