In my earlier put up<\/a>, Immediate Caching \u2014 what it’s, the way it works, and the way it can prevent some huge cash and time when working AI-powered apps with excessive visitors. In right this moment\u2019s put up, I stroll you thru implementing Immediate Caching particularly utilizing OpenAI\u2019s API, and we focus on some widespread pitfalls.<\/p>\n Earlier than getting our fingers soiled, let\u2019s briefly revisit what precisely the idea of Immediate Caching is. Immediate Caching is a performance offered in frontier mannequin API companies just like the OpenAI API<\/a> or Claude\u2019s API,<\/a> that enables caching and reusing components of the LLM\u2019s enter which might be repeated steadily. Such repeated components could also be system prompts or directions which might be handed to the mannequin each time when working an AI app, together with another variable content material, just like the consumer\u2019s question or data retrieved from a information base. To have the ability to hit cache with immediate caching, the repeated components of the immediate should be at the start of it, particularly, a immediate prefix<\/em><\/strong>. As well as, to ensure that immediate caching to be activated, this prefix should exceed a sure threshold <\/em><\/strong>(e.g., for OpenAI the prefix needs to be greater than 1,024 tokens, whereas Claude has totally different minimal cache lengths for various fashions). So far as these two situations are happy \u2014 repeated tokens as a prefix exceeding the dimensions threshold outlined by the API service and mannequin \u2014 caching may be activated to attain economies of scale when working AI apps.<\/p>\n In contrast to caching in different elements in a RAG or different AI app, immediate caching operates on the token stage, within the inner procedures of the LLM. Particularly, LLM inference takes place in two steps:<\/p>\n Briefly, immediate caching shops the computations that happen within the pre-fill stage, so the mannequin doesn\u2019t must recompute it once more when the identical prefix reappears. Any computations happening within the decoding iterations section, even when repeated, aren\u2019t going to be cached.<\/p>\n For the remainder of the put up, I can be focusing solely on using immediate caching within the OpenAI API.<\/p>\n In OpenAI\u2019s API, immediate caching was initially launched on the 1st of October 2024<\/a>. Initially, it supplied a 50% low cost on the cached tokens, however these days, this low cost goes as much as 90%. On prime of this, by hitting their immediate cache, further financial savings on latency may be achived as much as 80%.<\/p>\n When immediate caching is activated, the API service makes an attempt to hit the cache for a submitted request by routing the submitted immediate to an acceptable machine, the place the respective cache is predicted to exist. That is known as the Cache Routing, and to do that, the API service usually makes use of a\u00a0hash of the primary 256 tokens of the immediate. <\/p>\n Past this, their API additionally permits for explicitly defining a the\u00a0 As well as, OpenAI API offers two distinct sorts of caching with reference to length, outlined via the Now, with reference to how a lot all these value, OpenAI expenses the identical per enter (non cached) token, both we’ve immediate caching activated or not. If we handle to hit cache succesfully, we’re billed for the cached tokens at a drastically discounted value, with a reduction as much as 90%. Furthermore, the value per enter token stays the identical each for the in reminiscence and prolonged cache retention.<\/p>\n So, let\u2019s see how immediate caching truly works with a easy Python instance utilizing OpenAI\u2019s API service. Extra particularly, we’re going to do a sensible situation the place a lengthy system immediate (prefix)<\/strong> is reused throughout a number of requests. In case you are right here, I assume you have already got your OpenAI API key<\/a> in place and have put in the required libraries. So, the very first thing to do could be to import the then we will outline our prefix (the tokens which might be going to be repeated and we’re aiming to cache):<\/p>\n Discover how we artificially enhance the size (multiply with 200) to ensure the 1,024 token caching threshold is met. Then we additionally arrange a timer in order to measure our latency financial savings, and we’re lastly able to make our name:<\/p>\n So, what can we anticipate to occur from right here? For fashions from gpt-4o and newer, immediate caching is activated by default, and since our 4,616 enter tokens are properly above the 1,024 prefix token threshold, we’re good to go. Thus, what this request does is that it initially checks if the enter is a cache hit (it’s not, since that is the primary time we do a request with this prefix), and since it’s not, it processes your complete enter after which caches it. Subsequent time we ship an enter that matches the preliminary tokens of the cached enter to some extent, we’re going to get a cache hit. Let\u2019s test this in apply by making a second request with the identical prefix:<\/p>\n Certainly! The second request runs considerably sooner (23.31 vs 15.37 seconds). It is because the mannequin has already made the calculations for the cached prefix and solely must course of from scratch the brand new half, \u201cWhat’s regularization?\u201d. Consequently, through the use of immediate caching, we get considerably decrease latency and lowered value, since cached tokens are discounted.<\/p>\n One other factor talked about within the OpenAI documentation we\u2019ve already talked about is the \ud83e\udd14<\/p>\n It appears that evidently whereas Activating immediate caching and truly hitting the cache appears to be sort of easy from what we\u2019ve stated to this point. So, what might go improper, leading to us lacking the cache? Sadly, lots of issues. As easy as it’s, immediate caching requires lots of totally different assumptions to be in place. Lacking even a kind of conditions goes to end in a cache miss. However let\u2019s take a greater look!<\/p>\n One apparent miss is having a prefix that’s lower than the edge for activating immediate caching, particularly, lower than 1,024 tokens. Nonetheless, that is very simply solvable \u2014 we will at all times simply artificially enhance the prefix token depend by merely multiplying by an acceptable worth, as proven within the instance above.<\/p>\n One other factor could be silently breaking the prefix. Particularly, even once we use persistent directions and system prompts of acceptable measurement throughout all requests, we should be exceptionally cautious to not break the prefixes by including any variable content material at the start of the mannequin\u2019s enter, earlier than the prefix. That may be a assured approach to break the cache, irrespective of how lengthy and repeated the next prefix is. Regular suspects for falling into this pitfall are dynamic information, as an example, appending the consumer ID or timestamps at the start of the immediate. Thus, a greatest apply to observe throughout all AI app growth is that any dynamic content material ought to at all times be appended on the finish of the immediate \u2014 by no means at the start.<\/p>\n Finally, it’s price highlighting that immediate caching is simply concerning the pre-fill section \u2014 decoding is rarely cached. Which means even when we impose on the mannequin to generate responses following a particular template, that beggins with sure mounted tokens, these tokens aren\u2019t going to be cached, and we’re going to be billed for his or her processing as normal.<\/p>\n Conversely, for particular use instances, it doesn\u2019t actually make sense to make use of immediate caching. Such instances could be extremely dynamic prompts, like chatbots with little repetition, one-off requests, or real-time customized programs.<\/p>\n . . .<\/strong><\/p>\n Immediate caching can considerably enhance the efficiency of AI functions each by way of value and time. Particularly when trying to scale AI apps immediate caching comes extremelly useful, for sustaining value and latency in acceptable ranges. <\/p>\n For OpenAI\u2019s API immediate caching is activated by default and prices for enter, non-cached tokens are the identical both we activate immediate caching or not. Thus, one can solely win by activating immediate caching and aiming to hit it in each request, even when they don\u2019t succeed.<\/p>\n Claude additionally offers intensive performance on immediate caching via their API, which we’re going to be exploring intimately in a future put up.<\/p>\n Thanks for studying! \ud83d\ude42<\/p>\n . . .<\/strong><\/p>\n Cherished this put up? Let\u2019s be mates! Be part of me on: <\/em><\/p>\n \ud83d\udcf0Substack<\/a><\/em><\/strong> \ud83d\udc8c\u00a0Medium<\/a><\/strong><\/em> \ud83d\udcbcLinkedIn<\/a> <\/strong><\/em>\u2615Purchase me a espresso<\/a>!<\/strong><\/em><\/p>\n All pictures by the creator, besides talked about in any other case.<\/strong><\/em><\/p>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":" In my earlier put up, Immediate Caching \u2014 what it’s, the way it works, and the way it can prevent some huge cash and time when working AI-powered apps with excessive visitors. In right this moment\u2019s put up, I stroll you thru implementing Immediate Caching particularly utilizing OpenAI\u2019s API, and we focus on some widespread […]<\/p>\n","protected":false},"author":2,"featured_media":12988,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[664,7278,1813,660,82,152,1258,3028],"class_list":["post-12986","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-api","tag-caching","tag-full","tag-handson","tag-openai","tag-prompt","tag-python","tag-tutorial"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/12986","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=12986"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/12986\/revisions"}],"predecessor-version":[{"id":12987,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/12986\/revisions\/12987"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/12988"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=12986"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=12986"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=12986"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\nA quick reminder on Immediate Caching<\/h2>\n
\n
\nWhat concerning the OpenAI API?<\/h2>\n
prompt_cache_key<\/code><\/a> parameter within the API request to the mannequin. That may be a single key defining which cache we’re referring to, aiming to additional enhance the possibilities of our immediate being routed to the right machine and hitting cache.<\/p>\nprompt_cache_retention<\/code> parameter. These are:<\/p>\n\n
\nImmediate Caching in Observe<\/h2>\n
OpenAI<\/code> library, in addition to time<\/code> for capturing latency, and initialize an occasion of the OpenAI shopper:<\/p>\nfrom openai import OpenAI\nimport time\n\nshopper = OpenAI(api_key=\"your_api_key_here\")<\/code><\/pre>\nlong_prefix = \"\"\"\nYou're a extremely educated assistant specialised in machine studying.\nReply questions with detailed, structured explanations, together with examples when related.\n\n\"\"\" * 200 <\/code><\/pre>\nbegin = time.time()\n\nresponse1 = shopper.responses.create(\n mannequin=\"gpt-4.1-mini\",\n enter=long_prefix + \"What's overfitting in machine studying?\"\n)\n\nfinish = time.time()\n\nprint(\"First response time:\", spherical(finish - begin, 2), \"seconds\")\nprint(response1.output[0].content material[0].textual content)<\/code><\/pre>\n![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/03\/Screenshot-2026-03-18-at-10.44.32-PM.png\")
<\/figure>\nbegin = time.time()\n\nresponse2 = shopper.responses.create(\n mannequin=\"gpt-4.1-mini\",\n enter=long_prefix + \"What's regularization?\"\n)\n\nfinish = time.time()\n\nprint(\"Second response time:\", spherical(finish - begin, 2), \"seconds\")\nprint(response2.output[0].content material[0].textual content)<\/code><\/pre>\n![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/03\/Screenshot-2026-03-18-at-10.52.17-PM.png\")
<\/figure>\n
\nprompt_cache_key<\/code> parameter. Particularly, based on the documentation, we will explicitly outline a immediate cache key when making a request, and on this manner outline the requests that want to make use of the identical cache. Nonetheless, I attempted to incorporate it in my instance by appropriately adjusting the request parameters, however didn\u2019t have a lot luck:<\/p>\nresponse1 = shopper.responses.create(\n prompt_cache_key = 'prompt_cache_test1',\n mannequin=\"gpt-5.1\",\n enter=long_prefix + \"What's overfitting in machine studying?\"\n)<\/code><\/pre>\n![\"\"](\"https:\/\/contributor.insightmediagroup.io\/wp-content\/uploads\/2026\/03\/Screenshot-2026-03-19-at-7.32.31-AM-1024x57.png\")
<\/figure>\nprompt_cache_key<\/code> exists within the API capabilities, it’s not but uncovered within the Python SDK. In different phrases, we can’t explicitly management cache reuse but, however it’s slightly computerized and best-effort.<\/p>\n
\nSo, what can go improper?<\/h2>\n
On my thoughts<\/h2>\n