Semantic cache<\/a> is a complicated caching mechanism that differs from conventional caching, which depends on precise key phrase matching; it shops and retrieves information based mostly on semantic similarity. Redis LangCache\u00a0<\/strong>is a totally hosted semantic caching service that helps cache LLM prompts and responses semantically, thereby decreasing LLM utilization prices.<\/p>\n

On this tutorial, let’s learn to shortly create a easy utility and use LangCache for caching LLM queries. Additionally, see if we are able to mix fuzzy logic to enhance the responses. \u00a0<\/p>\n

Step 1: Set Up Redis LangCache<\/strong><\/h2>\nIf you happen to shouldn’t have an account but, create an account on https:\/\/cloud.redis.io\/<\/a>. Upon getting logged in,\u00a0<\/p>\n
\n
Navigate to databases and create a brand new database (I used the free plan right here).<\/li>\n
Click on on LangCache within the left menu and create an occasion of the LangCache service. I used the “Fast service creation” choice to create the LangCache service.\u00a0<\/li>\n
Copy the API key and hold it protected.<\/li>\n
Click on on the LangCache service you simply created. Once you click on the “Join” button, a fast join information will seem on the fitting with examples of how to hook up with your LangCache occasion.<\/li>\n<\/ol>\n
Step 2: Create a Easy Python Script<\/strong><\/h2>\n
Let’s create a easy Python script that can first verify the cache for a immediate. If a match is discovered, it returns the cached response. If not, it sends the immediate to the LLM, caches the response, and returns it.<\/p>\n
Connect with LangCache:<\/p>\n
\n
\n
\n
\n
lang_cache = LangCache( \n server_url=\"\",\n cache_id=\",\n)<\/get><\/langcache><\/code><\/pre>\n<\/p><\/div><\/div><\/div>\n<\/div>\nSearch the cache earlier than sending it to LLM:<\/p>\n
\n
\n
\n
\n
outcome = lang_cache.search(\n immediate=question,\n similarity_threshold=0.90,\n\u00a0 \u00a0 )<\/code><\/pre>\n<\/p><\/div><\/div><\/div>\n<\/div>\nThis similarity_threshold<\/code> determines how intently the immediate should match. The next worth means stricter matching.<\/p>\n Deal with a cache hit:<\/p>\n \n\n\n\nif outcome :\n for entry in outcome.information:\n print(\"Cache Hit!\")\n print(\"Cache Response:::\")\n print(f\"Immediate: {entry.immediate}\")\n print(f\"Response: {entry.response}\")\n print(f\"Rating: {entry.similarity}\")\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 return<\/code><\/pre>\n<\/p><\/div><\/div><\/div>\n<\/div>\nDeal with cache miss and retailer response:<\/p>\n \n\n\n\n# Calling LLM right here\nresponse = requests.put up(url, json=payload, headers=headers)\nresponse_json = response.json()\nresponse_text = response_json[\"choices\"][0][\"message\"][\"content\"]\n\n# --- Storing the reponse from LLM in LangCache ---\nsave_response = lang_cache.set(\n immediate=question,\n response=response_text,\n\u00a0 )<\/code><\/pre>\n<\/p><\/div><\/div><\/div>\n<\/div>\nAdvantages of Semantic Caching<\/strong><\/h2>\n\nFor related queries, responses are fetched from the cache, avoiding costly LLM calls.<\/li>\n Sooner response instances.<\/li>\n<\/ul>\nIssues to Watch Out For<\/strong><\/h2>\n\nSimilarity threshold<\/strong>: Set it thoughtfully. Too excessive and you will miss helpful matches. Too low and you will obtain irrelevant outcomes.<\/li>\n Accuracy<\/strong>: Even with an optimum threshold, outcomes might not at all times be excellent.<\/li>\n Knowledge privateness<\/strong>: In multi-tenant architectures, guarantee correct information partitioning so customers don\u2019t see one another\u2019s data.<\/li>\n Cache eviction<\/strong>: Know when and the way to evict cache entries.<\/li>\n<\/ul>\nLet\u2019s Run the Code<\/strong><\/h2>\nFirst Question: (Cache Is Empty)<\/strong><\/h3>\n\n\n\n\nQuestion : Temporary historical past on Capital of France\nResponse: \n \u00a0\u00a0\u00a0\u00a0Cache Miss!\n\u00a0 \u00a0 Redirecting to LLM<\/code><\/pre>\n<\/p><\/div><\/div><\/div>\nThe question and the response are actually saved within the cache.<\/p>\n<\/div>\n Modified Question<\/strong><\/h3>\n\n\n\n\nQuestion: ----Temporary historical past on Paris---\nResponse: \n\u00a0\u00a0\u00a0\u00a0Cache Hit!\n\u00a0\u00a0\u00a0\u00a0Cache Response:::\n\u00a0\u00a0\u00a0\u00a0Immediate: Temporary historical past on Capital of France\n\u00a0 \u00a0 Rating: 0.92440444<\/code><\/pre>\n<\/p><\/div><\/div><\/div>\nThough the question modified, the cache returned a semantically related outcome with a similarity rating of 0.92<\/p>\n<\/div>\n One other Variation<\/strong><\/h3>\n\n\n\n\nQuestion: \u00a0----Temporary historical past on France---\nResponse: \n\u00a0\u00a0\u00a0\u00a0Cache Hit!\n\u00a0\u00a0\u00a0\u00a0Cache Response:::\n\u00a0\u00a0\u00a0\u00a0Immediate: Temporary historical past on Capital of France\n\u00a0 \u00a0 Rating: 0.9121176<\/code><\/pre>\n<\/p><\/div><\/div><\/div>\nOops! We requested for the historical past of France, however we received the historical past of the capital of France. Although Paris performs a serious function in France’s historical past, the context is completely different. One is a metropolis, and the opposite is a rustic!<\/p>\n<\/div>\n Tuning the Threshold<\/strong><\/h3>\nLet\u2019s enhance the edge to 0.92 and clear the cache.<\/p>\n \n\n\n\nQuestion: ----Temporary historical past on Capital of France---\nResponse:\n\u00a0\u00a0\u00a0\u00a0Cache Miss!\n\u00a0\u00a0\u00a0\u00a0LLM Response: \n#####################################\n\nQuestion: ----Temporary historical past on Paris---\nResponse:\n\u00a0\u00a0\u00a0\u00a0Cache Hit!\n\u00a0\u00a0\u00a0\u00a0Cache Response:::\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Immediate: Temporary historical past on Capital of France\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Rating: 0.92440444\n#####################################\n\nQuestion: ----Temporary historical past on France---\nResponse:\n\u00a0\u00a0\u00a0\u00a0Cache Miss!\n\u00a0 \u00a0 LLM Response: <\/llm><\/code><\/pre>\n<\/p><\/div><\/div><\/div>\n<\/div>\nIt appears to be working higher!\u00a0<\/p>\n Efficiency Comparability<\/strong><\/h2>\nLet’s evaluate the time it takes to question from the cache<\/a> vs. querying from the LLM.<\/p>\n Semantic Cache Outcomes \u2014 Sorted by Time<\/h3>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n Question\n <\/p>\n<\/th>\n \n\n Consequence\n <\/p>\n<\/th>\n \n\n Matched Immediate\n <\/p>\n<\/th>\n \n\n Similarity Rating\n <\/p>\n<\/th>\n \n\n Response Supply\n <\/p>\n<\/th>\n \n\n Time (seconds)\n <\/p>\n<\/th>\n<\/tr>\n<\/thead>\n \n\n Temporary historical past on the Capital of France\n <\/p>\n<\/td>\n \n\n Cache Miss\n <\/p>\n<\/td>\n \n <\/td>\n \n <\/td>\n \n\n LLM\n <\/p>\n<\/td>\n \n\n 0.8499\n <\/p>\n<\/td>\n<\/tr>\n \n\n Temporary historical past on Paris\n <\/p>\n<\/td>\n \n\n Cache Hit\n <\/p>\n<\/td>\n \n\n Temporary historical past on the Capital of France\n <\/p>\n<\/td>\n \n\n 0.9244\n <\/p>\n<\/td>\n \n\n Cache\n <\/p>\n<\/td>\n \n\n 0.2705\n <\/p>\n<\/td>\n<\/tr>\n \n\n Temporary historical past on France\n <\/p>\n<\/td>\n \n\n Cache Miss\n <\/p>\n<\/td>\n \n <\/td>\n \n <\/td>\n \n\n LLM\n <\/p>\n<\/td>\n \n\n 1.2543\n <\/p>\n<\/td>\n<\/tr>\n \n\n Temporary historical past on the Capital of France\n <\/p>\n<\/td>\n \n\n Cache Hit\n <\/p>\n<\/td>\n \n\n Temporary historical past on the Capital of France\n <\/p>\n<\/td>\n \n\n 1.0\n <\/p>\n<\/td>\n \n\n Cache\n <\/p>\n<\/td>\n \n\n 1.1139\n <\/p>\n<\/td>\n<\/tr>\n \n\n Temporary historical past on Paris\n <\/p>\n<\/td>\n \n\n Cache Hit\n <\/p>\n<\/td>\n \n\n Temporary historical past on France\n <\/p>\n<\/td>\n \n\n 0.9386\n <\/p>\n<\/td>\n \n\n Cache\n <\/p>\n<\/td>\n \n\n 0.2761\n <\/p>\n<\/td>\n<\/tr>\n \n\n Temporary historical past on France\n <\/p>\n<\/td>\n \n\n Cache Hit\n <\/p>\n<\/td>\n \n\n Temporary historical past on France\n <\/p>\n<\/td>\n \n\n 1.0\n <\/p>\n<\/td>\n \n\n Cache\n <\/p>\n<\/td>\n \n\n 0.2798\n <\/p>\n<\/td>\n<\/tr>\n \n\n Temporary historical past on the \u00a0Capital of France\n <\/p>\n<\/td>\n \n\n Cache Hit\n <\/p>\n<\/td>\n \n\n Temporary historical past on the Capital of France\n <\/p>\n<\/td>\n \n\n 1.0\n <\/p>\n<\/td>\n \n\n Cache\n <\/p>\n<\/td>\n \n\n 1.0178\n <\/p>\n<\/td>\n<\/tr>\n \n\n Temporary historical past on Paris\n <\/p>\n<\/td>\n \n\n Cache Hit\n <\/p>\n<\/td>\n \n\n Temporary historical past on France\n <\/p>\n<\/td>\n \n\n 0.9386\n <\/p>\n<\/td>\n \n\n Cache\n <\/p>\n<\/td>\n \n\n 0.2806\n <\/p>\n<\/td>\n<\/tr>\n \n\n Temporary historical past on France\n <\/p>\n<\/td>\n \n\n Cache Hit\n <\/p>\n<\/td>\n \n\n Temporary historical past on France\n <\/p>\n<\/td>\n \n\n 1.0\n <\/p>\n<\/td>\n \n\n Cache\n <\/p>\n<\/td>\n \n\n 0.2778\n <\/p>\n<\/td>\n<\/tr>\n \n\n How does Langcache work? clarify\u2026\n <\/p>\n<\/td>\n \n\n Cache Hit\n <\/p>\n<\/td>\n \n\n How does Langcache work? clarify\u2026\n <\/p>\n<\/td>\n \n\n 1.0\n <\/p>\n<\/td>\n \n\n Cache\n <\/p>\n<\/td>\n \n\n 0.2930\n <\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n Observations:<\/p>\n \nAlthough there are some anomalies, the response from the cache is far quicker, which is apparent.<\/li>\n A excessive similarity threshold is held to reuse the cached response.<\/li>\n For distinct solutions, a better similarity threshold is really helpful.<\/li>\n Primarily based on the question and enterprise necessities, at all times tune and experiment, as outcomes fluctuate with embedding fashions, similarity thresholds, and so forth.<\/li>\n<\/ol>\nOne Extra Instance<\/strong><\/h3>\n\n\n\n\nQuestion: ----How does Langcache work---\nResponse:\n\u00a0\u00a0\u00a0\u00a0Cache Miss!\n\u00a0\u00a0\u00a0\u00a0LLM Response: < Offers a 20 strains response>\n\u00a0\u00a0\u00a0\u00a0Time to get response from LLM API: 0.8144 seconds\n----#####################################---\nQuestion: ----How does Langcache work---\nResponse: \n\u00a0\u00a0\u00a0\u00a0Cache Hit!\n\u00a0\u00a0\u00a0\u00a0Time for cache lookup: 0.3162 seconds\n\u00a0\u00a0\u00a0\u00a0Cache Response:::\n\u00a0\u00a0\u00a0\u00a0Immediate: How does Langcache work\n\u00a0\u00a0\u00a0\u00a0Response: < Offers the identical 20 strains response>\n\u00a0 \u00a0 Rating: 1.0<\/code><\/pre>\n<\/p><\/div><\/div><\/div>\nAll is properly until now. Let’s modify the question a bit!<\/p>\n<\/div>\n \n\n\n\n----#####################################---\nQuestion: ----How does Langcache work, clarify in 5 lines---\nResponse:\n\u00a0\u00a0\u00a0\u00a0Cache Hit!\n\u00a0\u00a0\u00a0\u00a0Time for cache lookup: 0.2719 seconds\n\u00a0\u00a0\u00a0\u00a0Cache Response:::\n\u00a0\u00a0\u00a0\u00a0Immediate: How does Langcache work\n\u00a0\u00a0\u00a0\u00a0Response: < Offers the identical 20 strains response>\n\u00a0\u00a0\u00a0\u00a0Rating: 0.9714471\n\n----#####################################---<\/code><\/pre>\n<\/p><\/div><\/div><\/div>\nWe requested for a 5-line response, however received the cached 20-line one. This highlights the significance of tuning and utilizing attributes to scope responses.<\/p>\n<\/div>\n Semantic Cache vs. Fuzzy Match\u00a0<\/strong><\/h2>\nFuzzy matching\u00a0<\/strong>works based mostly on approximate string matching. It really works finest for dealing with typos, spelling variants, and near-duplicate strings, whereas semantic match operates on the that means and context ranges.<\/p>\n Let’s have a look at the distinction between them in motion. Let’s evaluate the semantic rating (LangCache) with the fuzzy rating (Ratcliff\u2013Obershelp algorithm<\/a>) when matching two strings.<\/p>\n Querying for the primary time:<\/p>\n \n\n\nQuestion: Does Semantic cache work?\nResponse:\n Cache Miss!\n\u00a0\u00a0\u00a0\u00a0Cache Response:::\n Immediate: How does Semantic cache work?\n\u00a0 \u00a0 Response: Semantic caching shops question outcomes together with their semantic descriptions, enabling new queries to be answered partially or totally by reusing cached information.<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nQuerying the identical once more \u2014 discover that the semantic rating and fuzzy rating are fairly shut.<\/p>\n\n\n\nQuestion: How does Semantic cache work?\nResponse:\n Cache Hit!\n Cache Response:::\n Immediate: How does Semantic cache work?\n\u00a0 \u00a0 Response: Semantic caching shops question outcomes together with their semantic descriptions, enabling new queries to be answered partially or totally by reusing cached information.\n\u00a0 \u00a0 Semantic rating \u00a0(Langacache) : 0.95\n\u00a0 \u00a0 Fuzzy match rating : 0.93<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nLet’s attempt with a couple of extra variations:<\/p>\n\n\n\nQuestion: Will Semantic cache work?\nResponse:\n Cache Hit!\n Cache Response:::\n Immediate: How does Semantic cache work?\n\u00a0 \u00a0 Response: Semantic caching shops question outcomes together with their semantic descriptions, enabling new queries to be answered partially or totally by reusing cached information.\n\u00a0 \u00a0 Semantic rating \u00a0(Langacache) : 0.94\n\u00a0 \u00a0 Fuzzy match rating : 0.83<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\n\n\n\nQuestion: What does\u00a0Semantic cache imply?\nResponse:\n Cache Hit!\n Cache Response:::\n Immediate: How does Semantic cache work?\n\u00a0 \u00a0 Response: Semantic caching shops question outcomes together with their semantic descriptions, enabling new queries to be answered partially or totally by reusing cached information.\n\u00a0 \u00a0 Semantic rating \u00a0(Langacache) : 0.94\n\u00a0 \u00a0 Fuzzy match rating : 0.79<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nAs you’ll be able to see, fuzzy match focuses on “seems like,” whereas semantic matching focuses on “means like.”<\/p>\n Full implementation of the above is offered right here<\/a>.<\/p>\n Fuzzy match could be mixed with the semantic cache in a couple of methods:<\/p>\n \nRetailer the final ‘n’ prompts and do a fuzzy match on these. Use semantic caching provided that there isn’t a match discovered.\u00a0<\/li>\n When a excessive similarity threshold\/rating is used for the semantic cache (e.g., > 0.95), we find yourself caching prompts for each cache miss. This process will lead to a number of near-duplicates. We will use fuzzy match to establish these close to duplicates and retailer solely the prompts which are completely different<\/li>\n If the caching layer incorporates a number of close to duplicates, we are able to use fuzzy match for compaction.<\/li>\n<\/ol>\nUltimate Ideas<\/strong><\/h2>\nWhen constructing functions with semantic caching, efficient outcomes depend upon steady testing and context-aware tuning. Similarity thresholds, immediate patterns, and cache scope ought to be adjusted based mostly on workload habits and accuracy necessities. Redis LangCache allows fine-grained management via attributes that partition and scope cached responses. Semantic caching turns into much more environment friendly when fuzzy matching logic is added, placing a steadiness between accuracy and elevated cache hit charges. When mixed, these strategies can enhance latency, decrease LLM prices, and supply constant outcomes whereas sustaining accuracy and relevance.<\/p>\n Completely satisfied coding!<\/p>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":" Semantic cache is a complicated caching mechanism that differs from conventional caching, which depends on precise key phrase matching; it shops and retrieves information based mostly on semantic similarity. Redis LangCache\u00a0is a totally hosted semantic caching service that helps cache LLM prompts and responses semantically, thereby decreasing LLM utilization prices. On this tutorial, let’s learn […]<\/p>\n","protected":false},"author":2,"featured_media":10498,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[56],"tags":[7278,78,7279,185,3171,5527],"class_list":["post-10496","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-software","tag-caching","tag-guide","tag-langcache","tag-practical","tag-redis","tag-semantic"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/10496","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10496"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/10496\/revisions"}],"predecessor-version":[{"id":10497,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/10496\/revisions\/10497"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/10498"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10496"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10496"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10496"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}

\n \n Question\n <\/p>\n<\/th>\n	\n \n Consequence\n <\/p>\n<\/th>\n	\n \n Matched Immediate\n <\/p>\n<\/th>\n	\n \n Similarity Rating\n <\/p>\n<\/th>\n	\n \n Response Supply\n <\/p>\n<\/th>\n	\n \n Time (seconds)\n <\/p>\n<\/th>\n<\/tr>\n<\/thead>\n
\n \n Temporary historical past on the Capital of France\n <\/p>\n<\/td>\n	\n \n Cache Miss\n <\/p>\n<\/td>\n	\n <\/td>\n	\n <\/td>\n	\n \n LLM\n <\/p>\n<\/td>\n	\n \n 0.8499\n <\/p>\n<\/td>\n<\/tr>\n
\n \n Temporary historical past on Paris\n <\/p>\n<\/td>\n	\n \n Cache Hit\n <\/p>\n<\/td>\n	\n \n Temporary historical past on the Capital of France\n <\/p>\n<\/td>\n	\n \n 0.9244\n <\/p>\n<\/td>\n	\n \n Cache\n <\/p>\n<\/td>\n	\n \n 0.2705\n <\/p>\n<\/td>\n<\/tr>\n
\n \n Temporary historical past on France\n <\/p>\n<\/td>\n	\n \n Cache Miss\n <\/p>\n<\/td>\n	\n <\/td>\n	\n <\/td>\n	\n \n LLM\n <\/p>\n<\/td>\n	\n \n 1.2543\n <\/p>\n<\/td>\n<\/tr>\n
\n \n Temporary historical past on the Capital of France\n <\/p>\n<\/td>\n	\n \n Cache Hit\n <\/p>\n<\/td>\n	\n \n Temporary historical past on the Capital of France\n <\/p>\n<\/td>\n	\n \n 1.0\n <\/p>\n<\/td>\n	\n \n Cache\n <\/p>\n<\/td>\n	\n \n 1.1139\n <\/p>\n<\/td>\n<\/tr>\n
\n \n Temporary historical past on Paris\n <\/p>\n<\/td>\n	\n \n Cache Hit\n <\/p>\n<\/td>\n	\n \n Temporary historical past on France\n <\/p>\n<\/td>\n	\n \n 0.9386\n <\/p>\n<\/td>\n	\n \n Cache\n <\/p>\n<\/td>\n	\n \n 0.2761\n <\/p>\n<\/td>\n<\/tr>\n
\n \n Temporary historical past on France\n <\/p>\n<\/td>\n	\n \n Cache Hit\n <\/p>\n<\/td>\n	\n \n Temporary historical past on France\n <\/p>\n<\/td>\n	\n \n 1.0\n <\/p>\n<\/td>\n	\n \n Cache\n <\/p>\n<\/td>\n	\n \n 0.2798\n <\/p>\n<\/td>\n<\/tr>\n
\n \n Temporary historical past on the \u00a0Capital of France\n <\/p>\n<\/td>\n	\n \n Cache Hit\n <\/p>\n<\/td>\n	\n \n Temporary historical past on the Capital of France\n <\/p>\n<\/td>\n	\n \n 1.0\n <\/p>\n<\/td>\n	\n \n Cache\n <\/p>\n<\/td>\n	\n \n 1.0178\n <\/p>\n<\/td>\n<\/tr>\n
\n \n Temporary historical past on Paris\n <\/p>\n<\/td>\n	\n \n Cache Hit\n <\/p>\n<\/td>\n	\n \n Temporary historical past on France\n <\/p>\n<\/td>\n	\n \n 0.9386\n <\/p>\n<\/td>\n	\n \n Cache\n <\/p>\n<\/td>\n	\n \n 0.2806\n <\/p>\n<\/td>\n<\/tr>\n
\n \n Temporary historical past on France\n <\/p>\n<\/td>\n	\n \n Cache Hit\n <\/p>\n<\/td>\n	\n \n Temporary historical past on France\n <\/p>\n<\/td>\n	\n \n 1.0\n <\/p>\n<\/td>\n	\n \n Cache\n <\/p>\n<\/td>\n	\n \n 0.2778\n <\/p>\n<\/td>\n<\/tr>\n
\n \n How does Langcache work? clarify\u2026\n <\/p>\n<\/td>\n	\n \n Cache Hit\n <\/p>\n<\/td>\n	\n \n How does Langcache work? clarify\u2026\n <\/p>\n<\/td>\n	\n \n 1.0\n <\/p>\n<\/td>\n	\n \n Cache\n <\/p>\n<\/td>\n	\n \n 0.2930\n <\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n Observations:<\/p>\n \n Although there are some anomalies, the response from the cache is far quicker, which is apparent.<\/li>\n A excessive similarity threshold is held to reuse the cached response.<\/li>\n For distinct solutions, a better similarity threshold is really helpful.<\/li>\n Primarily based on the question and enterprise necessities, at all times tune and experiment, as outcomes fluctuate with embedding fashions, similarity thresholds, and so forth.<\/li>\n<\/ol>\n One Extra Instance<\/strong><\/h3>\n \n \n \n \n Question: ----How does Langcache work---\nResponse:\n\u00a0\u00a0\u00a0\u00a0Cache Miss!\n\u00a0\u00a0\u00a0\u00a0LLM Response: < Offers a 20 strains response>\n\u00a0\u00a0\u00a0\u00a0Time to get response from LLM API: 0.8144 seconds\n----#####################################---\nQuestion: ----How does Langcache work---\nResponse: \n\u00a0\u00a0\u00a0\u00a0Cache Hit!\n\u00a0\u00a0\u00a0\u00a0Time for cache lookup: 0.3162 seconds\n\u00a0\u00a0\u00a0\u00a0Cache Response:::\n\u00a0\u00a0\u00a0\u00a0Immediate: How does Langcache work\n\u00a0\u00a0\u00a0\u00a0Response: < Offers the identical 20 strains response>\n\u00a0 \u00a0 Rating: 1.0<\/code><\/pre>\n<\/p><\/div><\/div><\/div>\nAll is properly until now. Let’s modify the question a bit!<\/p>\n<\/div>\n \n\n\n\n----#####################################---\nQuestion: ----How does Langcache work, clarify in 5 lines---\nResponse:\n\u00a0\u00a0\u00a0\u00a0Cache Hit!\n\u00a0\u00a0\u00a0\u00a0Time for cache lookup: 0.2719 seconds\n\u00a0\u00a0\u00a0\u00a0Cache Response:::\n\u00a0\u00a0\u00a0\u00a0Immediate: How does Langcache work\n\u00a0\u00a0\u00a0\u00a0Response: < Offers the identical 20 strains response>\n\u00a0\u00a0\u00a0\u00a0Rating: 0.9714471\n\n----#####################################---<\/code><\/pre>\n<\/p><\/div><\/div><\/div>\nWe requested for a 5-line response, however received the cached 20-line one. This highlights the significance of tuning and utilizing attributes to scope responses.<\/p>\n<\/div>\n Semantic Cache vs. Fuzzy Match\u00a0<\/strong><\/h2>\nFuzzy matching\u00a0<\/strong>works based mostly on approximate string matching. It really works finest for dealing with typos, spelling variants, and near-duplicate strings, whereas semantic match operates on the that means and context ranges.<\/p>\n Let’s have a look at the distinction between them in motion. Let’s evaluate the semantic rating (LangCache) with the fuzzy rating (Ratcliff\u2013Obershelp algorithm<\/a>) when matching two strings.<\/p>\n Querying for the primary time:<\/p>\n \n\n\nQuestion: Does Semantic cache work?\nResponse:\n Cache Miss!\n\u00a0\u00a0\u00a0\u00a0Cache Response:::\n Immediate: How does Semantic cache work?\n\u00a0 \u00a0 Response: Semantic caching shops question outcomes together with their semantic descriptions, enabling new queries to be answered partially or totally by reusing cached information.<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nQuerying the identical once more \u2014 discover that the semantic rating and fuzzy rating are fairly shut.<\/p>\n\n\n\nQuestion: How does Semantic cache work?\nResponse:\n Cache Hit!\n Cache Response:::\n Immediate: How does Semantic cache work?\n\u00a0 \u00a0 Response: Semantic caching shops question outcomes together with their semantic descriptions, enabling new queries to be answered partially or totally by reusing cached information.\n\u00a0 \u00a0 Semantic rating \u00a0(Langacache) : 0.95\n\u00a0 \u00a0 Fuzzy match rating : 0.93<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nLet’s attempt with a couple of extra variations:<\/p>\n\n\n\nQuestion: Will Semantic cache work?\nResponse:\n Cache Hit!\n Cache Response:::\n Immediate: How does Semantic cache work?\n\u00a0 \u00a0 Response: Semantic caching shops question outcomes together with their semantic descriptions, enabling new queries to be answered partially or totally by reusing cached information.\n\u00a0 \u00a0 Semantic rating \u00a0(Langacache) : 0.94\n\u00a0 \u00a0 Fuzzy match rating : 0.83<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\n\n\n\nQuestion: What does\u00a0Semantic cache imply?\nResponse:\n Cache Hit!\n Cache Response:::\n Immediate: How does Semantic cache work?\n\u00a0 \u00a0 Response: Semantic caching shops question outcomes together with their semantic descriptions, enabling new queries to be answered partially or totally by reusing cached information.\n\u00a0 \u00a0 Semantic rating \u00a0(Langacache) : 0.94\n\u00a0 \u00a0 Fuzzy match rating : 0.79<\/code><\/pre>\n<\/p><\/div><\/div>\n<\/div>\nAs you’ll be able to see, fuzzy match focuses on “seems like,” whereas semantic matching focuses on “means like.”<\/p>\n Full implementation of the above is offered right here<\/a>.<\/p>\n Fuzzy match could be mixed with the semantic cache in a couple of methods:<\/p>\n \nRetailer the final ‘n’ prompts and do a fuzzy match on these. Use semantic caching provided that there isn’t a match discovered.\u00a0<\/li>\n When a excessive similarity threshold\/rating is used for the semantic cache (e.g., > 0.95), we find yourself caching prompts for each cache miss. This process will lead to a number of near-duplicates. We will use fuzzy match to establish these close to duplicates and retailer solely the prompts which are completely different<\/li>\n If the caching layer incorporates a number of close to duplicates, we are able to use fuzzy match for compaction.<\/li>\n<\/ol>\nUltimate Ideas<\/strong><\/h2>\nWhen constructing functions with semantic caching, efficient outcomes depend upon steady testing and context-aware tuning. Similarity thresholds, immediate patterns, and cache scope ought to be adjusted based mostly on workload habits and accuracy necessities. Redis LangCache allows fine-grained management via attributes that partition and scope cached responses. Semantic caching turns into much more environment friendly when fuzzy matching logic is added, placing a steadiness between accuracy and elevated cache hit charges. When mixed, these strategies can enhance latency, decrease LLM prices, and supply constant outcomes whereas sustaining accuracy and relevance.<\/p>\n Completely satisfied coding!<\/p>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":" Semantic cache is a complicated caching mechanism that differs from conventional caching, which depends on precise key phrase matching; it shops and retrieves information based mostly on semantic similarity. Redis LangCache\u00a0is a totally hosted semantic caching service that helps cache LLM prompts and responses semantically, thereby decreasing LLM utilization prices. On this tutorial, let’s learn […]<\/p>\n","protected":false},"author":2,"featured_media":10498,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[56],"tags":[7278,78,7279,185,3171,5527],"class_list":["post-10496","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-software","tag-caching","tag-guide","tag-langcache","tag-practical","tag-redis","tag-semantic"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/10496","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=10496"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/10496\/revisions"}],"predecessor-version":[{"id":10497,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/10496\/revisions\/10497"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/10498"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=10496"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=10496"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=10496"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}