, the usual \u201ctextual content in, textual content out\u201d paradigm will solely take you thus far.<\/p>\n

Actual purposes that ship precise worth ought to have the ability to look at visuals, cause by way of advanced issues, and produce outcomes that methods can truly use.<\/p>\n

On this put up, we\u2019ll design this stack by bringing collectively three highly effective capabilities: multimodal enter, reasoning, <\/strong>and structured output<\/strong>.<\/p>\n

For instance this, we\u2019ll stroll by way of a hands-on instance: constructing a time-series anomaly detection system<\/strong> for e-commerce order knowledge utilizing OpenAI\u2019s o3 mannequin<\/strong>. Particularly, we\u2019ll present find out how to pair o3\u2019s reasoning functionality with picture enter and emit validated JSON, in order that the downstream system can simply eat it.<\/p>\n

By the top, our app will:<\/p>\n

\n
See<\/strong>: analyze charts of e-commerce order quantity time collection <\/li>\n
Suppose<\/strong>: establish uncommon patterns <\/li>\n
Combine<\/strong>: output a structured anomaly report<\/li>\n<\/ul>\n
You\u2019ll go away with useful code you possibly can reuse for numerous use circumstances that transcend simply anomaly detection.<\/p>\n
Let\u2019s dive in.<\/p>\n
\n
Curious about studying the broader panorama of how LLMs are being utilized for anomaly detection? Take a look at my earlier put up: Boosting Your Anomaly Detection With LLMs<\/a><\/strong><\/em>, the place I summarized 7 rising utility patterns<\/em> that you simply shouldn\u2019t miss<\/em>.<\/p>\n<\/blockquote>\n
\n
1. Case Research<\/strong><\/p>\n
On this put up, we intention to construct an anomaly detection resolution for figuring out irregular patterns in e-commerce order time collection knowledge.<\/p>\n
For this case research, we generated three units of artificial <\/em>each day order knowledge. The datasets symbolize three totally different profiles of the each day order over roughly one month of time. To make seasonality apparent, now we have shaded the weekends. The x-axis exhibits the day of the week.<\/p>\n
$\"\"$
Determine 1. Dataset 1, with the shaded areas being the weekends. (Picture by writer)<\/figcaption><\/figure>\n
$\"\"$
Determine 2. Dataset 2, with the shaded areas being the weekends. (Picture by writer)<\/figcaption><\/figure>\n
$\"\"$
Determine 3. Dataset 3, with the shaded areas being the weekends. (Picture by writer)<\/figcaption><\/figure>\n
Every determine incorporates one particular sort of anomaly (can you discover them?<\/em>). We\u2019ll later use these figures to check our anomaly detection resolution and see if it may precisely recuperate these anomalies. <\/p>\n
2. Our Answer<\/strong><\/p>\n
2.1 Overview<\/strong><\/p>\n
Not like the normal machine studying approaches that require tedious function engineering and mannequin coaching, our present strategy is far easier. It really works with the next steps:<\/p>\n
\n
We put together the determine for visualizing the e-commerce order time collection knowledge.<\/li>\n
We immediate the reasoning mannequin o3, ask it to take a better take a look at the time collection picture we fed to it, and decide if an uncommon sample exists.<\/li>\n
The o3 mannequin will then output its findings in a pre-defined JSON format.<\/li>\n<\/ol>\n
And that\u2019s it. Easy. <\/p>\n
After all, to ship this resolution, we have to allow o3 mannequin to take picture enter and emit structured output. We’ll see how to try this shortly.<\/p>\n
2.2 Establishing the reasoning mannequin<\/strong><\/p>\n
As talked about earlier than, we\u2019ll use o3 mannequin, which is the flagship reasoning mannequin from OpenAI that may sort out advanced multi-step issues with state-of-the-art efficiency. Particularly, we\u2019ll use the Azure OpenAI endpoint to name the mannequin.<\/p>\n
Be sure you have put the endpoint, API key, and deployment identify in an .env<\/code> file, we are able to then proceed to establishing the LLM consumer:<\/p>\n
import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport matplotlib.dates as mdates\n\nfrom openai import AzureOpenAI\nfrom dotenv import load_dotenv\nimport os\n\nload_dotenv()\n\n# Setup LLM consumer\nendpoint = os.getenv(\"api_base\")\napi_key = os.getenv(\"o3_API_KEY\")\napi_version = \"2025-04-01-preview\"\nmodel_name = \"o3\"\ndeployment = os.getenv(\"deployment_name\")\n\nLLM_client = AzureOpenAI(\n api_key=api_key, \n api_version=api_version,\n azure_endpoint=endpoint\n)<\/code><\/pre>\nWe use the next instruction because the system message for the o3 mannequin (tuned by GPT-5):<\/p>\n instruction = f\"\"\"\n\n[Role]\nYou're a meticulous knowledge analyst.\n\n[Task]\nYou'll be given a line chart picture associated to each day e-commerce orders. \nYour activity is to establish outstanding anomalies within the knowledge.\n\n[Rules]\nThe anomaly sorts will be spike, drop, level_shift, or seasonal_outlier.\nA level_shift is a sustained baseline change (\u2265 5 consecutive days), not a single level.\nA seasonal_outlier occurs if a weekend\/weekday behaves in contrast to friends in its class. \nFor instance, weekend orders are often decrease than the weekdays'.\nLearn dates\/values from axes; should you can\u2019t learn precisely, snap to the closest tick and be aware uncertainty in clarification.\nThe weekends are shaded within the determine.\n\"\"\"<\/code><\/pre>\nWithin the above instruction, we clearly outlined the function of the LLM, the duty that the LLM ought to full, and the principles the LLM ought to comply with. <\/p>\n To restrict the complexity of our case research, we deliberately specified solely 4 anomaly sorts that LLM must establish. We additionally offered clear definitions of these anomaly sorts to take away ambiguity. <\/p>\n Lastly, we injected a little bit of area data about e-commerce patterns, i.e., decrease weekend orders are anticipated in comparison with weekdays. Incorporating area know-how is usually thought of good observe for guiding the mannequin\u2019s analytical course of.<\/p>\n Now that now we have our mannequin arrange, let\u2019s focus on find out how to put together the picture for o3 mannequin to eat.<\/p>\n 2.3 Picture preparation<\/strong> <\/p>\n To allow o3\u2019s multimodal capabilities, we have to present figures in a particular format, i.e., both publicly accessible internet URLs or as base64-encoded knowledge URLs. Since our figures are generated domestically, we\u2019ll use the second strategy.<\/p>\n \nWhat’s Base64 Encoding anyway?<\/strong> Base64 is a approach to symbolize binary knowledge (like our picture recordsdata) utilizing solely textual content characters which are secure to transmit over the web. It converts binary picture knowledge right into a string of letters, numbers, and some symbols.<\/em><\/p>\n And what about knowledge URL?<\/strong> An information URL is a kind of URL that embeds the file content material straight within the URL string, relatively than pointing to a file location.<\/em><\/p>\n<\/blockquote>\n We will use the next perform to deal with this conversion mechanically:<\/p>\n import io\nimport base64\n\ndef fig_to_data_url(fig, fmt=\"png\"):\n \"\"\"\n Converts a Matplotlib determine to a base64 knowledge URL with out saving to disk.\n\n Args:\n -----\n fig (matplotlib.determine.Determine): The determine to transform.\n fmt (str): The format of the picture (\"png\", \"jpeg\", and many others.)\n\n Returns:\n --------\n str: The info URL representing the determine.\n \"\"\"\n\n buf = io.BytesIO()\n fig.savefig(buf, format=fmt, bbox_inches=\"tight\")\n buf.search(0)\n \n base64_encoded_data = base64.b64encode(buf.learn()).decode(\"utf-8\")\n mime_type = f\"picture\/{fmt.decrease()}\"\n \n return f\"knowledge:{mime_type};base64,{base64_encoded_data}\"<\/code><\/pre>\nBasically, our perform first saves the matplotlib determine to a reminiscence buffer. It then encodes the binary PNG knowledge as base64 textual content and wraps it within the desired knowledge URL format.<\/p>\n Assuming now we have entry to the artificial each day order knowledge, we are able to use the next perform to generate the plot and convert it into a correct knowledge URL format in a single go:<\/p>\n def create_fig(df):\n \"\"\"\n Create a Matplotlib determine and convert it to a base64 knowledge URL.\n Weekends (Sat\u2013Solar) are shaded.\n\n Args:\n -----\n df: dataframe incorporates one profile of each day order time collection. \n dataframe has \"date\" and \"orders\" columns.\n\n Returns:\n --------\n image_url: The info URL representing the determine.\n \"\"\"\n\n df = df.copy()\n df['date'] = pd.to_datetime(df['date'])\n\n fig, ax = plt.subplots(figsize=(8, 4.5))\n ax.plot(df[\"date\"], df[\"orders\"], linewidth=2)\n ax.set_xlabel('Date', fontsize=14)\n ax.set_ylabel('Every day Orders', fontsize=14)\n\n # Weekend shading\n begin = df[\"date\"].min().normalize()\n finish = df[\"date\"].max().normalize()\n cur = begin\n whereas cur <= finish:\n if cur.weekday() == 5: # Saturday 00:00\n span_start = cur # Sat 00:00\n span_end = cur + pd.Timedelta(days=1) # Mon 00:00\n ax.axvspan(span_start, span_end, alpha=0.12, zorder=0)\n cur += pd.Timedelta(days=2) # skip Sunday \n else:\n cur += pd.Timedelta(days=1)\n\n # Title\n title = f'Every day Orders: {df[\"date\"].min():%b %d, %Y} - {df[\"date\"].max():%b %d, %Y}'\n ax.set_title(title, fontsize=16)\n\n # Format x-axis dates\n ax.xaxis.set_major_formatter(mdates.DateFormatter('%b %d')) \n ax.xaxis.set_major_locator(mdates.WeekdayLocator(interval=1))\n\n plt.tight_layout()\n\n # Receive url\n image_url = fig_to_data_url(fig)\n\n return image_url<\/code><\/pre>\nFigures 1-3 are generated by the above plotting routine.<\/p>\n 2.4 Structured output<\/strong><\/p>\n On this part, let\u2019s focus on how to make sure the o3 mannequin outputs a constant JSON format as a substitute of free-form textual content. That is what\u2019s referred to as \u201cstructured output,\u201d and it\u2019s one of many key enablers for integrating LLMs into present computerized workflows.<\/p>\n To attain that, we begin by defining the schema that governs the anticipated output construction. We\u2019ll be utilizing a Pydantic mannequin:<\/p>\n from pydantic import BaseModel, Discipline\nfrom typing import Literal\nfrom datetime import date\n\nAnomalyKind = Literal[\"spike\", \"drop\", \"level_shift\", \"seasonal_outlier\"]\n\nclass DateWindow(BaseModel):\n begin: date = Discipline(description=\"Earliest believable date the anomaly begins (ISO YYYY-MM-DD)\")\n finish: date = Discipline(description=\"Newest believable date the anomaly ends, inclusive (ISO YYYY-MM-DD)\")\n\nclass AnomalyReport(BaseModel):\n when: DateWindow = Discipline(\n description=(\n \"Minimal window that incorporates the anomaly. \"\n \"For single-point anomalies, use the interval that covers studying uncertainty, if the tick labels are unclear\"\n )\n )\n y: int = Discipline(description=\"Approx worth on the anomaly\u2019s most consultant day (peak\/lowest), rounded\")\n variety: AnomalyKind = Discipline(description=\"The kind of the anomaly\")\n why: str = Discipline(description=\"One-sentence cause for why this window is uncommon\")\n date_confidence: Literal[\"low\",\"medium\",\"high\"] = Discipline(\n default=\"medium\", description=\"Confidence that the window localization is appropriate\"\n )<\/code><\/pre>\nOur Pydantic schema tries to seize each the quantitative and qualitative elements of the detected anomalies. For every area, we specify its knowledge sort (e.g., int<\/code> for numerical values, Literal<\/code> for a set set of selections, and many others.). <\/p>\n Additionally, we use Discipline<\/code> perform to offer detailed descriptions of every key. These descriptions are particularly essential as they successfully function inline directions for o3, in order that it understands the semantic which means of every part.<\/p>\n Now, now we have lined the multimodal enter and structured output, time to place them collectively in a single LLM name.<\/p>\n 2.5 o3 mannequin invocation<\/strong><\/p>\n To work together with o3 utilizing multimodal enter and structured output, we use LLM_client.beta.chat.completions.parse()<\/code> API. A number of the key arguments embody:<\/p>\n \nmannequin<\/code>: the deployment identify;<\/li>\n messages<\/code>: the message object despatched to o3 mannequin;<\/li>\n max_completion_token<\/code>: the utmost variety of tokens the mannequin can generate in its closing response. Word that for reasoning fashions like o3, they may generate reasoning_tokens internally to \u201cassume by way of\u201d the issue. The present max_completion_token<\/code> solely limits the seen output tokens that customers obtain;<\/li>\n response_format<\/code>: the Pydantic mannequin that defines the anticipated JSON schema construction;<\/li>\n reasoning_effort<\/code>: a management knob that dictates how a lot computational effort o3 ought to use for reasoning. The accessible choices embody low, medium, and excessive.<\/li>\n<\/ul>\nWe will outline a helper perform to work together with the o3 mannequin:<\/p>\n def anomaly_detection(instruction, fig_path, \n response_format, immediate=None, \n deployment=\"o3\", reasoning_effort=\"excessive\"):\n\n # Compose messages\n messages=[\n { \"role\": \"system\", \"content\": instruction},\n { \"role\": \"user\", \"content\": [ \n { \n \"type\": \"image_url\",\n \"image_url\": {\n \"url\": fig_path,\n \"detail\": \"high\"\n }\n },\n ]} \n ]\n\n # Add immediate whether it is given\n if immediate just isn't None:\n messages[1][\"content\"].append({\"sort\": \"textual content\", \"textual content\": immediate})\n\n # Invoke LLM API\n response = LLM_client.beta.chat.completions.parse(\n mannequin=deployment,\n messages=messages,\n max_completion_tokens=4000,\n reasoning_effort=reasoning_effort,\n response_format=response_format\n )\n\n return response.selections[0].message.parsed.model_dump()<\/code><\/pre>\nWord that the messages<\/code> object accepts each textual content and picture content material. Since we\u2019ll solely use figures to immediate the mannequin, the textual content immediate is optionally available. <\/p>\n We set the \"element\": \"excessive\"<\/code> to allow high-resolution picture processing. For our present case research, that is more than likely essential as we’d like o3 to higher learn high quality particulars like axis tick labels, knowledge level values, and delicate visible patterns. Nonetheless, keep in mind that high-detail processing would incur extra tokens and better API prices. <\/p>\n Lastly, through the use of .parsed.model_dump()<\/code>, we flip the JSON output right into a typical Python dictionary. <\/p>\n That\u2019s it for the implementation. Let\u2019s see some outcomes subsequent.<\/p>\n \n3. Outcomes<\/strong><\/p>\n On this part, we\u2019ll enter the beforehand generated figures into the o3 mannequin and ask it to establish potential anomalies.<\/p>\n 3.1 Spike anomaly<\/strong><\/p>\n # df_spike_anomaly is the dataframe of the primary set of artificial knowledge (Determine 1)\nspike_anomaly_url = create_fig(df_spike_anomaly)\n\n# Anomaly detection\nconsequence = anomaly_detection(instruction,\n spike_anomaly_url,\n response_format=AnomalyReport,\n reasoning_effort=\"medium\")\nprint(consequence)<\/code><\/pre>\nWithin the name above, the spike_anomaly_url<\/code> is the info URL for Determine 1. The output of the result’s proven under:<\/p>\n {\n 'when': {'begin': datetime.date(2025, 8, 19), 'finish': datetime.date(2025, 8, 21)}, \n 'y': 166, \n 'variety': 'spike', \n 'why': 'Single day orders leap to ~166, far above adjoining days that sit close to 120\u2013130.', \n 'date_confidence': 'medium'\n}<\/code><\/pre>\nWe see that o3 mannequin faithfully returned the output precisely within the format we designed. Now, we are able to seize this consequence and generate a visualization programmatically:<\/p>\n # Create picture\nfig, ax = plt.subplots(figsize=(8, 4.5))\ndf_spike_anomaly['date'] = pd.to_datetime(df_spike_anomaly['date'])\nax.plot(df_spike_anomaly[\"date\"], df_spike_anomaly[\"orders\"], linewidth=2)\nax.set_xlabel('Date', fontsize=14)\nax.set_ylabel('Every day Orders', fontsize=14)\n\n# Format x-axis dates\nax.xaxis.set_major_formatter(mdates.DateFormatter('%b %d')) \nax.xaxis.set_major_locator(mdates.WeekdayLocator(interval=1)) \n\n# Add anomaly overlay\nstart_date = pd.to_datetime(consequence['when']['start'])\nend_date = pd.to_datetime(consequence['when']['end'])\n\n# Add shaded area\nax.axvspan(start_date, end_date, alpha=0.3, coloration='crimson', label=f\"Anomaly ({consequence['kind']})\")\n\n# Add textual content annotation\nmid_date = start_date + (end_date - start_date) \/ 2 # Center of anomaly window\nax.annotate(\n consequence['why'], \n xy=(mid_date, consequence['y']), \n xytext=(10, 20), # Offset from the purpose\n textcoords='offset factors',\n bbox=dict(boxstyle='spherical,pad=0.5', fc='yellow', alpha=0.7),\n arrowprops=dict(arrowstyle='->', connectionstyle='arc3,rad=0.1'),\n fontsize=10,\n wrap=True\n)\n\n# Add legend\nax.legend()\n\nplt.xticks(rotation=0)\nplt.tight_layout()<\/code><\/pre>\nThe generated visualization seems like this:<\/p>\n Determine 4. The anomaly detection outcomes for Determine 1. (Picture by writer)<\/figcaption><\/figure>\nWe will see that the o3 mannequin accurately recognized the spike anomaly offered on this first set of artificial knowledge. <\/p>\n Not unhealthy, particularly contemplating the truth that we didn\u2019t do any typical mannequin coaching, simply by prompting an LLM.<\/p>\n 3.2 Degree shift anomaly<\/strong><\/p>\n # df_level_shift_anomaly is the dataframe of the 2nd set of artificial knowledge (Determine 2)\nlevel_shift_anomaly_url = create_fig(df_level_shift_anomaly)\n\n# Anomaly detection\nconsequence = anomaly_detection(instruction,\n level_shift_anomaly_url,\n response_format=AnomalyReport,\n reasoning_effort=\"medium\")\nprint(consequence)<\/code><\/pre>\nThe output of the result’s proven under:<\/p>\n {\n 'when': {'begin': datetime.date(2025, 8, 26), 'finish': datetime.date(2025, 9, 2)}, \n 'y': 150, \n 'variety': 'level_shift', \n 'why': 'Orders all of the sudden leap from the 120-135 vary to ~150 on Aug 26 and stay elevated for all subsequent days, indicating a sustained baseline change.', \n 'date_confidence': 'excessive'\n}<\/code><\/pre>\nOnce more, we see that the mannequin precisely recognized {that a} \u201clevel_shift\u201d anomaly is current within the plot:<\/p>\n Determine 5. The anomaly detection outcomes for Determine 2. (Picture by writer)<\/figcaption><\/figure>\n3.3 Seasonality anomaly<\/strong><\/p>\n # df_seasonality_anomaly is the dataframe of the third set of artificial knowledge (Determine 3)\nseasonality_anomaly_url = create_fig(df_seasonality_anomaly)\n\n# Anomaly detection\nconsequence = anomaly_detection(instruction,\n seasonality_anomaly_url,\n response_format=AnomalyReport,\n reasoning_effort=\"medium\")\nprint(consequence)<\/code><\/pre>\nThe output of the result’s proven under:<\/p>\n {\n 'when': {'begin': datetime.date(2025, 8, 23), 'finish': datetime.date(2025, 8, 24)}, \n 'y': 132, \n 'variety': 'seasonal_outlier', \n 'why': 'Weekend of Aug 23-24 exhibits order volumes (~130+) on par with surrounding weekdays, whereas different weekends persistently drop to ~115, making it an out-of-season spike.', \n 'date_confidence': 'excessive'\n}<\/code><\/pre>\nThis can be a difficult case. However, our o3 mannequin managed to sort out it correctly, with correct localization and a transparent reasoning hint. Fairly spectacular:<\/p>\n Determine 6. The anomaly detection outcomes for Determine 3. (Picture by writer)<\/figcaption><\/figure>\n \n4. Abstract<\/strong><\/p>\n Congratulations! We\u2019ve efficiently constructed an anomaly detection resolution for time-series knowledge that labored solely by way of visualization and prompting. <\/p>\n By feeding each day order plots into the o3 reasoning mannequin and constraining its output to a JSON schema, the LLM managed to establish three totally different anomaly sorts with correct localization. All of this was achieved with out coaching any ML mannequin. Spectacular!<\/p>\n If we take a step again, we are able to see that the answer we constructed illustrates the broader sample of mixing three capabilities:<\/p>\n \nSee<\/strong>: multimodal enter to let the mannequin eat figures straight.<\/li>\n Suppose<\/strong>: step-by-step reasoning functionality to sort out advanced issues.<\/li>\n Combine<\/strong>: structured output that downstream methods can simply eat (e.g., producing visualizations).<\/li>\n<\/ul>\nThe mix of multimodal enter + reasoning + structured output actually creates a flexible basis for helpful LLM purposes. <\/p>\n You now have the constructing blocks prepared. What do you wish to construct subsequent?<\/p>\n<\/div>\n\n","protected":false},"excerpt":{"rendered":" , the usual \u201ctextual content in, textual content out\u201d paradigm will solely take you thus far. Actual purposes that ship precise worth ought to have the ability to look at visuals, cause by way of advanced issues, and produce outcomes that methods can truly use. On this put up, we\u2019ll design this stack by bringing […]<\/p>\n","protected":false},"author":2,"featured_media":6877,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[588,475,4905,2065,74,306,5200,1248],"class_list":["post-6875","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-apps","tag-building","tag-input","tag-integrate","tag-llm","tag-multimodal","tag-output","tag-structured"],"_links":{"self":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/6875","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6875"}],"version-history":[{"count":1,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/6875\/revisions"}],"predecessor-version":[{"id":6876,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/posts\/6875\/revisions\/6876"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=\/wp\/v2\/media\/6877"}],"wp:attachment":[{"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6875"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6875"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/techtrendfeed.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6875"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}