We are focused on improving the detection of egregious behavior in our LLM-powered chatbot for Amazon Customer Service AI (CSAI), both during customer interactions and in chatbot responses. Our goal is to protect customer experience by identifying and mitigating potential risks in real-time, ensuring that inappropriate or unsafe inputs are flagged, and that model outputs meet safety standards.

This project addresses two key areas:

• Inbound Risk: Preventing unsupported or harmful user input, such as inappropriate requests or non-relevant topics.

• Outbound Risk: Mitigating unacceptable outputs, including privacy breaches, inappropriate language, and hallucinations.

Our work focuses on developing robust anomaly detection methods to enhance the safety and reliability of Amazon's CSAI system, ensuring responsible AI usage in customer service interactions.

In April 2024, we deployed Large Language Models (LLMs) in our Keyword Recommendation Service and launched the LLM-powered Keyword Group, a new targeting control for Sponsored Products keyword-based campaigns, enhancing advertisers’ targeting strategies and campaign effectiveness. I, 

• Identified giftable products and conducted opportunity sizing for the Gifting Keyword Group to maximize advertising campaign effectiveness during key gifting seasons.

• Engaged in prompt engineering, integrating diverse data sources to refine and optimize LLM performance in keyword generation.

• Conducted evaluations of LLM-generated keywords, comparing their performance against organic search benchmarks, and ensuring relevance through manual audits.

See API release notes and developer guide for more details. 

In collaboration with my colleagues and under the supervision of Dr. Lihong Li,

• Investigated multi-level and multi-agent games with a large number of anonymous agents balancing their individual interests with collective goals set by a social planner.

• Developed a novel optimization framework, MESOB (Mean-field Equilibria & Social Op- timality Balancing), that applies bi-objective and mean-field interactions.

• Transformed MESOB into a single-objective optimization problem called MESOB-OMO using approximate Pareto efficiency and occupation measure optimization (OMO).

• Implemented and applied MESOB-OMO to simulated ad auctions and demonstrated its capability to enhance social welfare and reduce exploitability.

• Presented findings at SIAM Conference on Optimization, MarbleKDD, and Amazon Machine Learning Conference (AMLC).

Paper available on Amazon Science and arXiv.

In collaboration with Prof. Marciano Siniscalchi, Amazon Scholar, and colleagues in Auction team, 

• Analyzed modern online ad auctions, addressing key differences from traditional models.

• Modeled advertisers as agents using an adversarial bandit algorithm.

• Simulated ‘soft-floor’ auctions and compared revenue with that of optimal reserve prices. 

• Inferred advertiser value distributions based on bids observed on an e-commerce website. 

• Presented at AdKDD, and our work won the Sponsored Brand hackathon.

Paper available on Amazon Science and arXiv; Slides and Video (AdKDD).

In collaboration with a group of researchers at Microsoft Research, performance, and COGS execution teams at Microsoft Dynamics,

• Performed comprehensive feature engineering using F&O service telemetry signals.

• Built a machine learning model that predicts Azure SQL Database Transaction Unit (DTU) with a high accuracy and recommended optimal database tier.

• Achieved annual savings of $2.9M by optimizing tiers for over 300 F&O service production databases.

Under the supervision of Dr. Houssam Nassif and in consultation with Prof. Guido Imbens,

• Addressed two challenges in real-world applications: the absence of informative prior(s) in Bayesian settings and the inability to control parameter learning rates.

• Proposed a general framework to learn meta-prior from initial data using empirical Bayes.

• Implemented our proposed meta-prior framework and applied it to Generalized Linear Models.

• Performed experiments on a standard optimization problem as well as a contextual bandit setting in Amazon production system.

• Both during simulations and live experiments, our method showed marked improvements, especially in cases of small traffic.

• Published in Management Science Journal; Management Science 68(3):1737-1755. 

Paper also available on Amazon Science and arXiv.