Top suggestions for PortKey and Similar LLM Proxy Architecture Diagram |
- 420×420
+have+been+driving+a+new+wave+of+interactive+AI+applications+across+numerous+domains.+However%252C+efficiently+serving+LLM+inference+requests+is+challenging+due+to+their+unpredictable+execution+times+originating+from+the+autoregressive+nature+of+generative+models.+Existing+LLM+serving+systems+exploit+first-come-first-serve+(FCFS)+scheduling%252C+suffering+from+head-of-line+blocking+issues.+To+address+the+non-deterministic+nature+of+LLMs+and+enable+efficient+interactive+LLM+serving%252C+we+present+a+speculative+shortest-job-first+(SSJF)+scheduler+that+uses+a+light+proxy+model+to+predict+LLM+output+sequence+lengths.+Our+open-source+SSJF+implementation+does+not+require+changes+to+memory+management+or+batching+strategies.+Evaluations+on+real-world+datasets+and+production+workload+traces+show+that+SSJF+reduces+average+job+completion+times+by+30.5-39.6%2525+and+increases+throughput+by+2.2-3.6x+compared+to+FCFS+schedulers%252C+across+no+batching%252C+dynamic+batching%252C+and+continuous+batching+settings.&cdnurl=https%3a%2f%2fth.bing.com%2fth%2fid%2fR.965147946f39e73e1e0bb218bd658cac%3frik%3drjyzpnIH5QHQAQ%26pid%3dImgRaw%26r%3d0&exph=630&expw=1200&q=PortKey+and+Similar+LLM+Proxy+Architecture+Diagram&FORM=IRPRST&ck=759436B3C841AE9BB206E9D8BA577AA1&selectedIndex=17&itb=0)
Some results have been hidden because they may be inaccessible to you.Show inaccessible results
Top suggestions for PortKey and Similar LLM Proxy Architecture Diagram |
Feedback
