1. Introducción al patrón AI Gateway
Presenta el tema del webinar, al ponente y el contexto general de la evolución de las APIs hacia la IA. Explica por qué Kong ve el AI Gateway como una nueva pieza clave de su plataforma.
all right good morning good afternoon good evening to everyone wherever you're joining us from I'm very happy to have you here for our August user call the topic for today is the AI Gateway pattern we have syano herewith us he will introduce himself uh in just a little bit um this webinar is being recorded uh I will share the recording with you in a follow-up email afterwards uh with some additional resources you will also beable to find the recording on our YouTube channel um you can see on the bottom there is a chat and there is also a Q&A function so please whatever questions you want to ask silvano just put them inthe Q&A function um so we can keep track of the questions um he will answer all of your questions what right after he's done presenting so yeah San please please take it away from here hello everyone uh as
you know uh as you already know my name is silvano uh I joined Kong uh three and a half months ago uh as uh the director of product management uh for the run times so gway govern Randy smashetc etc uh this is my first webinar so super excited to be here um and talk yeah about like one of the uh most interesting thing that I've seen at Kong in the last since I joined uh whichis uh uh our effort uh in the in the a area uh with providing a um a Gateway uh and uh plugins uh for a better uh usage of AI in in an organization um one thing that isvery interesting is uh before coming to Kong I was working at shifi uh and I was working in an area uh where we were trying to use AI uh for a bunch of cool stuff uh and I really
experienced like firstand uh a lot of the problems that we are providing solutions for uh with the plugins uh so you know uh something that uh I really enjoyed uh seeing coming together um so yeah uh let's gothrough this presentation where we will talk about how you can accelerate uh AI transformation in your company in your organization uh using uh Kong uh AI plugins and the AI Gateway um as you may know here Kong ourmission is to power uh the API world so abis are really important for us and uh it's uh crazy uh to see like the the growth of usage of API and now this is driven by digital demand umand uh in this curve uh I still remember when uh uh between the 2007 of the mobile and Cloud uh I moved to California I joined Google and we were like in that transformation of everything needs to go
to Mobile so more and more apis are going out like Google itself uh was uh getting out so many apis for all sorts of developers and it was really like the the start of this API economy uh thatthen uh grew even more like with Cloud where everything is in AP if you're working in the cloud uh patterns and Architectural patterns like microservices for the simplification of monolith application into smaller pieces that are all uh apisand services like alling each other um and then really like the two more recent explosion uh covid you know uh everything goes even more online uh for uh for the needs of people um and then gen AI thatyou know I still remember the day like in uh uh when somebody told me like hey you you need to look at this thing it's really cool uh and it was open AI uh and it was indeed really
cool uh in a way like it made me think about uh 20 years earlier first time somebody showed me like Google search and you know I could actually find uh the things that I was looking for in theinternet um and and gen was you know really crazy and I think almost immediately like everybody was uh you know juming on what they could do uh in their own applications uh using uh J um which leads usto there's a lot of things that you can do uh applications uh are probably like a lot of applications today like they try to use uh different providers of uh large Lang language models in the cloud uh topower some new features to improve some existing feature uh people building AI agents are using like some sort of U selfhosted model uh for the AI agent that they are building um people are building chat Bots that use
all can use all of the above uh but also have some additional like requirements when it comes to improve the quality of the responses of the chat button some specific um domains with uh fine tuning like it's oneof the most diffused techniques um and then I think other use cases but all of these use cases are really starting to get into using uh Vector DBS uh for for different things right mostly uh retrieval augmented generationor you know adding some layers of semantic evaluation uh of what the users are doing for different reasons uh improve the quality of the results again on some specific domain or uh try to get better at the responsesgetting better at understanding like what is the the real meaning of what the user is trying to do uh and we will talk a little bit more about each of these uh in the uh in the rest of
the presentation now if we look at what all these different use cases need when they talk to all these apis basically uh because all of them are apis um that's different uh different use cases uh but a lotof them like are really repeated uh across uh and there's the need for some sort of transformation there's the need for limiting uh you know the how many tokens are sent to to to the llm providers uh ifyou tried using AI you know that it's uh it's it's powerful it's not really cheap uh and the more tokens send the more you pay um and and really like uh the traditional authentication and authorization to make surethat only authorized users and application can uh access your service um and analytics you know know what is working what it's not working in your app um these are the ones that are a little bit more traditional um
and then there's a bunch that are emerging that are more really tied to some specific of the AI uh and for example like the multiple llms providers like which one to use and how to make it easy toswitch uh between like different providers for evaluation or even like for fallback you know just to understand which one can perform better or uh if one is down or is not uh replying like use another one as afullback um other things that are pretty common are sanitization of the prompts like the moment that uh users can you know ask uh question that we'll have like uh replies that are generated by an llm uh some uhsome things you know uh might be better not to ask for in a certain way uh in a way like that's a a lot of production that is built in in all in in in all the providers of
llm uh but in an organization like there might be some specific topics uh that we want to make sure uh users using our apps cannot ask and so this is where really like the the concept of the AIGateway uh becomes like extremely useful in this type of scenario uh and especially like the Kong API Gateway uh because you know it's the it's the Gateway that can sit between all the applications all the use cases uhon on the left side uh and all the providers of the uh AI related service on the right side and then mediate access uh providing some of the tradition um AI Management areas that we talked about like authenticationsand rate limiting uh but also some of the functionalities that are and policies that are more specific uh to AI um and just to give you like a little bit of a sense of how this fits into Kong's
overall offering um Abbi gway is the is the traditional uh the most powerful and the most used like uh open Source Gateway uh in the world uh for abis um we have the Ingress controller uh which is auh Gateway like specifically like for kubernetes environment uh we have the pong mesh uh for eastwest traffic like between uh your services like with uh mtls observability uh and all the features of a modern mesh uh and nowthe AI Gateway like becomes the fourth pillar uh of our ability to let you manage and control traffic uh to a and services uh with these specific functionalities for for AI um and on top of all of this
2. Plugins y capacidades del AI Gateway
Describe cómo funciona el AI Gateway como un conjunto de plugins sobre Kong Gateway. Cubre proxy unificado para distintos proveedores, seguridad, observabilidad y transformaciones low-code con IA.
uh in case you are not already familiar with it uh sits conon connect uh that really like simplifies um the management uh of all of this uh through like a traditional uh SAS uh interace with re UI anda lot of things that uh we manage for you um so going a little bit back to our uh AI Gateway um this is really a set of plugins uh in Kong uh if you know already like howthe Kong Gateway Works uh Kong Gateway comes with a core uh that basically allows you to Define uh the basic concept of the API rooting like Services upstreams uh etc etc then there's a a lot of plugins thatallow you to extend uh the basic functionality of rooting with authentication and authorization uh rate limiting caching retries uh and a lot more services um and you know on top of the those standard plugins we now added uh
some plugins that are specific uh for for AI um and uh really at the very the very basic one like the uh the foundation of everything is the the AI proxy where through a unified uh API interface uhyou can access like the different llms providers uh with requests that are basically the same uh and where uh you know we take care of then uh transforming the request according to what is the spec uh for thespecific uh provider uh if you try to play a little bit with AI probably seeing the format of open AI when you want to send a request is like ly different than the format of all the others uhwe take care of uh uh giving you a common interface and then uh based on the provider that you're trying to reach uh create the the right request in the right format that they expect um and so this
is really about giving you one API uh to rule all uh the different providers um on top of this uh we have some uh specific AI security and obser availability plugins uh and and features that we provide uhnot necessarily with a specific plugin uh but for example like in the case of the credentials uh I've seen this firstand uh very few organization want every develop every developer in the organization to use their own uh individualum credentials to access a provider uh but because you go through the proxy then what you do is the the actual credential to access openai or ker or aour or any of the other provider is in the proyand then you give to your users uh credentials to access the proxy uh and so in this way they can access the llm provider without actually knowing the real credential uh to access the provider uh and on top
of this you know you can build logging so that you make sure that you know you can control all the requests uh that sure the developers and member of your organization are sending to the llm and you knowuh traffic control which is important because as we said uh these things are not cheap right um and then on the other side of functionality there's a compliance and abusee prevention uh which really goes a lot into guardingthe prompts uh and so like excluding like some filtering you know what the users and developers in your organization can ask an llm um but also in some cases with decorators or templates enforce uh the structure of thethe request that is going to go uh to the llm and finally like there's a couple of uh very cool plugin uh which uh are a little bit of uh a way of using uh AI without coding uh
because with the AI request Transformer uh request and response Transformer what you can do is you can apply some transformation to every request and every response uh and the way you do it is uh you express as aprompt what is the type of change that you would want to see uh and then we send it to an llm provider that you specify uh so that they can uh apply the transformation that you requested to yourrequest or response uh and so imagine like one example could be uh you want to uh transform every indication of temperature that it's a fahrenheit in celsius uh you know you just write that prompt we send the promptand the request for example to the llm provider uh and they provide the transformation um or you know in other cases you might want to filter out of uh a response anything that looks like a credit card same
thing uh you uh Express in the prompt want filter out everything that looks like a credit card we send that with the response to the llm that's the filtering and then we send back the filtered response um soyeah that's a uh in this one uh is just expanding the slides on uh uh really like uh all the things that can become better uh when uh uh when you start using AI uh and with you knowtraining of the models but also like uh retrieval ad manded generation to improve uh the quality of the response uh but even like conversions uh data enrichment transformation and sanitization uh can all go through uh the llm itselfuh and become better than just a developer uh writing in a procedural way you know what should be changed and how right According to some uh some definition of uh uh what's better uh and uh you know what
should be replaced with what um there's a some we start to get like in the slides in some features that are coming but we are almost there uh so uh it's really like behind the corner uh and this
3. Funciones de IA que mejoran aplicaciones
Explica capacidades emergentes como semantic caching, semantic routing y semantic firewalling. También resume los beneficios para productividad, cumplimiento, observabilidad y control de costes.
is where really like we start uh using llms uh to improve the quality uh of the of the application uh and the first one is semantic caching where um if you used uh any gen provider uh one thingthat you probably know this like beside the fact that they can be very expensive is uh that also they tend the reply tend to be slow um a lot of them uh have implemented like streaming in the responseso that at least when they start replying uh you see like pieces of the text appearing in a sequence uh but the times that it take to get there uh it's a long and the concept of semantic cachingis uh in many cases um the questions that a user might ask are similar and they have a similar response uh and so for example if you ask how to cook a steak or how to cook a re
by steak you probably end up with a reply that is not very different um and in those cases if you can through an AI understand what is the intent of the user what is that they're trying to doand establish a relation where you can say well these two questions are really similar I already have a response for the first question so rather than going back to the provider again and pay for the generation and waitfor the generation what I can do is just get the response that I have from a cache and serve it to the user uh because the uh the respon the question is pretty similar right uh and so thisis what sandic cash in is about uh and it's a feature that it's coming soon um technically like behind the hood this work with uh embeddings generation of embeddings if you're not familiar with it it's a basically a
representation of a vectorial representation of the uh of Concepts uh or sentences you know uh and then storing those embeddings like and and and storing like those representation in a vector um database where then for example when thenew question comes in you can create an embedding of the new question Vector representation do a vector search in the vector storage calculating is there anything that is near enough that I can consider it the same question andif yes what's the response fetch the response uh and give it back as a cache so you know pretty cool um semantic roading is also a similar concept that it's very interesting uh because imagine that you know youstart having uh some different providers or it could even be like uh you realize that some providers are really good uh at replying at certain like more generic question uh but then for some questions that might be more
specific uh to your domain uh you want to build a model uh that it's uh you know uh your own model that itself hosted uh and make sure that all the questions that are in that the main goto that model um you could manually build a map of all this question needs to go there all these other question needs to go uh to the self hosted model or you could use semantic routing where basically samething you try to understand what is the intention of the user if it's a more generic question routed to uh the llm uh if it's a question that is more specific to my domain route it to my uhcell pH model um and then you know uh security uh and semantic firewalling uh again uh really gets uh uh the power of llm like to the to the Max uh because if you think about that scenario where
we were talking about earlier where you might want to have some prompt filtering where you filter out like some words uh many cases like you would have to write very long lists of words and expression uh with semanticfirewalling you can just Express the intent right and so for example uh if if it's a question about recipes uh you might want to filter out any ingredients or uh any uh process that might lead to damage uhfor the people uh that are actually like cooking their recipes right uh and this would be easier than you know riding things like exclude nails or exclude glue or all the things that should never make it uh ina recipe um so when you s all of this and all that we've been talking about uh you really get to higher developer productivity uh very simple ways of enforcing governance and compliance because you know you have one
place uh where you can do it uh but also observability and ability to optimize your spend because all the a traffic is going through the gway uh and so like you can monitor everything you can enforce policies uhyou can measure uh where the money is going uh etc etc uh we talked about some of the uh features uh we released in May uh the first set of uh plugins uh and uh you know uh thefunctionality that you can see like on the on the left um in June we made some updates in terms of support of some specific LMS providers uh and again you know very soon uh you will get some uhuh some new stuff that it's uh really interesting uh I think especially for me like the old semantic AI plugins area is really uh when we start getting like more and more benefits from uh from using Ai and
4. Visión futura y primeros pasos
Plantea la evolución desde plataformas impulsadas por APIs hacia experiencias guiadas por IA. Luego muestra cómo empezar con el AI Gateway e instalarlo rápidamente con Docker.
U be before like uh we get into showing you a couple of Dots and a couple of some simple demos that I had ready uh you know our uh C and co-founder like Marco Paladino like presented this ideathat uh I think it's very interesting uh on the left you have what we were doing until a year ago uh which is uh apid driven platforms uh where really uh we've been able to provide better and betterexperiences to user uh by simplifying the way that we were building code uh and using apis and using different apis specialized in uh uh providing different services and then combining them in applications in the way that we thoughtwere more relevant and more interesting for the user now we are in the middle which is the AI in reach platform so now we have applications that use API and some of these API improve the quality of the
service uh by using llms behind the curtain right so that they can produce better and better results uh but really like tomorrow is going to be AI dreaming uh tomorrow like we're going to be able like to dothings where uh based on the intention of a user in an application uh thei can combine uh different services that are provided by uh different apis uh in uh in the best way possible for the user uh forme like I worked in developer tools and develop documentations for for a lot of years uh and one of the use case that very interesting is uh you know whenever you go in the documentation site can be thebest documentation site in the world but there's always like a lot of information and you know you really need to find your way based on assumption that whoever put the documentation together like made on on the reader what
if I could just say something like I know Java I don't I I don't like ja let's say I know rust uh and I want to learn how to build an extension on your platform that will run uhin an API that I'm going to deploy on cloud provider x uh and then you know the site itself compose all the information and all the uh all the additional like tutorials and remate code and B plate foryou basically everything that you need to accomplish that task um I think in the documentation side like we are starting to see that uh you know a lot of uh uh docks out there now have some gni featuresthat can really uh do this thing for you uh but the best thing I think is will be when uh you know through spec uh we will be able to see software that writes software uh where we are
literally saying I need you to connect from you know this uh language to this API and retrieve for me this information and transform it in this way uh and then an llm will generate all of that you knowuh it's going to be uh it's going to be become really like super interesting super cool and super fun um so yeah with all of that before we get into the Q&A uh let me just show you uha couple of other things uh and in order to do that I wanted first uh to show you oh sorry okay uh as I mention it's the first time I'm doing this I haven't presented in a while alsoum if you want to get started with the AI Gateway it's uh super easy you know just go you can just Google like Kong AI Gateway uh and you'll eventually land on this page uh the one thing that
is really cool that's a a script uh that as long as you have Docker uh you can just run this script uh depending on your internet connection uh it will take some time uh but that's really like theslower part uh and then it's the script is going to help you uh guiding you through like installing and setting up some services for you uh including like uh asking you uh a keys for uh some the providersthat we support or uh if you have a Mistral or a llama model that is s hosted uh helping you like configure access to that um and then you know you will find yourself with a uh with akway uh with the AI plugins enabled U and this is all available in the open source version um and the other thing that I wanted to show you is uh I had uh uh couple of uh examples here
5. Demostraciones de enrutamiento y filtros
Muestra ejemplos prácticos en Insomnia para cambiar de proveedor sin modificar la petición y para bloquear contenido con reglas de prompt guard. También enseña cómo configurar transformaciones de respuesta.
uh in insomnia uh and in the first one uh I'm really like just uh ignore like the the request it's I was playing with stuff earlier just to make sure that the reply to how to cook a steakand how to cook a Reby steak was actually similar uh and that I was not lying to you um it I set up and then I will show you the configuration uh well actually the script that script thatI was showing you set set all of these up for me uh and so now I'm trying to send uh this request to open AI where uh it's really like what is Kong uh and assume that my roleis a user uh and here like you could try to add any role that you would want the llm to assume while generating uh an AI I don't know if you ever tried this it makes a lot of
difference uh if you tell the AI like from which point of view it should generate a response so now if I send this you will see that uh I exceeded my uh water and uh it's actually long storyshort I don't I I cannot add the payment method to open a uh because I don't have a credit card at the moment right so what can I do I'm lost no because I can just change the urlthis is a road that is configured in Kong and I will show you to you in a second and now just by doing this I'm just going to send this to coh here right another LM provider as wediscussed earlier it's going to take some time uh but now I have my response and I didn't have to change anything I just had to change the url uh and this is something that you can also manage uh
in other ways like uh changing the OST in the Ed and configuring Kong uh to serch from a different OST um so if you want to see like how this uh works in the configuration just let me goback here uh where uh this is the Kong manager so this is the UI for the configuration of the uh of the Gateway uh I can you can see I'm not lying to you uh this is just thethe free one and if you go to the routes you would see that I have uh the one that I was trying to use that was failing that it's the one that routes to open AI is this oneuh and then I just changed it uh to this uh uh cohere chat and just by doing this uh I change the llm providers without changing at all my request right um and as I was mentioning like you
could also route in a different way by using for example uh the host header uh and assign a different value to open AI uh and go here um the second thing that I wanted to show you is uhI set up another uh plug-in to do one thing that is uh very important which is imagine that I want to ask do you like pineapple on pizza so now if I try to do this immediately says badrequest because this is a bad thing to to ask nobody should ask this you know there is no pineapple and pizza and there never will be um and so I'm just joking honestly like I'm Italian I need toassume the role of uh the person that Ates the pineapple on the pizza uh I actually like it in some cases but uh uh it's with jalapenos I like it jalapenos uh so what I did there is if
we get into uh the coher route now you can see that uh oops not here um I put it at at the service level so I configured this really all I did in this case was expressing a denypattern uh that now it's saying if anything contains pineapple and pizza don't let it go through right uh but this is also uh if you remember when I was talking about the doing this like with semantic firewalling thisis where it becomes interesting right uh because in this case like I have to write pineapple and pizza and a long list of ingredient s for example like chicken uh what I really like want to say is filterout anything that an Italian would never put on a pizza right and then you get pineapple barbecue sauce ketchup chicken uh the list is uh uh is really long um and I hope I'm not surprising you there um
if you want to know more about the AI plugins uh just go to docs con.com slab uh and you will find like all the AI plugins that are currently available uh the ones that I show you now arethe basic AI proxy so the thing that really allows you more easily uh to support uh different providers um and then this AI prompt guard uh but there's a bunch more and uh maybe like one thing that Ican show you uh just this AI response Transformer is the one one that I was telling you like allow you to do a little bit of low code changes uh and if you see the examples uh here uhfor example in this case uh The Prompt is for any city name put the country that's in in Brackets next to it reply with only the transformation uh with only the Json result uh and then here is how
you can configure like where to send it so uh for example like aour or open AI or uh whatever the whichever provider you want to use uh and then this will automatically uh apply the transformation to your response
6. Preguntas y respuestas finales
Responde dudas sobre filtrado de datos sensibles, fiabilidad de las transformaciones y enrutamiento entre modelos. Cierra aclarando el flujo entre aplicación, gateway y proveedores de IA.
so this is it uh and with this I'm now going to try to figure out how to share the Q&A uh amazing thank you so much syano I think we have one question currently in the Q&A
say uh so is it okay okay now I didn't now you should see it right yeah we see your screen yeah can we apply DLP uh bya AI Gateway I am blanking out on the acronym uh
uh might ask the question like uh if you can can you Pell out the
me you mean DLP oh yeah can we filter pii data yeah so uh the I mean in a in a sense uh you you would rely on the ability of the llm to to filter out the pii datauh but you know uh you could also like use uh create your own model uh that can do like that filtering uh in the right way uh and and then uh uh basically like direct to that model thisis the reply for today uh fing out pii especially like when it comes to um uh ABI uh is a a very important use cases uh use case so I would say you know stay tuned uh because maybeuh you'll see something
um you said that AI will trans AP request and response how can we be so sure it will be correct um yeah I mean that it's uh it's Al like a little bit of uh uh with AI likeyou never know exactly what you are getting uh which uh when I was working on trying to build products based on it uh one thing that was very surprising for me because it's non deterministic if you ask thesame thing 10 times there is no guarantee that you will get the same response 10 times um the I think with with the with the plugins like we are giving you like all the infrastructure that you need uhto perform an evaluation of the of the quality uh and then route based on where you see uh that you get uh better results and better responses uh and not just this but also like other other parameters like
at some point uh you might even want to find the tradeoff between quality and cost right and express like some level of threshold where uh you pay enough but not too much you get quality that is good uhand it's the best sort of compromise um do you think blockchain could be a better way to store AI information for more secure platform I one of the thing I really don't know almost anything uh about blockchain uhso I don't know I cannot really answer to this question
sorry we have I think one question in the chat as well from
Timothy I don't know how to get the zoom okay
here uh from the slides it appears that the K API Gateway can be configured to Issue request um calls based on the AI response uh to Downstream up can you confirm
uh so I'm trying to understand the question I'm not sure I I um request code based
uh um I mean the so I don't understand the part with the downstream apps uh in the sense that most of what I was showing is uh it goes the other way right so it's uh yeah okay yeahif you want to chat with the yes that that's good but the reply I can give you now it's uh it goes from the application to the llm and then the Gateway is in the middle like routing uhthe requesting routing filtering uh in different ways like to the different uh sem to the different providers that you can have like whether it's a llm it's essentially like red into different llms or selfhosted uh models and thenin between apply a series of uh uh optimizations or transformation based on some criteria that you express um but then the reply uh goes back to the to the application down stream so that that's the partner I'm not
sure I understanding and yeah please follow the suggestion join con community.com uh and uh any questions that uh I couldn't ask they're going to reply please ask it there exactly and somebody else will jump on
it okay and one more in theQ&A um
okay I don't see the Q name okayjust yeah so that's coming uh that how can we route to different models based on some context uh stay tuned that those features are coming it was the semantic crowding soon soon okay very soonamazing thank you Timothy as well yes please follow up um on the slack Channel and hopefully we can help you there cool any last questions for syano before we wrap it up you can expect the follow-up email uhin a few hours with some resources and the recording so yeah keep an eye on your emails and yeah if there are no other questions thank you so much syano for presenting for answering the questions we were veryhappy to have you here and thank thank you so much everyone for joining us today and uh yeah we hope to see you at our next webinars sometime I hope you have a good evening good day and uh