FriendliAI

An optimized serving engine behind the API

FriendliAI runs inference for open-weight and custom AI models. Its platform combines a serving engine with managed model APIs, dedicated endpoints and deployment options using customer infrastructure. The company describes optimizations including GPU kernels, batching, caching and speculative decoding. These concern how a model executes and shares compute, rather than a claim that FriendliAI trained every model it serves. [1]

The deployment choices address different operational needs. Shared endpoints let developers call a prepared model; dedicated deployments allocate resources for a particular workload; a container can run the serving engine in a private environment. Contemporary reporting described these distinct options when the serverless service launched. [3]

Adding shared model access

On January 4, 2024, FriendliAI launched Friendli Serverless Endpoints, offering API access to models including Llama 2 and Stable Diffusion. FriendliAI managed the infrastructure and GPU serving setup. Its speed and cost comparisons were vendor claims, not independently established properties of every model or workload. [2][3]