Performance index. Every figure, measured.

Under 300 s from launch to serving a 120B model. Under 1 s to first token for 95% of requests with 64 users at once. Up to 195 tokens/s for one user on a 120B model.

0 failed of 8,088 requests in an hour. Up to 4,212 tokens/s served by one GPU on a 120B model, and up to 14,391 tokens/s on a 3B model. Run the same test on your own endpoint.

Home · Pricing · Machines · Performance index · sales@calipercloud.com