Loading…
18
Serious question, did I mess up picking the open source model over the paid API for my side project?
So about 3 months back I had this little app idea, nothing crazy, just a tool that rewrites product descriptions for folks selling on Etsy. I hit the fork in the road everyone talks about, run a small open source model on my own rented GPU for like $40 a month, or just pay per call on one of the big hosted APIs. I went with the self hosted one because I liked the idea of owning the whole stack and not worrying about rate limits. First month was fine, maybe 200 users, model ran quick enough. Then I made a post on reddit that got some traction and traffic jumped to like 9,000 people in a week and my GPU box just fell over. Turns out serving a model to that many people at once is a whole different job than running it for yourself. I burned two weekends learning batching and caching just to get back to where the API version would have been on day one. The kicker is my total spend by now is basically the same as if I had just paid per call, except I did it with way more stress. So my question for anyone running models in production, is self hosting ever actually worth it at small scale, or did I just learn a $600 lesson the hard way?
1 comments
Log in to join the discussion
Log In1 Comment
iris_hall20h ago
Well, caching is less about the model choice and more about not paying for the same request twice.
1