The Evolution of Locobuzz Sentiment Analysis & AI
The master plan
Like all great plans, we, too, have humble beginnings. Our AI journey started with Sentiment Analysis.
Data is the new gold
Locobuzz has a well-managed terabyte scaled database. Using this, we’re able to accumulate a large dataset, but that was not enough. We needed variety, hence, we went to the internet and salvaged every bit of data we could get our hands-on. Through the hardships, we were able to accumulate a 9.3 million dataset spread across 38 global languages.
From crude oil to petrol
- ORM executives have contrasting perceptions of sentiment. Ex. “Nice” can be tagged as positive by one agent and neutral by another
- #LoveMyCountry is not understood by AI unless you split the word into “Love my country”
- The Thai language has no space between its words
At last, our efforts paid off. The Generic Preprocessor can process any kind of NLP-related data in more than 30 languages. It can resolve more than 50 issues in the data. This gives our NLP AI a significant boost in performance.
Making the smart kid
For Sentiment, we were looking for a smart algorithm. After months of experimentation, refinement, and retrials we were able to make not just one but two algorithms. We started using these two algorithms in hybrid mode. Locobuzz now has a bleeding-edge Hybrid Sentiment AI that is fast to train and is very accurate. Compared to before, new AI gives 76.3% accuracy on the same 7,000 datasets.
A good teacher and a good school
To perfect the training procedure, we required a cluster of high computationally intensive Graphics Processors (aka good school) that can handle the computations of 2.4 million parameters. These processors are extremely costly and are paid by minutes of usage. So we had to find the best training procedure with the least amount of trials. It’s a daunting task and our team did it very well.
Standing strong in the tsunami
Now, our challenge was the volume of sentiment requests. On a tsunami day, there are over 1500 sentiment requests per sec in about 30 different languages. We needed to create an API engine that can support such storms. Our team of data engineers created SentimentAPIEngine. This system supports multithreaded request handling and no-downtime updates. This allows it to function 24 hours/365 days without any downtime. As a result, we cater to our clients without any glitch.
Measure of success
As we receive a complaint, it goes through the verification and correction process. The model is updated every week and there are continuous revisions. We realized the value of what we’re able to deliver for our clients and that was all the motivation. Today, we have state-of-art sentiment analysis with great accuracy, and we are still pushing our limits to achieve zero tolerance.
A basket full of ai
Similar to our sentiment AI, we have carefully worked on each of these systems and tried our very best to make them zero complaints. These AI required a lot of effort to make them generalized for every client but our team’s ingenuity and creative ideas made it possible. We are very proud to say that each of these technologies is setting new industry standards.