Embedding models, the backbone of our AI’s understanding and generation capabilities, require text to be broken into manageable pieces or “chunks.” This necessity has pushed us to rethink traditional approaches extensively.
Traditionally, chunking strategies focus on dividing texts into segments that don’t exceed maximum size limits while ensuring logical continuity—think avoiding cutting sentences in half. This method is suitable when dealing with large documents like books, where each paragraph can stand alone contextually. However, in our case, where the AI needs access to entire news articles for comprehensive understanding and response generation, conventional chunking methods fall short.
Faced with the need to provide our AI with complete articles while adhering to size constraints of our embedding models, we were compelled to develop a custom chunking strategy. Our goal was not only to meet technical size limits but also to encapsulate the full text within each node created in our database. This approach ensures that every piece of text is available to the AI without additional fetch operations from our storage systems.
How we got there….
We leveraged availability of ContentFormatter interface (a tool that was pivotal in our original setup) as well as the fact that each Document can have registered ContentFormatter. But over time as Spring AI has evolved, newer versions of Neo4jVectorStore stopped using ContentFormatter.format call. This change could have derailed our data handling capabilities, but instead, it spurred innovation. We designed a custom Neo4jVectorStore class that incorporates a unique logic for node creation, continuing to utilize the ContentFormatter to ensure that full texts are embedded within nodes, not just the chunks used for generating embeddings.
This journey through the intricacies of chunking has not only been a technical challenge but a profound learning experience. It underscored the importance of adaptability and innovation in the face of evolving technologies. Our custom solution ensures that our AI systems are not just efficient but are also comprehensive in their ability to understand and interact with content.
As we continue to refine our approaches and enhance our systems, the lessons learned from tackling such challenges are invaluable. They remind us that at the heart of technological advancement lies the capacity to rethink, adapt, and innovate.
For more insights and updates on our journey with AI, stay tuned to our blog and feel free to share your thoughts or questions in the comments below!
