Vector Databases: A deeper dive

In the last blog of our series on vector databases, we talked about what they are, how they differ from traditional databases, why they are needed, and a brief overview of how they work. Our last part was everything introductory, moving on to the current and blog in this series we’ll be diving deeper into their way of working and the myriad of use cases they come packed with. 

Just to recap, vector databases are powerful tools for managing and querying high-dimensional vector data. They leverage advanced indexing techniques, optimization methods, and distributed computing to provide efficient similarity search capabilities. These databases support a wide range of applications, from recommendation systems and search engines to advanced AI applications, by enabling quick and accurate retrieval of similar items. As the volume of data and the demand for intelligent applications continue to grow, vector databases will play an increasingly important role in the data management landscape.

How do Vector Databases Work?

At the core of vector databases are vectors, which are essentially arrays of numbers. Each vector represents an object, and the dimensions of the vector encode various features of that object. For instance, in NLP, vectors can represent words, sentences, or documents, with each dimension capturing some aspect of the text, such as semantic meaning. These vectors are often generated using techniques like word embeddings (e.g., Word2Vec, GloVe) or more advanced models like BERT and GPT.

Vector databases store these high-dimensional vectors and support efficient querying, which is crucial for applications that require quick retrieval of similar items. The primary type of query in vector databases is the similarity search. This involves finding vectors in the database that are closest to a given query vector based on some distance metric, such as Euclidean distance, cosine similarity, or Manhattan distance. The choice of metric depends on the specific application and the nature of the data.

Vector databases unlike traditional databases store data in the form of vectors. Every object stored within the vector database be it a text, audio, video, or an image file is converter into their respective vectors for storage. The vectors are usually pretty long and are scattered across hundreds if not thousands of dimensions (for example OpenAI’s ada-002 model supports 1536 dimensions).

One of the widely used examples of vector databases is similarity searches. For example, Netflix, when you search for a movie on Netflix it suggests movies similar to the one you searched for based on the similarity between actors & directors in the movie, its genre, run time, and language, etc. Now, let’s dig a little deeper into the sequence of steps that take place when you search for something using a vector database. 

Starting with entering the query or prompt that you use as input. 

⬇️

Your query (prompt)  is converted into vectors using an embedding model (used differently for text, images, and videos) 

⬇️

These vector embeddings are stored within the vector database and the original content used to create them

⬇️

The vector database using the already stored information or vectors finally generates an output corresponding to the user’s query/ input.

⬇️

The output is generated using an Approximate Nearest Neighbor (ANN) search, where the closest vectors to the query embeddings are returned as the final result. If the user continues to make further queries, the same process is repeated, querying the database and producing a new output based on the updated query.

⬇️

Since vector databases rely on similarity indexing, accuracy and speed are key factors. While it may take a bit longer to return a range of results that best match the user's query, the focus is on delivering the most relevant options efficiently.

All in all, a vector database query goes through three major stages:

• Indexation – Before querying, the vector database indexes the vectors using techniques like tree-based structures (KD-trees, R-trees) or graph-based methods (HNSW, ANNOY). These structures help in efficiently organizing the vectors and speeding up the search process. Once vector embeddings are in the vector database, several techniques are used to translate a given vector embedding to data structures for a quicker search. 
• Inquiry – When a query vector is submitted, the database uses the indexed structures to quickly narrow down the search space. Instead of comparing the query vector with every vector in the database, the index helps in identifying a subset of vectors that are likely to be similar. After completing its search, the vector database compares the query vector to the indexed vectors, using the similarity metric to determine its nearest neighbor. 
• Post-production – In the final post-production step, the vector database generates output based on similar results. The vector embeddings are then converted into the appropriate format—such as text, image, or video: depending on what the application supports or the query requires.

The vectorization or embedding process typically uses techniques like Word2Vec, Doc2Vec, GloVe, or BERT, depending on the type of embedding needed. For instance, Word2Vec converts words into vectors so that words with similar meanings are represented by vectors that are close to each other.

Additional Considerations

Vector databases also incorporate various optimization techniques to handle large datasets and high query volumes. One such technique is dimensionality reduction, which aims to reduce the number of dimensions in the vectors while preserving their essential characteristics. Methods like Principal Component Analysis (PCA) or t-Distributed Stochastic Neighbor Embedding (t-SNE) can be applied to transform the data into a lower-dimensional space, making indexing and querying more efficient.

In addition to indexing and optimization, vector databases leverage distributed computing to scale horizontally. They can distribute the storage and processing of vectors across multiple nodes in a cluster, enabling the handling of massive datasets. Techniques like sharding, where the dataset is divided into smaller, more manageable pieces, and replication, where data is duplicated across nodes to ensure availability and fault tolerance, are commonly employed.

Vector Database: Use Cases

Vector databases’ primary use case lies in the field of generative AI, applications such as ChatGPT, CoPilot, and Bard are also powered by Vector databases. The recent hype around Generative AI and its use cases has also led to a surge in the use of vector databases. Below are some of the most popular areas where vector databases can play a crucial role and can be leveraged to the best of their abilities. 

1. Natural language processing (NLP): Natural Language Processing (NLP) relies heavily on vector databases to process unstructured data, such as videos, images, and large documents. Handling this unstructured data can be quite challenging without the support of vector databases. With vector databases, tasks like sentiment and contextual analysis, linguistic translation, and more become much easier and more nuanced through NLP. NLP is a widely discussed use case in machine learning and AI, as it extensively uses vector databases to create embeddings based on contextual and sentiment analysis. This enables real-time, human-like responses to user queries.

2. Anomaly and fraud detection: Another key use case for vector databases is anomaly and fraud detection, which can be applied in areas like online spam and fraud detection, cybercrimes, quality assurance, and control, etc. Vector databases are especially useful for detecting anomalies and fraud within large, unstructured, and multi-dimensional data sets. These databases facilitate quick and efficient detection by using similarity search algorithms to identify discrepancies or abnormalities within datasets easily.

3. Image and video recognition: Vector databases are also valuable for image and video recognition, as they can convert the pixels within an image or video into corresponding embeddings. These embeddings enable fast image or item searches on e-commerce platforms through efficient product tagging. Additionally, image and video recognition can be leveraged to flag inappropriate content for children under certain age limits, helping to enforce mandatory policy regulations and improve on traditional data management methods.

4. Autonomous vehicles: Autonomous vehicles rely on vector databases to process vast amounts of data from various sensors, such as radar, lidar, and video feeds from traffic signals. Since autonomous vehicles use reinforcement learning, this incoming data helps improve driving performance in real-life situations. By identifying and building patterns in on-road movements, vector databases play a crucial role in minimizing the risk of accidents or any other incident.

5. E-commerce product recommendations: E-commerce and shopping sites can use vector databases to personalized product recommendations for customers. By tracking and analyzing customer data, they can deliver tailored products and insights based on browsing history, leading to higher engagement and satisfaction. This personalization also enhances cross-selling opportunities. For example, customers can be shown a complete outfit—combining a shirt, trousers, and matching accessories like a watch, shoes, and sunglasses—by quickly identifying patterns and finding the right product matches in near real-time.

‍