# What is an embedding for AI?

When a question is presented to an Artificial Intelligence (AI) algorithm, it must be converted to a format that the algorithm can understand. It is often called “problem embedding” to use the verb form of a word. Scientists also use the term as a noun and talk about “embedding”.

In most cases, embeddings are a collection of numbers. They are often arranged in vectors to facilitate their presentation. Sometimes they are represented as square or rectangular matrices to enable some mathematical work.

Embeddings are made from raw data which can be numerical audio, video or text information. Almost any data from the experiment or sensor can be converted to embedding in some form.

In some cases, it is a clear process. Numbers like temperature or time can be copied very literally. They can also be spherical, converted into separate sets of units (say Fahrenheit to Celsius), to clear out common or simple errors.

In other cases, it is a combination of art and knowledge. Algorithms take raw information and find key features and patterns that can help answer the on-hand question for AI. For example, autonomous cars can detect octagonal patterns to identify stop signs. Similarly, a text algorithm can detect words that usually have an angry meaning so that it can measure the spirit of the statement.

## What is the structure of AI embedding?

The embedding algorithm converts these raw files into a simple collection of numbers. This numerical format for the problem is usually a deliberate simplification of the various components of the problem. It is designed in such a way that the details can be described with a very small set of numbers. Some scientists say that the embedding process goes from information-loose raw format to data-intensive format of embedding.

This short vector should not be confused with large raw data files, which are ultimately just a collection of numbers. All data is numerical in some form to anyone because computers are full of logic gates that can only make decisions based on numbers.

Embeddings are often some important numbers – a brief summary of important elements in the data. Analysis of a sports problem, for example, can reduce each entry of a player into height, weight, running speed, and vertical leap. The study of food can reduce every possible menu item in its composition of proteins, fats and carbohydrates.

Deciding what to include and what to leave out in embedding is both an art and a science. In many cases, this structure is a way for humans to add to their knowledge of the problem area and release outside information while guiding AI to the heart of the matter. For example, embedding can be structured so that athletes can exclude their eye color or number of tattoos.

In some cases, scientists deliberately start with as much information as possible and then let the algorithm find the most important details. Sometimes human guidance excludes useful details without implicit bias.

## How are embeddings biased?

Artificial intelligence algorithms are as good as their embedding in their training set and their embedding is as good as the data inside them. If the raw data collected is biased, the embeddings made from it – at least – will reflect that bias.

For example, if a dataset is collected from a town, it will only contain information about the people of that town and with it all the characteristics of the population. If the embeddings made from this data are used only in this town, the biases will fit the people. But if the data is used to fit the model used for many other towns, the biases can be very different.

Sometimes biases can enter the model through the process of creating embedding. Algorithms reduce the amount of information and make it easier. If this removes some of the critical element, the bias will increase.

There are several algorithms designed to reduce known biases. E.g. Maybe only some people responded to the request for information or maybe the data was just collected in a biased place. The embedded version can randomly exclude some extra set to restore some balance overall.

## Is there anything that can be done about bias?

In addition, there are several algorithms designed to add balance to a dataset. These algorithms use statistical techniques and AI to identify whether the dataset contains risky or biased correlations. Algorithms can then delete or resale the data and remove some bias.

A skilled scientist can also design embeddings to target the best answer. Men who create embedding algorithms can and do choose approaches that reduce the likelihood of bias. They can either omit certain data components or minimize their effects.

However, there are limitations to what they can do about incomplete datasets. In some cases, bias is a strong signal in the data stream.

## What are the most common formats for embedding?

Embeddings are designed for data-intensive representations of the dataset being studied. The most common format is the vector of floating-point numbers. Values ​​are measured, sometimes logically, so that each element of the vector has the same range of values. Some prefer values ​​between zero and one.

One goal is to ensure that the distance between the vectors represents the difference between the underlying elements. This may require some artistic decision making. Some data components may be truncated. Others can be measured or combined.

While there are some data components such as temperature or weight that are naturally floating-point numbers at full scale, many data components do not fit this directly. Some parameters are boolean values, for example, if a person owns a car. Drawn from a set of other standard values, such as car model, Mac and model year.

One real challenge is to convert unstructured text into embedded vectors. A common algorithm is to detect the presence or absence of unusual words. That is, words that are not the basic verbs, pronouns, or other adjectives used in each sentence. Some of the more complex algorithms include Word2vec, Latent Semantic Analysis (LSA), Latent Derichlet Allocation (LDA) and – Biterm Topic Model (BTM).

## Are there standards for embedding?

As AI has become more common and popular, scientists have created and shared some standard embedding algorithms. These versions, often secured by open-source licenses, are often developed by university researchers who share them to enhance knowledge.

Other algorithms come directly from companies. They are effectively selling not only their AI learning algorithms, but also embedding algorithms for data pre-processing.

Some of the more well-known standards are:

• Object2vec – From Amazon’s Sagemaker. This algorithm detects the most important parts of any data object and keeps them. It is designed to be highly customizable, so that the scientist can focus on important data fields.
• Word2vec – Google created Word2vec by analyzing the language and finding an algorithm that converts words into vector embeddings by analyzing contexts and creating embeddings that capture semantic and syntactic patterns. It is trained so that words with the same meaning end with the same vector embedding.
• Glove – Researchers at Stanford have developed an algorithm that seeks to analyze data from around the world. The name is short for global vector.
• Inception – This model uses a convoluted neural network to directly analyze images and then generate embeddings based on content. Its authors came from Google and many large universities.

## How are market leaders creating embeddings for their AI algorithms?

All major computing companies have strong investments in the tools needed to support artificial intelligence and algorithms. Pre-processing any data and creating customized embedding is a key step.

Amazon’s Sagemaker, for example, offers a powerful routine, Object2Vec, which converts data files into customizable embedding. The algorithm also learns as it progresses, adapting itself to the dataset to generate a continuous set of embedding vectors. They also support a number of algorithms that focus on unstructured data, such as blazing text, to extract useful embedding vectors from large text files.

Google’s TenserFlow project supports the Universal Sentence Encoder to provide a standard method for converting text into embeddings. Their image models are also pre-trained to handle some of the standard objects and features found in the image. Some use this as a basis for custom training on their specific set of image sets.

Microsoft’s AI research team provides extensive support for a number of universal embedding models for text. Their multitasking, deep neural network model, for example, aims to create robust models that are also compatible with the language used across different domains. Their debart model uses more than 1.5 billion dimensions to capture many of the complexities of natural language. Earlier versions are also integrated with the AutomatedML tool for easy use.

IBM supports a variety of embedding algorithms, including many standards. His quantum embedding algorithm was inspired by parts of the theory used to describe subatomic particles. It is designed to preserve logical concepts and structure during the process. Their MAX-Word approach uses swivel algorithms to preprocess text as part of their training for the Watson project.

## How do startups target AI embedding?

Startups focus on narrow areas of the process so they can make a difference. Some work on optimizing the embedding algorithm and others focus on specific domains or applied areas.

Building good search engines and databases for storing embeddings is an area of ​​great interest to make it easy to find close matches. Companies like Pinecone.io, Milvus, Zilliz and Elastic are building search engines that specialize in vector search so that they can be applied to vectors produced by embedding algorithms. They also simplify the embedding process, often using common open-source libraries and using embedding algorithms for natural language processing.

Intent AI seeks to unlock the power of network connections detected in first-party marketing data. Their embedding algorithms help marketers implement AI to optimize the process of matching buyers with sellers.

H20.ai builds an automated tool to help businesses apply AI to their products. The tool has a modeling pipeline with prebuilt embedding algorithms as a start. Scientists can also buy and sell model features used in embedding creations through their feature store.

The Rosette platform of BASIS technology provides pre-trained statistical models for identifying and tagging units in natural language. It integrates this model with indexer and translation software to provide a pan-language solution.

## Is there anything that can’t be embedded?

The process of converting data into numerical inputs for the AI ​​algorithm is usually slow. That is, it reduces the complexity and amount of detail. When some of this data destroys the required value, the entire training process may fail or at least fail to capture all the rich variations.

In some cases, the embedding process can carry all the bias with it. An excellent example of an AI training failure is when an algorithm is asked to distinguish between photos of two different types of objects. If one set of photos is taken on a sunny day and the other on a cloudy day, subtle differences in shading and color can be captured by the AI ​​training algorithm. If the embedding process goes through these differences, the whole experiment will produce an AI model that has learned to focus on lighting instead of the object.

There will also be some really complex datasets that cannot be reduced to a simpler, more systematic form. In these cases, different algorithms that do not use embeddings should be used.