← All courses ← Course home

Lesson 05 of 06

Utility audio, speech, vectors, and text

The rest of the utility catalog turns speech into text, media into searchable vectors (embeddings), or text into a useful form, or into changed text.

Agent brief (llms.md)

These models are not trying to be general chat assistants. Each one has a focused job. Ask what the app needs to store or return, then pick the matching utility task.

MediaSpeech, image, audio, or text
utilityFocused match or conversion
SpeechASR
Image or audioEmbedding vectors
TextEncoding or sequence-to-sequence output

ASR: speech to text

ASR means automatic speech recognition. It takes spoken audio and returns words. The registry lists whisper-large-v3-turbo, whisper-large-v3, and wav2vec2-base. Use this path for dictation, transcripts, or making spoken content searchable. It is not the same job as an LLM writing a reply after the transcript exists.

InputRecorded or live speech
asrwhisper or wav2vec2
OutputWords and transcript text

Embeddings: useful numbers for matching

An embedding turns an item into a vector, a list of numbers the app can compare with other vectors. You do not need the math to use the idea. Similar items should be easier to find near one another in the app's search or matching system. clip-vit-b32 gives image and text vectors. clap-htsat gives audio vectors.

Use CLIP when the app needs to connect pictures and words, such as finding pictures that fit a text description. Use CLAP when the app needs to compare or search sounds. These are useful numbers for a later app decision, not finished chat answers.

Encoding and text transformation

The encoding entries are roberta-base, t5-small, t5-base, and t5-large. RoBERTa is a text encoding utility. T5 entries are sequence-to-sequence style utilities. That means text goes in and changed text comes out. Keep them in the focused utility path unless the app really needs a general assistant.

NeedWords from sound
Typeasr
ResultTranscript
NeedCompare media or change text
Typeembedding or encoding
ResultVectors or text output

The architectures course helps you tell a language-writing shape from these narrower text utilities. The forward course is the image-classifier contrast. Continue in From the metal to the model when a utility recipe needs a closer look at runtime and the chip.

Keep the job visible. ASR returns words. Embeddings return numbers you can compare. Encoding utilities represent or change text. An LLM writes a broader language response.

Finish by choosing the type from the app's real input and output.

Key concepts

  • ASR turns speech into text: whisper-large-v3-turbo, whisper-large-v3, wav2vec2-base.
  • Embeddings return comparable vectors: clip-vit-b32 for image and text, clap-htsat for audio.
  • roberta-base encodes text. t5-small, t5-base, and t5-large change text sequence to sequence.
  • These are focused jobs, not general chat like an llm.

Takeaways

  • You can pick ASR when the app needs words from speech.
  • You can pick an embedding or encoding preset when the job is compare, represent, or change text.
  • You can say when an llm fits better than ASR, embedding, or encoding.