Ollo, Inc.

At Ollo, Inc., which builds video-analysis AI for manufacturing, I worked on research and development for image and video analysis as an AI/ML engineer.

The work focused on fine-tuning and improving vision-language models (VLMs), end-to-end development of a video-analysis pipeline using VLMs and MLMs, and raising analysis accuracy by improving existing algorithms.

After a problem and a goal were defined, I owned the technical approach, design, implementation, and evaluation on my own. I chose the next step from related research and from analysis of experiment results, then repeated implementation and verification.

Beyond improving a single model, I also built a video-analysis system that combines multiple models and improved data processing and inference. The internship covered both model research and implementation of the surrounding ML system.

Work

1. Fine-tuning and improving a vision-language model

The largest part of the internship was fine-tuning and improving a pretrained vision-language model.

The goal was to adapt a general pretrained model to a video-understanding task. I studied how to train the model, how to prepare its inputs, and how to evaluate it. I owned the research loop: designing the experiments, analyzing the results, and deciding what to change next, as well as training and evaluation themselves.

Implementing fine-tuning

Using PyTorch, Hugging Face, and Unsloth, I worked on parameter-efficient fine-tuning (PEFT) with LoRA and on supervised fine-tuning (SFT).

I decided how to handle the training data and which training conditions to use for the task, and implemented the training procedure. I then evaluated the model outputs, analyzed what the fine-tuning had changed, and used that analysis to guide the next improvement.

Long videos and input design

Unlike a single image, a video-understanding task has to represent information that continues over time. Long videos also have limits on how much can be passed into the model, so the choice of segment and of what to feed the model matters.

I investigated and tested several ways to split a video in time and to compose the model input, and looked for a processing method that fit the task. Rather than applying an existing VLM unchanged, I designed the input around both the model and the structure of the video.

Evaluation and improvement

Evaluation combined quantitative metrics with qualitative review of individual predictions. From those results I identified how the model behaved and where it failed, and formed hypotheses about training and input processing. I implemented the changes and re-evaluated them to see how each one affected performance.

This work confirmed an improvement over the base model. I learned the libraries and training methods used with large pretrained models, and practiced adapting a model to a specific task by following the experiment results.

2. End-to-end video-analysis pipeline using VLMs

I built an end-to-end video-analysis pipeline that takes video as input, runs model analysis, and returns the result.

The goal was not a standalone model, but a set of steps that work together as one analysis system. I designed and implemented a pipeline that combines video processing with model inference and runs from input through to the analysis output.

I implemented and checked the system with the role of each step and the handoff between models in mind, so that the whole system produced the intended result. Testing the full pipeline made the difference concrete: a model can run correctly on its own and still fail to meet the goal once it is part of the system.

3. Accuracy improvements through data augmentation and inference changes

I also worked on an existing video-analysis algorithm, with the goal of raising its accuracy. The changes were data augmentation and a different inference procedure, covering both data processing and inference.

Data augmentation

I used augmentation on video and image data to improve the existing analysis. I chose the processing from the data and the task, then measured the result. This showed that how the input is prepared affects prediction quality, not only the training algorithm.

Inference procedure

I revisited how the existing model was run at inference time. Instead of only replacing the model, I changed the inference procedure and compared the analysis output with the previous method. These changes improved the accuracy of the existing algorithm.

The approach was to improve the system by revising the current processing and the treatment of the data, not only by introducing a new model.

Independent development and experiment-driven problem solving

On each task, after the problem and the goal were set, I carried the technical approach, design, implementation, and evaluation myself.

I analyzed the problem, read papers, technical material, and existing implementations when needed, and then chose an approach. I implemented that approach and checked it with experiments. From the results I identified what was still wrong, and fed that into the next implementation.

I ran that loop independently: problem analysis, technical research, design, implementation, experiment and evaluation, then revision. I was responsible for both the technical decisions and the implementation. Findings and experiment results were shared with the team, and technical discussion sharpened my understanding of the problem.

What I learned

Building an ML system takes more than model training and architecture. Data processing, the inference procedure, how models connect, and how evaluation is designed all matter.

Fine-tuning a VLM and, separately, building an end-to-end pipeline that combines several models gave me both sides of the work: research on a single model, and development of an ML system that uses it.

Changing augmentation and inference showed that replacing the model is not the only way to improve performance. The data and the surrounding processing have to be considered together. Carrying each task independently was practice in setting a direction for an unfamiliar technical problem and improving it through implementation and experiments.

Technical stack

• Language: Python
• Deep learning: PyTorch, Hugging Face Transformers, Unsloth
• Computer vision: OpenCV, video processing
• Methods: vision-language models, LoRA, PEFT, supervised fine-tuning (SFT), data augmentation
• Environment: Linux