Journals Details

PERCEPTUAL BOUNDARY CONDITIONS IN VISION-LANGUAGE MODELS: A CONTROLLED STUDY OF ABSTRACTION IN HUMAN-GENERATED SKETCHES

Thomas Chen

Rye High School

ORCID: https://orcid.org/0009-0008-3869-9255

ABSTRACT

This study examines how a vision-language model interprets abstract and finished sketches. Although modern artificial intelligence systems can generate and classify complex images, less attention has been given to how they interpret simplified, hand-drawn, or stylistically reduced visual forms. This problem matters because many real-world images, including sketches, diagrams, icons, and student drawings, do not look like polished photographs. The study tested OpenCLIP on two matched sketch datasets: a 240-image synthetic dataset and a 120-image fully human dataset. Both datasets used the same twelve object categories and the same two drawing conditions: abstract and finished. The model evaluated each image by comparing it with a fixed set of text labels and recording Top-1 accuracy, decision margin, and category-level confusion patterns. OpenCLIP performed well overall, but it responded differently to synthetic and human drawings. The human-only dataset produced the highest Top-1 accuracy at 95.0%, compared with 87.1% for the synthetic dataset. However, the synthetic dataset produced higher average decision margins, about 0.081 compared with 0.056 for the human-only drawings, meaning that the model often made more decisive choices even when its accuracy was lower. Finished drawings improved recognition in both datasets. Category-level results showed important semantic drift patterns, especially for plant and shoe in the synthetic data. These findings show that abstraction can be used as a diagnostic tool for identifying when machine interpretation becomes unstable.

Keywords: Artificial intelligence; Computer vision; Vision-language models; Sketch recognition; Visual abstraction; Computational art

INTRODUCTION

Recent advances in artificial intelligence have changed how images are created and how researchers think about visual meaning. Diffusion models can generate images from noise, and vision-language models can connect images with words in a shared mathematical space [1]-[3]. These systems are important because they do more than produce images. They also make choices about what an image appears to mean.

Much public discussion asks whether AI can make art or whether AI is creative. This project asks a narrower and more testable question: how does an AI model interpret human-made drawings when visual detail is deliberately reduced? The question is important because a simplified sketch may still be clear to a human viewer while becoming uncertain or misleading for a machine. This difference can reveal what visual cues a model depends on.

The study focuses on abstraction, which means reducing or simplifying visual information. In art, abstraction is often treated as a style. In this project, abstraction is treated as a research variable. By comparing abstract sketches with more finished sketches, the study tests when a model keeps the intended object label and when it drifts toward another category.

The main research question is: How do abstract and finished sketches affect the way a vision-language model classifies hand-drawn and synthetic drawings? A secondary question is whether different object categories show different kinds of confusion. The expected result was that finished sketches would generally improve recognition, but that some categories would remain more difficult because the model depends on specific shapes or internal details.

METHODOLOGY

This study used a comparative zero-shot computer-vision evaluation design. Zero-shot means that the model was not trained or fine-tuned on the study drawings. Instead, it used its existing image-text knowledge to compare each sketch with a fixed list of possible labels.

The full dataset contained 360 model-input images. The synthetic corpus contained 240 file-level inputs, organized as 12 categories by 2 drawing conditions by 10 file-level repetitions. The fully human corpus contained 120 student-produced drawings, organized as the same 12 categories by the same 2 drawing conditions by 5 human-produced variations. The object categories were apple, block, books, bust, chair, hammer, keys, mug, plant, shoe, skull, and spoon.

The two drawing conditions were C1 abstract and C2 finished. C1 abstract sketches contained reduced visual information, simplified contour, and lower internal detail. C2 finished sketches contained more complete visual information and clearer internal structure. The comparison does not evaluate artistic quality. Each drawing is treated as a stimulus for testing model interpretation.

The model was OpenCLIP ViT-B-32 with the laion2b_s34b_b79k pretrained checkpoint [4]. The label set contained only the twelve object categories. For each label, the study used five text prompts: “a pencil sketch of a {label},” “a hand-drawn sketch of a {label},” “a simple line drawing of a {label},” “a black and white sketch of a {label},” and “a doodle of a {label}.” These prompts were averaged into one text representation for each label.

For each image, the model produced a ranked list of labels. The main outcome was Top-1 accuracy, which means whether the model’s first-choice label matched the true object category. The second outcome was decision margin, calculated as the Top-1 similarity score minus the Top-2 similarity score. A higher margin means the model separated its first choice clearly from its second choice. A low margin means the model was near a decision boundary, even if its first-choice label was correct. This two-part approach follows recent evaluation research showing that model performance should not be reduced to a single accuracy score [5], [8].

Because the synthetic and human datasets had different sizes, the results were reported as rates and averages rather than raw counts alone. The analysis compared accuracy, mean margin, and confusion patterns by source, condition, and object category. Student drawings were anonymized and used only as image stimuli; the study does not evaluate the students who made them.

DATASET STRUCTURE

Table 1. Dataset structure and model-evaluation outcomes

RESULTS

OpenCLIP performed well on both datasets, but the two sources produced different patterns. The fully human dataset produced the highest Top-1 accuracy: 95.0% overall. The synthetic dataset produced lower accuracy: 87.1% overall. Finished drawings improved performance in both sources, but the improvement was larger for the synthetic dataset.

Decision margin showed a different pattern from accuracy. The synthetic dataset produced higher average margins than the human dataset. This means the model often made more decisive choices on synthetic drawings, even when those choices were sometimes wrong. The human drawings were classified correctly more often, but the model’s first and second choices were closer together.

Table 2. Condition-level OpenCLIP performance by source

CATEGORY-LEVEL FINDINGS

The category results show that abstraction did not affect every object in the same way. In the synthetic dataset, most categories were classified correctly in both conditions. The largest changes appeared in plant and shoe. Plant increased from 0% accuracy in the abstract condition to 100% in the finished condition. This suggests that adding internal detail helped the model recover the correct label. Shoe remained at 0% accuracy in both synthetic conditions, showing a persistent category boundary problem.

The human-only dataset showed fewer extreme failures. Apple improved from 60% accuracy in the abstract condition to 100% in the finished condition, and plant improved from 80% to 100%. Hammer stayed at 80% in both conditions, while chair declined from 100% in the abstract condition to 80% in the finished condition. These results suggest that the model’s response depended on both object category and image source.

Table 3. Category-level Top-1 accuracy by source and condition

CONFUSION PATTERNS AND SEMANTIC DRIFT

The confusion patterns were especially important because errors were not random. In the synthetic abstract condition, all ten abstract plant images were classified as spoon, and all ten abstract shoe images were classified as bust. In the synthetic finished condition, most errors disappeared, but shoe remained difficult: nine finished shoe images were classified as skull and one as bust.

The human-only errors were fewer and less severe. In the abstract condition, two apple drawings were classified as block, one hammer drawing was classified as spoon, and one plant drawing was classified as mug. In the finished condition, one chair was classified as hammer and one hammer was classified as spoon. Several of these mistakes had very small decision margins. For example, one abstract human plant was classified as mug, with plant nearly tied as the second choice. This suggests a near-boundary decision rather than a strong wrong commitment.

These patterns support the idea of semantic drift. In this study, semantic drift means that a drawing moves away from its intended label and toward another label in the model’s image-text space, a way of thinking that is consistent with research on multimodal similarity and vector-space alignment [6], [7]. A drift zone appears when the model flips labels, when the margin becomes very small, or when many examples in one category move toward the same wrong label.

DISCUSSION

The main finding is that abstraction can reveal perceptual boundary conditions in a vision-language model. A perceptual boundary condition occurs when the model can no longer place an image clearly in its intended category. In this study, those boundary conditions appeared when important contours, proportions, or internal details were reduced.

The difference between accuracy and margin is also important. Human drawings had higher accuracy, but lower margins. This means OpenCLIP usually recognized them correctly, but it often remained close to choosing another label. Synthetic drawings had lower accuracy but higher margins, meaning the model was sometimes confidently wrong. This pattern shows why accuracy alone is not enough for studying machine perception.

The results also show a difference between human interpretation and machine interpretation. A human viewer can often recognize an object from an incomplete or uneven sketch because people use context, memory, intention, and flexible visual reasoning. OpenCLIP does not use those forms of understanding in the same way. It compares image features and text labels through learned patterns. When a simplified drawing loses the features the model expects, it may drift toward another label.

For art and design research, the study reframes drawing as a method of testing AI systems. The sketches are not decorative examples. They are controlled visual inputs that make the model’s strengths and limits visible. This approach is useful for student research because it combines studio production, dataset design, computer vision, and quantitative analysis without requiring the student to train a new AI model from scratch.

SIGNIFICANCE

This project has significance for AI interpretability, visual culture, and education. For AI interpretability, it shows that model errors can reveal how a system organizes meaning. Stable error routes, such as synthetic plant moving to spoon or synthetic shoe moving to skull, suggest that the model follows learned visual associations rather than guessing randomly.

For visual culture, the results show that AI recognition should be interpreted carefully when systems are used with stylized, hand-drawn, or simplified images. A model may appear accurate on average while still failing in specific categories or near certain visual boundaries. For education, the project models a realistic high-school-level research pathway: define a controlled visual variable, build a small dataset, run a reproducible model evaluation, and interpret the results with both numbers and visual reasoning.

LIMITATIONS

Several limitations should guide interpretation of the findings. First, the dataset was relatively small and included only twelve object categories. The results should not be generalized to all drawings, all forms of abstraction, or all visual recognition tasks. Second, the study used one model architecture and one pretrained checkpoint. A different vision-language model might classify the same drawings differently.

Third, the synthetic and human datasets were not identical in production process or sample size. The synthetic corpus contained 240 file-level inputs, while the human corpus contained 120 drawings. Some synthetic cells also included repeated inputs to complete a balanced file-level structure. For this reason, the study emphasizes rates, means, and patterns rather than raw totals alone.

Fourth, the model could choose only from the twelve available labels. This makes the experiment controlled, but it also means the model could not respond with labels outside the study vocabulary. Fifth, the study did not include a formal human-viewer comparison condition. Future research should compare machine labels with human labels on the same sketches.

CONCLUSION

This study examined how OpenCLIP classified synthetic and fully human sketch datasets across the same twelve object categories and two drawing conditions. The model performed well overall, but its behavior changed by image source, abstraction level, and object category. The human-only dataset produced higher Top-1 accuracy, while the synthetic dataset produced higher decision margins. Finished drawings improved recognition in both datasets, but the strongest findings appeared in categories with specific drift patterns, especially plant and shoe.

The larger conclusion is that abstraction can function as a computational measurement tool. By reducing visual information in controlled ways, a researcher can identify where machine interpretation becomes unstable. This makes abstraction valuable not only as an artistic strategy, but also as a diagnostic method for studying AI perception. The project shows how a high-school researcher can combine drawing, computer vision, and data analysis to study how machines interpret visual meaning.

DATA AVAILABILITY

Data are available upon request.

CONFLICTS OF INTEREST

The author declares no conflict of interest.

FUNDING STATEMENT

No external funding was received for this study.

REFERENCES

[1] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” arXiv:2006.11239, 2020.

[2] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10684–10695, 2022.

[3] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” arXiv:2103.00020, 2021.

[4] M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev, “Reproducible scaling laws for contrastive language-image learning,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818–2829, 2023.

[5] S. Hartwig, D. Engel, L. Sick, H. Kniesel, T. Payer, P. Poonam, M. Gloeckler, A. Baeuerle, and T. Ropinski, “A survey on quality metrics for text-to-image generation,” IEEE Transactions on Visualization and Computer Graphics, vol. 31, no. 10, pp. 9464–9483, 2025.

[6] B. Ihnaini, B. Abuhaija, E. Atta Mills, and M. Mahmuddin, “Semantic similarity on multimodal data: A comprehensive survey with applications,” Journal of King Saud University: Computer and Information Sciences, vol. 36, no. 10, 102263, 2024.

[7] J. Li, Y. Kementchedjhieva, C. Fierro, and A. Soegaard, “Do vision and language models share concepts? A vector space alignment study,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 1232–1249, 2024.

[8] M. Otani, R. Togashi, Y. Sawai, R. Ishigami, Y. Nakashima, E. Rahtu, J. Heikkilae, and S. Satoh, “Toward verifiable and reproducible human evaluation for text-to-image generation,” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14277–14286, 2023.

[9] P. Goodfellow, “The distributed authorship of art in the age of AI,” Arts, vol. 13, no. 5, 149, 2024.

[10] U.S. Copyright Office, Copyright and Artificial Intelligence, Part 2: Copyrightability, January 29, 2025.