>_ aaditya
GitHub

ML

Solving captchas with scikit-learn: a 99.7% pipeline

· · MLPythonscikit-learn

A few years ago I needed to read text captchas automatically for a scraping job. The captchas were simple by modern standards: fixed-length strings of letters and digits on a noisy background, all rendered in the same font. I did not want to call a paid solving service for something that regular, so I built a small pipeline with Python, Pillow and scikit-learn, and wrapped it in a PHP API so the rest of the project could call it over HTTP. On that one captcha style it reached 99.7% accuracy.

That number needs a caveat up front. It was measured on a single captcha style, and the test set was small and not formally held out the way I would do it today. Treat it as "it worked well enough for the job", not as a benchmark. What I think is more useful is the pipeline itself, because the same shape applies to most old-school text captchas.

What I built

The repo is captcha_ai. It has three parts:

I used matplotlib while developing to look at intermediate images, which turned out to be the most important tool of the whole project. You cannot debug a vision pipeline without looking at what each stage produces.

How it works

The pipeline is a fixed sequence: trim the image, split it into individual letters, rotate each letter upright, remove padding and resize to a fixed box, flatten the pixels into a vector, and classify. Every step is deliberately simple. The whole point is that a classic classifier on raw pixels is enough once the letters are isolated and normalised.

1. Trim

The captcha images had a margin of background around the text. I converted to greyscale, thresholded to black and white, and cropped to the bounding box of the dark pixels. This removes the variable margin so that later steps see only the text.

2. Split letters

Because the captcha length was fixed and the font was monospaced-ish, I could split on vertical projection: sum the dark pixels in each column, and the gaps between letters show up as columns with near-zero ink. Where letters touched, I fell back to splitting the trimmed width into equal slices. This is the step most likely to break on a different captcha style.

split.py
import numpy as np

def split_letters(binary, expected):
    # binary: 2D array, 1 = ink, 0 = background
    cols = binary.sum(axis=0)
    inked = cols > 0
    runs, start = [], None
    for x, on in enumerate(inked):
        if on and start is None:
            start = x
        elif not on and start is not None:
            runs.append((start, x)); start = None
    if len(runs) != expected:
        # letters touch: fall back to equal slices
        w = binary.shape[1] // expected
        runs = [(i * w, (i + 1) * w) for i in range(expected)]
    return [binary[:, a:b] for a, b in runs]

3. Rotate, remove padding, resize

The generator skewed each letter by a small random angle. I tried a handful of candidate rotations for each letter crop and kept the one with the narrowest bounding box, which is a cheap way to find "upright". Then I cropped the padding again and resized every letter to the same small square. After this step a letter is just a fixed-size grid of 0s and 1s, so the classifier never has to learn position or scale.

4. Pixels to features

Flatten the square into a one-dimensional vector. No edge detection, no histograms, no learned features. With normalised letters, raw pixels carry all the information a classifier needs, and the vector is small enough that training takes seconds.

5. Train and save

Each training captcha came with its label, so I had one labelled vector per letter. I tried a few scikit-learn classifiers with the same interface and kept the one that scored best on a held-back slice. The model is pickled to model.pkl and loaded once by the predictor.

train.py
import pickle
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier

X, y = load_letter_vectors("data/labelled/")
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.2, random_state=1)

clf = KNeighborsClassifier(n_neighbors=3)
clf.fit(X_tr, y_tr)
print("letter accuracy:", clf.score(X_te, y_te))

with open("model.pkl", "wb") as f:
    pickle.dump(clf, f)

One detail worth noting: letter accuracy and captcha accuracy are different numbers. A captcha is only solved if every letter is right, so a 99% per-letter model on a six-letter captcha solves roughly 94% of captchas. When I quote 99.7% I am quoting the figure I recorded for whole captchas on that one style, on a small test set, so the per-letter rate was higher still. On a harder style I would expect both numbers to fall sharply.

6. Serve it

The consumer of this model was a PHP project, so the simplest integration was a PHP endpoint that saves the uploaded image to a temp file, shells out to the Python predictor, and returns JSON. It is not elegant, and a Flask service would be the better design, but it meant zero changes on the PHP side beyond one HTTP call.

api/solve.php
<?php
$tmp = tempnam(sys_get_temp_dir(), 'cap');
move_uploaded_file($_FILES['image']['tmp_name'], $tmp);

$out = shell_exec('python3 predict.py ' . escapeshellarg($tmp));
unlink($tmp);

header('Content-Type: application/json');
echo json_encode(['text' => trim($out)]);

What I learned

The code is on GitHub. It is old and small, but the pipeline shape is still what I would start from for any fixed-style text captcha.

Source on GitHub >