Skip to work
← KINO LABKO

02 / DATA

Brand Website Crawler + Analysis

Crawls a brand's website into structured product data, then reads trends out of it

ANALYSIS

BRAND SITE — PRODUCT DETAIL

01All Clothing — T-Shirts & Tanks — PRODUCT A

02WEBSITE PAGE

03PRODUCT A$00Double-knit cotton crewneck, mid-weight

04ESSENTIAL 11 + LIMITED 3041 COLORS · #000000

05DETAILS — CONTENTSContent: 100% cotton / Care: Machine wash / Imported

01CATEGORY
The middle step of the breadcrumb is the category. It sits in the same place on every page, so Top / Dress / Pant come across untouched.
All Clothing — T-Shirts & Tanks — PRODUCT A
02WEBSITE PAGE
The main shot's address is kept so it opens straight from the spreadsheet. Numbers alone don't tell you what the garment is.
Website page → image URL
03NAME · PRICE · DESCRIPTION
List price and sale price are kept apart, and the one-line description is what the woven / knit call is later argued from.
PRODUCT A / $00 / "Double-knit cotton crewneck…"
04COLOR COUNT · STYLE NO.
Swatches are counted into a colour count. Essential and limited edition are listed apart but added together. The number top right is kept as the style identifier.
Essential 11 + Limited 30 = 41 colors / #000000
05CONTENTS
This area only opens when the details tab is pressed, so the tab is opened first and the text read after.
Content: 100% cotton / Care: Machine wash / Imported
  1. PUBLIC PRODUCT PAGE
  2. →COLLECT
  3. →CATEGORY / IMAGE / PRICE / COLOR / CONTENTS
  4. →NORMALIZE
  5. →WOVEN / KNIT / MATERIAL GROUP
  6. →ANALYSIS
  7. →TREND & INSIGHT

PROBLEM

From a vendor's seat there was no way to see which materials a brand actually uses, in what proportion, or how a line is put together. Internal data is not something a vendor receives, and while the public sales pages hold the information, it sits scattered across them in a form nothing can be done with.

THE OLD PROCESS

  1. 01Open one product page
  2. 02Find the values by eye
  3. 03Retype them into a spreadsheet
  4. 04Notation differs page to page, so nothing aggregates
  5. 05Fall back on instinct
  6. 06About a minute per product — roughly 12 hours for 721

WHAT I BUILT

  • Five fields — category, product name, price, colourway count, fibre composition — defined at fixed positions and extracted
  • Nothing is interpreted at collection time; the raw text is kept, so a change of criteria never means collecting again
  • Compositions that are identical but written differently are unified — bracketed notes, lining notation and comma placement normalized
  • Woven and knit, which the page never states, are determined by reading the description, category and composition together
  • Classified into 21 material groups by dominant fibre, one sheet per group

RESULT

  • 721 products · 14 fixed fields · 21 sheets by material group
  • Woven 425 / Knit 290 / 6 undetermined
  • Material mix, category share and season composition became numbers

Based on public product pages · automatic classification may carry some error