Hidradenitis suppurativa (HS) is a long-lasting skin disease with flares of painful boils in areas where skin meets skin, such as the armpits and groins. It affects somewhere between 1 and 40 people out of 1000 worldwide. When trying to measure the effect of a treatment in a clinical trial, so-called ‘outcome measurement instruments’ are used. These are ways of estimating ‘how much’ disease is present. It is obviously important that outcome measurement instruments measure the right thing and do not vary at random. It has previously been shown that outcome measurement instruments used in HS trials lack proof of their reliability. This study therefore aimed to find out how consistently 12 HS-experienced dermatologists would score 24 patients with HS, using nine different instruments. The study showed that one called Hurley staging was an acceptable instrument in terms of consistency of scores. However, Hurley staging is suited only for severity staging e.g. to guide the most appropriate treatment in each stage. For instruments designed to measure changes in disease severity, low agreement between the rating dermatologists was found. Therefore, the results question whether HS physical signs are best measured by traditional instruments based on a physician counting individual HS lesions (affected patches). As a result, other assessment methods of physical signs in HS, using technology such as ultrasound, should also be considered.