DEFCON 25 BRIEFING
This year was a milestone year in that we broke - no, we smashed the 2 Petabyte barrier! We were able to step up our communication this year and the results were clear. We took in over 500 drives for duplication - slightly more than we could handle give the time window and with our current equipment.
All the details are below.
General Intake
The team did a phenomenal job of accepting drives for copying this year! We pulled in 504 drives and (again) wiped out all of the available 6TB drives in Vegas this year. This is the breakdown of what we took in essentially by noon on Friday:
1600-2000 on Thursday:
- infocon.org archive requests: 136
- Hash Tables 1 of 2: 85
- Hash Tables 2 of 2: 77
1000-2000 on Friday:
- infocon.org archive requests: 92
- Hash Tables 1 of 2: 58
- Hash Tables 2 of 2: 56
DDV Drive content requests
Due to the limits of the flow of electrons and time available,
not all of these drives were able to be duplicated. More on that below.
Intake by Manufacturer
This year we were better prepared and took an extra few seconds per drive to identify some details of drives we took in. We weren't able to capture 100% of the drives but we got close. Here's the high-level breakdown:
Drives by Manufacturer Manufacturer Qty Fails Pct* -------------------------- --- ----- --- Seagate (mostly Ironwolf) 150 9 6% WD (mostly Blue) 147 2 1% Toshiba (mostly X300) 109 1 1% HGST (mostly Deskstar) 85 15 18% Mediamax (huh-what?) 7 3 43% -------------------------- --- ----- --- Total 498 30 6%
* Since not all of the drives taken in were processed, the actual fail percentages are slightly higher than what would have been captured if we processed the remaining 78 drives. Also - see below for more on percentages with small statistical samples.
The Mediamax was quite a surprise. With a little research, we identified these as refurbished WD drives. Yes, they failed a lot. No, I would never use one for production.
In addition to the manufacturer, we also tracked the model of the drives we took in - for the most part. Even though we specified 6TB drives some people brought in larger drives as well - we captured that too. The following breakdown shows us that information:
BY MODEL/TYPE: 6TB drives Make/Model Qty Fails Pct ----------------- --- ----- --- Seagate Ironwolf 79 2 3% Seagate Barracuda 24 2 8% Seagate Desktop 20 3 15% Seagate Skyhawk 7 0 0% WD Blue 42 0 0% WD Black 20 2 10% WD Red 17 0 0% WD Gold 38 0 0% WD Green 6 0 0% WD Yellow 5 0 0% WD Purple 2 0 0% Toshiba 109 1 1% HGST Deskstar/Pro 84 15 18% Mediamax 7 3 43% 8TB Drives (can’t follow directions) Make/Model Qty Fails Pct ----------------- --- ----- --- Seagate Barracuda 14 0 0% WD Red 13 0 0% WD Black 4 0 0% HGST Deskstar 1 0 0% 10TB (just trying to show off) Make/Model Qty Fails Pct ----------------- --- ----- --- Seagate Barracuda 5 0 0% Seagate Ironwolf 1 0 0%
Intake by Manufacturer
Drive failures
Do keep in mind that any failures with a small number of drives (i.e. 50 or less) can easily show a disproportionately higher failure rate than should be expected in real world conditions. You can follow the math here if you want to know more but the HGST drive failure percentage, for example, is calculated with 95% certainty to a +/-8.2% margin of error while the WD black drive failure percentage has a +/-186% margin of error - yes - that 10% failure is basically a useless number with such a small sample.
Making it happen
Obviously, Defcon only runs for a defined time. To that end, we had less than 68 hours to accept, tag, copy, and return everyone's drives (1600 Thursday to noon on Sunday). In addition, we only had our 14 original source drives and the 15 duplicators available to us (90 total drive copy slots). Each 6TB copy cycle takes about 15 hours to complete. Given those constraints, we buckled down to process as many of the 500 drives as we could.
Since our initial projections put us at 6 x 15 hour copy cycles for a total of 90 hours to process all the drives, standard duplicating wouldn't cut it like it did last year (only 335 drives in 2016). We had to find a better solution.
We looked closely at the infocon.org drives - more than 50% of the duplicating requests each year. The Dark Tangent had just re-encoded all of that content with the h.265 codec (thank you, DT) so the content actually shrunk in size this year - despite adding another year of talks and podcasts! This was our savior. We set the large duplicators to doing partition copies (3.1TB) which only took 7.5 - 8.5 hours! and had the remaining 50 slots cranking out the table drives. The duping broke down to look something like this:
- Infocon - 210 drives / 35 slots is 6 cycles at 8 hours each = 56 hours
- Tables - 150 drives / 50 slots is 4 cycles at 15 hours each = 60 hours
- Infocon - 19 drives / 5 slots for 4 cycle at 15 hours = 60 hours
- Tables - 41 drives / 41 slots for 1 cycle at 15 hours = 15 hours
With a bit of parallel processing, and never ending visits to the village at all hours to keep things moving quickly, we cranked them out and - even though we hit several failures, we were able to return 420 completed drives!
- Infocon.org archive: 229
- Hash Tables 1 of 2: 95
- Hash Tables 2 of 2: 96
- Grand Total: 420
One side note: all but one drive was returned at the conference and even that drive is now in the hands of its original owner.
Wrap up
In the end we were able to send over 2.5 Petabytes out the door to our community! A spectacular end result! We didn't get to some of the Hash Table duplications complete (1 of 2 and 2 of 2) so we're looking for those of you who did get them to help us out by offering to seed them. If you have the bandwidth, please let us know! You can reach out to us on the Defcon Forums and via Twitter. We're planning to have even more duplicators next year and possibly break the 3 PB barrier!
Big thanks to DT, Nikita, Charel, Zant, the QM Stores, the Goons, and dc408 for helping make this happen so seamlessly this year! See you in 2018!