OpenSOC @ DEF CON 28 Safe Mode
some of you may remember our last event, Camp COVID. that was the biggest event we had ever run.
UNTIL LAST WEEK: DEF CON 28
the stats speak for themselves... and so does the participant map above :)
8M graylog queries
91K+ scoreboard submissions
150GB+ endpoint telemetry
10K+ osquery queries
20+ hours of content
GLOBAL participation: AGAIN. this was epic.
a remote DEF CON meant people from everywhere were jumping in, and we're stoked that we were able to have that kind of reach and participation.
since DEF CON is a 3 day event for us, we've had luck in the last 2 years running 2 days of a general round, with finals on sundays. in this case, we let the top 20 teams compete in the sunday finals.
we opted to not run overnight, similar to last year. most of the team was toast, and after some discussion, i think going forward we will adhere to the con hours in order to remain at all functional.
we know this was a bit disheartening to players participating in other time zones, but we are a tiny team, and we need sleep too. especially since we had just come straight out of 4 days of Black Hat, and an IR prior to that.
this was heightened even more by the fact that we had discord tickets coming in non stop (to get approved on our network, for help with challenges, for kicking off velociraptor hunts, you name it). i fear that with less sleep, those tickets would've gotten a lot more colorful.
we decided to use discord this year. the con was already using it, BTV was using it, and we already use it internally for training events. SO. naturally, it made sense to move OpenSOC events to it as well.
as i mentioned--there were tickets. like 1500 of them. the YAGPDB bot handled all the things, and we had a process down for all of them coming in. it was just a matter of divide and conquer after that.
easier said than done during some parts of the day(s), but we managed! :)
if you didn't get to make it to this OpenSOC event, you should probably just join our discord anyway and get in on our future events :) just sayin.
similar to past years, we had everyone join our zerotier network in order to participate. this went really well, again, so kudos to zerotier for staying awesome.
one of the best parts of zerotier was that the zerotier ID associated to everyone we allowed in was tied to their discord ID (we'd know the IP either way, but this allowed us to follow up with folks in discord). SO... we could see every search/query/hunt/GET/POST/sneeze across the network tied to each participant.
keep that in mind for future events, nerds.
scenarios & validation
we ran 10 scenarios that made up the 500+ challenges mentioned above.
that's 500 challenges, that needed...
- to be validated after playbooks were run
- to have validation queries solidified and tested
- answers found/double checked/triple checked in the scoreboard
- regex's tested
this team was on FIRE staying ahead of the participants, while continuing to kick off nefarious red team activities and support everyone in the game.
and what's even MORE impressive are the months of research and development before this event that went into creating the scenarios themselves. this team has turned that process into an art.
if you are unfamiliar with how our scenarios work, check out this break down on the oldest and longest running (now retired) scenario, "Urgent IT Update!!!". get all the nerdy details (read: awesome details) from eric.
i say this every time: like every event we run, we inevitably have hiccups along the way. huge live environment, lots of moving parts, hundreds of people beating it up--things happen.
BUT. it makes us unbelievably happy to be able to say...
that aside from the #!?@#$* scoreboard, EVERYTHING HELD UP THROUGHOUT THE ENTIRE EVENT <3 throughout the pounding and querying and F5-ing and abuse, all the things stayed happy 99% of the time.
biggest lesson learned here: despite how much i thought was enough, throw EVEN MORE horsepower at elastic (this weekend it was doubled from Camp COVID) when there are EVEN MORE hundreds of angry nerds creating insane queries and kibana visualizations.
elastic got a little angry for a minute due to some folks not playing nice (like the person tossing splunk queries into elastic (WHY), or all the ridiculous bad regex), so we ended up taking away write access to kibana, and pointed some folks at (even more) lucene query docs. smooth sailing after that.
this was pretty much our biggest pain point besides the headaches of data validation. which, in the end, we'll take it!
unlike last time, we did not have CTFd host our scoreboard--we ran it on a massive instance in AWS. we had a handful of reasons for this, the biggest reason being we needed to be able to put it behind zerotier. another reason: plugins. another reason: the ability to view logs, which isn't an option on the enterprise version. couple of other things.
anyway, no matter how much we tweaked redis settings and allowed more connections, it continued to choke on itself and throw errors when it hadn't even hit 25% of that. or even put a strain on the system. if it isn't obvious, our priorities were on literally everything else.
we are still working on our own internal solution so we no longer need to rely on a 3rd party product, and this event only further solidified that decision.
we'd like to say thank you to folks who jumped in and mentored others throughout the entire event. and to everyone who volunteered.
there were several people who went out of their way to help other participants, whether it was with the challenges or just getting on the network. they spent hours that they could've spent doing literally anything else helping others troubleshoot and threat hunt.
these folks in particular:
there were also several teams comprised of total strangers prior to the event, and now they have a handful of new internet friends.
just a few of the many reasons we love being part of this community is seeing that kind of collaboration come to life.
and finally, huge kudos to our top teams!
- DEEZ TOTS
- ' OR 1=1#
and our top 3 solo players!
- Milagros Coldiron
we hope you all had a great time. we love running OpenSOC--it is really a labor of love for this team, and it takes a lot of it.
we thrive on giving back to a community that has provided us with so much of what we use and rely on, so thank you for helping us continue to grow that. especially during a year as crazy as this one.