Easy methods to Detect Uncommon Exercise in Okta Logs Utilizing the Databricks Lakehouse

With the latest social media reviews of an Okta incident by a 3rd celebration contractor, safety groups ran to their logs and requested distributors like Databricks for detection and analytics recommendation. Previous to being notified by Okta that we weren’t among the many probably impacted prospects, we used the Databricks Lakehouse for our personal investigation. We wish to present how we carried out that investigation and share each insights and technical particulars. We can even present notebooks that you could import into your Databricks deployment or our neighborhood version to ingest your Okta logs in order that by the tip of this weblog, you may carry out the identical evaluation on your firm.

Background

Okta is a market-leading cloud-based identification platform used for Single Signal-on (SSO) authentication and authorization, multi-factor authentication, and person administration companies with their prospects’ enterprises or enterprise purposes.

In January 2022 hackers gained entry to an endpoint (person system) owned and operated by a third-party group offering assist companies to Okta prospects. These actors have been probably capable of carry out actions as in the event that they have been the worker assigned to that endpoint. Like most organizations, Databricks instantly launched an investigation into the incident, analyzing a number of years of Okta information we’ve saved in our Lakehouse. We constructed our personal queries, however we additionally discovered great worth within the posts and tweets from others within the business.

Our favourite business weblog submit was from Cloudflare. Two statements specifically resonated with our safety crew:

“Regardless that logs can be found within the Okta console, we additionally retailer them in our personal programs. This provides an additional layer of safety as we’re capable of retailer logs longer than what is accessible within the Okta console. That additionally ensures {that a} compromise within the Okta platform can’t alter proof we’ve already collected and saved.”

Due to this method, they have been capable of “search the Okta System logs for any indicators of compromise (password modifications, {hardware} token modifications, and so forth.). Cloudflare reads the system Okta logs each 5 minutes and shops these in our SIEM in order that if we have been to expertise an incident akin to this one, we will look again additional than the 90 days supplied within the Okta dashboard.”

Determine 1. Quote from business submit from Cloudflare

Within the wake of the incident, lots of our prospects reached out to us asking if Databricks will help them ingest and analyze their Okta System Logs, and the reply is a powerful YES! The Databricks Lakehouse Platform enables you to retailer, course of and analyze your information at multi-petabyte scale, permitting for for much longer retention and lookback intervals and superior risk detection with information science and machine studying. What’s extra, you may even question them through your SIEM device, offering a 360 view of your safety occasions.

On this weblog submit, we are going to display how you can combine Okta System Logs together with your Databricks Lakehouse Platform, and acquire and monitor them. This integration permits your safety groups far larger visibility into the authentication and authorization behaviors of your purposes and end-users, and allows you to search for particular occasions tied to the latest Okta compromise.

In case your aim is to rapidly get began, you may skip studying the remainder of the weblog and use these notebooks in your individual Databricks deployment, referring to the feedback in every part of the pocket book if you happen to get caught.

Please learn on for a technical clarification of the mixing and the evaluation supplied within the notebooks.

About Okta System Logs

The Okta System Log information system occasions which can be associated to your group in an effort to present an audit path that can be utilized to grasp platform exercise and diagnose issues. The Okta System Log API offers close to real-time, read-only entry to your group’s system log. These logs present essential insights into person exercise, categorized by Okta occasion kind. Every occasion kind represents a particular exercise (e.g., login try, password reset, creating a brand new person). You possibly can search on occasion varieties and correlate exercise with different Okta log attributes such because the occasion end result (e.g., SUCCESS or FAILURE), IP handle, person title, browser kind, and geographic location.

There are a lot of strategies to ingest Okta System Log occasions into different programs, however we’re utilizing the System Log API to retrieve the newest System Log occasions.

Lakehouse structure for Okta System Logs

Databricks Lakehouse is an open structure that mixes the most effective parts of information lakes and information warehouses. We suggest the next lakehouse structure for cybersecurity workloads, akin to Okta System Log evaluation:

• Step 1: The Okta System Log information system occasions which can be associated to your group in an effort to present an audit path that can be utilized to grasp platform exercise and to diagnose issues.
• Step 2: The Okta System Log API offers close to real-time, read-only entry to your group’s system log.
• Step 3: You should use the pocket book supplied to hook up with Okta System Log API and ingest information into Databricks Delta routinely at brief intervals (optionally, schedule it as a Databricks job).
• Step 4: On the finish of this weblog, and with the notebooks supplied, you’ll be prepared to make use of the info for evaluation.



Within the subsequent sections, we’ll stroll by how one can ingest Okta log attributes to observe exercise throughout your purposes.

Ingesting Okta System Logs into Databricks Delta

In case you are following alongside at work or house (or as of late most frequently each) with this pocket book, we will likely be utilizing Delta Lake batch functionality to ingest the info utilizing Okta System Log API to a Delta desk by fetching the checklist of ordered log occasions out of your Okta group’s system log. We will likely be utilizing the bounded requests kind (bounded requests are for conditions when you realize the particular timeframe of logs you wish to retrieve).

For a request to be a bounded request, it should meet the next request parameter standards:

• since should be specified.
• till should be specified.

Bounded requests to the /api/v1/logs API have the next semantics:

• The returned occasions are time filtered by their related revealed area (in contrast to Polling Requests).
• The returned occasions are assured to be so as in response to the revealed area.
• They’ve a finite variety of pages. That’s, the final web page doesn’t include a subsequent hyperlink relation header.
• Not all occasions for the desired time vary could also be current — occasions could also be delayed. Such delays are uncommon however attainable.

For efficiency, we’re going to use an adaptive watermark method: i.e question for the final 72 hours to search out the newest ingest time; if we will’t discover one thing inside that timeframe, then we requery the entire desk to search out the newest ingest time. That is higher than querying the entire desk each time.

d = datetime.at the moment() - timedelta(days=3)
beginDate = d.strftime("%Y-%m-%d")

watermark = sql("SELECT coalesce(max(revealed)) FROM okta_demo.okta_system_logs WHERE date >= '{0}'".format(beginDate)).first()[0]
if not watermark:
  watermark = sql("SELECT coalesce(max(revealed)) FROM okta_demo.okta_system_logs").first()[0]

Determine 3. Cmd 3 of “2.Okta_Ingest_Logs” pocket book

We’ll assemble an API request as under through the use of the Okta API Token, and break aside the information into particular person JSON rows

headers = {'Authorization': 'SSWS ' + TOKEN}
url = URL_BASE + "api/v1/logs?restrict=" + str(LIMIT) + "&sortOrder=ASCENDING&since=" + SINCE
r = requests.get(url, headers=headers)
jsons = []
  jsons.lengthen([json.dumps(x) for x in r.json()])

Determine 4. Cmd 4 of “2.Okta_Ingest_Logs” pocket book

Rework the JSON rows right into a dataframe

df = (
    sc.parallelize([Row(recordJson=x) for x in jsons]).toDF()
    .withColumn("report", f.from_json(f.col("recordJson"), okta_schema))
    .withColumn("date", f.col("report.revealed").forged("date"))
    .choose(
      "date",
      "report.*",
 "recordJson",
    )
  )

Determine 5. Cmd 4 of “2.Okta_Ingest_Logs” pocket book

Persist the information into delta desk

df.write 
   .choice("mergeSchema", "true")
   .format('delta') 
   .mode('append') 
   .partitionBy("date") 
   .save(STORAGE_PATH)

Determine 6. Cmd 4 of “2.Okta_Ingest_Logs” pocket book

As proven above the Okta information assortment is lower than 50 traces of code and you’ll run that code routinely at brief intervals by scheduling it as a Databricks job.

Your Okta system logs are actually in Databricks. Let’s do some evaluation!

Analyzing Okta System Logs

For our evaluation, we will likely be referring to the “System Log queries for tried account takeover” data content material that the good of us at Okta revealed together with their docs.

Okta Impersonation Session Search

Reportedly, it seems an attacker compromised the endpoint for a third-party assist worker with elevated permissions (akin to the flexibility to drive a password reset on an Okta buyer account). Buyer safety groups could wish to begin searching for a couple of occasions within the logs for any indications of compromise to their Okta tenant.

Allow us to begin with administrator exercise. This question searches for impersonation occasions reportedly used within the LAPSUS$ exercise. Person.session.impersonation are uncommon occasions, usually triggered when an Okta assist individual requests admin entry for troubleshooting, so that you in all probability received’t see many.

SELECT
  eventType,
  depend(eventType)
from
  okta_demo.okta_system_logs
the place
  date >= date('2021-12-01')
  and eventType in (
    "person.session.impersonation.provoke",
    "person.session.impersonation.grant",
    "person.session.impersonation.lengthen",
 "person.session.impersonation.finish",
    "person.session.impersonation.revoke"
  )
group by eventType

Determine 7. Cmd 4 of “3.Okta_Analytics” pocket book

Within the outcomes, if you happen to see a person.session.impersonation.provoke occasion (triggered when a assist employees impersonates an admin) however no person.session.impersonation.grant occasion (triggered when an admin grants entry to assist), that’s trigger for a priority! We supplied an in depth question within the notebooks that detects “impersonation initiations” which can be lacking a corresponding “impersonation grant” or “impersonation finish”. You possibly can evaluation person.session.impersonation occasions and correlate that with reputable opened Okta assist tickets to find out if these are anomalous. See Okta API occasion varieties for documentation and Cloudflare’s investigation of the January 2022 Okta compromise for an actual world state of affairs.

Okta Latest Worker who had Reset their Password or Modified their MFA

Now, let’s search for any worker account who had their password reset or modified their multi issue authentication (MFA) in any approach since December 1. Occasion varieties inside Okta that assist with this search are: person.account.reset_password, person.mfa.issue.replace, system.mfa.issue.deactivate, person.mfa.attempt_bypass, or person.mfa.issue.reset_all (you may look into Okta docs to seize extra occasions to develop your evaluation as wanted). We’re searching for an “actor.alternateId” of system@okta.com that seems when the Okta assist group initiates a password reset. Be aware that though we’re additionally searching for the “Replace Password” occasion under, Okta’s assist representatives do not need the potential of updating passwords – they’ll solely reset them.

SELECT
  actor.alternateId,
  *
from
  okta_demo.okta_system_logs
the place
  (
    (
      eventType = "person.account.update_password"
      and actor.alternateId = "system@okta.com"
    )
    or (
      eventType = "person.account.reset_password"
      and actor.alternateId = "system@okta.com"
    )
    or eventType = "person.mfa.issue.replace"
    or eventType = "system.mfa.issue.deactivate"
    or eventType = "person.mfa.attempt_bypass"
    or eventType = "person.mfa.issue.reset_all"
  )
  and date >= date('2021-12-01') 
 

Determine 8. Cmd 8 of “3.Okta_Analytics” pocket book

If you happen to see outcomes from this question, you will have Okta person accounts which require additional investigation – particularly if they’re privileged or delicate customers.

MFA Fatigue Assaults

Multi issue authentication (MFA) is among the many simplest safety controls at stopping account takeovers, however it isn’t infallible. It was reportedly abused through the Solarwinds compromise and by LAPSUS$. This method known as an MFA fatigue assault or MFA immediate bombing. With it, an adversary makes use of beforehand stolen usernames and passwords to login into an account protected by push MFA and triggers many push notifications to the sufferer (sometimes to their telephone) till they tire of the alerts and approve a request. Easy! How would we detect these assaults? We took inspiration from this weblog submit by James Brodsky at Okta to handle simply that.

SELECT
    authenticationContext.externalSessionId externalSessionId, actor.alternateId, min(revealed) as firstTime, max(revealed) as lastTime,  
    depend(eventType) FILTER (the place eventType="system.push.send_factor_verify_push") pushes,
    depend(legacyEventType) FILTER (the place legacyEventType="core.person.issue.attempt_success") as successes, 
    depend(legacyEventType) FILTER (the place legacyEventType="core.person.issue.attempt_fail") as failures,  
    unix_timestamp(max(revealed)) -  unix_timestamp(min(revealed)) as elapsetime 
from
  okta_demo.okta_system_logs
the place
  eventType = "system.push.send_factor_verify_push" 
   OR 
  ((legacyEventType = "core.person.issue.attempt_success") AND (debugContext.debugData like "%OKTA_VERIFY_PUSH%"))
  OR 
  ((legacyEventType = "core.person.issue.attempt_fail") AND (debugContext.debugData like "%OKTA_VERIFY_PUSH%"))
  
group by authenticationContext.externalSessionId, actor.alternateId
having elapsetime 0 AND pushes>=3 and failures >= 1

Determine 9. Cmd 11 of “3.Okta_Analytics” pocket book

Here’s what the above question seems for: First, it reads in MFA push notification occasions and their matching success or failure occasions, per distinctive session ID and person. It then calculates the time elapsed through the login interval (restricted to 10 minutes), and calculates the variety of push notifications despatched, together with the variety of push notifications responded to affirmatively and negatively. Then it makes easy selections primarily based on the mixtures of outcomes returned. If greater than three pushes are seen, and a single profitable notification is seen, then this might be one thing value extra investigation.

Suggestions

In case you are an Okta buyer, we suggest reaching out to your account crew for additional info and steering.

We additionally recommend the next actions:

• Allow and strengthen MFA implementation for all person accounts.
• Ingest and retailer Okta logs in your Databricks Lakehouse.
• Examine and reply:
  • Repeatedly monitor uncommon support-initiated occasions, akin to Okta impersonation periods, utilizing Databricks jobs.
  • Monitor for suspicious password resets and MFA-related occasions.

Conclusion

On this weblog submit you realized how straightforward it’s to ingest Okta system logs into your Databricks Lakehouse. You additionally noticed a few evaluation examples to hunt for indicators of compromise inside your Okta occasions. Keep tuned for extra weblog posts that construct much more worth on this use case by making use of ML and utilizing Databricks SQL.

We invite you to log in to your individual Databricks account or in Databricks Group Version and run these notebooks. Please consult with the docs for detailed directions on importing the pocket book to run.

We sit up for your questions and solutions. You possibly can attain us at: cybersecurity@databricks.com. Additionally in case you are interested in how Databricks approaches safety, please evaluation our Safety & Belief Heart.

Acknowledgments

Thanks to all the employees throughout the business, at Okta, and at Databricks, who’ve been working to maintain everybody safe.