A Deep Dive into Psyberg: Stateless vs. Stateful Data Processing | Netflix Technology Blog

Let’s use the registration fact table as an example here. The workflow for this table runs hourly and the main input source is an Iceberg table that stores all raw registration events partitioned by login date, hour, and batch ID.

Here is a YAML snippet outlining the settings in Psyberg’s initialization step:

- job:
id: psyberg_session_init
type: Spark
spark:
app_args:
- --process_name=signup_fact_load
- --src_tables=raw_signups
- --psyberg_session_id=20230914061001
- --psyberg_hwm_table=high_water_mark_table
- --psyberg_session_table=psyberg_session_metadata
- --etl_pattern_id=1

Behind the scenes, Psyberg discovered that the pipeline was configured for stateless mode because etl_pattern_id=1.

Psyberg also uses the provided input to detect Iceberg snapshots that persist after the latest high watermark available in the watermark table.use Summary columns in snapshot metadata [see the Iceberg Metadata section in post 1 for more details]we parse out the partition information of each Iceberg snapshot of the source table.

Psyberg then retains these processing URIs (JSON string arrays containing landing date, hour, and batch ID combinations) determined by the snapshot changes.This information and other calculated metadata are stored in psyberg_session_f table.This stored data can then be used for subsequent operations Load fact table Jobs in workflows to leverage and use for analysis and debugging purposes.

Use stateful data processing when the output depends on a sequence of events on one or more input streams.

Let’s consider an example of building a cancellation fact table that takes as input:

  1. original cancellation event Indicates when the customer account was canceled
  2. Store the incoming fact table Customer requirements Cancel your subscription at the end of your billing cycle

These inputs help derive additional state analysis attributes such as type of churn, i.e. voluntary or involuntary, etc.

The initialization steps for stateful data processing are slightly different from those for stateless data processing. Psyberg offers additional configurations based on pipeline needs. The following is a YAML snippet outlining the configuration of unconfiguring the fact table during Psyberg’s initialization step:

- job:
id: psyberg_session_init
type: Spark
spark:
app_args:
- --process_name=cancel_fact_load
- --src_tables=raw_cancels|processing_ts,cancel_request_fact
- --psyberg_session_id=20230914061501
- --psyberg_hwm_table=high_water_mark_table
- --psyberg_session_table=psyberg_session_metadata
- --etl_pattern_id=2

Behind the scenes, Psyberg discovered that the pipeline was configured in stateful mode because etl_pattern_id It’s 2.

Note the additional details in the src_tables listing that correspond to raw_cancels above.this Process_ts This represents the event processing timestamp, which is different from the regular Iceberg snapshot submission timestamp, that is event_landing_ts As mentioned in Part 1 of this series.

It is important to capture the scope of a batch of merged events from all sources (i.e. raw_cancels and cancel_request_fact), taking into account late events. Changes to the source table snapshot can be tracked using different timestamp fields.Know which timestamp field to use, i.e. event_landing_ts or something similar Process_ts Helps avoid missing events.

Similar to the approach in stateless data processing, Psyberg uses the provided input to parse the partition information for each Iceberg snapshot of the source table.

Source link

from Tech Empire Solutions https://techempiresolutions.com/a-deep-dive-into-psyberg-stateless-vs-stateful-data-processing-netflix-technology-blog/
via https://techempiresolutions.com/

from Tech Empire Solutions https://techempiresolutions.wordpress.com/2024/02/10/a-deep-dive-into-psyberg-stateless-vs-stateful-data-processing-netflix-technology-blog/
via https://techempiresolutions.com/



from Paxton Willson https://paxtowillson.wordpress.com/2024/02/10/a-deep-dive-into-psyberg-stateless-vs-stateful-data-processing-netflix-technology-blog/
via https://techempiresolutions.com/

Comments

Post a Comment

Popular posts from this blog

How to view the canonical name of an AD object

Image
Learn how to view and edit the canonical names of Active Directory objects using the options described in this guide. Option 1: View the canonical name from ADUC 1. Open Active Directory Users and Computers and click view and select Advanced functions . After that, navigate to the AD container where the object is located, right-click on the object and select characteristic . Then click the Objects tab – the specification name appears on this tab. Option 2: Use PowerShell to view the spec name Open PowerShell on the computer that has access to the domain controller. Then, get the DistinguishedName of the object by executing the following command. This directive saves the result in the $distinguishedName variable. $distinguishedName = (Get-ADObject -Filter Name -eq 'Anthony Raj').DistinguishedName In the above command, “Anthony Raj” is the AD display name of the object whose CN I want to get. This is the result of the command. To confirm that it works, I execu...
Read more

Understanding the Laws of Robotics in a World of Artificial Intelligence

We all like the idea of ​​robots. Just look at how many appear on television, movies, books, and even music and social media. Since the advent of humanoid robots in the 1950s, our interest in the benefits they can bring to our lives has grown. But as we know from countless science fiction stories, we are also intrigued by the fear of robots being used for evil rather than for good or to rebel against human overlords. These are questions that scientists and civilians alike are constantly thinking about, especially with the rapid development of the use of artificial intelligence (AI) Robotic Process Automation (RPA) Software Perform routine tasks every day. To guide these discussions, some questions from the world of science fiction have become mainstream, known as the “Three Laws of Robotics.” What is the history of Robot Law? The Three Laws of Robotics were created by science fiction writer Isaac Asimov in 1942 to establish a moral system for the development of robotics technol...
Read more

Our review of the Beats Solo 4 headphones

Image
Beats today announced the Solo 4, a $200 set that looks almost identical to the Solo 3 but has major upgrades on the inside. The Solo 4 don’t have any active noise cancellation at this price point, and according to Billy Steele, who tested the new headphones, they sound a bit thin. However, the sound has been improved overall, and playtime has been bumped up to 50 hours (with USB-C), which is a significant improvement over its predecessor. They are available for purchase now. enjet Beats also surprised us with its new sub-$100 Solo headphones, wireless (non-ANC) headphones with 18 hours of playback. One focus is comfort, and ergonomic acoustic nozzles and vents help improve audio performance and reduce pressure on your ears. The Solo Buds will be available in June for $80. — Matt Smith The most important stories you may have missed Lorelei and Laser Eyes Preview: This Could Be My GOTY What to expect from Apple’s Let Loose iPad event The stunning and customizable Arc Bro...
Read more