A time series is a sequence of data points collected and ordered chronologically over time. It is characterized by its indexing in time, distinguishing it from other types of datasets.
Time series metrics represent data tracked at regular intervals, such as inventory sold in a store from one day to the next. In investing, time series tracks the movement of chosen data points, like a security's price, over a specified time with regularly recorded intervals.
Statistically, time series data is analyzed in two primary ways: to draw conclusions about the impact of one or more variables on a specific variable of interest across time or to predict future trends.
Components of Time Series:
The components of a time series include trends, seasonal variations, cyclic variations, and random or irregular movements. These elements collectively contribute to the pattern observed in the time series data.
Key Properties of Time Series Data:
- Sample Tracking Over Time: Time series data tracks a sample over successive periods, providing insights into the evolution of variables.
- Influence of Factors: Time series allows observing factors influencing specific variables over time, contributing to a comprehensive understanding of patterns.
- Use in Analysis: Time series analysis is valuable for examining how assets, securities, or economic variables change over time. It aids in both fundamental and technical analysis.
- Forecasting Methods: Forecasting methods using time series data predict future values, contributing to decision-making processes in various fields.
In essence, time series data captures the evolution of variables over time, and its analysis, including forecasting methods, is instrumental in understanding and predicting patterns in diverse domains such as finance, economics, and inventory management.
Types of outliers:
Anomalies in time series data, called outliers or novelties, represent data points or patterns that deviate significantly from the expected or normal behavior. These anomalies can arise for various reasons, including errors, unusual events, or changes in the underlying process being monitored. These can be categorized into two broad groups -
Unwanted Data:
Unwanted data in time series refers to unintentional irregularities or disturbances that often stem from technical issues or errors in the data collection process. These anomalies are typically considered noise and can result from various sources, such as measurement errors, sensor malfunctions, or inaccuracies during data entry. Unwanted data is characterized by its randomness and lack of meaningful information. If not addressed appropriately, these anomalies can introduce time series analysis and interpretation inaccuracies. Detection and removal of unwanted data anomalies are essential, and statistical techniques and filtering methods are commonly employed to ensure the reliability of the time series data for further analysis.
Event of Interest:
In contrast, events of interest involve purposeful deviations in the time series that are actively sought or considered highly meaningful. These anomalies represent occurrences that are specifically targeted for detection and analysis. Events of interest anomalies may be triggered by planned events, special occurrences, or phenomena that hold particular importance in the time series context. Unlike unwanted data anomalies, events of interest anomalies are often crucial for decision-making or gaining insights into the underlying processes being monitored. Detection methods for these anomalies are tailored to identify specific patterns or deviations deemed necessary, making them the focal point of the analysis. The intentional pursuit of events of interest anomalies is driven by the desire to uncover and understand specific occurrences within the time series data that have particular significance or relevance to the analysis or application at hand.
Outliers in Time Series data:
1. Point Outlier in Time Series Data:
A point outlier in time series data refers to an individual data point significantly deviating from the expected or normal pattern within the series. This deviation can manifest as an unusually high or low value compared to the surrounding data points. Point outliers are often characterized by their isolated nature, representing a singular instance of divergence from the overall trend.
- Characteristics:
- Isolation: Point outliers stand alone in their deviation from the expected pattern, isolated from neighboring data points.
- Magnitude: These outliers exhibit a substantial magnitude of difference compared to the surrounding values.
- Single Occurrence: Point outliers are typically singular occurrences, influencing only one specific data point within the time series.
- Detection Methods:
- Z-Score: Identify points with z-scores beyond a certain threshold.
- Grubbs' Test: Apply statistical tests to detect outliers in univariate datasets.
- Visual Inspection: Plotting the time series allows for visual identification of individual points deviating significantly from the overall trend.
2. Subsequent Outlier in Time Series Data:
A subsequent outlier in time series data refers to a pattern of anomalous behavior that extends beyond a single data point. Unlike point outliers, subsequent outliers involve a sequence of consecutively deviant data points, indicating a sustained deviation from the expected trend.
- Characteristics:
- Sequential Nature: Subsequent outliers manifest as a sequence of data points that collectively deviate from the expected pattern.
- Duration: These outliers may persist for an extended period, influencing a series of consecutive time points.
- Aggregate Effect: The impact of subsequent outliers is cumulative, affecting the overall trend of the time series.
- Detection Methods:
- Moving Average or Exponential Smoothing: Identify deviations from smoothed trends that persist over multiple time points.
- Change Point Detection Algorithms: Algorithms designed to detect shifts or changes in the underlying distribution of the time series.
- Cluster Analysis: Identify clusters or patterns of anomalous behavior over consecutive time points.
Identifying and Treating Point Outliers:
Identifying and treating point outliers in time series data is crucial for ensuring the accuracy and reliability of analyses. Here are a few methods to get started with.
1. Z-Score Method:
- Method:
- Calculate the z-score for each data point using the mean and standard deviation of the time series.
- Identify points with z-scores beyond a certain threshold (e.g., 2 or 3) as potential outliers.
- Treatment:
- Once identified, assess the context and nature of the outlier.
- Consider imputing the outlier with a more typical value or removing it, depending on the impact on the analysis and the underlying reasons for the outlier.
2. Tukey's Fences (Interquartile Range Method):
- Method:
- Calculate the interquartile range (IQR) of the time series.
- Define lower and upper fences as Q1 - k * IQR and Q3 + k * IQR, respectively (typically, k is set to 1.5).
- Identify points outside the fences as potential outliers.
- Treatment:
- Similar to the Z-Score method, assess the nature and context of the outlier.
- Consider replacing or removing the outlier based on the impact and the goals of the analysis.
3. Moving Average or Exponential Smoothing:
- Method:
- Smooth the time series using a moving average or exponential smoothing technique.
- Compare each data point with its smoothed value to identify deviations.
- Points significantly deviating from the smoothed trend may be considered outliers.
- Treatment:
- Analyze the outliers in the context of the smoothed trend.
- Adjust the time series by imputing or removing outliers, considering the impact on the overall analysis.
Example Use Case - using Z-Score Method:
Consider a financial time series tracking daily stock prices. An unexpected surge in stock price might be identified using the Z-Score method:
- Scenario:
- The stock price experienced a sudden spike that seemed unusual compared to historical data.
- Z-Score Analysis:
- Calculate the z-score for each daily stock price.
- Identify days with z-scores exceeding a threshold (e.g., 2 or 3).
- Treatment:
- Investigate the context of the outlier (e.g., news, events).
- If the surge is deemed anomalous and not justified by known factors, consider adjusting or removing the outlier in the analysis.
In this use case, the Z-Score method helps pinpoint days where stock prices deviate significantly from the expected behavior, enabling a more informed decision on how to treat these point outliers in the financial time series.
Identifying and Treating Subsequent Outliers:
Identifying and treating subsequent outliers in time series data involves detecting patterns of sustained abnormal behavior. Here are a few that can be started with.
1. Moving Average or Exponential Smoothing:
- Method:
- Smooth the time series using a moving average or exponential smoothing technique.
- Analyze deviations of each data point from the smoothed trend over consecutive time points.
- Identify periods where the deviation persists as subsequent outliers.
- Treatment:
- Examine the duration and impact of the subsequent outliers.
- Adjust the time series by imputing or removing the outliers based on their influence on the overall trend.
2. Change Point Detection Algorithms:
- Method:
- Utilize algorithms designed to detect shifts or changes in the underlying distribution of the time series.
- Identify points where the distribution changes are significant, which indicates the presence of subsequent outliers.
- Treatment:
- Assess the context of detected change points and their impact on the time series.
- Modify the time series by addressing or mitigating the effects of the subsequent outliers, depending on their significance.
3. Cluster Analysis:
- Method:
- Apply cluster analysis techniques to identify clusters or patterns of anomalous behavior over consecutive time points.
- Identify clusters that represent sustained deviations from the expected pattern.
- Treatment:
- Examine the characteristics and duration of the identified clusters.
- Adjust the time series by addressing or removing the clusters of subsequent outliers based on their impact on the analysis.
Example Use Case - using Moving Average or Exponential Smoothing:
Consider energy consumption time series data for a smart building. An extended period of unusually high energy consumption might be identified through moving average or exponential smoothing:
- Scenario:
- Energy consumption shows a sustained increase over several consecutive days.
- Z-Score Analysis:
- Smooth the time series using a moving average or exponential smoothing.
- Identify periods where the actual energy consumption consistently deviates from the smoothed trend.
- Treatment:
- Investigate the reasons behind the prolonged increase in energy consumption.
- Modify the time series by addressing or mitigating the effects of the subsequent outliers, ensuring a more accurate representation of the building's energy usage over time.
In this use case, the moving average or exponential smoothing method helps identify and address a sustained increase in energy consumption, allowing for a more informed treatment of the subsequent outliers in the smart building's time series data.
Identifying and Treating Real-life Data Outliers:
Data can have any point, subsequent, or combined outliers in real life. Relying on a single methodology may often produce ineffective and unreliable outputs. An ensemble approach leverages the collective intelligence of multiple models, making it more adaptive to various patterns and ensuring a more accurate identification of anomalies. This synergistic collaboration improves the overall accuracy and increases the reliability of anomaly detection systems, making them better suited to handle the complexity and diversity of real-world time series datasets.
Importance of Domain Knowledge:Along with ensemble, there are methods in time series analysis that are versatile in their ability to identify point outliers and subsequent outliers. These methods are designed to capture deviations from the expected pattern, whether they occur as isolated data points or manifest as sustained abnormal behavior over consecutive time points. Moving Average or Exponential Smoothing and Change Point Detection Algorithms are two such methods, among many.
Importance of Domain Knowledge:
In the realm of time series data analysis, the significance of domain knowledge cannot be overstated, particularly in the treatment of outliers. While statistical methods offer valuable insights, integrating domain expertise elevates the process by providing a contextual understanding of the subject matter. Domain experts bring a wealth of knowledge that aids in distinguishing genuine anomalies from expected variations, assessing data quality, and understanding the impact of outliers on analyses. Their insights extend to recognizing seasonal and cyclical patterns, incorporating awareness of external factors, and optimizing treatment strategies based on the specific goals and constraints within the domain. This collaborative approach ensures that the identification and treatment of outliers align with the intricacies of the domain, resulting in more informed and contextually relevant outlier treatment strategies that contribute to effective decision-making processes.
Conclusion:
In conclusion, a time series is a powerful and versatile tool for capturing the evolution of variables over time, providing insights into patterns, trends, and influencing factors. Its components collectively contribute to the observed pattern, including trends, seasonal variations, cyclic variations, and random movements. Time series data plays a pivotal role in various analyses, aiding in fundamental and technical approaches, and its forecasting methods contribute to decision-making processes in diverse fields. The presence of outliers in time series data, categorized into unwanted data anomalies and events of interest anomalies, necessitates careful identification and treatment. Point outliers, represented by individual data points deviating significantly from the expected pattern, and subsequent outliers, reflecting sustained abnormal behavior, require distinct methodologies for detection and treatment. Utilizing methods such as Z-Score, Tukey's Fences, Moving Average, Exponential Smoothing, Change Point Detection Algorithms, and Cluster Analysis allows a comprehensive approach to identify and treat outliers effectively. Moreover, acknowledging the real-life complexity of data, an ensemble of anomaly detection methods and versatile techniques capable of handling both point and subsequent outliers offer robust solutions for accurate analyses in diverse applications.
Topic Tags
Detailed Case Study
Enabled Data-Ops on Cloud for a North American Telecom Giant
Learn how a Tredence client integrated all its data into a single data lake with our 4-phase migration approach, saving $50K/month! Reach out to us to know more.
Detailed Case Study
MIGRATING LEGACY APPLICATIONS TO A MODERN SUPPLY CHAIN PLATFORM FOR A LEADING $15 BILLION WATER, SANITATION, AND INFECTION PREVENTION SOLUTIONS PROVIDER
Learn how a Tredence client integrated all its data into a single data lake with our 4-phase migration approach, saving $50K/month! Reach out to us to know more.
