Hi, this is Ricky Sun, a PI Consultant. In this post, I’ll address a problem that keeps coming back for advanced PI Analytics users--the analytic flatline. This issue occurs when calculations stop evaluating, leaving anomaly detection and monitoring systems blind. If you only use the PI Server as a historian, this issue won’t affect you. But if your organization relies heavily on PI Analytics—especially for anomaly detection across thousands of calculations—this post is for you.

What Causes the Analytic Flatline?
The flatline issue I experienced involves the interaction between the PI Analytics service and the PI Server. Here’s what happens:

- Busy PI Server Timeout: The PI Server becomes overloaded and can’t respond in time, causing analytics to stop pulling data. Even restarting the PI Server results in analytics stalling.

- Template Changes in Asset Framework (AF): If you change a template used by many assets, all related elements restart. For example, in my case:

- 2,000 wells
- 150 attributes per well (like pressure, temperature, and levels)
- This leads to 300,000 attributes being affected by any single template change.

Each time we updated the template (e.g., by adding or changing a calculation), all these assets restarted, stalling analytics and triggering reconciliation issues. Analytics wouldn’t resume until we manually restarted the service, which caused even more backfill problems.

How It Impacts Operations
Our remote operations team relies on anomaly detection using PI Analytics to respond to field conditions quickly. When analytics stalls:

- Anomaly detection fails, leaving operators blind to critical events.
- Event frames stop working, disrupting workflows.
- Operators can't take action without real-time insights, which can lead to delays and risks in production.

Identifying the Problem
When the flatline first occurred, I asked my team if anyone had made changes to the AF templates or PI Data Archive. Sure enough, we discovered that modifying templates with many elements or attributes caused a restart across the system.

I also found documentation that confirmed similar issues:

“After editing a template with many associated elements, the PI Analytics service may stop processing calculations. This creates performance issues and temporarily halts evaluations.”

This overload happens because the PI Update Manager processes too many requests at once. This affects not only analytics but also other applications connected to the PI Server.

Proposed Solutions (And Why They Aren't Ideal)
​Here are some of the solutions we tried and considered, along with their drawbacks:

1. Stop and Restart Analytics Services

- Approach: Stop the PI Analytics service, make template changes, and then restart.
- Drawback: This requires backfilling data and risks reconciliation issues.

2. Splitting Element Templates

- Approach: Break large templates into smaller ones to reduce the impact of changes.
- Drawback: Managing multiple templates adds complexity. For example, splitting a well template into sections like "front" and "back" isn’t practical.

3. Batch Changes to Minimize Restarts

- Approach: Make all changes at once and apply them in batches.
- Drawback: While this reduces the number of restarts, it doesn’t eliminate the issue entirely.

4. Implementing a Testing Environment

- Approach: Test changes in a staging environment before deploying them in production.
- Drawback: This adds operational overhead and isn’t always feasible for real-time operations.

What Needs to Change?
In my opinion, AVEVA should prioritize fixing this issue. Heavy users of PI Analytics rely on the system for real-time monitoring, and these flatline problems cause major disruptions. Developing new features is great, but the core functionality of analytics and anomaly detection must work reliably.

If AF templates and analytics can’t support high-volume operations smoothly, it raises questions about the value of the PI System. Why invest in a platform if it can’t meet the needs of its most intensive users?

Conclusion
Unfortunately, we don’t have a perfect solution for the analytic flatline issue yet. If you’ve encountered this problem, I’d love to hear your feedback and ideas. Please share your experiences and let me know how you’ve managed it.

​Thank you for reading!