Optimizing Data Retention for Specific Tables in Azure Log Analytics Workspace

Application Insights plays a crucial role in applications hosted on Microsoft Azure. It is important to periodically review the associated Log Analytics Workspace instances to ensure they are properly governed and optimized.

The following areas should be examined for these resources in alignment with the Azure Well-Architected Framework.

• Reliability
• Security
• Cost Optimization
• Operational Excellence
• Performance Efficiency

The following analysis aligns with Cost Optimization, Operational Excellence, and Performance Efficiency pillars of the well architected framework.

Recently, I noticed an increase in app usage for a client, leading to higher log volumes. 

We can easily analyze this pattern by navigating to the Log Analytics Workspace and checking the Usage & Estimated Costs section.

It is evident that AppTraces have increased recently, which may negatively impact overall query performance.

My Log Analytics Workspace is set to the default 90-day data retention period, which I prefer to keep unchanged to achieve cost optimization.

Since the outlier is only in the AppTraces table, I want to reduce the retention period specifically for that table. The expectation is to maintain a lower data volume in that container while keeping the overall workspace retention unchanged.

Following was the approach I used.

Navigate to Tables in the Log Analytics Workspace and select the AppTraces table.

Click on Manage Table to access the table settings.

Change the retention period to 60 days, which overrides the default retention setting for this table only.

That's it! The AppTraces table will now retain data for 60 days, reducing log volume while keeping the overall workspace retention unchanged.

Let's consider the opposite approach: How can you increase data retention while still optimizing costs?

You can also use this approach to optimize costs. For example, if you need to increase the retention period from 90 to 120 days, instead of applying the change to all tables, you can selectively extend retention only for the tables that truly require it. This helps balance cost efficiency and data availability.

Additional retention will incur costs based on the amount of storage consumed.

By strategically planning data retention for specific tables, you can effectively optimize costs.