Confirmed users
632
edits
Loop/Data Collection: Difference between revisions
Jump to navigation
Jump to search
→Data Purging
| Line 354: | Line 354: | ||
=== Data Purging === | === Data Purging === | ||
At the end of each week, a data retention job will locate containers with a name indicating that they exceed the data retention policy for these reports (five weeks), and remove the containers (including all contained reports). | At the end of each week, a data retention job will locate containers with a name indicating that they exceed the data retention policy for these reports (five weeks), and remove the containers (including all contained reports). Note that this means that raw data will be kept for between 28 and 35 days, depending on the day of the week on which they are submitted. | ||
This job also removes data from the index table older than its retention policy (which may be somewhat longer than the retention policy for the raw data; this is still TBD). (TODO: table removal can probably be optimized through the use of partitions -- I need to look into Azure table storage a bit more). | This job also removes data from the index table older than its retention policy (which may be somewhat longer than the retention policy for the raw data; this is still TBD). (TODO: table removal can probably be optimized through the use of partitions -- I need to look into Azure table storage a bit more). | ||