Cleanup of Queues and Logs Older Than Three Months: A Practical Guide to Log Archiving on Qeasy

What This Strategy Solves (Scenario and Value)

In one real project, the customer's supply chain integration between an order system and a mid-platform ERP was running on the Qeasy data integration platform. After half a year, the platform's own queue and log tables had accumulated hundreds of millions of rows. The ops team reported slow scheduling pages, SQL timeouts during troubleshooting, and disk alarms. We later added a dedicated strategy, "Cleanup of queues and logs older than three months," which trims historical data on a cycle and makes the platform light again. It solves the platform's own data bloat, not business data, but without it, business sync will eventually be dragged down.

Data Flow and Field Mapping

Both the source and target of this strategy live inside the Qeasy platform itself. Logically, it is a "platform-internal cleanup" action.

target_1 is an object-type input parameter that defines the list of schemes to clean in this run, supporting batch input across multiple schemes. The datetime and params fields in the response are auto-filled for audit tracing.

How to Configure It on Qeasy

Configuring this strategy on Qeasy is essentially about wrapping the "cleanup action" into a schedulable WebAPI scheme.

Step 1: Define the API metadata. The source uses the platform's built-in DeleteExceptData (POST, QUERY semantics) to filter queues and logs older than three months. The target uses Write Null Operation (POST, EXECUTE semantics) with empty request and response. It looks pointless, but it actually triggers an internal cleanup transaction receipt, keeping the scheduling chain complete and observable.

Step 2: Set scheme parameters. Maintain target_1 (schemes to process) in the source API's request, leaving it empty by default. At runtime the scheduling script injects the concrete list of scheme codes. This supports both batch cleanup of all schemes and future filtering by business domain.

Step 3: Isolate and whitelist. A common pattern among Qeasy customers is "centralized code mapping management," which applies here too: keep scheme codes in one config table, and let the cleanup strategy only read from it. Avoid hardcoding in scripts.

Step 4: Turn on auto-fill. Keep autoFillResponse on so the response field (especially datetime) is automatically written with a timestamp, making it easy to audit "when did this cleanup run" afterwards.

Implementation Steps

We recommend splitting this strategy into three phases.

Phase 1: Incremental starting point. When first going live, run a "full trigger" to clean up everything older than three months at once. Record the cleanup duration and remaining data volume as a baseline.

Phase 2: Scheduling frequency. Set the source strategy's crontab to 3 2 * * * (run at 02:03 every day) and the target placeholder strategy to 1 1 1 1 1 as a weekly/monthly trigger slot. A typical mistake is to "run a full cleanup every day," which can easily delete logs that are still being used for incident investigation. The safe approach is the "incremental + full dual track" pattern: incremental cleanup daily, full verification on weekends.

Phase 3: Monitoring and rollback. After each cleanup, compare the remaining queue count and disk usage; if the week-over-week change exceeds 20%, trigger manual review automatically. A common pitfall here is "the cleanup script colliding with the business peak," so always place the execution window in the early-morning low-traffic period.

Lessons Learned from Real Rollouts

Pitfall 1: Setting the cleanup window during the day. In one customer site we set crontab to 10:00 AM, which happened to be the peak for sales order sync. The cleanup locked the table and caused downstream backlog. We later moved everything to after 02:00 AM.

Pitfall 2: Omitting the audit receipt. In an early version someone removed the Write Null Operation step, thinking "there is no business data anyway." As a result we could not later check whether the cleanup ran and which schemes it covered. The placeholder API may look redundant, but it is the audit anchor.

Pitfall 3: Hardcoding the three-month threshold. Financial businesses require 180-day retention, while retail businesses only need 90 days. The safe approach is to abstract "retention days" into a scheme parameter rather than hardcoding it in the script.

Pitfall 4: Scheme codes scattered everywhere. When multiple environments (dev, test, prod) share one cleanup strategy, not centralizing target_1 can easily lead to the test scheme's cleanup action mistakenly hitting production.

Pitfall 5: Ignoring disk monitoring. Cleanup itself is a large transaction that writes redo/undo logs instantaneously. If the disk is already above 85%, the cleanup may fail directly. Expanding the disk first and cleaning later saves more effort than the other way around.

Applicable and Non-Applicable Scenarios

Applicable: Projects where Qeasy has been running for more than six months, queue and log rows are in the hundreds of millions, disk usage stays above 70% long term, and the scheduling page is noticeably slow; retail, e-commerce, and trading businesses with compliance requiring only 90+ days of retention.

Not applicable: Strong-compliance scenarios such as finance and healthcare (which require 180 days or more of retention); projects where the platform has just gone live and historical data does not yet reach three months; and early-stage environments where header/body phasing has not yet been done and cleanup granularity cannot converge to "per scheme."