ABAP Developmentadvanced

Performance Part 9: Package Processing for Large Data Volumes

Key takeaway

Process millions of records safely: SELECT … PACKAGE SIZE with ENDSELECT, OPEN CURSOR WITH HOLD / FETCH NEXT CURSOR, committing between packages, restartability, memory limits, and parallel processing with background jobs or aRFC for very large runs.

Explanation

📌 KEY POINTS
✅ Reading millions of rows into one internal table can exhaust memory (TSV_TNEW_PAGE_ALLOC_FAILED); process in packages instead.
✅ SELECT … INTO TABLE … PACKAGE SIZE n … ENDSELECT processes n rows per pass; OPEN CURSOR / FETCH NEXT CURSOR … PACKAGE SIZE gives more control.
✅ COMMIT WORK between packages closes normal cursors — use OPEN CURSOR WITH HOLD when you must commit while reading.
⚠️ Design for restart: if a job fails after 3 of 10 million records, it must continue without duplicating work (status flags, last processed key, logs).

━━━━━━━━━━━━━━━━━━━━
▶ WHERE WE ARE
Parts 4–8 made each read efficient. For very large volumes, the program structure itself must change.

━━━━━━━━━━━━━━━━━━━━
▶ 1. WHY PACKAGES

Memory: bounded by package size.
Commits: long LUWs hold locks and rollback segments; packages allow commits.
Progress and restart points.

━━━━━━━━━━━━━━━━━━━━
▶ 2. SELECT PACKAGE SIZE

SELECT … INTO TABLE lt_pkg PACKAGE SIZE 10000 … ENDSELECT: lt_pkg is overwritten each pass.
No COMMIT inside (it would close the cursor → dump DBIF_RSQL_INVALID_CURSOR).

━━━━━━━━━━━━━━━━━━━━
▶ 3. OPEN CURSOR WITH HOLD

OPEN CURSOR WITH HOLD lv_cursor FOR SELECT …; DO. FETCH NEXT CURSOR lv_cursor INTO TABLE lt_pkg PACKAGE SIZE n. IF sy-subrc <> 0. EXIT. ENDIF. process; COMMIT WORK. ENDDO. CLOSE CURSOR.
WITH HOLD keeps the cursor open across database commits (COMMIT WORK in newer releases; check release notes — DB commit via DB_COMMIT is the classic case).

━━━━━━━━━━━━━━━━━━━━
▶ 4. RESTARTABILITY

Process in key order; store the last processed key in a status table after each package.
Mark processed records (status field) or write idempotent updates.
Application log (SLG1) per package.

━━━━━━━━━━━━━━━━━━━━
▶ 5. PARALLEL PROCESSING

Split the key range into intervals and schedule several jobs, or use aRFC (CALL FUNCTION … STARTING NEW TASK … DESTINATION IN GROUP) with a server group.
Avoid lock conflicts between parallel parts (partition by company code/plant/key range).

━━━━━━━━━━━━━━━━━━━━
▶ STEP BY STEP

Rewrite a full-table read to SELECT … PACKAGE SIZE.
Add OPEN CURSOR WITH HOLD with a commit per package (test system).
Store the last processed key and test a restart.

✅ CHECKPOINT: you can process very large volumes with bounded memory and restart capability.

Code example

ABAP Code

REPORT zperf_packages. PARAMETERS p_size TYPE i DEFAULT 10000. TYPES: BEGIN OF ty_doc,         belnr TYPE belnr_d,         bukrs TYPE bukrs,         gjahr TYPE gjahr,       END OF ty_doc. DATA: lt_pkg    TYPE STANDARD TABLE OF ty_doc,      lv_cursor TYPE cursor,      lv_total  TYPE i,      lv_lines  TYPE i. START-OF-SELECTION.* ===== Option A: SELECT ... PACKAGE SIZE (no COMMIT inside!) =====  SELECT belnr bukrs gjahr FROM bkpf    INTO TABLE lt_pkg    PACKAGE SIZE p_size    WHERE gjahr = '2026'.    DESCRIBE TABLE lt_pkg LINES lv_lines.    lv_total = lv_total + lv_lines.    " process this package (lt_pkg is replaced on each pass)  ENDSELECT.  WRITE: / 'Processed (A):', lv_total. * ===== Option B: cursor WITH HOLD, commit per package =====  CLEAR lv_total.  OPEN CURSOR WITH HOLD lv_cursor FOR    SELECT belnr bukrs gjahr FROM bkpf      WHERE gjahr = '2026'      ORDER BY PRIMARY KEY.                 " stable order for restart logic  DO.    FETCH NEXT CURSOR lv_cursor INTO TABLE lt_pkg PACKAGE SIZE p_size.    IF sy-subrc <> 0.      EXIT.    ENDIF.    DESCRIBE TABLE lt_pkg LINES lv_lines.    lv_total = lv_total + lv_lines.    " process, write results, store last key in a status table    CALL FUNCTION 'DB_COMMIT'.              " database commit keeps WITH HOLD cursor open  ENDDO.  CLOSE CURSOR lv_cursor.  WRITE: / 'Processed (B):', lv_total.

Real project scenario

A data-cleansing job read 45 million accounting lines into one table and dumped with TSV_TNEW_PAGE_ALLOC_FAILED after 2 hours. The next attempt used PACKAGE SIZE but called COMMIT WORK inside SELECT…ENDSELECT and dumped with an invalid cursor.
The final version used OPEN CURSOR WITH HOLD, a database commit per 20,000 rows, a status table with the last processed key, and 8 parallel jobs split by company code. It ran in 70 minutes and could be restarted after a system interruption.

Common mistakes

Reading everything into one table.
COMMIT inside SELECT…ENDSELECT.
No restart logic.
Parallel jobs locking the same objects.
Package sizes too small (overhead) or too large (memory).

Best practices

Package processing for large volumes.
Cursor WITH HOLD when committing.
Store progress for restarts.
Partition parallel work without overlap.
Tune package size by measurement.

Interview angle

Questions: "How do you process millions of records?", "Why does COMMIT inside SELECT…ENDSELECT dump?", "How do you make jobs restartable?"
Strong answer: PACKAGE SIZE or OPEN CURSOR/FETCH NEXT CURSOR PACKAGE SIZE for bounded memory; commits close cursors → OPEN CURSOR WITH HOLD with DB commit; restart via key order, status table, idempotent updates, SLG1; parallelise by key ranges via jobs or aRFC server groups.

Practice this lesson

Quick check

1. Why does COMMIT WORK inside SELECT … PACKAGE SIZE … ENDSELECT cause a dump?

2. Why is package processing useful for very large datasets?

3. Parallel processing in ABAP is typically done with:

Interview questions

1. How would you use parallel processing (aRFC or bgRFC) to shorten a batch run, and what risks do you plan for?

Answer it in your own words first, then compare.

2. A nightly job loads more than ten million records into one internal table and fails with a memory dump. How do you redesign it?

Answer it in your own words first, then compare.

Practice more with Interview AI →

Reference-verified: the steps are checked against official SAP documentation and tutorials, and object and field names in the sample code are checked for consistency. The code has not been compiled or run on an SAP system by ERPClimb. Syntax marked release-dependent varies by SAP release, so check it in your own system.